Skip to content

Commit 91adbd6

Browse files
pieternpietern
authored
postgres: Support purge_on_delete on postgres_branches (#5801)
Adds `purge_on_delete` to Lakebase `postgres_branches`, mirroring `postgres_projects` (#5414). When `true`, destroy calls `DeleteBranch` with `?purge=true` (hard delete); otherwise the branch is soft-deleted and recoverable within its retention window. The flag is input-only, kept in state so `DoDelete` can apply it on destroy. `DoUpdate` strips it from the field mask so toggling it is a state-only refresh. Acceptance tests assert the `?purge=true` vs plain `DELETE` contrast on both engines. This pull request and its description were written by Isaac.
1 parent 3a01ff0 commit 91adbd6

25 files changed

Lines changed: 498 additions & 12 deletions

File tree

‎acceptance/bundle/refschema/out.fields.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2954,6 +2954,7 @@ resources.postgres_branches.*.modified_status string INPUT
29542954
resources.postgres_branches.*.name string REMOTE
29552955
resources.postgres_branches.*.no_expiry bool ALL
29562956
resources.postgres_branches.*.parent string ALL
2957+
resources.postgres_branches.*.purge_on_delete bool INPUT STATE
29572958
resources.postgres_branches.*.replace_existing bool INPUT STATE
29582959
resources.postgres_branches.*.source_branch string ALL
29592960
resources.postgres_branches.*.source_branch_lsn string ALL

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/databricks.yml.tmpl‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
bundle:
2+
name: deploy-postgres-branch-purge-$UNIQUE_NAME
3+
4+
sync:
5+
paths: []
6+
7+
# A project with two branches side by side so the recorded destroy requests
8+
# make the purge_on_delete contrast obvious:
9+
# - hard_delete: purge_on_delete: true -> DELETE …?purge=true
10+
# - soft_delete: field omitted (default) -> DELETE … (no purge query)
11+
resources:
12+
postgres_projects:
13+
my_project:
14+
project_id: test-pg-proj-$UNIQUE_NAME
15+
display_name: "Test Project for Branch purge_on_delete"
16+
pg_version: 16
17+
18+
postgres_branches:
19+
hard_delete:
20+
parent: ${resources.postgres_projects.my_project.id}
21+
branch_id: test-pg-branch-hard-$UNIQUE_NAME
22+
no_expiry: true
23+
purge_on_delete: true
24+
25+
soft_delete:
26+
parent: ${resources.postgres_projects.my_project.id}
27+
branch_id: test-pg-branch-soft-$UNIQUE_NAME
28+
no_expiry: true

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/out.destroy.txt‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
The following resources will be deleted:
2+
delete resources.postgres_branches.hard_delete
3+
delete resources.postgres_branches.soft_delete
4+
delete resources.postgres_projects.my_project
5+
6+
This action will result in the deletion of the following Lakebase projects along with
7+
all their branches, databases, and endpoints. All data stored in them will be permanently lost:
8+
delete resources.postgres_projects.my_project
9+
10+
This action will result in the deletion of the following Lakebase branches.
11+
All data stored in them will be permanently lost:
12+
delete resources.postgres_branches.hard_delete
13+
delete resources.postgres_branches.soft_delete
14+
15+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-branch-purge-[UNIQUE_NAME]/default
16+
17+
Deleting files...
18+
Destroy complete!

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/out.requests.deploy.direct.json‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
{
2+
"method": "GET",
3+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-hard-[UNIQUE_NAME]"
4+
}
5+
{
6+
"method": "GET",
7+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-soft-[UNIQUE_NAME]"
8+
}
9+
{
10+
"method": "POST",
11+
"path": "/api/2.0/postgres/projects",
12+
"q": {
13+
"project_id": "test-pg-proj-[UNIQUE_NAME]"
14+
},
15+
"body": {
16+
"spec": {
17+
"display_name": "Test Project for Branch purge_on_delete",
18+
"pg_version": 16
19+
}
20+
}
21+
}
22+
{
23+
"method": "POST",
24+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches",
25+
"q": {
26+
"branch_id": "test-pg-branch-hard-[UNIQUE_NAME]"
27+
},
28+
"body": {
29+
"spec": {
30+
"no_expiry": true
31+
}
32+
}
33+
}
34+
{
35+
"method": "POST",
36+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches",
37+
"q": {
38+
"branch_id": "test-pg-branch-soft-[UNIQUE_NAME]"
39+
},
40+
"body": {
41+
"spec": {
42+
"no_expiry": true
43+
}
44+
}
45+
}

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/out.requests.deploy.terraform.json‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
{
2+
"method": "GET",
3+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-hard-[UNIQUE_NAME]"
4+
}
5+
{
6+
"method": "GET",
7+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-soft-[UNIQUE_NAME]"
8+
}
9+
{
10+
"method": "POST",
11+
"path": "/api/2.0/postgres/projects",
12+
"q": {
13+
"project_id": "test-pg-proj-[UNIQUE_NAME]"
14+
},
15+
"body": {
16+
"spec": {
17+
"display_name": "Test Project for Branch purge_on_delete",
18+
"pg_version": 16
19+
}
20+
}
21+
}
22+
{
23+
"method": "POST",
24+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches",
25+
"q": {
26+
"branch_id": "test-pg-branch-hard-[UNIQUE_NAME]"
27+
},
28+
"body": {
29+
"parent": "projects/test-pg-proj-[UNIQUE_NAME]",
30+
"spec": {
31+
"no_expiry": true
32+
}
33+
}
34+
}
35+
{
36+
"method": "POST",
37+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches",
38+
"q": {
39+
"branch_id": "test-pg-branch-soft-[UNIQUE_NAME]"
40+
},
41+
"body": {
42+
"parent": "projects/test-pg-proj-[UNIQUE_NAME]",
43+
"spec": {
44+
"no_expiry": true
45+
}
46+
}
47+
}

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/out.requests.destroy.json‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"method": "DELETE",
3+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]"
4+
}
5+
{
6+
"method": "DELETE",
7+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-hard-[UNIQUE_NAME]",
8+
"q": {
9+
"purge": "true"
10+
}
11+
}
12+
{
13+
"method": "DELETE",
14+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-soft-[UNIQUE_NAME]"
15+
}
16+
{
17+
"method": "GET",
18+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]"
19+
}
20+
{
21+
"method": "GET",
22+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-hard-[UNIQUE_NAME]"
23+
}
24+
{
25+
"method": "GET",
26+
"path": "/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-soft-[UNIQUE_NAME]"
27+
}

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/out.test.toml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/output.txt‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
2+
>>> [CLI] bundle validate
3+
Name: deploy-postgres-branch-purge-[UNIQUE_NAME]
4+
Target: default
5+
Workspace:
6+
User: [USERNAME]
7+
Path: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-branch-purge-[UNIQUE_NAME]/default
8+
9+
Validation OK!
10+
11+
>>> [CLI] bundle deploy
12+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-branch-purge-[UNIQUE_NAME]/default/files...
13+
Deploying resources...
14+
Updating deployment state...
15+
Deployment complete!
16+
17+
>>> [CLI] postgres get-branch projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-hard-[UNIQUE_NAME]
18+
{
19+
"branch_id": "test-pg-branch-hard-[UNIQUE_NAME]",
20+
"name": "projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-hard-[UNIQUE_NAME]",
21+
"parent": "projects/test-pg-proj-[UNIQUE_NAME]",
22+
"status": {
23+
"branch_id": "test-pg-branch-hard-[UNIQUE_NAME]",
24+
"current_state": "READY",
25+
"default": false,
26+
"is_protected": false,
27+
"source_branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
28+
"source_branch_lsn": "[LSN]",
29+
"source_branch_time": "[TIMESTAMP]",
30+
"state_change_time": "[TIMESTAMP]"
31+
},
32+
"uid": "[BRANCH_UID]"
33+
}
34+
35+
>>> [CLI] postgres get-branch projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-soft-[UNIQUE_NAME]
36+
{
37+
"branch_id": "test-pg-branch-soft-[UNIQUE_NAME]",
38+
"name": "projects/test-pg-proj-[UNIQUE_NAME]/branches/test-pg-branch-soft-[UNIQUE_NAME]",
39+
"parent": "projects/test-pg-proj-[UNIQUE_NAME]",
40+
"status": {
41+
"branch_id": "test-pg-branch-soft-[UNIQUE_NAME]",
42+
"current_state": "READY",
43+
"default": false,
44+
"is_protected": false,
45+
"source_branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
46+
"source_branch_lsn": "[LSN]",
47+
"source_branch_time": "[TIMESTAMP]",
48+
"state_change_time": "[TIMESTAMP]"
49+
},
50+
"uid": "[BRANCH_UID]"
51+
}
52+
53+
>>> print_requests.py --del-body project_id,branch_id,endpoint_id,database_id,role_id,catalog_id,synced_table_id --sort --get //postgres ^//workspace-files/ ^//workspace/ ^//telemetry-ext ^//operations/
54+
55+
=== bundle destroy
56+
>>> print_requests.py --del-body project_id,branch_id,endpoint_id,database_id,role_id,catalog_id,synced_table_id --sort --get //postgres ^//workspace-files/ ^//workspace/ ^//telemetry-ext ^//operations/

‎acceptance/bundle/resources/postgres_branches/purge_on_delete/script‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
envsubst < databricks.yml.tmpl > databricks.yml
2+
3+
project_name="projects/test-pg-proj-${UNIQUE_NAME}"
4+
hard_branch="${project_name}/branches/test-pg-branch-hard-${UNIQUE_NAME}"
5+
soft_branch="${project_name}/branches/test-pg-branch-soft-${UNIQUE_NAME}"
6+
7+
cleanup() {
8+
# Belt-and-braces in case bundle destroy was skipped or partially failed.
9+
# The soft-delete case leaves a record in the trash; --purge clears it.
10+
$CLI postgres delete-branch --purge "${hard_branch}" 2>>LOG.delete-branch || true
11+
$CLI postgres delete-branch --purge "${soft_branch}" 2>>LOG.delete-branch || true
12+
$CLI postgres delete-project --purge "${project_name}" 2>>LOG.delete-project || true
13+
rm -f out.requests.txt
14+
}
15+
trap cleanup EXIT
16+
17+
trace $CLI bundle validate
18+
19+
rm -f out.requests.txt
20+
trace $CLI bundle deploy
21+
22+
trace $CLI postgres get-branch "${hard_branch}" | jq 'del(.create_time, .update_time, .status.logical_size_bytes)'
23+
trace $CLI postgres get-branch "${soft_branch}" | jq 'del(.create_time, .update_time, .status.logical_size_bytes)'
24+
25+
# Deploy requests are split per-engine: Terraform sends "parent" in the branch
26+
# create body, the direct engine sends it only as a query parameter.
27+
trace print_requests.py --del-body project_id,branch_id,endpoint_id,database_id,role_id,catalog_id,synced_table_id --sort --get '//postgres' '^//workspace-files/' '^//workspace/' '^//telemetry-ext' '^//operations/' > out.requests.deploy.$DATABRICKS_BUNDLE_ENGINE.json
28+
29+
# bundle destroy should send ?purge=true on the hard_delete branch and no
30+
# purge query on the soft_delete branch. Both DELETEs land in the recorded
31+
# requests so the contrast is visible in the diff.
32+
title "bundle destroy"
33+
$CLI bundle destroy --auto-approve > out.destroy.txt 2>&1 || true
34+
35+
trace print_requests.py --del-body project_id,branch_id,endpoint_id,database_id,role_id,catalog_id,synced_table_id --sort --get '//postgres' '^//workspace-files/' '^//workspace/' '^//telemetry-ext' '^//operations/' > out.requests.destroy.json

‎acceptance/bundle/resources/postgres_branches/purge_on_delete_transitions/databricks.yml.tmpl‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
bundle:
2+
name: pg-branch-purge-transitions-$UNIQUE_NAME
3+
4+
sync:
5+
paths: []
6+
7+
# Walks purge_on_delete through unset -> true -> false -> unset, deploying
8+
# at each step and inspecting the persisted state so reviewers can see that
9+
# state tracks the user's latest intent. The final destroy is a soft delete
10+
# (state has purge_on_delete unset), recorded for regression coverage.
11+
resources:
12+
postgres_projects:
13+
proj:
14+
project_id: test-pg-proj-$UNIQUE_NAME
15+
display_name: "Transitions test"
16+
pg_version: 16
17+
18+
postgres_branches:
19+
branch:
20+
parent: ${resources.postgres_projects.proj.id}
21+
branch_id: test-pg-branch-$UNIQUE_NAME
22+
no_expiry: true
23+
# PURGE_ON_DELETE

0 commit comments

Comments
 (0)