Skip to content

Commit 36bd5cc

Browse files
pieternpietern
committed
postgres: Support purge_on_delete on postgres_projects
Adds support for `purge_on_delete` on Lakebase `postgres_projects` so bundles can hard-delete a project on destroy. The flag is passed to the DeleteProject API call as `?purge=true`; when unset, the backend performs a soft delete that can be undone via `databricks postgres undelete-project` within the project's retention window. The field is input-only — it is not modeled by the backend resource for projects, and the GET API never returns it. We store it in state purely so DoDelete can apply it on destroy: by that point the configuration for the resource is gone, so state is the only place it can live. PrepareState preserves `input.ForceSendFields` so the structdiff comparison correctly distinguishes "explicit false" from the fictional remote zero — otherwise toggling `true -> false` would be classified as no change, state would stay `true`, and the next destroy would still emit `?purge=true`. DoUpdate strips `purge_on_delete` from the API field mask so a state-only flip doesn't fire an unnecessary remote write. Acceptance tests under `acceptance/bundle/resources/postgres_projects/`: - `purge_on_delete/`: deploys a `hard_delete` and a `soft_delete` project side by side and asserts the destroy emits `?purge=true` and a plain DELETE respectively, on both engines. - `purge_on_delete_transitions/`: direct-engine only. Walks `purge_on_delete` through unset -> true -> false -> unset and records the persisted value at each step; final destroy is a plain DELETE. Regression coverage for the FSF-preservation fix. Manually verified against dogfood with ephemeral projects on both engines (deploy -> flip -> destroy; GET on project/branches/endpoints in soft vs hard cases; native `postgres undelete-project` restoration semantics — captured in DECO-27233). Co-authored-by: Isaac
1 parent 865787a commit 36bd5cc

26 files changed

Lines changed: 431 additions & 13 deletions

File tree

‎NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@
1111
* Retry transient HTTP 5xx and 408 errors in direct deployment engine ([#5349](https://github.com/databricks/cli/pull/5349), [#5364](https://github.com/databricks/cli/pull/5364)).
1212
* Preserve `.designer.ipynb` suffix when translating notebook task paths so Lakeflow Designer files referenced from a `notebook_task` resolve correctly in the workspace ([#5370](https://github.com/databricks/cli/pull/5370)).
1313
* Fix script output dropping last line without trailing newline ([#4995](https://github.com/databricks/cli/pull/4995)).
14+
* Support `purge_on_delete: true` on `postgres_projects` so bundles can hard-delete a Lakebase project on destroy (skipping the soft-delete retention window).
1415

1516
### Dependency updates

‎acceptance/bundle/refschema/out.fields.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2915,6 +2915,7 @@ resources.postgres_projects.*.modified_status string INPUT
29152915
resources.postgres_projects.*.name string REMOTE
29162916
resources.postgres_projects.*.pg_version int ALL
29172917
resources.postgres_projects.*.project_id string ALL
2918+
resources.postgres_projects.*.purge_on_delete bool INPUT STATE
29182919
resources.postgres_projects.*.purge_time *time.Time REMOTE
29192920
resources.postgres_projects.*.status *postgres.ProjectStatus REMOTE
29202921
resources.postgres_projects.*.status.branch_logical_size_limit_bytes int64 REMOTE

‎acceptance/bundle/resources/postgres_projects/purge_on_delete/databricks.yml.tmpl‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
bundle:
2+
name: deploy-postgres-purge-$UNIQUE_NAME
3+
4+
sync:
5+
paths: []
6+
7+
# Two projects side by side so the recorded destroy requests make the
8+
# purge_on_delete contrast obvious:
9+
# - hard_delete: purge_on_delete: true -> DELETE …?purge=true
10+
# - soft_delete: field omitted (default) -> DELETE … (no purge query)
11+
resources:
12+
postgres_projects:
13+
hard_delete:
14+
project_id: test-pg-proj-hard-$UNIQUE_NAME
15+
display_name: "purge_on_delete = true"
16+
pg_version: 16
17+
purge_on_delete: true
18+
19+
soft_delete:
20+
project_id: test-pg-proj-soft-$UNIQUE_NAME
21+
display_name: "purge_on_delete unset (soft delete)"
22+
pg_version: 16

‎acceptance/bundle/resources/postgres_projects/purge_on_delete/out.destroy.txt‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
The following resources will be deleted:
2+
delete resources.postgres_projects.hard_delete
3+
delete resources.postgres_projects.soft_delete
4+
5+
This action will result in the deletion of the following Lakebase projects along with
6+
all their branches, databases, and endpoints. All data stored in them will be permanently lost:
7+
delete resources.postgres_projects.hard_delete
8+
delete resources.postgres_projects.soft_delete
9+
10+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-purge-[UNIQUE_NAME]/default
11+
12+
Deleting files...
13+
Destroy complete!

‎acceptance/bundle/resources/postgres_projects/purge_on_delete/out.requests.deploy.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"method": "GET",
3+
"path": "/api/2.0/postgres/projects/test-pg-proj-hard-[UNIQUE_NAME]"
4+
}
5+
{
6+
"method": "GET",
7+
"path": "/api/2.0/postgres/projects/test-pg-proj-soft-[UNIQUE_NAME]"
8+
}
9+
{
10+
"method": "POST",
11+
"path": "/api/2.0/postgres/projects",
12+
"q": {
13+
"project_id": "test-pg-proj-hard-[UNIQUE_NAME]"
14+
},
15+
"body": {
16+
"spec": {
17+
"display_name": "purge_on_delete = true",
18+
"pg_version": 16
19+
}
20+
}
21+
}
22+
{
23+
"method": "POST",
24+
"path": "/api/2.0/postgres/projects",
25+
"q": {
26+
"project_id": "test-pg-proj-soft-[UNIQUE_NAME]"
27+
},
28+
"body": {
29+
"spec": {
30+
"display_name": "purge_on_delete unset (soft delete)",
31+
"pg_version": 16
32+
}
33+
}
34+
}

‎acceptance/bundle/resources/postgres_projects/purge_on_delete/out.requests.destroy.json‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
{
2+
"method": "DELETE",
3+
"path": "/api/2.0/postgres/projects/test-pg-proj-hard-[UNIQUE_NAME]",
4+
"q": {
5+
"purge": "true"
6+
}
7+
}
8+
{
9+
"method": "DELETE",
10+
"path": "/api/2.0/postgres/projects/test-pg-proj-soft-[UNIQUE_NAME]"
11+
}
12+
{
13+
"method": "GET",
14+
"path": "/api/2.0/postgres/projects/test-pg-proj-hard-[UNIQUE_NAME]"
15+
}
16+
{
17+
"method": "GET",
18+
"path": "/api/2.0/postgres/projects/test-pg-proj-hard-[UNIQUE_NAME]"
19+
}
20+
{
21+
"method": "GET",
22+
"path": "/api/2.0/postgres/projects/test-pg-proj-soft-[UNIQUE_NAME]"
23+
}
24+
{
25+
"method": "GET",
26+
"path": "/api/2.0/postgres/projects/test-pg-proj-soft-[UNIQUE_NAME]"
27+
}
28+
{
29+
"method": "POST",
30+
"path": "/api/2.0/postgres/projects",
31+
"q": {
32+
"project_id": "test-pg-proj-hard-[UNIQUE_NAME]"
33+
},
34+
"body": {
35+
"spec": {
36+
"display_name": "purge_on_delete = true",
37+
"pg_version": 16
38+
}
39+
}
40+
}
41+
{
42+
"method": "POST",
43+
"path": "/api/2.0/postgres/projects",
44+
"q": {
45+
"project_id": "test-pg-proj-soft-[UNIQUE_NAME]"
46+
},
47+
"body": {
48+
"spec": {
49+
"display_name": "purge_on_delete unset (soft delete)",
50+
"pg_version": 16
51+
}
52+
}
53+
}

‎acceptance/bundle/resources/postgres_projects/purge_on_delete/out.test.toml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/resources/postgres_projects/purge_on_delete/output.txt‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
2+
>>> [CLI] bundle validate
3+
Name: deploy-postgres-purge-[UNIQUE_NAME]
4+
Target: default
5+
Workspace:
6+
User: [USERNAME]
7+
Path: /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-purge-[UNIQUE_NAME]/default
8+
9+
Validation OK!
10+
11+
>>> [CLI] bundle deploy
12+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-postgres-purge-[UNIQUE_NAME]/default/files...
13+
Deploying resources...
14+
Updating deployment state...
15+
Deployment complete!
16+
17+
>>> [CLI] postgres get-project projects/test-pg-proj-hard-[UNIQUE_NAME]
18+
{
19+
"name": "projects/test-pg-proj-hard-[UNIQUE_NAME]",
20+
"status": {
21+
"branch_logical_size_limit_bytes": [NUMID],
22+
"default_branch": "projects/test-pg-proj-hard-[UNIQUE_NAME]/branches/production",
23+
"display_name": "purge_on_delete = true",
24+
"enable_pg_native_login": false,
25+
"owner": "[USERNAME]",
26+
"pg_version": 16,
27+
"project_id": "test-pg-proj-hard-[UNIQUE_NAME]",
28+
"synthetic_storage_size_bytes": 0
29+
},
30+
"uid": "[UUID]"
31+
}
32+
33+
>>> [CLI] postgres get-project projects/test-pg-proj-soft-[UNIQUE_NAME]
34+
{
35+
"name": "projects/test-pg-proj-soft-[UNIQUE_NAME]",
36+
"status": {
37+
"branch_logical_size_limit_bytes": [NUMID],
38+
"default_branch": "projects/test-pg-proj-soft-[UNIQUE_NAME]/branches/production",
39+
"display_name": "purge_on_delete unset (soft delete)",
40+
"enable_pg_native_login": false,
41+
"owner": "[USERNAME]",
42+
"pg_version": 16,
43+
"project_id": "test-pg-proj-soft-[UNIQUE_NAME]",
44+
"synthetic_storage_size_bytes": 0
45+
},
46+
"uid": "[UUID]"
47+
}
48+
49+
>>> print_requests.py --keep --sort --get //postgres ^//workspace-files/ ^//workspace/ ^//telemetry-ext ^//operations/
50+
51+
=== bundle destroy
52+
>>> print_requests.py --keep --sort --get //postgres ^//workspace-files/ ^//workspace/ ^//telemetry-ext ^//operations/

‎acceptance/bundle/resources/postgres_projects/purge_on_delete/script‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
envsubst < databricks.yml.tmpl > databricks.yml
2+
3+
cleanup() {
4+
# Belt-and-braces in case bundle destroy was skipped or partially failed.
5+
# The soft-delete case leaves a record in the trash; --purge clears it.
6+
$CLI postgres delete-project --purge "projects/test-pg-proj-hard-${UNIQUE_NAME}" 2>/dev/null || true
7+
$CLI postgres delete-project --purge "projects/test-pg-proj-soft-${UNIQUE_NAME}" 2>/dev/null || true
8+
rm -f out.requests.txt
9+
}
10+
trap cleanup EXIT
11+
12+
trace $CLI bundle validate
13+
14+
rm -f out.requests.txt
15+
trace $CLI bundle deploy
16+
17+
trace $CLI postgres get-project "projects/test-pg-proj-hard-${UNIQUE_NAME}" | jq 'del(.create_time, .update_time)'
18+
trace $CLI postgres get-project "projects/test-pg-proj-soft-${UNIQUE_NAME}" | jq 'del(.create_time, .update_time)'
19+
20+
trace print_requests.py --keep --sort --get '//postgres' '^//workspace-files/' '^//workspace/' '^//telemetry-ext' '^//operations/' > out.requests.deploy.json
21+
22+
# bundle destroy should send ?purge=true on the hard_delete project and no
23+
# purge query on the soft_delete project. Both DELETEs land in the recorded
24+
# requests so the contrast is visible in the diff.
25+
title "bundle destroy"
26+
$CLI bundle destroy --auto-approve > out.destroy.txt 2>&1 || true
27+
28+
trace print_requests.py --keep --sort --get '//postgres' '^//workspace-files/' '^//workspace/' '^//telemetry-ext' '^//operations/' > out.requests.destroy.json

‎acceptance/bundle/resources/postgres_projects/purge_on_delete_transitions/databricks.yml.tmpl‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
bundle:
2+
name: pg-purge-transitions-$UNIQUE_NAME
3+
4+
sync:
5+
paths: []
6+
7+
# Walks purge_on_delete through unset -> true -> false -> unset, deploying
8+
# at each step and inspecting the persisted state so reviewers can see that
9+
# state tracks the user's latest intent. The final destroy is a soft delete
10+
# (state has purge_on_delete unset), recorded for regression coverage.
11+
resources:
12+
postgres_projects:
13+
proj:
14+
project_id: test-pg-proj-$UNIQUE_NAME
15+
display_name: "Transitions test"
16+
pg_version: 16
17+
# PURGE_ON_DELETE

0 commit comments

Comments
 (0)