Skip to content

Commit 87c99e8

Browse files
shreyas-goenkashreyas-goenka
committed
bundle: record deployment history in DMS alongside the file lock
Records each deploy/destroy as a version with the Deployment Metadata Service (DMS), gated by experimental.record_deployment_history. DMS recording is a separate concern from the deployment lock: it lives in its own libs/dms package and does not acquire any lock. The existing workspace-filesystem lock (lock.Acquire / lock.Release) is used as-is; the DMS version is created after the lock is acquired and completed before it is released. - libs/dms: Recorder with CreateVersion / CompleteVersion. The deployment ID is the bundle's state lineage, so a bundle deployment maps one-to-one to a DMS deployment record. GetDeployment first, CreateDeployment only when missing, then create the next version (server validates version_id == last+1). A heartbeat goroutine keeps the version's lease alive; CompleteVersion records success/failure and, for destroy, deletes the deployment record on success. Independent of bundle/lock — takes the SDK client, deployment ID, target, and version type directly. A nil Recorder is a no-op. - direct/dstate: GetOrInitLineage returns the state lineage, generating and storing one on the first deploy so DMS and the deployment engine use the same lineage value (DMS records the version at lock time, before the engine assigns the lineage at plan-apply time). - phases: newDeploymentRecorder builds the recorder from the bundle (nil unless the flag is set); deploy/destroy create a version after acquiring the lock and complete it in the deferred release. - libs/testserver: in-memory DMS handlers under /api/2.0/bundle/... - acceptance/bundle/dms: deploy/redeploy/destroy record versions and apply the file lock; redeploy after deleting .databricks recovers the lineage from remote state; enabling the flag after a plain deploy creates a new deployment. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
1 parent 33a489c commit 87c99e8

15 files changed

Lines changed: 491 additions & 1 deletion

File tree

‎acceptance/bundle/dms/enable-after-deploy/databricks.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
bundle:
2+
name: dms-enable-after-deploy
3+
4+
resources:
5+
jobs:
6+
foo:
7+
name: foo

‎acceptance/bundle/dms/enable-after-deploy/out.test.toml‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/dms/enable-after-deploy/output.txt‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
2+
=== Deploy without the feature: no DMS calls are made
3+
>>> [CLI] bundle deploy
4+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-enable-after-deploy/default/files...
5+
Deploying resources...
6+
Updating deployment state...
7+
Deployment complete!
8+
9+
>>> print_requests.py //api/2.0/bundle --keep
10+
11+
>>> print_requests.py //deploy.lock
12+
{
13+
"method": "POST",
14+
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/dms-enable-after-deploy/default/state/deploy.lock",
15+
"q": {
16+
"overwrite": "false"
17+
},
18+
"body": {
19+
"ID": "[UUID]",
20+
"AcquisitionTime": "[TIMESTAMP]",
21+
"IsForced": false,
22+
"User": "[USERNAME]"
23+
}
24+
}
25+
26+
=== Enable experimental.record_deployment_history
27+
=== Delete local cache (.databricks) so the next deploy cannot rely on it
28+
=== Next deploy creates a DMS deployment + version 1 (lineage recovered from remote state, not local cache)
29+
>>> [CLI] bundle deploy
30+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-enable-after-deploy/default/files...
31+
Deploying resources...
32+
Updating deployment state...
33+
Deployment complete!
34+
35+
>>> print_requests.py //api/2.0/bundle --keep
36+
{
37+
"method": "POST",
38+
"path": "/api/2.0/bundle/deployments",
39+
"q": {
40+
"deployment_id": "[UUID]"
41+
},
42+
"body": {
43+
"target_name": "default"
44+
}
45+
}
46+
{
47+
"method": "POST",
48+
"path": "/api/2.0/bundle/deployments/[UUID]/versions",
49+
"q": {
50+
"version_id": "1"
51+
},
52+
"body": {
53+
"cli_version": "[DEV_VERSION]",
54+
"target_name": "default",
55+
"version_type": "VERSION_TYPE_DEPLOY"
56+
}
57+
}
58+
{
59+
"method": "POST",
60+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/1/complete",
61+
"body": {
62+
"completion_reason": "VERSION_COMPLETE_SUCCESS"
63+
}
64+
}
65+
66+
>>> print_requests.py //deploy.lock
67+
{
68+
"method": "POST",
69+
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/dms-enable-after-deploy/default/state/deploy.lock",
70+
"q": {
71+
"overwrite": "false"
72+
},
73+
"body": {
74+
"ID": "[UUID]",
75+
"AcquisitionTime": "[TIMESTAMP]",
76+
"IsForced": false,
77+
"User": "[USERNAME]"
78+
}
79+
}
80+
81+
>>> [CLI] bundle destroy --auto-approve
82+
The following resources will be deleted:
83+
delete resources.jobs.foo
84+
85+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-enable-after-deploy/default
86+
87+
Deleting files...
88+
Destroy complete!

‎acceptance/bundle/dms/enable-after-deploy/script‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
cleanup() {
2+
trace $CLI bundle destroy --auto-approve
3+
rm -f out.requests.txt
4+
}
5+
trap cleanup EXIT
6+
7+
title "Deploy without the feature: no DMS calls are made"
8+
trace $CLI bundle deploy
9+
trace print_requests.py //api/2.0/bundle --keep
10+
trace print_requests.py //deploy.lock
11+
12+
title "Enable experimental.record_deployment_history"
13+
cat >> databricks.yml <<'YAML'
14+
15+
experimental:
16+
record_deployment_history: true
17+
YAML
18+
19+
title "Delete local cache (.databricks) so the next deploy cannot rely on it"
20+
rm -rf .databricks
21+
22+
title "Next deploy creates a DMS deployment + version 1 (lineage recovered from remote state, not local cache)"
23+
trace $CLI bundle deploy
24+
trace print_requests.py //api/2.0/bundle --keep
25+
trace print_requests.py //deploy.lock

‎acceptance/bundle/dms/record/databricks.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
bundle:
2+
name: dms-record
3+
4+
experimental:
5+
record_deployment_history: true
6+
7+
resources:
8+
jobs:
9+
foo:
10+
name: foo

‎acceptance/bundle/dms/record/out.test.toml‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/dms/record/output.txt‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
2+
=== Deploy: a DMS deployment + version are recorded while the file lock is held
3+
>>> [CLI] bundle deploy
4+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-record/default/files...
5+
Deploying resources...
6+
Updating deployment state...
7+
Deployment complete!
8+
9+
=== DMS API calls made during deploy (create deployment, create + complete version)
10+
>>> print_requests.py //api/2.0/bundle --keep
11+
{
12+
"method": "POST",
13+
"path": "/api/2.0/bundle/deployments",
14+
"q": {
15+
"deployment_id": "[UUID]"
16+
},
17+
"body": {
18+
"target_name": "default"
19+
}
20+
}
21+
{
22+
"method": "POST",
23+
"path": "/api/2.0/bundle/deployments/[UUID]/versions",
24+
"q": {
25+
"version_id": "1"
26+
},
27+
"body": {
28+
"cli_version": "[DEV_VERSION]",
29+
"target_name": "default",
30+
"version_type": "VERSION_TYPE_DEPLOY"
31+
}
32+
}
33+
{
34+
"method": "POST",
35+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/1/complete",
36+
"body": {
37+
"completion_reason": "VERSION_COMPLETE_SUCCESS"
38+
}
39+
}
40+
41+
=== The workspace-filesystem lock is applied alongside DMS (deploy.lock written)
42+
>>> print_requests.py //deploy.lock
43+
{
44+
"method": "POST",
45+
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/dms-record/default/state/deploy.lock",
46+
"q": {
47+
"overwrite": "false"
48+
},
49+
"body": {
50+
"ID": "[UUID]",
51+
"AcquisitionTime": "[TIMESTAMP]",
52+
"IsForced": false,
53+
"User": "[USERNAME]"
54+
}
55+
}
56+
57+
=== Redeploy after deleting local cache (.databricks): the lineage is recovered from remote state, so the same deployment is reused and the version increments (no new CreateDeployment)
58+
>>> [CLI] bundle deploy
59+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-record/default/files...
60+
Deploying resources...
61+
Updating deployment state...
62+
Deployment complete!
63+
64+
>>> print_requests.py //api/2.0/bundle --keep
65+
{
66+
"method": "POST",
67+
"path": "/api/2.0/bundle/deployments/[UUID]/versions",
68+
"q": {
69+
"version_id": "2"
70+
},
71+
"body": {
72+
"cli_version": "[DEV_VERSION]",
73+
"target_name": "default",
74+
"version_type": "VERSION_TYPE_DEPLOY"
75+
}
76+
}
77+
{
78+
"method": "POST",
79+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/2/complete",
80+
"body": {
81+
"completion_reason": "VERSION_COMPLETE_SUCCESS"
82+
}
83+
}
84+
85+
>>> print_requests.py //deploy.lock
86+
{
87+
"method": "POST",
88+
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/dms-record/default/state/deploy.lock",
89+
"q": {
90+
"overwrite": "false"
91+
},
92+
"body": {
93+
"ID": "[UUID]",
94+
"AcquisitionTime": "[TIMESTAMP]",
95+
"IsForced": false,
96+
"User": "[USERNAME]"
97+
}
98+
}
99+
100+
=== Destroy: a DMS version is recorded while the file lock is held
101+
>>> [CLI] bundle destroy --auto-approve
102+
The following resources will be deleted:
103+
delete resources.jobs.foo
104+
105+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-record/default
106+
107+
Deleting files...
108+
Destroy complete!
109+
110+
>>> print_requests.py //api/2.0/bundle --keep
111+
{
112+
"method": "POST",
113+
"path": "/api/2.0/bundle/deployments/[UUID]/versions",
114+
"q": {
115+
"version_id": "3"
116+
},
117+
"body": {
118+
"cli_version": "[DEV_VERSION]",
119+
"target_name": "default",
120+
"version_type": "VERSION_TYPE_DESTROY"
121+
}
122+
}
123+
{
124+
"method": "POST",
125+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/3/complete",
126+
"body": {
127+
"completion_reason": "VERSION_COMPLETE_SUCCESS"
128+
}
129+
}
130+
{
131+
"method": "DELETE",
132+
"path": "/api/2.0/bundle/deployments/[UUID]"
133+
}
134+
135+
>>> print_requests.py //deploy.lock
136+
{
137+
"method": "POST",
138+
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/dms-record/default/state/deploy.lock",
139+
"q": {
140+
"overwrite": "false"
141+
},
142+
"body": {
143+
"ID": "[UUID]",
144+
"AcquisitionTime": "[TIMESTAMP]",
145+
"IsForced": false,
146+
"User": "[USERNAME]"
147+
}
148+
}

‎acceptance/bundle/dms/record/script‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
cleanup() {
2+
title "Destroy: a DMS version is recorded while the file lock is held"
3+
trace $CLI bundle destroy --auto-approve
4+
trace print_requests.py //api/2.0/bundle --keep
5+
trace print_requests.py //deploy.lock
6+
rm -f out.requests.txt
7+
}
8+
trap cleanup EXIT
9+
10+
title "Deploy: a DMS deployment + version are recorded while the file lock is held"
11+
trace $CLI bundle deploy
12+
13+
title "DMS API calls made during deploy (create deployment, create + complete version)"
14+
trace print_requests.py //api/2.0/bundle --keep
15+
16+
title "The workspace-filesystem lock is applied alongside DMS (deploy.lock written)"
17+
trace print_requests.py //deploy.lock
18+
19+
title "Redeploy after deleting local cache (.databricks): the lineage is recovered from remote state, so the same deployment is reused and the version increments (no new CreateDeployment)"
20+
rm -rf .databricks
21+
trace $CLI bundle deploy
22+
trace print_requests.py //api/2.0/bundle --keep
23+
trace print_requests.py //deploy.lock

‎acceptance/bundle/dms/test.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Deployment Metadata Service (DMS) recording is only meaningful in the direct
2+
# engine, where the deployment ID is derived from the state lineage.
3+
EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]
4+
5+
RecordRequests = true

‎bundle/direct/dstate/state.go‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,26 @@ func (db *DeploymentState) GetResourceID(key string) string {
147147
return db.stateIDs[key]
148148
}
149149

150+
// GetOrInitLineage returns the state lineage, generating and storing a new one
151+
// in memory if the state does not have a lineage yet (fresh deployment). Storing
152+
// it in Data.Lineage means a subsequent UpgradeToWrite persists the same value,
153+
// so the lineage observed here matches the one the deployment engine later writes
154+
// to the state file.
155+
//
156+
// This lets DMS use the same lineage value as the deployment engine for the
157+
// deployment ID. It is necessary because DMS creates the version/deployment at
158+
// lock time, which is before the engine assigns the lineage at plan-apply time;
159+
// without seeding it here the two would diverge on the first deploy.
160+
func (db *DeploymentState) GetOrInitLineage() string {
161+
db.mu.Lock()
162+
defer db.mu.Unlock()
163+
164+
if db.Data.Lineage == "" {
165+
db.Data.Lineage = uuid.New().String()
166+
}
167+
return db.Data.Lineage
168+
}
169+
150170
type (
151171
// If true, then Open reads the WAL and merges it in the state. If false, and WAL is present, Open returns an error.
152172
WithRecovery bool

0 commit comments

Comments
 (0)