Skip to content

Commit 63b717b

Browse files
denikdenik
authored
direct: Support partial plan/deploy with --select (#5413)
## Changes New --select option for plan & deploy that allows selecting one or more resources to deploy. The dependencies are included as well. File uploads are done in full as with usual deploy. Direct engine only. ## Why If you're working on a single resource from the target and target contains many jobs, sometimes you don't want to pollute your workspace with (/ wait for) other resources. ## Tests New acceptance and unit tests.
1 parent 8651cac commit 63b717b

32 files changed

Lines changed: 563 additions & 6 deletions

‎NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,6 @@
1111
* Retry transient HTTP 5xx and 408 errors in direct deployment engine ([#5349](https://github.com/databricks/cli/pull/5349), [#5364](https://github.com/databricks/cli/pull/5364)).
1212
* Preserve `.designer.ipynb` suffix when translating notebook task paths so Lakeflow Designer files referenced from a `notebook_task` resolve correctly in the workspace ([#5370](https://github.com/databricks/cli/pull/5370)).
1313
* Fix script output dropping last line without trailing newline ([#4995](https://github.com/databricks/cli/pull/4995)).
14+
* Add `--select` flag to `bundle plan` and `bundle deploy` to plan/deploy a subset of resources (e.g. `--select my_job` or `--select jobs.my_job`); resources referenced by the selection are included transitively. Direct engine only ([#5413](https://github.com/databricks/cli/pull/5413)).
1415

1516
### Dependency updates

‎acceptance/bundle/help/bundle-deploy/output.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Flags:
2020
--force-lock Force acquisition of deployment lock.
2121
-h, --help help for deploy
2222
--plan string Path to a JSON plan file to apply instead of planning (direct engine only).
23+
--select strings Deploy only the specified resource (e.g. 'my_job' or 'jobs.my_job'). Can be repeated or comma-separated.
2324

2425
Global Flags:
2526
--debug enable debug logging

‎acceptance/bundle/select/ambiguous/databricks.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
bundle:
2+
name: select-ambiguous
3+
4+
# "thing" is defined as both a job and a pipeline. This is what an ambiguous
5+
# --select selector would resolve to, but UniqueResourceKeys rejects it at load
6+
# time, so the bundle never reaches selector resolution.
7+
resources:
8+
jobs:
9+
thing:
10+
name: job-thing
11+
pipelines:
12+
thing:
13+
name: pipeline-thing

‎acceptance/bundle/select/ambiguous/out.test.toml‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/select/ambiguous/output.txt‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
2+
>>> musterr [CLI] bundle plan --select thing
3+
Error: multiple resources or scripts have been defined with the same key: thing
4+
at resources.jobs.thing
5+
resources.pipelines.thing
6+
in databricks.yml:10:7
7+
databricks.yml:13:7
8+
9+
10+
>>> musterr [CLI] bundle deploy --select thing
11+
Error: multiple resources or scripts have been defined with the same key: thing
12+
at resources.jobs.thing
13+
resources.pipelines.thing
14+
in databricks.yml:10:7
15+
databricks.yml:13:7
16+

‎acceptance/bundle/select/ambiguous/script‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
trace musterr $CLI bundle plan --select thing
2+
trace musterr $CLI bundle deploy --select thing

‎acceptance/bundle/select/basic/databricks.yml.tmpl‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
bundle:
2+
name: select-$UNIQUE_NAME
3+
4+
resources:
5+
jobs:
6+
bar:
7+
name: bar-$UNIQUE_NAME
8+
9+
foo:
10+
name: foo-$UNIQUE_NAME
11+
tasks:
12+
- task_key: run_bar
13+
run_job_task:
14+
job_id: ${resources.jobs.bar.id}
15+
16+
baz:
17+
name: baz-$UNIQUE_NAME

‎acceptance/bundle/select/basic/out.test.toml‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/select/basic/output.txt‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
2+
>>> [CLI] bundle plan --select bar
3+
create jobs.bar
4+
5+
Plan: 1 to add, 0 to change, 0 to delete, 0 unchanged
6+
7+
>>> [CLI] bundle plan --select jobs.bar --select jobs.baz
8+
create jobs.bar
9+
create jobs.baz
10+
11+
Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged
12+
13+
>>> [CLI] bundle plan --select jobs.bar,jobs.baz
14+
create jobs.bar
15+
create jobs.baz
16+
17+
Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged
18+
19+
>>> [CLI] bundle plan --select jobs.foo
20+
create jobs.bar
21+
create jobs.foo
22+
23+
Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged
24+
25+
=== bundle deploy --select jobs.foo
26+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/files...
27+
Deploying resources...
28+
Updating deployment state...
29+
Deployment complete!
30+
31+
=== Telemetry:
32+
select_used true
33+
34+
>>> print_requests.py --sort //jobs
35+
{
36+
"method": "POST",
37+
"path": "/api/2.2/jobs/create",
38+
"body": {
39+
"deployment": {
40+
"kind": "BUNDLE",
41+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/state/metadata.json"
42+
},
43+
"edit_mode": "UI_LOCKED",
44+
"format": "MULTI_TASK",
45+
"max_concurrent_runs": 1,
46+
"name": "bar-[UNIQUE_NAME]",
47+
"queue": {
48+
"enabled": true
49+
}
50+
}
51+
}
52+
{
53+
"method": "POST",
54+
"path": "/api/2.2/jobs/create",
55+
"body": {
56+
"deployment": {
57+
"kind": "BUNDLE",
58+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/state/metadata.json"
59+
},
60+
"edit_mode": "UI_LOCKED",
61+
"format": "MULTI_TASK",
62+
"max_concurrent_runs": 1,
63+
"name": "foo-[UNIQUE_NAME]",
64+
"queue": {
65+
"enabled": true
66+
},
67+
"tasks": [
68+
{
69+
"run_job_task": {
70+
"job_id": [NUMID]
71+
},
72+
"task_key": "run_bar"
73+
}
74+
]
75+
}
76+
}
77+
78+
>>> [CLI] bundle summary
79+
Name: select-[UNIQUE_NAME]
80+
Target: default
81+
Workspace:
82+
User: [USERNAME]
83+
Path: /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default
84+
Resources:
85+
Jobs:
86+
bar:
87+
Name: bar-[UNIQUE_NAME]
88+
URL: [DATABRICKS_URL]/jobs/[NUMID]?w=[NUMID]
89+
baz:
90+
Name: baz-[UNIQUE_NAME]
91+
URL: (not deployed)
92+
foo:
93+
Name: foo-[UNIQUE_NAME]
94+
URL: [DATABRICKS_URL]/jobs/[NUMID]?w=[NUMID]
95+
96+
=== Full plan after partial deploy
97+
>>> [CLI] bundle plan
98+
create jobs.baz
99+
100+
Plan: 1 to add, 0 to change, 0 to delete, 2 unchanged
101+
102+
=== Full deploy
103+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/files...
104+
Deploying resources...
105+
Updating deployment state...
106+
Deployment complete!
107+
108+
>>> print_requests.py --sort //jobs
109+
{
110+
"method": "POST",
111+
"path": "/api/2.2/jobs/create",
112+
"body": {
113+
"deployment": {
114+
"kind": "BUNDLE",
115+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default/state/metadata.json"
116+
},
117+
"edit_mode": "UI_LOCKED",
118+
"format": "MULTI_TASK",
119+
"max_concurrent_runs": 1,
120+
"name": "baz-[UNIQUE_NAME]",
121+
"queue": {
122+
"enabled": true
123+
}
124+
}
125+
}
126+
127+
=== Full plan again
128+
>>> [CLI] bundle plan
129+
Plan: 0 to add, 0 to change, 0 to delete, 3 unchanged
130+
131+
=== Destroy
132+
>>> [CLI] bundle destroy --auto-approve
133+
The following resources will be deleted:
134+
delete resources.jobs.bar
135+
delete resources.jobs.baz
136+
delete resources.jobs.foo
137+
138+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/select-[UNIQUE_NAME]/default
139+
140+
Deleting files...
141+
Destroy complete!

‎acceptance/bundle/select/basic/script‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
envsubst '$UNIQUE_NAME' < databricks.yml.tmpl > databricks.yml
2+
3+
cleanup() {
4+
title "Destroy"
5+
trace $CLI bundle destroy --auto-approve
6+
rm -f out.requests.txt
7+
}
8+
trap cleanup EXIT
9+
10+
# --- selector resolution ---
11+
# Unqualified name, unique across resource types.
12+
trace $CLI bundle plan --select bar
13+
# Repeated --select.
14+
trace $CLI bundle plan --select jobs.bar --select jobs.baz
15+
# Comma-separated --select (equivalent to repeating the flag).
16+
trace $CLI bundle plan --select jobs.bar,jobs.baz
17+
# Qualified name; foo pulls in its dependency bar but not the independent baz.
18+
trace $CLI bundle plan --select jobs.foo
19+
20+
# --- partial deploy ---
21+
# Serialize the filtered plan to a temp file (kept out of the recorded output: the
22+
# JSON embeds remote state that differs between local and cloud), then deploy it:
23+
# inline, or via --plan (READPLAN=1). The deploy is not traced because readplanarg
24+
# varies the command line between READPLAN variants, which must produce identical output.
25+
$CLI bundle plan --select jobs.foo -o json > plan.json
26+
title "bundle deploy --select jobs.foo\n"
27+
$CLI bundle deploy --select jobs.foo $(readplanarg plan.json)
28+
# The deploy reports that --select was used via telemetry.
29+
title "Telemetry:\n"
30+
print_telemetry_bool_values | grep '^select_used '
31+
# Only bar and foo were created, never baz.
32+
trace print_requests.py --sort //jobs
33+
# Summary after the partial deploy: foo and bar are deployed, baz is not.
34+
trace $CLI bundle summary
35+
36+
# --- full plan/deploy after the partial deploy ---
37+
# foo and bar are already deployed, so only baz remains to create.
38+
title "Full plan after partial deploy"
39+
trace $CLI bundle plan
40+
$CLI bundle plan -o json > plan-full.json
41+
title "Full deploy\n"
42+
$CLI bundle deploy $(readplanarg plan-full.json)
43+
# Only baz is created this time.
44+
trace print_requests.py --sort //jobs
45+
# Everything is deployed now: no changes.
46+
title "Full plan again"
47+
trace $CLI bundle plan

0 commit comments

Comments
 (0)