Skip to content

Commit 3bff4ca

Browse files
acceptance: replace TestGenerateFromExistingJobAndDeploy (#5190)
Moves `TestGenerateFromExistingJobAndDeploy` from `integration/` to `acceptance/`. Acceptance tests are the only path that wires up terraform installs now, so the integration variant can't run end-to-end. The new test runs the same flow (upload notebook → create job → `bundle generate job` → `bundle deploy` → `bundle destroy`) on both modes: - `localupdate`: ~5s - `cloudupdate` (azure-prod-ucws, terraform + direct): ~3-4m Also fixes a Windows path-separator bug in `bundle/generate/downloader.go` that the new test caught: `filepath.Rel` returned `..\src\test_notebook.py` on Windows, so the generated YAML had OS-native separators. Mirrors the existing `filepath.ToSlash` pattern in `pipeline.go`. This pull request was AI-assisted by Isaac. We did not have coverage for generate + deploy in our existing tests, that's why I translated it instead. --------- Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
1 parent 9f7db0a commit 3bff4ca

7 files changed

Lines changed: 89 additions & 116 deletions

File tree

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
bundle:
2+
name: python_job_and_deploy

acceptance/bundle/generate/python_job_and_deploy/out.test.toml

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
2+
=== Upload notebook to a workspace path
3+
>>> [CLI] workspace import /Workspace/Users/[USERNAME]/test_notebook.py --file test_notebook.py --format AUTO --overwrite
4+
5+
=== Create a job that references the notebookCreated job
6+
7+
=== Generate bundle config from the job
8+
>>> [CLI] bundle generate job --existing-job-id [JOB_ID] --key out --config-dir resources --source-dir src --force
9+
File successfully saved to src/test_notebook.py
10+
Job configuration successfully saved to resources/out.job.yml
11+
12+
=== Verify generated yaml has expected fields
13+
=== Deploy the generated bundle
14+
>>> [CLI] bundle deploy
15+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/python_job_and_deploy/default/files...
16+
Deploying resources...
17+
Deployment complete!
18+
19+
=== Destroy the deployed bundle
20+
>>> [CLI] bundle destroy --auto-approve
21+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/python_job_and_deploy/default
22+
23+
Deleting files...
24+
Destroy complete!
25+
26+
=== Cleanup: delete the original job and notebook
27+
>>> errcode [CLI] jobs delete [JOB_ID]
28+
29+
>>> errcode [CLI] workspace delete /Workspace/Users/[USERNAME]/test_notebook
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
title "Upload notebook to a workspace path"
2+
trace $CLI workspace import "/Workspace/Users/${CURRENT_USER_NAME}/test_notebook.py" --file test_notebook.py --format AUTO --overwrite
3+
4+
title "Create a job that references the notebook"
5+
JOB_ID=$($CLI jobs create --json '{
6+
"name": "test-job",
7+
"max_concurrent_runs": 1,
8+
"queue": {"enabled": true},
9+
"tasks": [
10+
{
11+
"task_key": "test_task",
12+
"notebook_task": {
13+
"notebook_path": "/Workspace/Users/'${CURRENT_USER_NAME}'/test_notebook"
14+
}
15+
}
16+
]
17+
}' | jq -r '.job_id')
18+
echo "Created job"
19+
# Disable MSYS_NO_PATHCONV when invoking python scripts: with it set, Git Bash on Windows
20+
# fails to translate the script path so the python interpreter can't find the file.
21+
env -u MSYS_NO_PATHCONV add_repl.py "$JOB_ID" JOB_ID
22+
23+
cleanup() {
24+
title "Cleanup: delete the original job and notebook"
25+
trace errcode $CLI jobs delete "$JOB_ID"
26+
trace errcode $CLI workspace delete "/Workspace/Users/${CURRENT_USER_NAME}/test_notebook"
27+
}
28+
trap cleanup EXIT
29+
30+
title "Generate bundle config from the job"
31+
trace $CLI bundle generate job --existing-job-id "$JOB_ID" --key out --config-dir resources --source-dir src --force
32+
33+
title "Verify generated yaml has expected fields"
34+
cat resources/out.job.yml | env -u MSYS_NO_PATHCONV contains.py "task_key: test_task" "notebook_task:" "notebook_path: ../src/test_notebook.py" > /dev/null
35+
36+
title "Deploy the generated bundle"
37+
trace $CLI bundle deploy
38+
39+
title "Destroy the deployed bundle"
40+
trace $CLI bundle destroy --auto-approve
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
Local = true
2+
Cloud = true
3+
4+
Ignore = [
5+
"databricks.yml",
6+
"resources/*",
7+
"src/*",
8+
".databricks",
9+
]
10+
11+
[Env]
12+
# MSYS2 automatically converts absolute paths on Windows; disable for the workspace path.
13+
MSYS_NO_PATHCONV = "1"
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Databricks notebook source
2+
print("Hello, World!")

integration/bundle/generate_job_test.go

Lines changed: 0 additions & 116 deletions
This file was deleted.

0 commit comments

Comments
 (0)