Skip to content

Commit e86aeba

Browse files
filer: switch workspace upload from import-file to /workspace/import
Replace `POST /api/2.0/workspace-files/import-file/{path}?overwrite=…` with the multipart variant of `POST /api/2.0/workspace/import` (via the SDK's `Workspace.Upload` + `format=AUTO`). The legacy endpoint is being deprecated and the new endpoint is the strategic replacement. The multipart variant is required because the JSON body of /workspace/import is server-capped at 10 MiB; multipart accepts the same sizes import-file did (verified up to 250 MiB against a real workspace), so DAB users shipping wheels/jars/large files keep working. Error mapping uses SDK sentinels via errors.Is rather than raw status/error code comparisons: - ErrNotFound → noSuchDirectoryError (or mkdir+retry under CreateParentDirectories mode); also catches RESOURCE_DOES_NOT_EXIST. - ErrResourceAlreadyExists → fileAlreadyExistsError (the new endpoint reliably sets error_code RESOURCE_ALREADY_EXISTS). - ErrInvalidParameterValue + "Requested node type" message → fileAlreadyExistsError (existing object's node type doesn't match the upload — file vs notebook collision). - ErrPermissionDenied → permissionError. Apply the same sentinel-based pattern to Delete, ReadDir, and Stat for 404 detection, matching the existing usage in bundle/direct/util.go and following AGENTS.md's rule against branching on err.Error() string content. DIRECTORY_NOT_EMPTY in Delete keeps an explicit ErrorCode check since the SDK has no sentinel for it. Test plan: - libs/filer/workspace_files_client_test.go covers the new error mapping. - libs/testserver/handlers.go extended with a multipart handler at /workspace/import that surfaces 409s from the shared fake as 400 + RESOURCE_ALREADY_EXISTS to match real-workspace shape. - acceptance/internal/prepare_server.go normalizes multipart bodies (sorted form fields, file parts recorded as {filename, size}) so request fixtures stay deterministic. - ~70 acceptance fixtures regenerated for the new request shape. - End-to-end verified against a real workspace for files, all notebook types (.py / .sql / .ipynb / .lvdash.json / .scala / .r), dashboards, and a 60 MB binary upload. - Integration test TestFilerWorkspaceNotebook assertion updated to assert the path with extension (tc.name) — matches the absPath returned in fileAlreadyExistsError. Same change as #5106. Co-authored-by: Isaac
1 parent 796a01d commit e86aeba

84 files changed

Lines changed: 612 additions & 972 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

acceptance/acceptance_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,7 @@ func testAccept(t *testing.T, inprocessMode bool, singleTest string) int {
250250
// Skip building yamlfmt when running on workspace filesystem (DBR).
251251
// This fails today on DBR. Can be looked into and fixed as a follow-up
252252
// as and when needed.
253-
if !WorkspaceTmpDir {
253+
if !WorkspaceTmpDir && os.Getenv("SKIP_BUILD_YAMLFMT") == "" {
254254
BuildYamlfmt(t)
255255
}
256256

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,12 @@
11
{
22
"method": "POST",
3-
"path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/app/app.yml",
4-
"q": {
5-
"overwrite": "true"
6-
},
7-
"raw_body": "command:\n - python\n - app.py\n"
3+
"path": "/api/2.0/workspace/import",
4+
"body": {
5+
"multipart_form": {
6+
"content": "[content]",
7+
"format": "AUTO",
8+
"overwrite": "true",
9+
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/app/app.yml"
10+
}
11+
}
812
}

acceptance/bundle/apps/app_yaml/output.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Deploying resources...
1919
Updating deployment state...
2020
Deployment complete!
2121

22-
>>> jq select(.path | test("app.yml")) out.requests.txt
22+
>>> jq select(.body.multipart_form.path | strings | test("app.yml")) out.requests.txt
2323

2424
>>> [CLI] bundle destroy --auto-approve
2525
The following resources will be deleted:

acceptance/bundle/apps/app_yaml/script

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
trace $CLI bundle validate
22
trace $CLI bundle plan
33
trace $CLI bundle deploy
4-
trace jq 'select(.path | test("app.yml"))' out.requests.txt | sed 's/\\r//g' > out.app.yml.txt
4+
trace jq 'select(.body.multipart_form.path | strings | test("app.yml"))' out.requests.txt | sed 's/\\r//g' > out.app.yml.txt
55
#trace print_requests.py //apps # currently fails due to TF inserting description=""
66
rm out.requests.txt
77

acceptance/bundle/artifacts/artifact_upload_for_workspace/output.txt

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,9 +50,9 @@ Deployment complete!
5050
]
5151

5252
=== Expecting wheel to be uploaded
53-
>>> jq .path
54-
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source.whl"
55-
"/api/2.0/workspace-files/import-file/Workspace/foo/bar/artifacts/.internal/source.whl"
53+
>>> jq -r .body.multipart_form.path | strings
54+
/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source.whl
55+
/Workspace/foo/bar/artifacts/.internal/source.whl
5656

5757
=== Expecting environment dependencies to be updated
5858
>>> jq -s .[] | select(.path=="/api/2.2/jobs/create") | .body.environments out.requests.txt

acceptance/bundle/artifacts/artifact_upload_for_workspace/script

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ title "Expecting 2 wheels in libraries section in /jobs/create"
88
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body.tasks' out.requests.txt
99

1010
title "Expecting wheel to be uploaded"
11-
trace jq .path < out.requests.txt | grep import | grep whl | sort
11+
trace jq -r '.body.multipart_form.path | strings' < out.requests.txt | grep whl | sort
1212

1313
title "Expecting environment dependencies to be updated"
1414
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body.environments' out.requests.txt

acceptance/bundle/artifacts/artifact_upload_with_no_library_reference/output.txt

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,9 @@ Deploying resources...
66
Deployment complete!
77

88
=== Expecting wheel to be uploaded
9-
>>> jq .path
10-
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source.whl"
11-
"/api/2.0/workspace-files/import-file/Workspace/foo/bar/artifacts/.internal/source.whl"
9+
>>> jq -r .body.multipart_form.path | strings
10+
/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source.whl
11+
/Workspace/foo/bar/artifacts/.internal/source.whl
1212

1313
=== Expecting delete request to artifact_path/.internal folder
1414
>>> jq -s .[] | select(.path=="/api/2.0/workspace/delete") | select(.body.path | test(".*/artifacts/.internal")) out.requests.txt

acceptance/bundle/artifacts/artifact_upload_with_no_library_reference/script

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ echo "test wheel content" > whl/source.whl
55
trace $CLI bundle deploy
66

77
title "Expecting wheel to be uploaded"
8-
trace jq .path < out.requests.txt | grep import | grep whl | sort
8+
trace jq -r '.body.multipart_form.path | strings' < out.requests.txt | grep whl | sort
99

1010
title "Expecting delete request to artifact_path/.internal folder"
1111
trace jq -s '.[] | select(.path=="/api/2.0/workspace/delete") | select(.body.path | test(".*/artifacts/.internal"))' out.requests.txt

acceptance/bundle/artifacts/upload_multiple_libraries/output.txt

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -40,15 +40,15 @@ Deployment complete!
4040
]
4141

4242
=== Expecting 4 wheels to be uploaded
43-
>>> jq .path
44-
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source1.whl"
45-
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source2.whl"
46-
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source3.whl"
47-
"/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source4.whl"
48-
"/api/2.0/workspace-files/import-file/Workspace/foo/bar/artifacts/.internal/source1.whl"
49-
"/api/2.0/workspace-files/import-file/Workspace/foo/bar/artifacts/.internal/source2.whl"
50-
"/api/2.0/workspace-files/import-file/Workspace/foo/bar/artifacts/.internal/source3.whl"
51-
"/api/2.0/workspace-files/import-file/Workspace/foo/bar/artifacts/.internal/source4.whl"
43+
>>> jq -r .body.multipart_form.path | strings
44+
/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source1.whl
45+
/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source2.whl
46+
/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source3.whl
47+
/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files/whl/source4.whl
48+
/Workspace/foo/bar/artifacts/.internal/source1.whl
49+
/Workspace/foo/bar/artifacts/.internal/source2.whl
50+
/Workspace/foo/bar/artifacts/.internal/source3.whl
51+
/Workspace/foo/bar/artifacts/.internal/source4.whl
5252

5353
=== Expecting environment dependencies to be updated
5454
>>> jq -s .[] | select(.path=="/api/2.2/jobs/create") | .body.environments out.requests.txt

acceptance/bundle/artifacts/upload_multiple_libraries/script

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ title "Expecting 5 wheels in libraries section in /jobs/create"
1111
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body.tasks' out.requests.txt
1212

1313
title "Expecting 4 wheels to be uploaded"
14-
trace jq .path < out.requests.txt | grep import | grep whl | sort
14+
trace jq -r '.body.multipart_form.path | strings' < out.requests.txt | grep whl | sort
1515

1616
title "Expecting environment dependencies to be updated"
1717
trace jq -s '.[] | select(.path=="/api/2.2/jobs/create") | .body.environments' out.requests.txt

0 commit comments

Comments
 (0)