Skip to content

Commit 6f39c1b

Browse files
authored
fix: pass --s3-no-head-object to rclone for dataset uploads (#146)
rclone v1.72 changed how its S3 backend handles a 403 from the FS-init HeadObject probe (rclone/rclone#8975, commit rclone/rclone@6440052f, 2025-11-18): non-NotFound errors now propagate as fatal NewFs failures instead of being silently treated as "this is a directory". For BlobTree dataset uploads the platform's session credentials are scoped to <prefix>/<version>/* and HEAD on the bare <version> key is denied — so once Rclone_jll resolved to >= v1.72 (currently v1.73.5), every BlobTree upload aborted with `JuliaHubError: Data upload failed`. The destination is always a directory in our usage, so the probe serves no purpose; suppress it with --s3-no-head-object. Verified locally against internal.juliahub.com with Rclone_jll v1.73.5: the live datasets testset now passes end-to-end, where it previously failed at datasets-live.jl:70 (BlobTree upload). The flag has existed in rclone since v1.55 (2021), so no compat impact.
1 parent 5a4e3de commit 6f39c1b

1 file changed

Lines changed: 10 additions & 4 deletions

File tree

src/datasets.jl

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -814,18 +814,24 @@ function _upload_dataset(upload_config, local_path; progress::Bool)
814814
remote_path = "$bucket/$prefix"
815815

816816
# --s3-no-check-bucket - don't check the bucket exists
817+
# --s3-no-head - skip the post-upload HeadObject verification
818+
# --s3-no-head-object - skip the FS-init HeadObject probe rclone
819+
# uses to detect "is this destination an existing file?". The
820+
# platform's session credentials are scoped to <prefix>/<version>/*
821+
# for BlobTree uploads and reject HEAD on the bare <version> key
822+
# with 403. From rclone v1.72 (commit rclone/rclone@6440052f,
823+
# issue rclone/rclone#8975) that 403 propagates as a fatal NewFs
824+
# error instead of being swallowed; older versions silently
825+
# treated it as "destination is a directory" and proceeded.
817826
# --no-check-dest - don't check whether the file exists before uploading
818827
#
819828
# Additional useful options not included here:
820-
# * For restricted permissions, --s3-no-head avoids using HeadObject to
821-
# check file upload success.
822829
# * To force multipart upload at a smaller threshold use something like
823830
# --s3-upload-cutoff 1M --s3-chunk-size 5M
824-
825-
# FIXME: remove `--s3-no-head` once policies are figured out (again)
826831
args = [
827832
"--s3-no-check-bucket",
828833
"--s3-no-head",
834+
"--s3-no-head-object",
829835
"--no-check-dest",
830836
]
831837

0 commit comments

Comments
 (0)