[Storage] Add Hugging Face Buckets and repos as a storage backend

[XciD]

This is a continuation of #9418 by @nikhiljha, opened directly against master to ease the review process. The original commits are preserved in history.

Summary

Adds a new hf storage type that wraps Hugging Face Buckets (Xet-backed S3-like object storage, read-write) and, via the same handle, also supports mounting HF model/dataset/space repos read-only.

All bucket lifecycle operations (create/upload/download/delete) go through huggingface_hub (>=1.10). MOUNT / MOUNT_CACHED modes install and invoke hf-mount with its FUSE backend, which aligns with SkyPilot's existing FUSE-based mount infrastructure (gcsfuse / blobfuse2 / rclone mount / goofys).

URL formats

hf://buckets/<ns>/<bucket>[/<sub>]        (bucket, rw)
hf://<ns>/<model>[@<rev>][/<sub>]         (repo, ro)
hf://datasets/<ns>/<ds>[@<rev>][/<sub>]   (repo, ro)
hf://spaces/<ns>/<sp>[@<rev>][/<sub>]     (repo, ro)

Other notes

• Auth: reads HF_TOKEN from env or the HF CLI token file and propagates it to clusters via the existing credential-file-mounts pipeline.
• Fail-loud uploads: uploading to a repo with a local source raises StorageUploadError instead of silently no-op'ing.
• glibc constraint: hf-mount v0.6.5 Linux binaries are linked against glibc >= 2.34, so MOUNT-mode tasks on the default SkyPilot Kubernetes image (Ubuntu 20.04, glibc 2.31) need an explicit newer base image via resources.image_id (e.g. docker:mirror.gcr.io/ubuntu:22.04). COPY mode works on any image. See docs/source/reference/storage.rst.

hf-mount bump (v0.3.1 → v0.6.5)

The originally referenced hf-mount v0.3.1 could not mount in unprivileged Kubernetes pods (the default SkyPilot k8s execution context). Two bugs blocked it, both now fixed upstream:

1. fuser hard-failed before reaching the fusermount fallback: fuser's direct /dev/fuse open path didn't fall back to fusermount when the device wasn't openable, so the mount failed before the setuid fusermount-shim could ever be invoked. Fixed in hf-mount#173 (picking up fuser#1). Released as v0.6.3.

2. clone_fd opened /dev/fuse N-1 more times after the initial mount: after the initial mount succeeded via the proxy, fuser's session loop opened /dev/fuse an additional n_threads - 1 times to clone per-thread fds. Those direct opens hit EPERM in the unprivileged pod and the daemon crashed with FUSE session error: Operation not permitted (os error 1) right after the FUSE_INIT handshake. Fixed in hf-mount#174: probe /dev/fuse openability once at startup and disable clone_fd if denied. Released as v0.6.4. v0.6.5 is a maintenance release on top.

Also corrects the inline comment's glibc minimum: the binaries actually require glibc 2.34 (verified with strings hf-mount-fuse-x86_64-linux | grep GLIBC_ against the published release), not 2.32 as previously documented.

[zpoint]

This seems to be a copy from #9418. Could you address these comments?

[XciD]

This seems to be a copy from #9418. Could you address these comments?

Addressed here: 1b30839

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 19 additional findings in Devin Review.

[devin-ai-integration]

🟡 Temp directory leaked in make_sync_file_command for HF repo file downloads

In HFCloudStorage.make_sync_file_command for repo (non-bucket) sources, a temporary directory is created via tempfile.mkdtemp() but is never cleaned up after the file is copied. The generated Python code downloads the file into tmp_dir, copies it to dest with shutil.copy2, and then exits — leaving tmp_dir and its contents on disk.

The comparable make_sync_dir_command at sky/cloud_stores.py:859-872 correctly wraps the download+move in a try/finally block that calls shutil.rmtree(tmp_dir, ignore_errors=True). This file-sync variant is missing the same cleanup, so every repo file download accumulates a leaked temp directory on the remote VM.

Suggested change

	code = '\n'.join([
	'import os',
	'import shutil',
	'import tempfile',
	'from huggingface_hub import hf_hub_download',
	self._TOKEN_HELPER,
	f'dest = os.path.expanduser({destination!r})',
	'tmp_dir = tempfile.mkdtemp()',
	(f'downloaded = hf_hub_download(repo_id={repo_id!r}, '
	f'repo_type={repo_type!r}, revision={revision!r}, '
	f'filename={path!r}, local_dir=tmp_dir, token=token)'),
	'os.makedirs(os.path.dirname(dest) or ".", exist_ok=True)',
	'shutil.copy2(downloaded, dest)',
	])
	code = '\n'.join([
	'import os',
	'import shutil',
	'import tempfile',
	'from huggingface_hub import hf_hub_download',
	self._TOKEN_HELPER,
	f'dest = os.path.expanduser({destination!r})',
	'tmp_dir = tempfile.mkdtemp()',
	'try:',
	(f' downloaded = hf_hub_download(repo_id={repo_id!r}, '
	f'repo_type={repo_type!r}, revision={revision!r}, '
	f'filename={path!r}, local_dir=tmp_dir, token=token)'),
	' os.makedirs(os.path.dirname(dest) or ".", exist_ok=True)',
	' shutil.copy2(downloaded, dest)',
	'finally:',
	' shutil.rmtree(tmp_dir, ignore_errors=True)',
	])

---

Was this helpful? React with 👍 or 👎 to provide feedback.