Skip to content

Commit 5873e81

Browse files
Add SkyPilot to storage bucket integrations (#2644)
* docs: add SkyPilot to storage bucket integrations Add a SkyPilot section to storage-buckets-integrations.md (store: hf file mounts for buckets and repos) and a SkyPilot row to the access-method table in storage-buckets-access.md. * Update docs/hub/storage-buckets-access.md --------- Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com>
1 parent 049a281 commit 5873e81

2 files changed

Lines changed: 44 additions & 0 deletions

File tree

docs/hub/storage-buckets-access.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ Beyond the [CLI and Python SDK](./storage-buckets#managing-files), there are sev
1212
| **CLI sync** | Batch transfers, backups | [Sync docs](./storage-buckets#syncing-directories) |
1313
| **S3 API** | Existing S3 tooling (AWS CLI, boto3, s5cmd) | [S3-Compatible API](./storage-buckets-s3) |
1414

15+
For tools that use buckets as a backend (SkyPilot, Inspect, …), see [Integrations](./storage-buckets-integrations).
16+
1517
## Mount as a Local Filesystem
1618

1719
[hf-mount](https://github.com/huggingface/hf-mount) lets you mount buckets (and repos) as local filesystems via NFS (recommended) or FUSE. Files are fetched lazily — only the bytes your code reads hit the network.

docs/hub/storage-buckets-integrations.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,48 @@ inspect view
7979

8080
See [Inspect's eval logs guide](https://inspect.aisi.org.uk/eval-logs.html#sec-hugging-face-storage-buckets) for details.
8181

82+
## SkyPilot
83+
84+
[SkyPilot](https://docs.skypilot.co/) runs AI workloads across 20+ clouds, Kubernetes, and on-prem, and can use Hugging Face storage as a backend — so one bucket is readable from every cloud with no per-cloud copies. Set `store: hf` on a `file_mounts` entry to mount a bucket read-write or a repo read-only:
85+
86+
```yaml
87+
# qwen-sft.yaml — launch anywhere: sky launch qwen-sft.yaml --infra aws|gcp|...
88+
resources:
89+
accelerators: H100:1
90+
91+
file_mounts:
92+
/base-model:
93+
source: hf://Qwen/Qwen2.5-3B # model repo, read-only
94+
store: hf
95+
mode: MOUNT
96+
/data:
97+
source: hf://datasets/username/my-data@v1.0 # dataset repo, pinned to a tag, read-only
98+
store: hf
99+
mode: MOUNT
100+
/checkpoints:
101+
source: hf://buckets/username/qwen-sft # bucket, read-write — checkpoints sync back
102+
store: hf
103+
mode: MOUNT
104+
105+
run: |
106+
python train.py --model /base-model --output_dir /checkpoints
107+
```
108+
109+
Authenticate once - `hf auth login` (or `export HF_TOKEN=<your-token>`) is all SkyPilot needs. It forwards your local Hugging Face token to every cloud, so the bucket and repo mounts authenticate automatically:
110+
111+
```bash
112+
pip install "skypilot[huggingface]"
113+
hf auth login # or: export HF_TOKEN=<your-token>
114+
sky launch qwen-sft.yaml
115+
```
116+
117+
If your own `run` code pulls gated repos, add `--secret HF_TOKEN` to the launch command to also expose the token as an env var.
118+
119+
> [!TIP]
120+
> `MOUNT` and `MOUNT_CACHED` behave identically for `hf` and use the [hf-mount](https://github.com/huggingface/hf-mount) FUSE backend, which needs a base image with glibc ≥ 2.34 and `/dev/fuse`. Bare-VM clouds provide both. SkyPilot's default Kubernetes image ships older glibc, so set a newer `image_id` (e.g. `docker:mirror.gcr.io/ubuntu:22.04`). See the [SkyPilot storage docs](https://docs.skypilot.co/en/latest/reference/storage.html) for the current environment requirements.
121+
122+
See the [SkyPilot + Hugging Face storage blog post](https://huggingface.co/blog/skypilot-hf-storage) for benchmarks and a full walkthrough.
123+
82124
## Filesystem operations
83125

84126
For direct file operations, `huggingface_hub` exposes a pre-instantiated [filesystem object](/docs/huggingface_hub/guides/hf_file_system), `hffs`:

0 commit comments

Comments
 (0)