feat: add docs for buckets s3 compatibility api

[jgodlew]

Storage Buckets now expose an S3-compatible API via a gateway at https://s3.hf.co, so users can reach their buckets with existing S3 tooling (AWS CLI, boto3, s5cmd, etc.).

This PR documents that feature and its limitations/differences to AWS S3 in a new Storage Buckets -> S3 Compatibility page.

---

Note

Low Risk
Documentation-only changes with no runtime, auth, or data-handling code modifications.

Overview
Documents the S3-compatible gateway (https://s3.hf.co) for Storage Buckets so users can use AWS CLI, boto3, s5cmd, and similar tools without rewriting workflows.

Adds a new S3 Compatibility Hub doc page covering credential generation from access tokens, required client settings (path-style addressing, us-east-1, checksum workarounds for newer AWS clients), namespace/bucket addressing patterns, AWS S3 differences/limitations, and an rclone import example. The Storage Buckets section in _toctree.yml now links this page; Access Patterns lists S3 API as an option (replacing the prior “not supported” note), and the main Storage Buckets tip points readers to the new guide.

^{Reviewed by Cursor Bugbot for commit b858ad3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[julien-c]

really cool!

[julien-c]

proxies the data through itself

ouch, this sounds costly 😓

[jgodlew]

Yeah, it is unfortunate that the aws-cli/boto libraries don't follow redirects. We might want to think about integrating this API with the Hugging Face CDN if we find that a significant amount of downloads are being proxied.

[Pierrci]

🔥

[XciD]

We need to link to this doc in moon-landing when we create the S3 secrets I guess.

[pcuenca]

🔥

[Wauplin]

Very nice!

Wondering if we should add a real use case example already or if that should rather live in a separate doc. Here is a CC suggestion on how to import data using rclone:

## Example: import data using `rclone`

[`rclone`](https://rclone.org/) is a convenient way to copy data between two S3-compatible stores, so it's a good fit for moving an existing AWS S3 bucket (or any S3-compatible source) into a Storage Bucket.

The idea is to declare two remotes — your source bucket and the Hugging Face gateway — and let `rclone` stream the objects between them. Add both remotes to `~/.config/rclone/rclone.conf`. The first points at your existing S3 bucket; adjust it to match your source (here, plain AWS S3):

```ini
[aws]
type = s3
provider = AWS
access_key_id = AKIA...
secret_access_key = ...
region = us-east-1
```

The second points at the Hugging Face gateway. As with any other client, scope the endpoint to your [namespace](#addressing-buckets), use `path` addressing, and set large multipart sizes so uploads use as few parts as possible:

```ini
[hf]
type = s3
provider = Other
endpoint = https://s3.hf.co/<namespace>
access_key_id = HFAK...
secret_access_key = ...
region = us-east-1
force_path_style = true
upload_cutoff = 2G
chunk_size = 2G
```

Note: replace the `<namespace>` above with the username or organization your buckets are stored in, and use the [S3 credentials](#generating-s3-credentials) generated from your access token. The destination bucket must already exists under that namespace.

Now, let's copy a source bucket into your Storage Bucket using `rclone`:

```bash
rclone copy aws:my-source-bucket hf:my-bucket --progress
```

`rclone copy` only transfers objects that are missing or changed at the destination, so it's safe to re-run to resume an interrupted import or pick up new objects. To make the destination an exact mirror of the source — deleting objects at the destination that no longer exist at the source — use `rclone sync` instead:

```bash
rclone sync aws:my-source-bucket hf:my-bucket --progress
```

> [!TIP]
> For large imports, add `--transfers` and `--checkers` to raise the concurrency (e.g. `--transfers 16 --checkers 16`), and run `rclone check aws:my-source-bucket hf:my-bucket` afterwards to confirm every object made it across.

(can be done in a future PR)

[julien-c]

@Wauplin Yes, I was wondering about that as well.

[davanstrien]

Wondering if we should add a real use case example already or if that should rather live in a separate doc. Here is a CC suggestion on how to import data using rclone:

Had Claude Code run a bunch of S3 tooling and it found that the [hf] remote needs list_version = 2. With provider = Other, rclone defaults to ListObjects V1, which the gateway returns a 501 for, so listing inside a bucket fails. Since rclone copy/sync list the destination to compute the diff, the import would hit that too.

So the [hf] remote wants:

[hf]
type = s3
provider = Other
endpoint = https://s3.hf.co/<namespace>
access_key_id = HFAK...
secret_access_key = ...
region = us-east-1
force_path_style = true
upload_cutoff = 2G
chunk_size = 2G
list_version = 2

[davanstrien]

very nice! It could be worth adding a longer intergrations supported via S3 section at some point but can iterate on that later and focus on most used to start

[davanstrien]

🔥

[jgodlew]

Wondering if we should add a real use case example already

Yeah, I agree that is a good idea. I've added an Examples section to the doc with the suggested Import data using rclone as the first example (incorporating the list_version change @davanstrien indicated). I think we can probably add more examples as we find more use-cases.

[Wauplin]

Thank you!