|
| 1 | +# S3 Compatibility |
| 2 | + |
| 3 | +Storage Buckets can be accessed through an **S3-compatible API**, letting you use existing S3 tooling — the AWS CLI, `boto3`, `s5cmd`, and most other S3 SDKs — against your buckets without changing your code. |
| 4 | +Requests go through a gateway service at `https://s3.hf.co`. |
| 5 | + |
| 6 | +> [!NOTE] |
| 7 | +> The S3 API works only with [Storage Buckets](./storage-buckets). It does not expose other Hugging Face repository types (models, datasets, Spaces). |
| 8 | +
|
| 9 | +## Generating S3 Credentials |
| 10 | + |
| 11 | +The gateway authenticates with AWS-style access keys derived from a Hugging Face [User Access Token](./security-tokens). |
| 12 | + |
| 13 | +1. Go to your [Access Tokens settings](https://huggingface.co/settings/tokens). Create a token with the **Create new token** button if you don't already have one. The token's permissions become the S3 credentials' permissions — choose **Read** for read-only access to your buckets, or **Write** for read and write access. |
| 14 | +2. Find the token in the list, open its dropdown menu, and choose **Generate S3 credentials**. |
| 15 | + |
| 16 | + <div class="flex justify-center"> |
| 17 | + <img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/bucket-generate-s3-credentials-light.png" alt="Generate S3 credentials option in access token's dropdown menu"/> |
| 18 | + <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/buckets/bucket-generate-s3-credentials-dark.png" alt="Generate S3 credentials option in access token's dropdown menu"/> |
| 19 | + </div> |
| 20 | + |
| 21 | +3. Copy the generated **access key ID** (prefixed `HFAK…`) and **secret access key** somewhere safe — the secret is shown only once. |
| 22 | + |
| 23 | +The S3 credentials inherit the permissions of the underlying access token. For fine-grained tokens, scope them to only the namespaces and buckets you intend to use. |
| 24 | + |
| 25 | +## Configuring a Client |
| 26 | + |
| 27 | +Point your S3 client at the gateway endpoint and set a few required options. Use your Hugging Face **namespace** (your username or an organization name) in the endpoint URL — see [Addressing buckets](#addressing-buckets) below. |
| 28 | + |
| 29 | +| Setting | Value | Why | |
| 30 | +|--------------------------------|--------------------------------|---------------------------------------------------------------| |
| 31 | +| `endpoint_url` | `https://s3.hf.co/<namespace>` | The gateway, scoped to your namespace | |
| 32 | +| `region` | `us-east-1` | Required; the gateway is currently single-region | |
| 33 | +| `s3.addressing_style` | `path` | Buckets are addressed as path segments, not subdomains | |
| 34 | +| `request_checksum_calculation` | `when_required` | Prevents recent clients from sending trailing checksums | |
| 35 | +| `response_checksum_validation` | `when_required` | Prevents recent clients from expecting checksums in responses | |
| 36 | + |
| 37 | +The two checksum settings matter for recent clients: AWS CLI ≥ 2.23 and recent `boto3` versions send trailing CRC32 checksums (via `aws-chunked` framing) by default, which the gateway does not parse. |
| 38 | +These settings tell the client to send checksums only when an operation strictly requires them. |
| 39 | + |
| 40 | +The following are optional but recommended so large uploads use as few multipart parts as possible: |
| 41 | + |
| 42 | +| Setting | Value | |
| 43 | +|--------------------------|-------| |
| 44 | +| `s3.multipart_threshold` | `2GB` | |
| 45 | +| `s3.multipart_chunksize` | `2GB` | |
| 46 | + |
| 47 | +### Example: AWS CLI profile |
| 48 | + |
| 49 | +Add a profile to `~/.aws/config`: |
| 50 | + |
| 51 | +```ini |
| 52 | +[profile hf] |
| 53 | +region = us-east-1 |
| 54 | +endpoint_url = https://s3.hf.co/<namespace> |
| 55 | +s3 = |
| 56 | + addressing_style = path |
| 57 | + multipart_threshold = 2GB |
| 58 | + multipart_chunksize = 2GB |
| 59 | +request_checksum_calculation = when_required |
| 60 | +response_checksum_validation = when_required |
| 61 | +``` |
| 62 | +Note: replace the `<namespace>` above with the username or organization your buckets are stored in. |
| 63 | + |
| 64 | +Add the matching credentials for the profile to `~/.aws/credentials`: |
| 65 | + |
| 66 | +```ini |
| 67 | +[hf] |
| 68 | +aws_access_key_id = HFAK... |
| 69 | +aws_secret_access_key = ... |
| 70 | +``` |
| 71 | + |
| 72 | +Then use any S3 command with the profile: |
| 73 | + |
| 74 | +```bash |
| 75 | +aws --profile hf s3 ls |
| 76 | +aws --profile hf s3 mb s3://my-bucket |
| 77 | +aws --profile hf s3 cp ./model.safetensors s3://my-bucket/models/model.safetensors |
| 78 | +``` |
| 79 | + |
| 80 | +## Addressing Buckets |
| 81 | + |
| 82 | +AWS S3 uses a single flat, globally-unique space of bucket names, and SDKs expect a bucket name to be a plain string with no `/`. Hugging Face buckets are instead identified as `namespace/bucket`, where the `namespace` is your username or organization. That extra level introduces a mismatch with S3 clients — many won't accept a `/` in a bucket name, or will URL-escape it incorrectly. There are two ways to work around it: |
| 83 | + |
| 84 | +**1. Put the namespace in the endpoint URL** (recommended for most cases). This scopes every operation to that namespace, so the bucket name passed to the client is just the HF bucket name. It works well for your own buckets or buckets in a single org, but breaks down across namespaces — e.g. a server-side copy from a personal bucket into an org bucket. |
| 85 | + |
| 86 | +```bash |
| 87 | +aws --endpoint-url https://s3.hf.co/my-org s3api get-object \ |
| 88 | + --bucket my-bucket --key some/object.txt ./object.txt |
| 89 | +``` |
| 90 | + |
| 91 | +**2. Treat the namespace as the bucket** and prepend the HF bucket name to the object key. This works for object-level operations (uploads, downloads) but has issues with bucket-level operations such as creating or deleting buckets. |
| 92 | + |
| 93 | +```bash |
| 94 | +aws --endpoint-url https://s3.hf.co s3api get-object \ |
| 95 | + --bucket my-org --key my-bucket/some/object.txt ./object.txt |
| 96 | +``` |
| 97 | + |
| 98 | +## Limitations and Differences from AWS S3 |
| 99 | + |
| 100 | +Because Storage Buckets don't model every S3 concept, some behaviors differ or aren't supported. |
| 101 | + |
| 102 | +### Object downloads |
| 103 | + |
| 104 | +The gateway is currently single-region. To improve download performance, `GetObject` typically responds with an HTTP 302 redirect to the nearest Hugging Face CDN edge rather than serving bytes directly. |
| 105 | + |
| 106 | +Some SDKs don't follow redirects from an S3 endpoint, so the gateway detects clients identifying as `aws-cli`, `botocore` (which covers `boto3`), or `aws-sdk-rust` and proxies the data through itself for them. All other clients (`rclone`, `s5cmd`, `curl`, the AWS Go SDK, etc.) receive the 302 and follow it natively, keeping the gateway out of the data path for faster downloads. |
| 107 | + |
| 108 | +### Object key naming |
| 109 | + |
| 110 | +Bucket object keys are more restricted than S3. A key must **not**: |
| 111 | + |
| 112 | +- start or end with `/` |
| 113 | +- contain consecutive slashes (`//`) |
| 114 | +- contain `../` sequences |
| 115 | +- start with `./` |
| 116 | +- end with `..` |
| 117 | +- contain backslashes (`\`) or null bytes (`\0`) |
| 118 | + |
| 119 | +### ListObjects |
| 120 | + |
| 121 | +- `ListObjectsV1` is not supported — use `ListObjectsV2`. Note that some clients (like rclone) |
| 122 | + may need to be configured to use `ListObjectsV2` exclusively. |
| 123 | +- Only `/` is allowed as the delimiter. |
| 124 | + |
| 125 | +### Other API differences |
| 126 | + |
| 127 | +- **Object metadata**: arbitrary user metadata (`x-amz-meta-*`) is not stored or returned. `Content-Type` is supported. |
| 128 | +- **Unsupported features**: ACLs, bucket policies, object tagging, object versioning, lifecycle rules, server-side encryption (SSE), and bucket notifications are not supported. Objects are always reported with the `STANDARD` storage class. Related request headers and parameters are accepted but ignored. |
| 129 | +- **CopyObject**: server-side copy works only within a single namespace. Cross-namespace copy and `UploadPartCopy` (copying a part from an existing object into a multipart upload) are not supported. |
| 130 | +- **Conditional requests**: `If-Match` / `If-None-Match` preconditions are honored on `PutObject` and on the copy-source of `CopyObject`, but not on `GetObject`. |
| 131 | +- **Multipart upload expiry**: in-flight multipart uploads that are never completed or aborted are automatically expired and cleaned up after 7 days. |
| 132 | + |
| 133 | +## Examples |
| 134 | + |
| 135 | +Real-world recipes for common tasks. Each builds on the [client configuration](#configuring-a-client) above. |
| 136 | + |
| 137 | +### Import data using `rclone` |
| 138 | + |
| 139 | +[`rclone`](https://rclone.org/) is a convenient way to copy data between two S3-compatible stores, so it's a good fit for moving an existing AWS S3 bucket (or any S3-compatible source) into a Storage Bucket. |
| 140 | + |
| 141 | +The idea is to declare two remotes — your source bucket and the Hugging Face gateway — and let `rclone` stream the objects between them. Add both remotes to `~/.config/rclone/rclone.conf`. The first points at your existing S3 bucket; adjust it to match your source (here, plain AWS S3): |
| 142 | + |
| 143 | +```ini |
| 144 | +[aws] |
| 145 | +type = s3 |
| 146 | +provider = AWS |
| 147 | +access_key_id = AKIA... |
| 148 | +secret_access_key = ... |
| 149 | +region = us-east-1 |
| 150 | +``` |
| 151 | + |
| 152 | +The second points at the Hugging Face gateway. As with any other client, scope the endpoint to your [namespace](#addressing-buckets), use `path` addressing, force ListObjectsV2 (the only listing version the gateway supports), and set large multipart sizes so uploads use as few parts as possible: |
| 153 | + |
| 154 | +```ini |
| 155 | +[hf] |
| 156 | +type = s3 |
| 157 | +provider = Other |
| 158 | +endpoint = https://s3.hf.co/<namespace> |
| 159 | +access_key_id = HFAK... |
| 160 | +secret_access_key = ... |
| 161 | +region = us-east-1 |
| 162 | +force_path_style = true |
| 163 | +list_version = 2 |
| 164 | +upload_cutoff = 2G |
| 165 | +chunk_size = 2G |
| 166 | +``` |
| 167 | + |
| 168 | +Note: replace the `<namespace>` above with the username or organization your buckets are stored in, and use the [S3 credentials](#generating-s3-credentials) generated from your access token. The destination bucket must already exist under that namespace. |
| 169 | + |
| 170 | +Now copy a source bucket into your Storage Bucket: |
| 171 | + |
| 172 | +```bash |
| 173 | +rclone copy aws:my-source-bucket hf:my-bucket --progress |
| 174 | +``` |
| 175 | + |
| 176 | +`rclone copy` only transfers objects that are missing or changed at the destination, so it's safe to re-run to resume an interrupted import or pick up new objects. To make the destination an exact mirror of the source — deleting objects at the destination that no longer exist at the source — use `rclone sync` instead: |
| 177 | + |
| 178 | +```bash |
| 179 | +rclone sync aws:my-source-bucket hf:my-bucket --progress |
| 180 | +``` |
| 181 | + |
| 182 | +> [!TIP] |
| 183 | +> For large imports, add `--transfers` and `--checkers` to raise the concurrency (e.g. `--transfers 16 --checkers 16`), and run `rclone check aws:my-source-bucket hf:my-bucket` afterwards to confirm every object made it across. |
0 commit comments