|
3 | 3 | Storage Buckets can be accessed through an **S3-compatible API**, letting you use existing S3 tooling — the AWS CLI, `boto3`, `s5cmd`, and most other S3 SDKs — against your buckets without changing your code. |
4 | 4 | Requests go through a gateway service at `https://s3.hf.co`. |
5 | 5 |
|
| 6 | +The S3 API is one of several ways to reach bucket data — for Hugging Face-native access (the `hf` CLI, `hf://` paths, and filesystem mounts) without separate S3 credentials, see [Access Patterns](./storage-buckets-access). |
| 7 | + |
6 | 8 | > [!NOTE] |
7 | 9 | > The S3 API works only with [Storage Buckets](./storage-buckets). It does not expose other Hugging Face repository types (models, datasets, Spaces). |
8 | 10 |
|
@@ -134,6 +136,54 @@ Bucket object keys are more restricted than S3. A key must **not**: |
134 | 136 |
|
135 | 137 | Real-world recipes for common tasks. Each builds on the [client configuration](#configuring-a-client) above. |
136 | 138 |
|
| 139 | +### Read and write with `boto3` |
| 140 | + |
| 141 | +[`boto3`](https://docs.aws.amazon.com/boto3/latest/) works against the gateway with the [client settings](#configuring-a-client) above. Replace `<namespace>` with your username or organization: |
| 142 | + |
| 143 | +```python |
| 144 | +import boto3 |
| 145 | +from botocore.config import Config |
| 146 | + |
| 147 | +s3 = boto3.client( |
| 148 | + "s3", |
| 149 | + endpoint_url="https://s3.hf.co/<namespace>", |
| 150 | + aws_access_key_id="HFAK...", |
| 151 | + aws_secret_access_key="...", |
| 152 | + config=Config( |
| 153 | + region_name="us-east-1", |
| 154 | + s3={"addressing_style": "path"}, |
| 155 | + request_checksum_calculation="when_required", |
| 156 | + response_checksum_validation="when_required", |
| 157 | + ), |
| 158 | +) |
| 159 | + |
| 160 | +s3.upload_file("model.safetensors", "my-bucket", "models/model.safetensors") |
| 161 | +s3.download_file("my-bucket", "models/model.safetensors", "model.safetensors") |
| 162 | +``` |
| 163 | + |
| 164 | +### Query a bucket with DuckDB |
| 165 | + |
| 166 | +With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket: |
| 167 | + |
| 168 | +```sql |
| 169 | +INSTALL httpfs; |
| 170 | +LOAD httpfs; |
| 171 | + |
| 172 | +CREATE SECRET hf ( |
| 173 | + TYPE s3, |
| 174 | + KEY_ID 'HFAK...', |
| 175 | + SECRET '...', |
| 176 | + ENDPOINT 's3.hf.co/<namespace>', |
| 177 | + URL_STYLE 'path', |
| 178 | + REGION 'us-east-1' |
| 179 | +); |
| 180 | + |
| 181 | +SELECT * FROM read_parquet('s3://my-bucket/data.parquet'); |
| 182 | +``` |
| 183 | + |
| 184 | +> [!NOTE] |
| 185 | +> `URL_STYLE 'path'` is required. Without it, DuckDB uses virtual-hosted-style addressing (`my-bucket.s3.hf.co`), which the gateway does not serve — you will see a "Could not resolve hostname" error. |
| 186 | +
|
137 | 187 | ### Import data using `rclone` |
138 | 188 |
|
139 | 189 | [`rclone`](https://rclone.org/) is a convenient way to copy data between two S3-compatible stores, so it's a good fit for moving an existing AWS S3 bucket (or any S3-compatible source) into a Storage Bucket. |
|
0 commit comments