Skip to content

Commit cd66d41

Browse files
davanstrienclaude
andauthored
Add boto3 and DuckDB examples to the S3 docs (#2608)
Add two verified examples to the Examples section of the Storage Buckets S3 page (boto3 read/write, DuckDB query a bucket), plus a one-line pointer to Access Patterns so S3 reads as one of several access methods. The DuckDB note documents the required URL_STYLE 'path' setting (without it, virtual- hosted-style addressing fails to resolve). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9833a2e commit cd66d41

1 file changed

Lines changed: 50 additions & 0 deletions

File tree

docs/hub/storage-buckets-s3.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33
Storage Buckets can be accessed through an **S3-compatible API**, letting you use existing S3 tooling — the AWS CLI, `boto3`, `s5cmd`, and most other S3 SDKs — against your buckets without changing your code.
44
Requests go through a gateway service at `https://s3.hf.co`.
55

6+
The S3 API is one of several ways to reach bucket data — for Hugging Face-native access (the `hf` CLI, `hf://` paths, and filesystem mounts) without separate S3 credentials, see [Access Patterns](./storage-buckets-access).
7+
68
> [!NOTE]
79
> The S3 API works only with [Storage Buckets](./storage-buckets). It does not expose other Hugging Face repository types (models, datasets, Spaces).
810
@@ -134,6 +136,54 @@ Bucket object keys are more restricted than S3. A key must **not**:
134136

135137
Real-world recipes for common tasks. Each builds on the [client configuration](#configuring-a-client) above.
136138

139+
### Read and write with `boto3`
140+
141+
[`boto3`](https://docs.aws.amazon.com/boto3/latest/) works against the gateway with the [client settings](#configuring-a-client) above. Replace `<namespace>` with your username or organization:
142+
143+
```python
144+
import boto3
145+
from botocore.config import Config
146+
147+
s3 = boto3.client(
148+
"s3",
149+
endpoint_url="https://s3.hf.co/<namespace>",
150+
aws_access_key_id="HFAK...",
151+
aws_secret_access_key="...",
152+
config=Config(
153+
region_name="us-east-1",
154+
s3={"addressing_style": "path"},
155+
request_checksum_calculation="when_required",
156+
response_checksum_validation="when_required",
157+
),
158+
)
159+
160+
s3.upload_file("model.safetensors", "my-bucket", "models/model.safetensors")
161+
s3.download_file("my-bucket", "models/model.safetensors", "model.safetensors")
162+
```
163+
164+
### Query a bucket with DuckDB
165+
166+
With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket:
167+
168+
```sql
169+
INSTALL httpfs;
170+
LOAD httpfs;
171+
172+
CREATE SECRET hf (
173+
TYPE s3,
174+
KEY_ID 'HFAK...',
175+
SECRET '...',
176+
ENDPOINT 's3.hf.co/<namespace>',
177+
URL_STYLE 'path',
178+
REGION 'us-east-1'
179+
);
180+
181+
SELECT * FROM read_parquet('s3://my-bucket/data.parquet');
182+
```
183+
184+
> [!NOTE]
185+
> `URL_STYLE 'path'` is required. Without it, DuckDB uses virtual-hosted-style addressing (`my-bucket.s3.hf.co`), which the gateway does not serve — you will see a "Could not resolve hostname" error.
186+
137187
### Import data using `rclone`
138188

139189
[`rclone`](https://rclone.org/) is a convenient way to copy data between two S3-compatible stores, so it's a good fit for moving an existing AWS S3 bucket (or any S3-compatible source) into a Storage Bucket.

0 commit comments

Comments
 (0)