Skip to content

Commit c5a1aa1

Browse files
committed
docs: document S3 and S3-compatible storage for the s3 filesystem
1 parent 8746518 commit c5a1aa1

1 file changed

Lines changed: 29 additions & 0 deletions

File tree

docs/overview.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,35 @@ Integrations with many systems and cloud vendors include (but not limited to):
1919
- Microsoft Azure Storage
2020
- Alibaba Cloud OSS etc.
2121

22+
## S3 and S3-compatible object storage
23+
24+
The `s3://` file system is registered when you `import tensorflow_io`. It reads
25+
the standard AWS credential and region environment variables
26+
(`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`), so it works with
27+
Amazon S3 out of the box:
28+
29+
```python
30+
import tensorflow as tf
31+
import tensorflow_io as tfio # registers the s3:// file system
32+
33+
dataset = tf.data.TFRecordDataset("s3://my-bucket/train/shard-00000.tfrecord")
34+
```
35+
36+
It also works with any S3-compatible object store, such as Backblaze B2,
37+
Cloudflare R2, MinIO, and others, by pointing the client at the provider
38+
endpoint with the `S3_ENDPOINT` environment variable. The client uses path-style
39+
addressing by default, so no extra configuration is required:
40+
41+
```bash
42+
export AWS_ACCESS_KEY_ID=<access_key_id>
43+
export AWS_SECRET_ACCESS_KEY=<secret_access_key>
44+
export AWS_REGION=us-west-004
45+
export S3_ENDPOINT=https://s3.us-west-004.backblazeb2.com
46+
```
47+
48+
The same `s3://` paths work for reading and writing checkpoints and SavedModel
49+
artifacts through `tf.io.gfile`, `tf.train.Checkpoint`, and `tf.saved_model`.
50+
2251
## Community
2352

2453
* SIG IO [Google Group](https://groups.google.com/a/tensorflow.org/forum/#!forum/io) and mailing list: [io@tensorflow.org](io@tensorflow.org)

0 commit comments

Comments
 (0)