Skip to content

Add a DVC example to the Storage Buckets S3 page#2625

Merged
davanstrien merged 2 commits into
huggingface:mainfrom
davanstrien:docs-dvc-s3-example
Jul 7, 2026
Merged

Add a DVC example to the Storage Buckets S3 page#2625
davanstrien merged 2 commits into
huggingface:mainfrom
davanstrien:docs-dvc-s3-example

Conversation

@davanstrien

@davanstrien davanstrien commented Jul 6, 2026

Copy link
Copy Markdown
Member

Adds a Version data with DVC recipe to the Storage Buckets S3 page's Examples section, alongside the existing boto3 / DuckDB / rclone examples.

DVC can use a bucket as a DVC remote through the S3-compatible gateway — code and .dvc pointers stay in git, the data lives in the bucket.

Follows the existing example pattern; references the page's own Credentials / Client / Addressing sections instead of restating them. One section, no new page, no _toctree.yml change.

🤖 Generated with Claude Code


Note

Low Risk
Documentation-only change with no code, config, or runtime behavior impact.

Overview
Adds a Version data with DVC example to the Storage Buckets S3 Examples section, next to the existing boto3, DuckDB, and rclone recipes.

The new section explains using a bucket as a DVC S3 remote via the gateway: pip install 'dvc[s3]', dvc remote add / endpointurl / region, credentials via env or --local, and a short dvc add → commit → dvc push flow. It links to the page’s Configuring a client, Generating S3 credentials, and Addressing buckets sections instead of duplicating them, and notes putting the namespace in endpointurl and the bare bucket name in the s3:// URL.

Reviewed by Cursor Bugbot for commit cca2d7d. Bugbot is set up for automated code reviews on this repo. Configure here.

davanstrien and others added 2 commits July 6, 2026 16:43
Documents using a bucket as a DVC remote via the S3-compatible gateway. Verified end-to-end (init -> add -> push -> fresh clone -> pull) with DVC 3.67.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The checksum-error workaround did not reproduce against the gateway with DVC's client stack (push succeeds without the overrides, verified end-to-end incl. a real GitHub round-trip). The general client-config section already documents the setting for anyone who needs it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@davanstrien

Copy link
Copy Markdown
Member Author

Marking this ready — the example is verified end-to-end against the live gateway (DVC 3.67.1), not just eyeballed:

  • dvc init → add → push → fresh clone → pull, including a real GitHub repo round-trip (code + .dvc pointer on GitHub, data in the bucket; a second clone pulls it back)
  • credentials both ways: env vars and dvc remote modify --local (git-ignored .dvc/config.local)
  • a 1 GB single file (multipart) and a 2000-file dataset (train/val dirs) — whole-dataset sha256 integrity matched on both push and pull
  • iterative update (add files → push → teammate pull) round-trips cleanly

One change from my first draft: I removed a checksum-error [!NOTE] because it did not reproduce on DVC's client stack (push succeeds without the when_required overrides), and the general client-config section already covers that setting.

The example is intentionally minimal — mirrors the existing boto3 / DuckDB / rclone recipes and links the page's own Credentials / Client / Addressing sections rather than restating them. Happy to adjust wording or placement.

@davanstrien
davanstrien marked this pull request as ready for review July 7, 2026 09:10

@lhoestq lhoestq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

btw dvc was acquired by lakefs and will stay open source but I don't expect it to grow more

@davanstrien
davanstrien merged commit 7c0de66 into huggingface:main Jul 7, 2026
2 checks passed
@davanstrien
davanstrien deleted the docs-dvc-s3-example branch July 7, 2026 09:45
@julien-c

Copy link
Copy Markdown
Member

and if you use DVC, it's probably better to just switch to a HF git repo no? 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants