Skip to content

Commit c83b9ad

Browse files
davanstrienpcuencajulien-c
authored
docs: make server-side repo→bucket copy easier to find (#2519)
* docs: surface server-side repo→bucket copy (hf buckets cp) Cross-link the instant copy from datasets-ingesting.md and the Managing Files intro, and tie it to the Xet dedup explanation. Feature already existed but was buried at the bottom of storage-buckets.md. * docs: note server-side copy is same-region only Per review (Wauplin), folded into the existing copy note. * Update docs/hub/storage-buckets.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Update docs/hub/datasets-ingesting.md Co-authored-by: Julien Chaumond <julien@huggingface.co> --------- Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Julien Chaumond <julien@huggingface.co>
1 parent cf8a226 commit c83b9ad

3 files changed

Lines changed: 15 additions & 2 deletions

File tree

docs/hub/datasets-ingesting.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,14 @@ See the list of [Libraries supported by the Datasets Hub](./datasets-libraries)
8484

8585
If you are ingesting raw data that need further curation before being published as AI-ready datasets or if you need an S3-like experience, consider ingesting them to [Hugging Face Storage Buckets](./storage-buckets).
8686

87+
If your data already lives in a dataset, model, or Space repository, you can copy it into a bucket **server-side** — no download or re-upload — with `hf buckets cp`:
88+
89+
```bash
90+
hf cp hf://datasets/username/my-dataset/ hf://buckets/username/my-bucket/
91+
```
92+
93+
Only the Xet content hashes are migrated, so even very large files copy instantly. See [Copying files between repos and buckets](./storage-buckets#copying-files-between-repos-and-buckets).
94+
8795
## Scheduled ingestion
8896

8997
There are some limitations when updating the same file on the Hub thousands of times.

docs/hub/storage-buckets.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,8 @@ hf buckets list julien-c/my-training-bucket --tree -h -R
154154

155155
You can upload and download files directly from the bucket page on the Hub, or use the CLI and Python API for programmatic access. Bucket files are referenced using `hf://buckets/` paths (e.g., `hf://buckets/username/my-bucket/path/to/file`). The `hf buckets cp` command handles individual file transfers while `hf buckets sync` is better suited for directories. All commands work in both directions — local-to-remote and remote-to-local.
156156

157+
If your data already lives in a model, dataset, or Space repository (or another bucket), you can copy it **server-side** with `hf buckets cp` — no download or re-upload required. See [Copying files between repos and buckets](#copying-files-between-repos-and-buckets).
158+
157159
### Uploading files
158160

159161
For quick uploads, you can drag and drop files directly on the bucket page in your browser. For programmatic use, `hf buckets cp` copies individual files into a bucket. The source is a local path and the destination is an `hf://buckets/` path. You can also pipe data from stdin, which is handy for programmatically generated content.
@@ -281,10 +283,10 @@ For more deletion options (pattern-based filtering, recursive removal, etc.), se
281283

282284
### Copying files between repos and buckets
283285

284-
You can copy [Xet](./xet/index)-tracked files from any repository (model, dataset, Space) or bucket into a destination bucket without re-uploading the data. The copy is server-side: only the Xet content hashes are migrated, so even very large files are copied instantly.
286+
You can copy [Xet](./xet/index)-tracked files from any repository (model, dataset, Space) or bucket into a destination bucket without re-uploading the data. The copy is server-side: only the Xet content hashes are migrated, so even very large files are copied instantly thanks to [chunk-level deduplication](./xet/deduplication).
285287

286288
> [!NOTE]
287-
> Only Xet-tracked files are copied server-to-server. Small non-Xet files (e.g., config files and READMEs) are automatically downloaded and re-uploaded.
289+
> Only Xet-tracked files are copied server-to-server. Small non-Xet files (e.g., config files and READMEs) are automatically downloaded and re-uploaded. Server-side copy also requires the source and destination to be in the same storage region.
288290
289291
**CLI:**
290292
```bash

docs/hub/xet/deduplication.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22

33
Xet-enabled repositories utilize [content-defined chunking (CDC)](https://huggingface.co/blog/from-files-to-chunks) to deduplicate on the level of bytes (~64KB of data, also referred to as a "chunk"). Each chunk is identified by a rolling hash that determines chunk boundaries based on the actual file contents, making it resilient to insertions or deletions anywhere in the file. When a file is uploaded to a Xet-backed repository using a Xet-aware client, its contents are broken down into these variable-sized chunks. Only new chunks not already present in Xet storage are kept after chunking, everything else is discarded.
44

5+
> [!TIP]
6+
> Chunk-level deduplication is also what makes **server-side copies instant**: `hf buckets cp` between repositories and buckets migrates only content hashes, with no data re-upload. See [Copying files between repos and buckets](../storage-buckets#copying-files-between-repos-and-buckets).
7+
58
## How Content-Defined Chunking Works
69

710
To understand content-defined chunking, imagine a file as a long passage of text. The system scans the data using a rolling hash — a small mathematical function that slides over the bytes. Whenever the hash hits a special pattern, a chunk boundary is placed at that position. Because the boundaries are determined by the *content itself* (not by fixed positions), identical regions of data always produce the same chunks, even if surrounding content changes.

0 commit comments

Comments
 (0)