From aee934d441898b547c80342e3b726afe2ddf7883 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 2 Jun 2026 10:58:44 +0100 Subject: [PATCH 1/4] =?UTF-8?q?docs:=20surface=20server-side=20repo?= =?UTF-8?q?=E2=86=92bucket=20copy=20(hf=20buckets=20cp)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-link the instant copy from datasets-ingesting.md and the Managing Files intro, and tie it to the Xet dedup explanation. Feature already existed but was buried at the bottom of storage-buckets.md. --- docs/hub/datasets-ingesting.md | 8 ++++++++ docs/hub/storage-buckets.md | 4 +++- docs/hub/xet/deduplication.md | 3 +++ 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/hub/datasets-ingesting.md b/docs/hub/datasets-ingesting.md index 7384e47f1b..d62d768b1c 100644 --- a/docs/hub/datasets-ingesting.md +++ b/docs/hub/datasets-ingesting.md @@ -84,6 +84,14 @@ See the list of [Libraries supported by the Datasets Hub](./datasets-libraries) If you are ingesting raw data that need further curation before being published as AI-ready datasets or if you need an S3-like experience, consider ingesting them to [Hugging Face Storage Buckets](./storage-buckets). +If your data already lives in a dataset, model, or Space repository, you can copy it into a bucket **server-side** — no download or re-upload — with `hf buckets cp`: + +```bash +hf buckets cp hf://datasets/username/my-dataset/ hf://buckets/username/my-bucket/ +``` + +Only the Xet content hashes are migrated, so even very large files copy instantly. See [Copying files between repos and buckets](./storage-buckets#copying-files-between-repos-and-buckets). + ## Scheduled ingestion There are some limitations when updating the same file on the Hub thousands of times. diff --git a/docs/hub/storage-buckets.md b/docs/hub/storage-buckets.md index d367d5305e..0344d2c529 100644 --- a/docs/hub/storage-buckets.md +++ b/docs/hub/storage-buckets.md @@ -154,6 +154,8 @@ hf buckets list julien-c/my-training-bucket --tree -h -R You can upload and download files directly from the bucket page on the Hub, or use the CLI and Python API for programmatic access. Bucket files are referenced using `hf://buckets/` paths (e.g., `hf://buckets/username/my-bucket/path/to/file`). The `hf buckets cp` command handles individual file transfers while `hf buckets sync` is better suited for directories. All commands work in both directions — local-to-remote and remote-to-local. +If your data already lives in a model, dataset, or Space repository (or another bucket), you can copy it in **server-side** with `hf buckets cp` — no download or re-upload required. See [Copying files between repos and buckets](#copying-files-between-repos-and-buckets). + ### Uploading files For quick uploads, you can drag and drop files directly on the bucket page in your browser. For programmatic use, `hf buckets cp` copies individual files into a bucket. The source is a local path and the destination is an `hf://buckets/` path. You can also pipe data from stdin, which is handy for programmatically generated content. @@ -281,7 +283,7 @@ For more deletion options (pattern-based filtering, recursive removal, etc.), se ### Copying files between repos and buckets -You can copy [Xet](./xet/index)-tracked files from any repository (model, dataset, Space) or bucket into a destination bucket without re-uploading the data. The copy is server-side: only the Xet content hashes are migrated, so even very large files are copied instantly. +You can copy [Xet](./xet/index)-tracked files from any repository (model, dataset, Space) or bucket into a destination bucket without re-uploading the data. The copy is server-side: only the Xet content hashes are migrated, so even very large files are copied instantly thanks to [chunk-level deduplication](./xet/deduplication). > [!NOTE] > Only Xet-tracked files are copied server-to-server. Small non-Xet files (e.g., config files and READMEs) are automatically downloaded and re-uploaded. diff --git a/docs/hub/xet/deduplication.md b/docs/hub/xet/deduplication.md index d64123e8af..b13a0348cc 100644 --- a/docs/hub/xet/deduplication.md +++ b/docs/hub/xet/deduplication.md @@ -2,6 +2,9 @@ Xet-enabled repositories utilize [content-defined chunking (CDC)](https://huggingface.co/blog/from-files-to-chunks) to deduplicate on the level of bytes (~64KB of data, also referred to as a "chunk"). Each chunk is identified by a rolling hash that determines chunk boundaries based on the actual file contents, making it resilient to insertions or deletions anywhere in the file. When a file is uploaded to a Xet-backed repository using a Xet-aware client, its contents are broken down into these variable-sized chunks. Only new chunks not already present in Xet storage are kept after chunking, everything else is discarded. +> [!TIP] +> Chunk-level deduplication is also what makes **server-side copies instant**: `hf buckets cp` between repositories and buckets migrates only content hashes, with no data re-upload. See [Copying files between repos and buckets](../storage-buckets#copying-files-between-repos-and-buckets). + ## How Content-Defined Chunking Works To understand content-defined chunking, imagine a file as a long passage of text. The system scans the data using a rolling hash — a small mathematical function that slides over the bytes. Whenever the hash hits a special pattern, a chunk boundary is placed at that position. Because the boundaries are determined by the *content itself* (not by fixed positions), identical regions of data always produce the same chunks, even if surrounding content changes. From 4e2122e15d74f2353762a45c889b71d59ef425ad Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 2 Jun 2026 11:46:21 +0100 Subject: [PATCH 2/4] docs: note server-side copy is same-region only Per review (Wauplin), folded into the existing copy note. --- docs/hub/storage-buckets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/storage-buckets.md b/docs/hub/storage-buckets.md index 0344d2c529..cefcb2a5f1 100644 --- a/docs/hub/storage-buckets.md +++ b/docs/hub/storage-buckets.md @@ -286,7 +286,7 @@ For more deletion options (pattern-based filtering, recursive removal, etc.), se You can copy [Xet](./xet/index)-tracked files from any repository (model, dataset, Space) or bucket into a destination bucket without re-uploading the data. The copy is server-side: only the Xet content hashes are migrated, so even very large files are copied instantly thanks to [chunk-level deduplication](./xet/deduplication). > [!NOTE] -> Only Xet-tracked files are copied server-to-server. Small non-Xet files (e.g., config files and READMEs) are automatically downloaded and re-uploaded. +> Only Xet-tracked files are copied server-to-server. Small non-Xet files (e.g., config files and READMEs) are automatically downloaded and re-uploaded. Server-side copy also requires the source and destination to be in the same storage region. **CLI:** ```bash From dcf35ea588539e120f4930b0935588147a265f53 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 2 Jun 2026 14:15:51 +0100 Subject: [PATCH 3/4] Update docs/hub/storage-buckets.md Co-authored-by: Pedro Cuenca --- docs/hub/storage-buckets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/storage-buckets.md b/docs/hub/storage-buckets.md index cefcb2a5f1..867ef71248 100644 --- a/docs/hub/storage-buckets.md +++ b/docs/hub/storage-buckets.md @@ -154,7 +154,7 @@ hf buckets list julien-c/my-training-bucket --tree -h -R You can upload and download files directly from the bucket page on the Hub, or use the CLI and Python API for programmatic access. Bucket files are referenced using `hf://buckets/` paths (e.g., `hf://buckets/username/my-bucket/path/to/file`). The `hf buckets cp` command handles individual file transfers while `hf buckets sync` is better suited for directories. All commands work in both directions — local-to-remote and remote-to-local. -If your data already lives in a model, dataset, or Space repository (or another bucket), you can copy it in **server-side** with `hf buckets cp` — no download or re-upload required. See [Copying files between repos and buckets](#copying-files-between-repos-and-buckets). +If your data already lives in a model, dataset, or Space repository (or another bucket), you can copy it **server-side** with `hf buckets cp` — no download or re-upload required. See [Copying files between repos and buckets](#copying-files-between-repos-and-buckets). ### Uploading files From a760b69314468d3cbc47e466449f972e52e6f97e Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Sun, 7 Jun 2026 12:29:36 +0100 Subject: [PATCH 4/4] Update docs/hub/datasets-ingesting.md Co-authored-by: Julien Chaumond --- docs/hub/datasets-ingesting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/datasets-ingesting.md b/docs/hub/datasets-ingesting.md index d62d768b1c..a61cb72687 100644 --- a/docs/hub/datasets-ingesting.md +++ b/docs/hub/datasets-ingesting.md @@ -87,7 +87,7 @@ If you are ingesting raw data that need further curation before being published If your data already lives in a dataset, model, or Space repository, you can copy it into a bucket **server-side** — no download or re-upload — with `hf buckets cp`: ```bash -hf buckets cp hf://datasets/username/my-dataset/ hf://buckets/username/my-bucket/ +hf cp hf://datasets/username/my-dataset/ hf://buckets/username/my-bucket/ ``` Only the Xet content hashes are migrated, so even very large files copy instantly. See [Copying files between repos and buckets](./storage-buckets#copying-files-between-repos-and-buckets).