diff --git a/docs/hub/datasets-ingesting.md b/docs/hub/datasets-ingesting.md index 7384e47f1b..a61cb72687 100644 --- a/docs/hub/datasets-ingesting.md +++ b/docs/hub/datasets-ingesting.md @@ -84,6 +84,14 @@ See the list of [Libraries supported by the Datasets Hub](./datasets-libraries) If you are ingesting raw data that need further curation before being published as AI-ready datasets or if you need an S3-like experience, consider ingesting them to [Hugging Face Storage Buckets](./storage-buckets). +If your data already lives in a dataset, model, or Space repository, you can copy it into a bucket **server-side** — no download or re-upload — with `hf buckets cp`: + +```bash +hf cp hf://datasets/username/my-dataset/ hf://buckets/username/my-bucket/ +``` + +Only the Xet content hashes are migrated, so even very large files copy instantly. See [Copying files between repos and buckets](./storage-buckets#copying-files-between-repos-and-buckets). + ## Scheduled ingestion There are some limitations when updating the same file on the Hub thousands of times. diff --git a/docs/hub/storage-buckets.md b/docs/hub/storage-buckets.md index d367d5305e..867ef71248 100644 --- a/docs/hub/storage-buckets.md +++ b/docs/hub/storage-buckets.md @@ -154,6 +154,8 @@ hf buckets list julien-c/my-training-bucket --tree -h -R You can upload and download files directly from the bucket page on the Hub, or use the CLI and Python API for programmatic access. Bucket files are referenced using `hf://buckets/` paths (e.g., `hf://buckets/username/my-bucket/path/to/file`). The `hf buckets cp` command handles individual file transfers while `hf buckets sync` is better suited for directories. All commands work in both directions — local-to-remote and remote-to-local. +If your data already lives in a model, dataset, or Space repository (or another bucket), you can copy it **server-side** with `hf buckets cp` — no download or re-upload required. See [Copying files between repos and buckets](#copying-files-between-repos-and-buckets). + ### Uploading files For quick uploads, you can drag and drop files directly on the bucket page in your browser. For programmatic use, `hf buckets cp` copies individual files into a bucket. The source is a local path and the destination is an `hf://buckets/` path. You can also pipe data from stdin, which is handy for programmatically generated content. @@ -281,10 +283,10 @@ For more deletion options (pattern-based filtering, recursive removal, etc.), se ### Copying files between repos and buckets -You can copy [Xet](./xet/index)-tracked files from any repository (model, dataset, Space) or bucket into a destination bucket without re-uploading the data. The copy is server-side: only the Xet content hashes are migrated, so even very large files are copied instantly. +You can copy [Xet](./xet/index)-tracked files from any repository (model, dataset, Space) or bucket into a destination bucket without re-uploading the data. The copy is server-side: only the Xet content hashes are migrated, so even very large files are copied instantly thanks to [chunk-level deduplication](./xet/deduplication). > [!NOTE] -> Only Xet-tracked files are copied server-to-server. Small non-Xet files (e.g., config files and READMEs) are automatically downloaded and re-uploaded. +> Only Xet-tracked files are copied server-to-server. Small non-Xet files (e.g., config files and READMEs) are automatically downloaded and re-uploaded. Server-side copy also requires the source and destination to be in the same storage region. **CLI:** ```bash diff --git a/docs/hub/xet/deduplication.md b/docs/hub/xet/deduplication.md index d64123e8af..b13a0348cc 100644 --- a/docs/hub/xet/deduplication.md +++ b/docs/hub/xet/deduplication.md @@ -2,6 +2,9 @@ Xet-enabled repositories utilize [content-defined chunking (CDC)](https://huggingface.co/blog/from-files-to-chunks) to deduplicate on the level of bytes (~64KB of data, also referred to as a "chunk"). Each chunk is identified by a rolling hash that determines chunk boundaries based on the actual file contents, making it resilient to insertions or deletions anywhere in the file. When a file is uploaded to a Xet-backed repository using a Xet-aware client, its contents are broken down into these variable-sized chunks. Only new chunks not already present in Xet storage are kept after chunking, everything else is discarded. +> [!TIP] +> Chunk-level deduplication is also what makes **server-side copies instant**: `hf buckets cp` between repositories and buckets migrates only content hashes, with no data re-upload. See [Copying files between repos and buckets](../storage-buckets#copying-files-between-repos-and-buckets). + ## How Content-Defined Chunking Works To understand content-defined chunking, imagine a file as a long passage of text. The system scans the data using a rolling hash — a small mathematical function that slides over the bytes. Whenever the hash hits a special pattern, a chunk boundary is placed at that position. Because the boundaries are determined by the *content itself* (not by fixed positions), identical regions of data always produce the same chunks, even if surrounding content changes.