From 76906c2ada91c7b60fa92616008afd3271cb7de3 Mon Sep 17 00:00:00 2001 From: Assaf Vayner Date: Thu, 14 May 2026 14:30:45 -0700 Subject: [PATCH 1/2] docs(xet): update broken xet-core links after repo re-org The xet-core repository was reorganized: the per-crate top-level directories (cas_types, cas_client, mdb_shard, merklehash, deduplication, data, hf_xet_thin_wasm, cas_object) were merged into a smaller set of larger crates (xet_client, xet_core_structures, xet_data, wasm/hf_xet_thin_wasm). Update all references in docs/xet/ to point at the new paths, and shift the line ranges in hashing.md to the corresponding code in the new locations. --- docs/xet/hashing.md | 8 ++++---- docs/xet/index.md | 16 ++++++++-------- docs/xet/xorb.md | 2 +- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/xet/hashing.md b/docs/xet/hashing.md index 6b2c3ab7c9..dc936e8e2f 100644 --- a/docs/xet/hashing.md +++ b/docs/xet/hashing.md @@ -21,7 +21,7 @@ After cutting a chunk of data, the chunk hash is computed via a blake3 keyed has ] ``` -[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/data_hash.rs#L308-L311) +[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/merklehash/data_hash.rs#L310-L313) ## Xorb Hashes @@ -39,7 +39,7 @@ The hash function used to compute internal node hashes is as follows: - finally a newline `\n` character - Then take the bytes from this string and compute a blake3 keyed hash with the following key (INTERNAL_NODE_KEY) -[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/aggregated_hashes.rs#L103-L109) +[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/merklehash/aggregated_hashes.rs#L123-L136) ### INTERNAL_NODE_KEY @@ -95,7 +95,7 @@ After chunking a whole file, to compute the file hash, follow the same procedure This means create a MerkleTree using the same hashing functions described in the previous section. Then take the root node's hash and compute a blake3 keyed hash with the key being 32 0-value bytes. -[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/aggregated_hashes.rs#L123-L125) +[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/merklehash/aggregated_hashes.rs#L191-L195) ## Term Verification Hashes @@ -117,7 +117,7 @@ To generate this hash, take the chunk hashes for the specific range of chunks th The result of the blake3 keyed hash is the verification hash that MUST be used in the FileVerificationEntry for the term. -[reference implementation](https://github.com/huggingface/xet-core/blob/main/mdb_shard/src/chunk_verification.rs#L4-L16) +[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/metadata_shard/chunk_verification.rs#L4-L16) ### Example Python code for the verification hash diff --git a/docs/xet/index.md b/docs/xet/index.md index 18faa58822..8e98cea090 100644 --- a/docs/xet/index.md +++ b/docs/xet/index.md @@ -32,13 +32,13 @@ Implementors can create their own clients, SDKs, and tools that speak the Xet pr The primary reference implementation of the protocol written in Rust 🦀 lives in the [xet-core](https://github.com/huggingface/xet-core) repository under multiple crates: -- [cas_types](https://github.com/huggingface/xet-core/tree/main/cas_types) - Common re-usable types for interacting with CAS API's -- [cas_client](https://github.com/huggingface/xet-core/tree/main/cas_client) - Client interface that calls CAS API's, including comprehensive implementation of download protocol. -- [mdb_shard](https://github.com/huggingface/xet-core/tree/main/mdb_shard) - Library for interacting with shards and the shard binary format. -- [deduplication](https://github.com/huggingface/xet-core/tree/main/deduplication) - Exposes interfaces to deduplicate chunks locally and using global deduplication - - [deduplication/src/chunking.rs](https://github.com/huggingface/xet-core/blob/main/deduplication/src/chunking.rs) - The reference implementation of the chunking algorithm. -- [merklehash](https://github.com/huggingface/xet-core/tree/main/merklehash) - Exports a `MerkleHash` type extensively used to represent hashes. Exports functions to compute the different hashes used to track chunks, xorbs and files. -- [data](https://github.com/huggingface/xet-core/tree/main/data) - Comprehensive package exposing interfaces to upload and download contents +- [cas_types](https://github.com/huggingface/xet-core/tree/main/xet_client/src/cas_types) - Common re-usable types for interacting with CAS API's +- [cas_client](https://github.com/huggingface/xet-core/tree/main/xet_client/src/cas_client) - Client interface that calls CAS API's, including comprehensive implementation of download protocol. +- [metadata_shard](https://github.com/huggingface/xet-core/tree/main/xet_core_structures/src/metadata_shard) - Library for interacting with shards and the shard binary format. +- [deduplication](https://github.com/huggingface/xet-core/tree/main/xet_data/src/deduplication) - Exposes interfaces to deduplicate chunks locally and using global deduplication + - [xet_data/src/deduplication/chunking.rs](https://github.com/huggingface/xet-core/blob/main/xet_data/src/deduplication/chunking.rs) - The reference implementation of the chunking algorithm. +- [merklehash](https://github.com/huggingface/xet-core/tree/main/xet_core_structures/src/merklehash) - Exports a `MerkleHash` type extensively used to represent hashes. Exports functions to compute the different hashes used to track chunks, xorbs and files. +- [xet_data](https://github.com/huggingface/xet-core/tree/main/xet_data) - Comprehensive package exposing interfaces to upload and download contents - [hf_xet](https://github.com/huggingface/xet-core/tree/main/hf_xet) - Python bindings to use the Xet protocol for uploads and downloads with the Hugging Face Hub. - [git-xet](https://github.com/huggingface/xet-core/tree/main/git_xet) - git lfs custom transfer agent that uploads files using the xet protocol to the Hugging Face Hub. @@ -48,4 +48,4 @@ There is also a second reference implementation in Huggingface.js that can be us - Download uses the `XetBlob` that can be found in [XetBlob.ts](https://github.com/huggingface/huggingface.js/blob/main/packages/hub/src/utils/XetBlob.ts). - The upload implementation is more comprehensive but the root of it begins in [uploadShards](https://github.com/huggingface/huggingface.js/blob/main/packages/hub/src/utils/uploadShards.ts). - - The upload process uses xet-core constructs compiled from Rust to WebAssembly, particularly all functions exported from the [hf_xet_thin_wasm](https://github.com/huggingface/xet-core/tree/main/hf_xet_thin_wasm) crate. + - The upload process uses xet-core constructs compiled from Rust to WebAssembly, particularly all functions exported from the [hf_xet_thin_wasm](https://github.com/huggingface/xet-core/tree/main/wasm/hf_xet_thin_wasm) crate. diff --git a/docs/xet/xorb.md b/docs/xet/xorb.md index e043b129ec..57ebd7f015 100644 --- a/docs/xet/xorb.md +++ b/docs/xet/xorb.md @@ -111,7 +111,7 @@ Note that a Xorb MAY contain chunks that utilize different compression schemes. 2. **Best Effort Prediction** In `xet-core`, to predict if BG4 will be useful we maximum KL divergence between the distribution of per-byte pop-counts on a sample of each of the 4 groups that would be formed. - You can read more about it in [bg4_prediction.rs](https://github.com/huggingface/xet-core/blob/main/cas_object/src/byte_grouping/bg4_prediction.rs) and accompanying scripts. + You can read more about it in [bg4_prediction.rs](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/xorb_object/byte_grouping/bg4_prediction.rs) and accompanying scripts. If the predictor does not show that BG4 will be better, we use Lz4 and in either case we will store the chunk as the uncompressed version if the compression scheme used does not show any benefit. From 49af07db94abc2f8e5224ab8d1a4c4ddd885bdda Mon Sep 17 00:00:00 2001 From: Assaf Vayner Date: Fri, 15 May 2026 11:33:25 -0700 Subject: [PATCH 2/2] docs(xet): pin line-anchored xet-core links to a fixed commit The four "reference implementation" links in hashing.md include line anchors (#L...). Pin them to xet-core commit c3c726b so the line numbers don't drift as upstream code changes. --- docs/xet/hashing.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/xet/hashing.md b/docs/xet/hashing.md index dc936e8e2f..54dac31d84 100644 --- a/docs/xet/hashing.md +++ b/docs/xet/hashing.md @@ -21,7 +21,7 @@ After cutting a chunk of data, the chunk hash is computed via a blake3 keyed has ] ``` -[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/merklehash/data_hash.rs#L310-L313) +[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/merklehash/data_hash.rs#L310-L313) ## Xorb Hashes @@ -39,7 +39,7 @@ The hash function used to compute internal node hashes is as follows: - finally a newline `\n` character - Then take the bytes from this string and compute a blake3 keyed hash with the following key (INTERNAL_NODE_KEY) -[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/merklehash/aggregated_hashes.rs#L123-L136) +[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/merklehash/aggregated_hashes.rs#L123-L136) ### INTERNAL_NODE_KEY @@ -95,7 +95,7 @@ After chunking a whole file, to compute the file hash, follow the same procedure This means create a MerkleTree using the same hashing functions described in the previous section. Then take the root node's hash and compute a blake3 keyed hash with the key being 32 0-value bytes. -[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/merklehash/aggregated_hashes.rs#L191-L195) +[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/merklehash/aggregated_hashes.rs#L191-L195) ## Term Verification Hashes @@ -117,7 +117,7 @@ To generate this hash, take the chunk hashes for the specific range of chunks th The result of the blake3 keyed hash is the verification hash that MUST be used in the FileVerificationEntry for the term. -[reference implementation](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/metadata_shard/chunk_verification.rs#L4-L16) +[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/metadata_shard/chunk_verification.rs#L4-L16) ### Example Python code for the verification hash