Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/xet/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ After cutting a chunk of data, the chunk hash is computed via a blake3 keyed has
]
```

[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/data_hash.rs#L308-L311)
[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/merklehash/data_hash.rs#L310-L313)

## Xorb Hashes

Expand All @@ -39,7 +39,7 @@ The hash function used to compute internal node hashes is as follows:
- finally a newline `\n` character
- Then take the bytes from this string and compute a blake3 keyed hash with the following key (INTERNAL_NODE_KEY)

[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/aggregated_hashes.rs#L103-L109)
[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/merklehash/aggregated_hashes.rs#L123-L136)

### INTERNAL_NODE_KEY

Expand Down Expand Up @@ -95,7 +95,7 @@ After chunking a whole file, to compute the file hash, follow the same procedure
This means create a MerkleTree using the same hashing functions described in the previous section.
Then take the root node's hash and compute a blake3 keyed hash with the key being 32 0-value bytes.

[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/aggregated_hashes.rs#L123-L125)
[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/merklehash/aggregated_hashes.rs#L191-L195)

## Term Verification Hashes

Expand All @@ -117,7 +117,7 @@ To generate this hash, take the chunk hashes for the specific range of chunks th

The result of the blake3 keyed hash is the verification hash that MUST be used in the FileVerificationEntry for the term.

[reference implementation](https://github.com/huggingface/xet-core/blob/main/mdb_shard/src/chunk_verification.rs#L4-L16)
[reference implementation](https://github.com/huggingface/xet-core/blob/c3c726bed5cf54ded92a63fab892cfb7857c751a/xet_core_structures/src/metadata_shard/chunk_verification.rs#L4-L16)

### Example Python code for the verification hash

Expand Down
16 changes: 8 additions & 8 deletions docs/xet/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,13 @@ Implementors can create their own clients, SDKs, and tools that speak the Xet pr

The primary reference implementation of the protocol written in Rust 🦀 lives in the [xet-core](https://github.com/huggingface/xet-core) repository under multiple crates:

- [cas_types](https://github.com/huggingface/xet-core/tree/main/cas_types) - Common re-usable types for interacting with CAS API's
- [cas_client](https://github.com/huggingface/xet-core/tree/main/cas_client) - Client interface that calls CAS API's, including comprehensive implementation of download protocol.
- [mdb_shard](https://github.com/huggingface/xet-core/tree/main/mdb_shard) - Library for interacting with shards and the shard binary format.
- [deduplication](https://github.com/huggingface/xet-core/tree/main/deduplication) - Exposes interfaces to deduplicate chunks locally and using global deduplication
- [deduplication/src/chunking.rs](https://github.com/huggingface/xet-core/blob/main/deduplication/src/chunking.rs) - The reference implementation of the chunking algorithm.
- [merklehash](https://github.com/huggingface/xet-core/tree/main/merklehash) - Exports a `MerkleHash` type extensively used to represent hashes. Exports functions to compute the different hashes used to track chunks, xorbs and files.
- [data](https://github.com/huggingface/xet-core/tree/main/data) - Comprehensive package exposing interfaces to upload and download contents
- [cas_types](https://github.com/huggingface/xet-core/tree/main/xet_client/src/cas_types) - Common re-usable types for interacting with CAS API's
- [cas_client](https://github.com/huggingface/xet-core/tree/main/xet_client/src/cas_client) - Client interface that calls CAS API's, including comprehensive implementation of download protocol.
- [metadata_shard](https://github.com/huggingface/xet-core/tree/main/xet_core_structures/src/metadata_shard) - Library for interacting with shards and the shard binary format.
- [deduplication](https://github.com/huggingface/xet-core/tree/main/xet_data/src/deduplication) - Exposes interfaces to deduplicate chunks locally and using global deduplication
- [xet_data/src/deduplication/chunking.rs](https://github.com/huggingface/xet-core/blob/main/xet_data/src/deduplication/chunking.rs) - The reference implementation of the chunking algorithm.
- [merklehash](https://github.com/huggingface/xet-core/tree/main/xet_core_structures/src/merklehash) - Exports a `MerkleHash` type extensively used to represent hashes. Exports functions to compute the different hashes used to track chunks, xorbs and files.
- [xet_data](https://github.com/huggingface/xet-core/tree/main/xet_data) - Comprehensive package exposing interfaces to upload and download contents
- [hf_xet](https://github.com/huggingface/xet-core/tree/main/hf_xet) - Python bindings to use the Xet protocol for uploads and downloads with the Hugging Face Hub.
- [git-xet](https://github.com/huggingface/xet-core/tree/main/git_xet) - git lfs custom transfer agent that uploads files using the xet protocol to the Hugging Face Hub.

Expand All @@ -48,4 +48,4 @@ There is also a second reference implementation in Huggingface.js that can be us

- Download uses the `XetBlob` that can be found in [XetBlob.ts](https://github.com/huggingface/huggingface.js/blob/main/packages/hub/src/utils/XetBlob.ts).
- The upload implementation is more comprehensive but the root of it begins in [uploadShards](https://github.com/huggingface/huggingface.js/blob/main/packages/hub/src/utils/uploadShards.ts).
- The upload process uses xet-core constructs compiled from Rust to WebAssembly, particularly all functions exported from the [hf_xet_thin_wasm](https://github.com/huggingface/xet-core/tree/main/hf_xet_thin_wasm) crate.
- The upload process uses xet-core constructs compiled from Rust to WebAssembly, particularly all functions exported from the [hf_xet_thin_wasm](https://github.com/huggingface/xet-core/tree/main/wasm/hf_xet_thin_wasm) crate.
2 changes: 1 addition & 1 deletion docs/xet/xorb.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Note that a Xorb MAY contain chunks that utilize different compression schemes.
2. **Best Effort Prediction**

In `xet-core`, to predict if BG4 will be useful we maximum KL divergence between the distribution of per-byte pop-counts on a sample of each of the 4 groups that would be formed.
You can read more about it in [bg4_prediction.rs](https://github.com/huggingface/xet-core/blob/main/cas_object/src/byte_grouping/bg4_prediction.rs) and accompanying scripts.
You can read more about it in [bg4_prediction.rs](https://github.com/huggingface/xet-core/blob/main/xet_core_structures/src/xorb_object/byte_grouping/bg4_prediction.rs) and accompanying scripts.

If the predictor does not show that BG4 will be better, we use Lz4 and in either case we will store the chunk as the uncompressed version if the compression scheme used does not show any benefit.

Expand Down
Loading