From 396630787d5ceddac49568380ff32418b1479d7c Mon Sep 17 00:00:00 2001 From: HuggingFaceInfra <148469759+HuggingFaceInfra@users.noreply.github.com> Date: Thu, 25 Sep 2025 11:03:06 +0200 Subject: [PATCH 01/19] Update Inference Providers documentation (automated) (#1953) Co-authored-by: Wauplin <11801849+Wauplin@users.noreply.github.com> --- docs/inference-providers/providers/cerebras.md | 7 +++++-- docs/inference-providers/providers/cohere.md | 7 +++++-- docs/inference-providers/providers/fal-ai.md | 7 +++++-- docs/inference-providers/providers/featherless-ai.md | 7 +++++-- docs/inference-providers/providers/fireworks-ai.md | 7 +++++-- docs/inference-providers/providers/groq.md | 7 +++++-- docs/inference-providers/providers/hf-inference.md | 11 +++++++---- docs/inference-providers/providers/hyperbolic.md | 7 +++++-- docs/inference-providers/providers/nebius.md | 7 +++++-- docs/inference-providers/providers/novita.md | 9 ++++++--- docs/inference-providers/providers/nscale.md | 7 +++++-- docs/inference-providers/providers/publicai.md | 7 +++++-- docs/inference-providers/providers/replicate.md | 7 +++++-- docs/inference-providers/providers/sambanova.md | 7 +++++-- docs/inference-providers/providers/scaleway.md | 7 +++++-- docs/inference-providers/providers/together.md | 7 +++++-- .../inference-providers/tasks/audio-classification.md | 7 +++++-- .../tasks/automatic-speech-recognition.md | 7 +++++-- docs/inference-providers/tasks/chat-completion.md | 2 +- docs/inference-providers/tasks/feature-extraction.md | 9 ++++++--- docs/inference-providers/tasks/fill-mask.md | 9 ++++++--- .../inference-providers/tasks/image-classification.md | 7 +++++-- docs/inference-providers/tasks/image-segmentation.md | 7 +++++-- docs/inference-providers/tasks/image-text-to-text.md | 9 ++++++--- docs/inference-providers/tasks/image-to-image.md | 7 +++++-- docs/inference-providers/tasks/object-detection.md | 7 +++++-- docs/inference-providers/tasks/question-answering.md | 7 +++++-- docs/inference-providers/tasks/summarization.md | 7 +++++-- .../tasks/table-question-answering.md | 7 +++++-- docs/inference-providers/tasks/text-classification.md | 10 ++++++++-- docs/inference-providers/tasks/text-generation.md | 7 +++++-- docs/inference-providers/tasks/text-to-image.md | 7 +++++-- docs/inference-providers/tasks/text-to-video.md | 7 +++++-- .../inference-providers/tasks/token-classification.md | 7 +++++-- docs/inference-providers/tasks/translation.md | 8 +++++--- .../tasks/zero-shot-classification.md | 7 +++++-- 36 files changed, 185 insertions(+), 78 deletions(-) diff --git a/docs/inference-providers/providers/cerebras.md b/docs/inference-providers/providers/cerebras.md index 4235ba8f0e..10a030964b 100644 --- a/docs/inference-providers/providers/cerebras.md +++ b/docs/inference-providers/providers/cerebras.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Cerebras -> [!TIP] -> All supported Cerebras models can be found [here](https://huggingface.co/models?inference_provider=cerebras&sort=trending) + + +All supported Cerebras models can be found [here](https://huggingface.co/models?inference_provider=cerebras&sort=trending) + +
diff --git a/docs/inference-providers/providers/cohere.md b/docs/inference-providers/providers/cohere.md index ed9f1df06f..bce401ec01 100644 --- a/docs/inference-providers/providers/cohere.md +++ b/docs/inference-providers/providers/cohere.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Cohere -> [!TIP] -> All supported Cohere models can be found [here](https://huggingface.co/models?inference_provider=cohere&sort=trending) + + +All supported Cohere models can be found [here](https://huggingface.co/models?inference_provider=cohere&sort=trending) + +
diff --git a/docs/inference-providers/providers/fal-ai.md b/docs/inference-providers/providers/fal-ai.md index ab9a3b8057..8001427650 100644 --- a/docs/inference-providers/providers/fal-ai.md +++ b/docs/inference-providers/providers/fal-ai.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Fal -> [!TIP] -> All supported Fal models can be found [here](https://huggingface.co/models?inference_provider=fal-ai&sort=trending) + + +All supported Fal models can be found [here](https://huggingface.co/models?inference_provider=fal-ai&sort=trending) + +
diff --git a/docs/inference-providers/providers/featherless-ai.md b/docs/inference-providers/providers/featherless-ai.md index eccb16fce7..2f89a71fb4 100644 --- a/docs/inference-providers/providers/featherless-ai.md +++ b/docs/inference-providers/providers/featherless-ai.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Featherless AI -> [!TIP] -> All supported Featherless AI models can be found [here](https://huggingface.co/models?inference_provider=featherless-ai&sort=trending) + + +All supported Featherless AI models can be found [here](https://huggingface.co/models?inference_provider=featherless-ai&sort=trending) + +
diff --git a/docs/inference-providers/providers/fireworks-ai.md b/docs/inference-providers/providers/fireworks-ai.md index dd3d6e9e52..458b4a7954 100644 --- a/docs/inference-providers/providers/fireworks-ai.md +++ b/docs/inference-providers/providers/fireworks-ai.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Fireworks AI -> [!TIP] -> All supported Fireworks AI models can be found [here](https://huggingface.co/models?inference_provider=fireworks-ai&sort=trending) + + +All supported Fireworks AI models can be found [here](https://huggingface.co/models?inference_provider=fireworks-ai&sort=trending) + +
diff --git a/docs/inference-providers/providers/groq.md b/docs/inference-providers/providers/groq.md index 397376accd..80df0503b5 100644 --- a/docs/inference-providers/providers/groq.md +++ b/docs/inference-providers/providers/groq.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Groq -> [!TIP] -> All supported Groq models can be found [here](https://huggingface.co/models?inference_provider=groq&sort=trending) + + +All supported Groq models can be found [here](https://huggingface.co/models?inference_provider=groq&sort=trending) + +
diff --git a/docs/inference-providers/providers/hf-inference.md b/docs/inference-providers/providers/hf-inference.md index 438022dc98..661f85fd61 100644 --- a/docs/inference-providers/providers/hf-inference.md +++ b/docs/inference-providers/providers/hf-inference.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # HF Inference -> [!TIP] -> All supported HF Inference models can be found [here](https://huggingface.co/models?inference_provider=hf-inference&sort=trending) + + +All supported HF Inference models can be found [here](https://huggingface.co/models?inference_provider=hf-inference&sort=trending) + +
@@ -70,7 +73,7 @@ Find out more about Feature Extraction [here](../tasks/feature_extraction). @@ -80,7 +83,7 @@ Find out more about Fill Mask [here](../tasks/fill_mask). diff --git a/docs/inference-providers/providers/hyperbolic.md b/docs/inference-providers/providers/hyperbolic.md index 4d468d41b9..b9c6fbbf2a 100644 --- a/docs/inference-providers/providers/hyperbolic.md +++ b/docs/inference-providers/providers/hyperbolic.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Hyperbolic: The On-Demand AI Cloud -> [!TIP] -> All supported Hyperbolic models can be found [here](https://huggingface.co/models?inference_provider=hyperbolic&sort=trending) + + +All supported Hyperbolic models can be found [here](https://huggingface.co/models?inference_provider=hyperbolic&sort=trending) + +
diff --git a/docs/inference-providers/providers/nebius.md b/docs/inference-providers/providers/nebius.md index 72fc631754..7d7ab29cc7 100644 --- a/docs/inference-providers/providers/nebius.md +++ b/docs/inference-providers/providers/nebius.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Nebius -> [!TIP] -> All supported Nebius models can be found [here](https://huggingface.co/models?inference_provider=nebius&sort=trending) + + +All supported Nebius models can be found [here](https://huggingface.co/models?inference_provider=nebius&sort=trending) + +
diff --git a/docs/inference-providers/providers/novita.md b/docs/inference-providers/providers/novita.md index 1ea629f607..ff76a00172 100644 --- a/docs/inference-providers/providers/novita.md +++ b/docs/inference-providers/providers/novita.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Novita -> [!TIP] -> All supported Novita models can be found [here](https://huggingface.co/models?inference_provider=novita&sort=trending) + + +All supported Novita models can be found [here](https://huggingface.co/models?inference_provider=novita&sort=trending) + +
@@ -59,7 +62,7 @@ Find out more about Chat Completion (VLM) [here](../tasks/chat-completion). diff --git a/docs/inference-providers/providers/nscale.md b/docs/inference-providers/providers/nscale.md index fbb3a3960b..f84d741cb4 100644 --- a/docs/inference-providers/providers/nscale.md +++ b/docs/inference-providers/providers/nscale.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Nscale -> [!TIP] -> All supported Nscale models can be found [here](https://huggingface.co/models?inference_provider=nscale&sort=trending) + + +All supported Nscale models can be found [here](https://huggingface.co/models?inference_provider=nscale&sort=trending) + +
diff --git a/docs/inference-providers/providers/publicai.md b/docs/inference-providers/providers/publicai.md index 21a7627cbb..5d578408ca 100644 --- a/docs/inference-providers/providers/publicai.md +++ b/docs/inference-providers/providers/publicai.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # PublicAI -> [!TIP] -> All models supported by the PublicAI Inference Utility can be found [here](https://huggingface.co/models?inference_provider=publicai&sort=trending) + + +All models supported by the PublicAI Inference Utility can be found [here](https://huggingface.co/models?inference_provider=publicai&sort=trending) + +
diff --git a/docs/inference-providers/providers/replicate.md b/docs/inference-providers/providers/replicate.md index de9832ff4d..677f10d319 100644 --- a/docs/inference-providers/providers/replicate.md +++ b/docs/inference-providers/providers/replicate.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Replicate -> [!TIP] -> All supported Replicate models can be found [here](https://huggingface.co/models?inference_provider=replicate&sort=trending) + + +All supported Replicate models can be found [here](https://huggingface.co/models?inference_provider=replicate&sort=trending) + +
diff --git a/docs/inference-providers/providers/sambanova.md b/docs/inference-providers/providers/sambanova.md index df463faf5c..fefd34b9b2 100644 --- a/docs/inference-providers/providers/sambanova.md +++ b/docs/inference-providers/providers/sambanova.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # SambaNova -> [!TIP] -> All supported SambaNova models can be found [here](https://huggingface.co/models?inference_provider=sambanova&sort=trending) + + +All supported SambaNova models can be found [here](https://huggingface.co/models?inference_provider=sambanova&sort=trending) + +
diff --git a/docs/inference-providers/providers/scaleway.md b/docs/inference-providers/providers/scaleway.md index b44591e367..1dbf5cc4cd 100644 --- a/docs/inference-providers/providers/scaleway.md +++ b/docs/inference-providers/providers/scaleway.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Scaleway -> [!TIP] -> All supported Scaleway models can be found [here](https://huggingface.co/models?inference_provider=scaleway&sort=trending) + + +All supported Scaleway models can be found [here](https://huggingface.co/models?inference_provider=scaleway&sort=trending) + +
diff --git a/docs/inference-providers/providers/together.md b/docs/inference-providers/providers/together.md index f73670045e..9b5035ef77 100644 --- a/docs/inference-providers/providers/together.md +++ b/docs/inference-providers/providers/together.md @@ -19,8 +19,11 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Together -> [!TIP] -> All supported Together models can be found [here](https://huggingface.co/models?inference_provider=together&sort=trending) + + +All supported Together models can be found [here](https://huggingface.co/models?inference_provider=together&sort=trending) + +
diff --git a/docs/inference-providers/tasks/audio-classification.md b/docs/inference-providers/tasks/audio-classification.md index 379f763546..fcc01910fa 100644 --- a/docs/inference-providers/tasks/audio-classification.md +++ b/docs/inference-providers/tasks/audio-classification.md @@ -21,8 +21,11 @@ Example applications: * Identifying a speaker * Detecting the genre of a song -> [!TIP] -> For more details about the `audio-classification` task, check out its [dedicated page](https://huggingface.co/tasks/audio-classification)! You will find examples and related materials. + + +For more details about the `audio-classification` task, check out its [dedicated page](https://huggingface.co/tasks/audio-classification)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/automatic-speech-recognition.md b/docs/inference-providers/tasks/automatic-speech-recognition.md index f7d0f2d73b..5e04d75486 100644 --- a/docs/inference-providers/tasks/automatic-speech-recognition.md +++ b/docs/inference-providers/tasks/automatic-speech-recognition.md @@ -21,8 +21,11 @@ Example applications: * Building a voice assistant * Generating subtitles for a video -> [!TIP] -> For more details about the `automatic-speech-recognition` task, check out its [dedicated page](https://huggingface.co/tasks/automatic-speech-recognition)! You will find examples and related materials. + + +For more details about the `automatic-speech-recognition` task, check out its [dedicated page](https://huggingface.co/tasks/automatic-speech-recognition)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/chat-completion.md b/docs/inference-providers/tasks/chat-completion.md index 01eb79e44e..e2362ac95e 100644 --- a/docs/inference-providers/tasks/chat-completion.md +++ b/docs/inference-providers/tasks/chat-completion.md @@ -74,7 +74,7 @@ conversational /> diff --git a/docs/inference-providers/tasks/feature-extraction.md b/docs/inference-providers/tasks/feature-extraction.md index feeaf95478..1292052509 100644 --- a/docs/inference-providers/tasks/feature-extraction.md +++ b/docs/inference-providers/tasks/feature-extraction.md @@ -21,8 +21,11 @@ Example applications: * Reranking a list of documents based on their similarity to a query. * Calculating the similarity between two sentences. -> [!TIP] -> For more details about the `feature-extraction` task, check out its [dedicated page](https://huggingface.co/tasks/feature-extraction)! You will find examples and related materials. + + +For more details about the `feature-extraction` task, check out its [dedicated page](https://huggingface.co/tasks/feature-extraction)! You will find examples and related materials. + + ### Recommended models @@ -35,7 +38,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/fill-mask.md b/docs/inference-providers/tasks/fill-mask.md index 55000d75b3..0e7894c7f8 100644 --- a/docs/inference-providers/tasks/fill-mask.md +++ b/docs/inference-providers/tasks/fill-mask.md @@ -16,8 +16,11 @@ For more details, check out: Mask filling is the task of predicting the right word (token to be precise) in the middle of a sequence. -> [!TIP] -> For more details about the `fill-mask` task, check out its [dedicated page](https://huggingface.co/tasks/fill-mask)! You will find examples and related materials. + + +For more details about the `fill-mask` task, check out its [dedicated page](https://huggingface.co/tasks/fill-mask)! You will find examples and related materials. + + ### Recommended models @@ -30,7 +33,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/image-classification.md b/docs/inference-providers/tasks/image-classification.md index 97ac6c361a..6c3864f8d7 100644 --- a/docs/inference-providers/tasks/image-classification.md +++ b/docs/inference-providers/tasks/image-classification.md @@ -16,8 +16,11 @@ For more details, check out: Image classification is the task of assigning a label or class to an entire image. Images are expected to have only one class for each image. -> [!TIP] -> For more details about the `image-classification` task, check out its [dedicated page](https://huggingface.co/tasks/image-classification)! You will find examples and related materials. + + +For more details about the `image-classification` task, check out its [dedicated page](https://huggingface.co/tasks/image-classification)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/image-segmentation.md b/docs/inference-providers/tasks/image-segmentation.md index 48bfdb28d9..1ceca0e685 100644 --- a/docs/inference-providers/tasks/image-segmentation.md +++ b/docs/inference-providers/tasks/image-segmentation.md @@ -16,8 +16,11 @@ For more details, check out: Image Segmentation divides an image into segments where each pixel in the image is mapped to an object. -> [!TIP] -> For more details about the `image-segmentation` task, check out its [dedicated page](https://huggingface.co/tasks/image-segmentation)! You will find examples and related materials. + + +For more details about the `image-segmentation` task, check out its [dedicated page](https://huggingface.co/tasks/image-segmentation)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/image-text-to-text.md b/docs/inference-providers/tasks/image-text-to-text.md index 5ad5a4dcdc..d9bb7e10ad 100644 --- a/docs/inference-providers/tasks/image-text-to-text.md +++ b/docs/inference-providers/tasks/image-text-to-text.md @@ -16,8 +16,11 @@ For more details, check out: Image-text-to-text models take in an image and text prompt and output text. These models are also called vision-language models, or VLMs. The difference from image-to-text models is that these models take an additional text input, not restricting the model to certain use cases like image captioning, and may also be trained to accept a conversation as input. -> [!TIP] -> For more details about the `image-text-to-text` task, check out its [dedicated page](https://huggingface.co/tasks/image-text-to-text)! You will find examples and related materials. + + +For more details about the `image-text-to-text` task, check out its [dedicated page](https://huggingface.co/tasks/image-text-to-text)! You will find examples and related materials. + + ### Recommended models @@ -30,7 +33,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/image-to-image.md b/docs/inference-providers/tasks/image-to-image.md index 4515d9d164..3926142052 100644 --- a/docs/inference-providers/tasks/image-to-image.md +++ b/docs/inference-providers/tasks/image-to-image.md @@ -21,8 +21,11 @@ Example applications: * Colorizing a black and white image * Increasing the resolution of an image -> [!TIP] -> For more details about the `image-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/image-to-image)! You will find examples and related materials. + + +For more details about the `image-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/image-to-image)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/object-detection.md b/docs/inference-providers/tasks/object-detection.md index d484cb1df5..3c36c4081b 100644 --- a/docs/inference-providers/tasks/object-detection.md +++ b/docs/inference-providers/tasks/object-detection.md @@ -16,8 +16,11 @@ For more details, check out: Object Detection models allow users to identify objects of certain defined classes. These models receive an image as input and output the images with bounding boxes and labels on detected objects. -> [!TIP] -> For more details about the `object-detection` task, check out its [dedicated page](https://huggingface.co/tasks/object-detection)! You will find examples and related materials. + + +For more details about the `object-detection` task, check out its [dedicated page](https://huggingface.co/tasks/object-detection)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/question-answering.md b/docs/inference-providers/tasks/question-answering.md index 2f4a4a436e..2f13300148 100644 --- a/docs/inference-providers/tasks/question-answering.md +++ b/docs/inference-providers/tasks/question-answering.md @@ -16,8 +16,11 @@ For more details, check out: Question Answering models can retrieve the answer to a question from a given text, which is useful for searching for an answer in a document. -> [!TIP] -> For more details about the `question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/question-answering)! You will find examples and related materials. + + +For more details about the `question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/question-answering)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/summarization.md b/docs/inference-providers/tasks/summarization.md index 1149568500..6d39944062 100644 --- a/docs/inference-providers/tasks/summarization.md +++ b/docs/inference-providers/tasks/summarization.md @@ -16,8 +16,11 @@ For more details, check out: Summarization is the task of producing a shorter version of a document while preserving its important information. Some models can extract text from the original input, while other models can generate entirely new text. -> [!TIP] -> For more details about the `summarization` task, check out its [dedicated page](https://huggingface.co/tasks/summarization)! You will find examples and related materials. + + +For more details about the `summarization` task, check out its [dedicated page](https://huggingface.co/tasks/summarization)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/table-question-answering.md b/docs/inference-providers/tasks/table-question-answering.md index 4834bc808d..358287c697 100644 --- a/docs/inference-providers/tasks/table-question-answering.md +++ b/docs/inference-providers/tasks/table-question-answering.md @@ -16,8 +16,11 @@ For more details, check out: Table Question Answering (Table QA) is the answering a question about an information on a given table. -> [!TIP] -> For more details about the `table-question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/table-question-answering)! You will find examples and related materials. + + +For more details about the `table-question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/table-question-answering)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/text-classification.md b/docs/inference-providers/tasks/text-classification.md index 5274c67ab5..db519fb91d 100644 --- a/docs/inference-providers/tasks/text-classification.md +++ b/docs/inference-providers/tasks/text-classification.md @@ -16,13 +16,19 @@ For more details, check out: Text Classification is the task of assigning a label or class to a given text. Some use cases are sentiment analysis, natural language inference, and assessing grammatical correctness. -> [!TIP] -> For more details about the `text-classification` task, check out its [dedicated page](https://huggingface.co/tasks/text-classification)! You will find examples and related materials. + + +For more details about the `text-classification` task, check out its [dedicated page](https://huggingface.co/tasks/text-classification)! You will find examples and related materials. + + ### Recommended models - [distilbert/distilbert-base-uncased-finetuned-sst-2-english](https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english): A robust model trained for sentiment analysis. - [ProsusAI/finbert](https://huggingface.co/ProsusAI/finbert): A sentiment analysis model specialized in financial sentiment. +- [cardiffnlp/twitter-roberta-base-sentiment-latest](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment-latest): A sentiment analysis model specialized in analyzing tweets. +- [papluca/xlm-roberta-base-language-detection](https://huggingface.co/papluca/xlm-roberta-base-language-detection): A model that can classify languages. +- [meta-llama/Prompt-Guard-86M](https://huggingface.co/meta-llama/Prompt-Guard-86M): A model that can classify text generation attacks. Explore all available models and find the one that suits you best [here](https://huggingface.co/models?inference=warm&pipeline_tag=text-classification&sort=trending). diff --git a/docs/inference-providers/tasks/text-generation.md b/docs/inference-providers/tasks/text-generation.md index f929e1cb07..3cf64a02bc 100644 --- a/docs/inference-providers/tasks/text-generation.md +++ b/docs/inference-providers/tasks/text-generation.md @@ -18,8 +18,11 @@ Generate text based on a prompt. If you are interested in a Chat Completion task, which generates a response based on a list of messages, check out the [`chat-completion`](./chat_completion) task. -> [!TIP] -> For more details about the `text-generation` task, check out its [dedicated page](https://huggingface.co/tasks/text-generation)! You will find examples and related materials. + + +For more details about the `text-generation` task, check out its [dedicated page](https://huggingface.co/tasks/text-generation)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/text-to-image.md b/docs/inference-providers/tasks/text-to-image.md index 2b485f7b7a..08eaa0b37d 100644 --- a/docs/inference-providers/tasks/text-to-image.md +++ b/docs/inference-providers/tasks/text-to-image.md @@ -16,8 +16,11 @@ For more details, check out: Generate an image based on a given text prompt. -> [!TIP] -> For more details about the `text-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-image)! You will find examples and related materials. + + +For more details about the `text-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-image)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/text-to-video.md b/docs/inference-providers/tasks/text-to-video.md index 3b17a8e572..9d8625958e 100644 --- a/docs/inference-providers/tasks/text-to-video.md +++ b/docs/inference-providers/tasks/text-to-video.md @@ -16,8 +16,11 @@ For more details, check out: Generate an video based on a given text prompt. -> [!TIP] -> For more details about the `text-to-video` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-video)! You will find examples and related materials. + + +For more details about the `text-to-video` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-video)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/token-classification.md b/docs/inference-providers/tasks/token-classification.md index 5bbcfb447b..3636602eb9 100644 --- a/docs/inference-providers/tasks/token-classification.md +++ b/docs/inference-providers/tasks/token-classification.md @@ -16,8 +16,11 @@ For more details, check out: Token classification is a task in which a label is assigned to some tokens in a text. Some popular token classification subtasks are Named Entity Recognition (NER) and Part-of-Speech (PoS) tagging. -> [!TIP] -> For more details about the `token-classification` task, check out its [dedicated page](https://huggingface.co/tasks/token-classification)! You will find examples and related materials. + + +For more details about the `token-classification` task, check out its [dedicated page](https://huggingface.co/tasks/token-classification)! You will find examples and related materials. + + ### Recommended models diff --git a/docs/inference-providers/tasks/translation.md b/docs/inference-providers/tasks/translation.md index 94ecc0e5c5..6366a4807d 100644 --- a/docs/inference-providers/tasks/translation.md +++ b/docs/inference-providers/tasks/translation.md @@ -16,12 +16,14 @@ For more details, check out: Translation is the task of converting text from one language to another. -> [!TIP] -> For more details about the `translation` task, check out its [dedicated page](https://huggingface.co/tasks/translation)! You will find examples and related materials. + + +For more details about the `translation` task, check out its [dedicated page](https://huggingface.co/tasks/translation)! You will find examples and related materials. + + ### Recommended models -- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base): A general-purpose Transformer that can be used to translate from English to German, French, or Romanian. Explore all available models and find the one that suits you best [here](https://huggingface.co/models?inference=warm&pipeline_tag=translation&sort=trending). diff --git a/docs/inference-providers/tasks/zero-shot-classification.md b/docs/inference-providers/tasks/zero-shot-classification.md index 08851fdf11..6a9f4768a6 100644 --- a/docs/inference-providers/tasks/zero-shot-classification.md +++ b/docs/inference-providers/tasks/zero-shot-classification.md @@ -16,8 +16,11 @@ For more details, check out: Zero-shot text classification is super useful to try out classification with zero code, you simply pass a sentence/paragraph and the possible labels for that sentence, and you get a result. The model has not been necessarily trained on the labels you provide, but it can still predict the correct label. -> [!TIP] -> For more details about the `zero-shot-classification` task, check out its [dedicated page](https://huggingface.co/tasks/zero-shot-classification)! You will find examples and related materials. + + +For more details about the `zero-shot-classification` task, check out its [dedicated page](https://huggingface.co/tasks/zero-shot-classification)! You will find examples and related materials. + + ### Recommended models From 766101e85deac44159ab1da47b985177345ab48d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?c=C3=A9lina?= Date: Thu, 25 Sep 2025 14:40:36 +0200 Subject: [PATCH 02/19] update Inference Providers templates (#1955) --- docs/inference-providers/providers/cerebras.md | 7 ++----- docs/inference-providers/providers/cohere.md | 7 ++----- docs/inference-providers/providers/fal-ai.md | 7 ++----- docs/inference-providers/providers/featherless-ai.md | 7 ++----- docs/inference-providers/providers/fireworks-ai.md | 7 ++----- docs/inference-providers/providers/groq.md | 7 ++----- docs/inference-providers/providers/hf-inference.md | 9 +++------ docs/inference-providers/providers/hyperbolic.md | 7 ++----- docs/inference-providers/providers/nebius.md | 7 ++----- docs/inference-providers/providers/novita.md | 9 +++------ docs/inference-providers/providers/nscale.md | 7 ++----- docs/inference-providers/providers/publicai.md | 7 ++----- docs/inference-providers/providers/replicate.md | 7 ++----- docs/inference-providers/providers/sambanova.md | 7 ++----- docs/inference-providers/providers/scaleway.md | 7 ++----- docs/inference-providers/providers/together.md | 7 ++----- docs/inference-providers/tasks/audio-classification.md | 6 ++---- .../tasks/automatic-speech-recognition.md | 6 ++---- docs/inference-providers/tasks/chat-completion.md | 2 +- docs/inference-providers/tasks/feature-extraction.md | 8 +++----- docs/inference-providers/tasks/fill-mask.md | 6 ++---- docs/inference-providers/tasks/image-classification.md | 6 ++---- docs/inference-providers/tasks/image-segmentation.md | 6 ++---- docs/inference-providers/tasks/image-text-to-text.md | 6 ++---- docs/inference-providers/tasks/image-to-image.md | 6 ++---- docs/inference-providers/tasks/object-detection.md | 6 ++---- docs/inference-providers/tasks/question-answering.md | 6 ++---- docs/inference-providers/tasks/summarization.md | 7 ++----- .../tasks/table-question-answering.md | 6 ++---- docs/inference-providers/tasks/text-classification.md | 6 ++---- docs/inference-providers/tasks/text-generation.md | 6 ++---- docs/inference-providers/tasks/text-to-image.md | 6 ++---- docs/inference-providers/tasks/text-to-video.md | 6 ++---- docs/inference-providers/tasks/token-classification.md | 6 ++---- docs/inference-providers/tasks/translation.md | 7 +++---- .../tasks/zero-shot-classification.md | 6 ++---- scripts/inference-providers/scripts/generate.ts | 8 +++----- .../templates/providers/cerebras.handlebars | 7 ++----- .../templates/providers/cohere.handlebars | 7 ++----- .../templates/providers/fal-ai.handlebars | 7 ++----- .../templates/providers/featherless-ai.handlebars | 7 ++----- .../templates/providers/fireworks-ai.handlebars | 7 ++----- .../templates/providers/groq.handlebars | 7 ++----- .../templates/providers/hf-inference.handlebars | 7 ++----- .../templates/providers/hyperbolic.handlebars | 7 ++----- .../templates/providers/nebius.handlebars | 7 ++----- .../templates/providers/novita.handlebars | 7 ++----- .../templates/providers/nscale.handlebars | 7 ++----- .../templates/providers/publicai.handlebars | 7 ++----- .../templates/providers/replicate.handlebars | 7 ++----- .../templates/providers/sambanova.handlebars | 7 ++----- .../templates/providers/scaleway.handlebars | 7 ++----- .../templates/providers/together.handlebars | 7 ++----- 53 files changed, 110 insertions(+), 246 deletions(-) diff --git a/docs/inference-providers/providers/cerebras.md b/docs/inference-providers/providers/cerebras.md index 10a030964b..4235ba8f0e 100644 --- a/docs/inference-providers/providers/cerebras.md +++ b/docs/inference-providers/providers/cerebras.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Cerebras - - -All supported Cerebras models can be found [here](https://huggingface.co/models?inference_provider=cerebras&sort=trending) - - +> [!TIP] +> All supported Cerebras models can be found [here](https://huggingface.co/models?inference_provider=cerebras&sort=trending)
diff --git a/docs/inference-providers/providers/cohere.md b/docs/inference-providers/providers/cohere.md index bce401ec01..ed9f1df06f 100644 --- a/docs/inference-providers/providers/cohere.md +++ b/docs/inference-providers/providers/cohere.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Cohere - - -All supported Cohere models can be found [here](https://huggingface.co/models?inference_provider=cohere&sort=trending) - - +> [!TIP] +> All supported Cohere models can be found [here](https://huggingface.co/models?inference_provider=cohere&sort=trending)
diff --git a/docs/inference-providers/providers/fal-ai.md b/docs/inference-providers/providers/fal-ai.md index 8001427650..ab9a3b8057 100644 --- a/docs/inference-providers/providers/fal-ai.md +++ b/docs/inference-providers/providers/fal-ai.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Fal - - -All supported Fal models can be found [here](https://huggingface.co/models?inference_provider=fal-ai&sort=trending) - - +> [!TIP] +> All supported Fal models can be found [here](https://huggingface.co/models?inference_provider=fal-ai&sort=trending)
diff --git a/docs/inference-providers/providers/featherless-ai.md b/docs/inference-providers/providers/featherless-ai.md index 2f89a71fb4..eccb16fce7 100644 --- a/docs/inference-providers/providers/featherless-ai.md +++ b/docs/inference-providers/providers/featherless-ai.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Featherless AI - - -All supported Featherless AI models can be found [here](https://huggingface.co/models?inference_provider=featherless-ai&sort=trending) - - +> [!TIP] +> All supported Featherless AI models can be found [here](https://huggingface.co/models?inference_provider=featherless-ai&sort=trending)
diff --git a/docs/inference-providers/providers/fireworks-ai.md b/docs/inference-providers/providers/fireworks-ai.md index 458b4a7954..dd3d6e9e52 100644 --- a/docs/inference-providers/providers/fireworks-ai.md +++ b/docs/inference-providers/providers/fireworks-ai.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Fireworks AI - - -All supported Fireworks AI models can be found [here](https://huggingface.co/models?inference_provider=fireworks-ai&sort=trending) - - +> [!TIP] +> All supported Fireworks AI models can be found [here](https://huggingface.co/models?inference_provider=fireworks-ai&sort=trending)
diff --git a/docs/inference-providers/providers/groq.md b/docs/inference-providers/providers/groq.md index 80df0503b5..397376accd 100644 --- a/docs/inference-providers/providers/groq.md +++ b/docs/inference-providers/providers/groq.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Groq - - -All supported Groq models can be found [here](https://huggingface.co/models?inference_provider=groq&sort=trending) - - +> [!TIP] +> All supported Groq models can be found [here](https://huggingface.co/models?inference_provider=groq&sort=trending)
diff --git a/docs/inference-providers/providers/hf-inference.md b/docs/inference-providers/providers/hf-inference.md index 661f85fd61..29ac5d82e5 100644 --- a/docs/inference-providers/providers/hf-inference.md +++ b/docs/inference-providers/providers/hf-inference.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # HF Inference - - -All supported HF Inference models can be found [here](https://huggingface.co/models?inference_provider=hf-inference&sort=trending) - - +> [!TIP] +> All supported HF Inference models can be found [here](https://huggingface.co/models?inference_provider=hf-inference&sort=trending)
@@ -73,7 +70,7 @@ Find out more about Feature Extraction [here](../tasks/feature_extraction). diff --git a/docs/inference-providers/providers/hyperbolic.md b/docs/inference-providers/providers/hyperbolic.md index b9c6fbbf2a..4d468d41b9 100644 --- a/docs/inference-providers/providers/hyperbolic.md +++ b/docs/inference-providers/providers/hyperbolic.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Hyperbolic: The On-Demand AI Cloud - - -All supported Hyperbolic models can be found [here](https://huggingface.co/models?inference_provider=hyperbolic&sort=trending) - - +> [!TIP] +> All supported Hyperbolic models can be found [here](https://huggingface.co/models?inference_provider=hyperbolic&sort=trending)
diff --git a/docs/inference-providers/providers/nebius.md b/docs/inference-providers/providers/nebius.md index 7d7ab29cc7..72fc631754 100644 --- a/docs/inference-providers/providers/nebius.md +++ b/docs/inference-providers/providers/nebius.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Nebius - - -All supported Nebius models can be found [here](https://huggingface.co/models?inference_provider=nebius&sort=trending) - - +> [!TIP] +> All supported Nebius models can be found [here](https://huggingface.co/models?inference_provider=nebius&sort=trending)
diff --git a/docs/inference-providers/providers/novita.md b/docs/inference-providers/providers/novita.md index ff76a00172..47cf4d6f69 100644 --- a/docs/inference-providers/providers/novita.md +++ b/docs/inference-providers/providers/novita.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Novita - - -All supported Novita models can be found [here](https://huggingface.co/models?inference_provider=novita&sort=trending) - - +> [!TIP] +> All supported Novita models can be found [here](https://huggingface.co/models?inference_provider=novita&sort=trending)
@@ -52,7 +49,7 @@ Find out more about Chat Completion (LLM) [here](../tasks/chat-completion). diff --git a/docs/inference-providers/providers/nscale.md b/docs/inference-providers/providers/nscale.md index f84d741cb4..fbb3a3960b 100644 --- a/docs/inference-providers/providers/nscale.md +++ b/docs/inference-providers/providers/nscale.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Nscale - - -All supported Nscale models can be found [here](https://huggingface.co/models?inference_provider=nscale&sort=trending) - - +> [!TIP] +> All supported Nscale models can be found [here](https://huggingface.co/models?inference_provider=nscale&sort=trending)
diff --git a/docs/inference-providers/providers/publicai.md b/docs/inference-providers/providers/publicai.md index 5d578408ca..21a7627cbb 100644 --- a/docs/inference-providers/providers/publicai.md +++ b/docs/inference-providers/providers/publicai.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # PublicAI - - -All models supported by the PublicAI Inference Utility can be found [here](https://huggingface.co/models?inference_provider=publicai&sort=trending) - - +> [!TIP] +> All models supported by the PublicAI Inference Utility can be found [here](https://huggingface.co/models?inference_provider=publicai&sort=trending)
diff --git a/docs/inference-providers/providers/replicate.md b/docs/inference-providers/providers/replicate.md index 677f10d319..de9832ff4d 100644 --- a/docs/inference-providers/providers/replicate.md +++ b/docs/inference-providers/providers/replicate.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Replicate - - -All supported Replicate models can be found [here](https://huggingface.co/models?inference_provider=replicate&sort=trending) - - +> [!TIP] +> All supported Replicate models can be found [here](https://huggingface.co/models?inference_provider=replicate&sort=trending)
diff --git a/docs/inference-providers/providers/sambanova.md b/docs/inference-providers/providers/sambanova.md index fefd34b9b2..df463faf5c 100644 --- a/docs/inference-providers/providers/sambanova.md +++ b/docs/inference-providers/providers/sambanova.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # SambaNova - - -All supported SambaNova models can be found [here](https://huggingface.co/models?inference_provider=sambanova&sort=trending) - - +> [!TIP] +> All supported SambaNova models can be found [here](https://huggingface.co/models?inference_provider=sambanova&sort=trending)
diff --git a/docs/inference-providers/providers/scaleway.md b/docs/inference-providers/providers/scaleway.md index 1dbf5cc4cd..b44591e367 100644 --- a/docs/inference-providers/providers/scaleway.md +++ b/docs/inference-providers/providers/scaleway.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Scaleway - - -All supported Scaleway models can be found [here](https://huggingface.co/models?inference_provider=scaleway&sort=trending) - - +> [!TIP] +> All supported Scaleway models can be found [here](https://huggingface.co/models?inference_provider=scaleway&sort=trending)
diff --git a/docs/inference-providers/providers/together.md b/docs/inference-providers/providers/together.md index 9b5035ef77..f73670045e 100644 --- a/docs/inference-providers/providers/together.md +++ b/docs/inference-providers/providers/together.md @@ -19,11 +19,8 @@ For more details, check out the `generate.ts` script: https://github.com/hugging # Together - - -All supported Together models can be found [here](https://huggingface.co/models?inference_provider=together&sort=trending) - - +> [!TIP] +> All supported Together models can be found [here](https://huggingface.co/models?inference_provider=together&sort=trending)
diff --git a/docs/inference-providers/tasks/audio-classification.md b/docs/inference-providers/tasks/audio-classification.md index fcc01910fa..3c31cb593c 100644 --- a/docs/inference-providers/tasks/audio-classification.md +++ b/docs/inference-providers/tasks/audio-classification.md @@ -21,11 +21,9 @@ Example applications: * Identifying a speaker * Detecting the genre of a song - +> [!TIP] +> For more details about the `audio-classification` task, check out its [dedicated page](https://huggingface.co/tasks/audio-classification)! You will find examples and related materials. -For more details about the `audio-classification` task, check out its [dedicated page](https://huggingface.co/tasks/audio-classification)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/automatic-speech-recognition.md b/docs/inference-providers/tasks/automatic-speech-recognition.md index 5e04d75486..bf8b65e14e 100644 --- a/docs/inference-providers/tasks/automatic-speech-recognition.md +++ b/docs/inference-providers/tasks/automatic-speech-recognition.md @@ -21,11 +21,9 @@ Example applications: * Building a voice assistant * Generating subtitles for a video - +> [!TIP] +> For more details about the `automatic-speech-recognition` task, check out its [dedicated page](https://huggingface.co/tasks/automatic-speech-recognition)! You will find examples and related materials. -For more details about the `automatic-speech-recognition` task, check out its [dedicated page](https://huggingface.co/tasks/automatic-speech-recognition)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/chat-completion.md b/docs/inference-providers/tasks/chat-completion.md index e2362ac95e..8da832fb2f 100644 --- a/docs/inference-providers/tasks/chat-completion.md +++ b/docs/inference-providers/tasks/chat-completion.md @@ -64,7 +64,7 @@ The API supports: diff --git a/docs/inference-providers/tasks/feature-extraction.md b/docs/inference-providers/tasks/feature-extraction.md index 1292052509..cbfa46c9e4 100644 --- a/docs/inference-providers/tasks/feature-extraction.md +++ b/docs/inference-providers/tasks/feature-extraction.md @@ -21,11 +21,9 @@ Example applications: * Reranking a list of documents based on their similarity to a query. * Calculating the similarity between two sentences. - +> [!TIP] +> For more details about the `feature-extraction` task, check out its [dedicated page](https://huggingface.co/tasks/feature-extraction)! You will find examples and related materials. -For more details about the `feature-extraction` task, check out its [dedicated page](https://huggingface.co/tasks/feature-extraction)! You will find examples and related materials. - - ### Recommended models @@ -38,7 +36,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/fill-mask.md b/docs/inference-providers/tasks/fill-mask.md index 0e7894c7f8..3e68afc3e7 100644 --- a/docs/inference-providers/tasks/fill-mask.md +++ b/docs/inference-providers/tasks/fill-mask.md @@ -16,11 +16,9 @@ For more details, check out: Mask filling is the task of predicting the right word (token to be precise) in the middle of a sequence. - +> [!TIP] +> For more details about the `fill-mask` task, check out its [dedicated page](https://huggingface.co/tasks/fill-mask)! You will find examples and related materials. -For more details about the `fill-mask` task, check out its [dedicated page](https://huggingface.co/tasks/fill-mask)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/image-classification.md b/docs/inference-providers/tasks/image-classification.md index 6c3864f8d7..8fa784a7d6 100644 --- a/docs/inference-providers/tasks/image-classification.md +++ b/docs/inference-providers/tasks/image-classification.md @@ -16,11 +16,9 @@ For more details, check out: Image classification is the task of assigning a label or class to an entire image. Images are expected to have only one class for each image. - +> [!TIP] +> For more details about the `image-classification` task, check out its [dedicated page](https://huggingface.co/tasks/image-classification)! You will find examples and related materials. -For more details about the `image-classification` task, check out its [dedicated page](https://huggingface.co/tasks/image-classification)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/image-segmentation.md b/docs/inference-providers/tasks/image-segmentation.md index 1ceca0e685..eeeab94cff 100644 --- a/docs/inference-providers/tasks/image-segmentation.md +++ b/docs/inference-providers/tasks/image-segmentation.md @@ -16,11 +16,9 @@ For more details, check out: Image Segmentation divides an image into segments where each pixel in the image is mapped to an object. - +> [!TIP] +> For more details about the `image-segmentation` task, check out its [dedicated page](https://huggingface.co/tasks/image-segmentation)! You will find examples and related materials. -For more details about the `image-segmentation` task, check out its [dedicated page](https://huggingface.co/tasks/image-segmentation)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/image-text-to-text.md b/docs/inference-providers/tasks/image-text-to-text.md index d9bb7e10ad..d31105008f 100644 --- a/docs/inference-providers/tasks/image-text-to-text.md +++ b/docs/inference-providers/tasks/image-text-to-text.md @@ -16,11 +16,9 @@ For more details, check out: Image-text-to-text models take in an image and text prompt and output text. These models are also called vision-language models, or VLMs. The difference from image-to-text models is that these models take an additional text input, not restricting the model to certain use cases like image captioning, and may also be trained to accept a conversation as input. - +> [!TIP] +> For more details about the `image-text-to-text` task, check out its [dedicated page](https://huggingface.co/tasks/image-text-to-text)! You will find examples and related materials. -For more details about the `image-text-to-text` task, check out its [dedicated page](https://huggingface.co/tasks/image-text-to-text)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/image-to-image.md b/docs/inference-providers/tasks/image-to-image.md index 3926142052..a78612bbca 100644 --- a/docs/inference-providers/tasks/image-to-image.md +++ b/docs/inference-providers/tasks/image-to-image.md @@ -21,11 +21,9 @@ Example applications: * Colorizing a black and white image * Increasing the resolution of an image - +> [!TIP] +> For more details about the `image-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/image-to-image)! You will find examples and related materials. -For more details about the `image-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/image-to-image)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/object-detection.md b/docs/inference-providers/tasks/object-detection.md index 3c36c4081b..42c8517bad 100644 --- a/docs/inference-providers/tasks/object-detection.md +++ b/docs/inference-providers/tasks/object-detection.md @@ -16,11 +16,9 @@ For more details, check out: Object Detection models allow users to identify objects of certain defined classes. These models receive an image as input and output the images with bounding boxes and labels on detected objects. - +> [!TIP] +> For more details about the `object-detection` task, check out its [dedicated page](https://huggingface.co/tasks/object-detection)! You will find examples and related materials. -For more details about the `object-detection` task, check out its [dedicated page](https://huggingface.co/tasks/object-detection)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/question-answering.md b/docs/inference-providers/tasks/question-answering.md index 2f13300148..7851001241 100644 --- a/docs/inference-providers/tasks/question-answering.md +++ b/docs/inference-providers/tasks/question-answering.md @@ -16,11 +16,9 @@ For more details, check out: Question Answering models can retrieve the answer to a question from a given text, which is useful for searching for an answer in a document. - +> [!TIP] +> For more details about the `question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/question-answering)! You will find examples and related materials. -For more details about the `question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/question-answering)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/summarization.md b/docs/inference-providers/tasks/summarization.md index 6d39944062..b1e43a71e3 100644 --- a/docs/inference-providers/tasks/summarization.md +++ b/docs/inference-providers/tasks/summarization.md @@ -16,16 +16,13 @@ For more details, check out: Summarization is the task of producing a shorter version of a document while preserving its important information. Some models can extract text from the original input, while other models can generate entirely new text. - +> [!TIP] +> For more details about the `summarization` task, check out its [dedicated page](https://huggingface.co/tasks/summarization)! You will find examples and related materials. -For more details about the `summarization` task, check out its [dedicated page](https://huggingface.co/tasks/summarization)! You will find examples and related materials. - - ### Recommended models - [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn): A strong summarization model trained on English news articles. Excels at generating factual summaries. -- [Falconsai/medical_summarization](https://huggingface.co/Falconsai/medical_summarization): A summarization model trained on medical articles. Explore all available models and find the one that suits you best [here](https://huggingface.co/models?inference=warm&pipeline_tag=summarization&sort=trending). diff --git a/docs/inference-providers/tasks/table-question-answering.md b/docs/inference-providers/tasks/table-question-answering.md index 358287c697..42d9d87679 100644 --- a/docs/inference-providers/tasks/table-question-answering.md +++ b/docs/inference-providers/tasks/table-question-answering.md @@ -16,11 +16,9 @@ For more details, check out: Table Question Answering (Table QA) is the answering a question about an information on a given table. - +> [!TIP] +> For more details about the `table-question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/table-question-answering)! You will find examples and related materials. -For more details about the `table-question-answering` task, check out its [dedicated page](https://huggingface.co/tasks/table-question-answering)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/text-classification.md b/docs/inference-providers/tasks/text-classification.md index db519fb91d..f1234fca51 100644 --- a/docs/inference-providers/tasks/text-classification.md +++ b/docs/inference-providers/tasks/text-classification.md @@ -16,11 +16,9 @@ For more details, check out: Text Classification is the task of assigning a label or class to a given text. Some use cases are sentiment analysis, natural language inference, and assessing grammatical correctness. - +> [!TIP] +> For more details about the `text-classification` task, check out its [dedicated page](https://huggingface.co/tasks/text-classification)! You will find examples and related materials. -For more details about the `text-classification` task, check out its [dedicated page](https://huggingface.co/tasks/text-classification)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/text-generation.md b/docs/inference-providers/tasks/text-generation.md index 3cf64a02bc..12b47fb921 100644 --- a/docs/inference-providers/tasks/text-generation.md +++ b/docs/inference-providers/tasks/text-generation.md @@ -18,11 +18,9 @@ Generate text based on a prompt. If you are interested in a Chat Completion task, which generates a response based on a list of messages, check out the [`chat-completion`](./chat_completion) task. - +> [!TIP] +> For more details about the `text-generation` task, check out its [dedicated page](https://huggingface.co/tasks/text-generation)! You will find examples and related materials. -For more details about the `text-generation` task, check out its [dedicated page](https://huggingface.co/tasks/text-generation)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/text-to-image.md b/docs/inference-providers/tasks/text-to-image.md index 08eaa0b37d..94b29b35bf 100644 --- a/docs/inference-providers/tasks/text-to-image.md +++ b/docs/inference-providers/tasks/text-to-image.md @@ -16,11 +16,9 @@ For more details, check out: Generate an image based on a given text prompt. - +> [!TIP] +> For more details about the `text-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-image)! You will find examples and related materials. -For more details about the `text-to-image` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-image)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/text-to-video.md b/docs/inference-providers/tasks/text-to-video.md index 9d8625958e..3a9d7fde6f 100644 --- a/docs/inference-providers/tasks/text-to-video.md +++ b/docs/inference-providers/tasks/text-to-video.md @@ -16,11 +16,9 @@ For more details, check out: Generate an video based on a given text prompt. - +> [!TIP] +> For more details about the `text-to-video` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-video)! You will find examples and related materials. -For more details about the `text-to-video` task, check out its [dedicated page](https://huggingface.co/tasks/text-to-video)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/token-classification.md b/docs/inference-providers/tasks/token-classification.md index 3636602eb9..10070760c4 100644 --- a/docs/inference-providers/tasks/token-classification.md +++ b/docs/inference-providers/tasks/token-classification.md @@ -16,11 +16,9 @@ For more details, check out: Token classification is a task in which a label is assigned to some tokens in a text. Some popular token classification subtasks are Named Entity Recognition (NER) and Part-of-Speech (PoS) tagging. - +> [!TIP] +> For more details about the `token-classification` task, check out its [dedicated page](https://huggingface.co/tasks/token-classification)! You will find examples and related materials. -For more details about the `token-classification` task, check out its [dedicated page](https://huggingface.co/tasks/token-classification)! You will find examples and related materials. - - ### Recommended models diff --git a/docs/inference-providers/tasks/translation.md b/docs/inference-providers/tasks/translation.md index 6366a4807d..4464262506 100644 --- a/docs/inference-providers/tasks/translation.md +++ b/docs/inference-providers/tasks/translation.md @@ -16,14 +16,13 @@ For more details, check out: Translation is the task of converting text from one language to another. - +> [!TIP] +> For more details about the `translation` task, check out its [dedicated page](https://huggingface.co/tasks/translation)! You will find examples and related materials. -For more details about the `translation` task, check out its [dedicated page](https://huggingface.co/tasks/translation)! You will find examples and related materials. - - ### Recommended models +- [google-t5/t5-base](https://huggingface.co/google-t5/t5-base): A general-purpose Transformer that can be used to translate from English to German, French, or Romanian. Explore all available models and find the one that suits you best [here](https://huggingface.co/models?inference=warm&pipeline_tag=translation&sort=trending). diff --git a/docs/inference-providers/tasks/zero-shot-classification.md b/docs/inference-providers/tasks/zero-shot-classification.md index 6a9f4768a6..80874e7ed3 100644 --- a/docs/inference-providers/tasks/zero-shot-classification.md +++ b/docs/inference-providers/tasks/zero-shot-classification.md @@ -16,11 +16,9 @@ For more details, check out: Zero-shot text classification is super useful to try out classification with zero code, you simply pass a sentence/paragraph and the possible labels for that sentence, and you get a result. The model has not been necessarily trained on the labels you provide, but it can still predict the correct label. - +> [!TIP] +> For more details about the `zero-shot-classification` task, check out its [dedicated page](https://huggingface.co/tasks/zero-shot-classification)! You will find examples and related materials. -For more details about the `zero-shot-classification` task, check out its [dedicated page](https://huggingface.co/tasks/zero-shot-classification)! You will find examples and related materials. - - ### Recommended models diff --git a/scripts/inference-providers/scripts/generate.ts b/scripts/inference-providers/scripts/generate.ts index 6a04012b02..46963ec5d8 100644 --- a/scripts/inference-providers/scripts/generate.ts +++ b/scripts/inference-providers/scripts/generate.ts @@ -384,11 +384,9 @@ function processPayloadSchema(schema: any): JsonObject[] { //// Inline templates //// ////////////////////////// -const TIP_LINK_TO_TASK_PAGE_TEMPLATE = Handlebars.compile(` - -For more details about the \`{{task}}\` task, check out its [dedicated page](https://huggingface.co/tasks/{{task}})! You will find examples and related materials. - -`); +const TIP_LINK_TO_TASK_PAGE_TEMPLATE = Handlebars.compile(`> [!TIP] +> For more details about the \`{{task}}\` task, check out its [dedicated page](https://huggingface.co/tasks/{{task}})! You will find examples and related materials. +`); const TIP_LIST_MODELS_LINK_TEMPLATE = Handlebars.compile( `Explore all available models and find the one that suits you best [here](https://huggingface.co/models?inference=warm&pipeline_tag={{task}}&sort=trending).`, diff --git a/scripts/inference-providers/templates/providers/cerebras.handlebars b/scripts/inference-providers/templates/providers/cerebras.handlebars index 76b3569e41..5847d36191 100644 --- a/scripts/inference-providers/templates/providers/cerebras.handlebars +++ b/scripts/inference-providers/templates/providers/cerebras.handlebars @@ -1,10 +1,7 @@ # Cerebras - - -All supported Cerebras models can be found [here](https://huggingface.co/models?inference_provider=cerebras&sort=trending) - - +> [!TIP] +> All supported Cerebras models can be found [here](https://huggingface.co/models?inference_provider=cerebras&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/cohere.handlebars b/scripts/inference-providers/templates/providers/cohere.handlebars index 5d22bc2c15..2a64093100 100644 --- a/scripts/inference-providers/templates/providers/cohere.handlebars +++ b/scripts/inference-providers/templates/providers/cohere.handlebars @@ -1,10 +1,7 @@ # Cohere - - -All supported Cohere models can be found [here](https://huggingface.co/models?inference_provider=cohere&sort=trending) - - +> [!TIP] +> All supported Cohere models can be found [here](https://huggingface.co/models?inference_provider=cohere&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/fal-ai.handlebars b/scripts/inference-providers/templates/providers/fal-ai.handlebars index 9a4ac0fc07..32a17a3d93 100644 --- a/scripts/inference-providers/templates/providers/fal-ai.handlebars +++ b/scripts/inference-providers/templates/providers/fal-ai.handlebars @@ -1,10 +1,7 @@ # Fal - - -All supported Fal models can be found [here](https://huggingface.co/models?inference_provider=fal-ai&sort=trending) - - +> [!TIP] +> All supported Fal models can be found [here](https://huggingface.co/models?inference_provider=fal-ai&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/featherless-ai.handlebars b/scripts/inference-providers/templates/providers/featherless-ai.handlebars index 48d4086aaa..24ea2b0565 100644 --- a/scripts/inference-providers/templates/providers/featherless-ai.handlebars +++ b/scripts/inference-providers/templates/providers/featherless-ai.handlebars @@ -1,10 +1,7 @@ # Featherless AI - - -All supported Featherless AI models can be found [here](https://huggingface.co/models?inference_provider=featherless-ai&sort=trending) - - +> [!TIP] +> All supported Featherless AI models can be found [here](https://huggingface.co/models?inference_provider=featherless-ai&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/fireworks-ai.handlebars b/scripts/inference-providers/templates/providers/fireworks-ai.handlebars index ad343dbec6..8b59ca57fd 100644 --- a/scripts/inference-providers/templates/providers/fireworks-ai.handlebars +++ b/scripts/inference-providers/templates/providers/fireworks-ai.handlebars @@ -1,10 +1,7 @@ # Fireworks AI - - -All supported Fireworks AI models can be found [here](https://huggingface.co/models?inference_provider=fireworks-ai&sort=trending) - - +> [!TIP] +> All supported Fireworks AI models can be found [here](https://huggingface.co/models?inference_provider=fireworks-ai&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/groq.handlebars b/scripts/inference-providers/templates/providers/groq.handlebars index d2494feb3f..58d355895f 100644 --- a/scripts/inference-providers/templates/providers/groq.handlebars +++ b/scripts/inference-providers/templates/providers/groq.handlebars @@ -1,10 +1,7 @@ # Groq - - -All supported Groq models can be found [here](https://huggingface.co/models?inference_provider=groq&sort=trending) - - +> [!TIP] +> All supported Groq models can be found [here](https://huggingface.co/models?inference_provider=groq&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/hf-inference.handlebars b/scripts/inference-providers/templates/providers/hf-inference.handlebars index cd63f1423d..604d080e13 100644 --- a/scripts/inference-providers/templates/providers/hf-inference.handlebars +++ b/scripts/inference-providers/templates/providers/hf-inference.handlebars @@ -1,10 +1,7 @@ # HF Inference - - -All supported HF Inference models can be found [here](https://huggingface.co/models?inference_provider=hf-inference&sort=trending) - - +> [!TIP] +> All supported HF Inference models can be found [here](https://huggingface.co/models?inference_provider=hf-inference&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/hyperbolic.handlebars b/scripts/inference-providers/templates/providers/hyperbolic.handlebars index 42a522d57d..65dbc68c0a 100644 --- a/scripts/inference-providers/templates/providers/hyperbolic.handlebars +++ b/scripts/inference-providers/templates/providers/hyperbolic.handlebars @@ -1,10 +1,7 @@ # Hyperbolic: The On-Demand AI Cloud - - -All supported Hyperbolic models can be found [here](https://huggingface.co/models?inference_provider=hyperbolic&sort=trending) - - +> [!TIP] +> All supported Hyperbolic models can be found [here](https://huggingface.co/models?inference_provider=hyperbolic&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/nebius.handlebars b/scripts/inference-providers/templates/providers/nebius.handlebars index 18bd63e921..8efd3524d9 100644 --- a/scripts/inference-providers/templates/providers/nebius.handlebars +++ b/scripts/inference-providers/templates/providers/nebius.handlebars @@ -1,10 +1,7 @@ # Nebius - - -All supported Nebius models can be found [here](https://huggingface.co/models?inference_provider=nebius&sort=trending) - - +> [!TIP] +> All supported Nebius models can be found [here](https://huggingface.co/models?inference_provider=nebius&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/novita.handlebars b/scripts/inference-providers/templates/providers/novita.handlebars index 12458b0481..118b44a98a 100644 --- a/scripts/inference-providers/templates/providers/novita.handlebars +++ b/scripts/inference-providers/templates/providers/novita.handlebars @@ -1,10 +1,7 @@ # Novita - - -All supported Novita models can be found [here](https://huggingface.co/models?inference_provider=novita&sort=trending) - - +> [!TIP] +> All supported Novita models can be found [here](https://huggingface.co/models?inference_provider=novita&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/nscale.handlebars b/scripts/inference-providers/templates/providers/nscale.handlebars index 3276f68098..751a184a40 100644 --- a/scripts/inference-providers/templates/providers/nscale.handlebars +++ b/scripts/inference-providers/templates/providers/nscale.handlebars @@ -1,10 +1,7 @@ # Nscale - - -All supported Nscale models can be found [here](https://huggingface.co/models?inference_provider=nscale&sort=trending) - - +> [!TIP] +> All supported Nscale models can be found [here](https://huggingface.co/models?inference_provider=nscale&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/publicai.handlebars b/scripts/inference-providers/templates/providers/publicai.handlebars index be4a687fff..b97dbbb4ee 100644 --- a/scripts/inference-providers/templates/providers/publicai.handlebars +++ b/scripts/inference-providers/templates/providers/publicai.handlebars @@ -1,10 +1,7 @@ # PublicAI - - -All models supported by the PublicAI Inference Utility can be found [here](https://huggingface.co/models?inference_provider=publicai&sort=trending) - - +> [!TIP] +> All models supported by the PublicAI Inference Utility can be found [here](https://huggingface.co/models?inference_provider=publicai&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/replicate.handlebars b/scripts/inference-providers/templates/providers/replicate.handlebars index 8e9ce34217..073a2b7239 100644 --- a/scripts/inference-providers/templates/providers/replicate.handlebars +++ b/scripts/inference-providers/templates/providers/replicate.handlebars @@ -1,10 +1,7 @@ # Replicate - - -All supported Replicate models can be found [here](https://huggingface.co/models?inference_provider=replicate&sort=trending) - - +> [!TIP] +> All supported Replicate models can be found [here](https://huggingface.co/models?inference_provider=replicate&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/sambanova.handlebars b/scripts/inference-providers/templates/providers/sambanova.handlebars index b9578352f8..733e557319 100644 --- a/scripts/inference-providers/templates/providers/sambanova.handlebars +++ b/scripts/inference-providers/templates/providers/sambanova.handlebars @@ -1,10 +1,7 @@ # SambaNova - - -All supported SambaNova models can be found [here](https://huggingface.co/models?inference_provider=sambanova&sort=trending) - - +> [!TIP] +> All supported SambaNova models can be found [here](https://huggingface.co/models?inference_provider=sambanova&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/scaleway.handlebars b/scripts/inference-providers/templates/providers/scaleway.handlebars index ca6198f5a2..fd2630e01d 100644 --- a/scripts/inference-providers/templates/providers/scaleway.handlebars +++ b/scripts/inference-providers/templates/providers/scaleway.handlebars @@ -1,10 +1,7 @@ # Scaleway - - -All supported Scaleway models can be found [here](https://huggingface.co/models?inference_provider=scaleway&sort=trending) - - +> [!TIP] +> All supported Scaleway models can be found [here](https://huggingface.co/models?inference_provider=scaleway&sort=trending) {{{logoSection}}} diff --git a/scripts/inference-providers/templates/providers/together.handlebars b/scripts/inference-providers/templates/providers/together.handlebars index add7b1fffa..61de6ec211 100644 --- a/scripts/inference-providers/templates/providers/together.handlebars +++ b/scripts/inference-providers/templates/providers/together.handlebars @@ -1,10 +1,7 @@ # Together - - -All supported Together models can be found [here](https://huggingface.co/models?inference_provider=together&sort=trending) - - +> [!TIP] +> All supported Together models can be found [here](https://huggingface.co/models?inference_provider=together&sort=trending) {{{logoSection}}} From 8de5913796d3d8e19d9c07d0886a04906c30c6f5 Mon Sep 17 00:00:00 2001 From: HuggingFaceInfra <148469759+HuggingFaceInfra@users.noreply.github.com> Date: Fri, 26 Sep 2025 09:17:26 +0200 Subject: [PATCH 03/19] Update Inference Providers documentation (automated) (#1957) Co-authored-by: Wauplin <11801849+Wauplin@users.noreply.github.com> --- docs/inference-providers/providers/fal-ai.md | 2 +- docs/inference-providers/providers/replicate.md | 2 +- docs/inference-providers/tasks/image-to-image.md | 2 +- docs/inference-providers/tasks/summarization.md | 1 + docs/inference-providers/tasks/text-to-image.md | 2 +- 5 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/inference-providers/providers/fal-ai.md b/docs/inference-providers/providers/fal-ai.md index ab9a3b8057..07c03757ac 100644 --- a/docs/inference-providers/providers/fal-ai.md +++ b/docs/inference-providers/providers/fal-ai.md @@ -67,7 +67,7 @@ Find out more about Text To Image [here](../tasks/text_to_image). diff --git a/docs/inference-providers/providers/replicate.md b/docs/inference-providers/providers/replicate.md index de9832ff4d..c3af798347 100644 --- a/docs/inference-providers/providers/replicate.md +++ b/docs/inference-providers/providers/replicate.md @@ -57,7 +57,7 @@ Find out more about Image To Image [here](../tasks/image_to_image). diff --git a/docs/inference-providers/tasks/image-to-image.md b/docs/inference-providers/tasks/image-to-image.md index a78612bbca..ff27e4875c 100644 --- a/docs/inference-providers/tasks/image-to-image.md +++ b/docs/inference-providers/tasks/image-to-image.md @@ -37,7 +37,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/summarization.md b/docs/inference-providers/tasks/summarization.md index b1e43a71e3..d3219f6239 100644 --- a/docs/inference-providers/tasks/summarization.md +++ b/docs/inference-providers/tasks/summarization.md @@ -23,6 +23,7 @@ Summarization is the task of producing a shorter version of a document while pre ### Recommended models - [facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn): A strong summarization model trained on English news articles. Excels at generating factual summaries. +- [Falconsai/medical_summarization](https://huggingface.co/Falconsai/medical_summarization): A summarization model trained on medical articles. Explore all available models and find the one that suits you best [here](https://huggingface.co/models?inference=warm&pipeline_tag=summarization&sort=trending). diff --git a/docs/inference-providers/tasks/text-to-image.md b/docs/inference-providers/tasks/text-to-image.md index 94b29b35bf..24c53bb34c 100644 --- a/docs/inference-providers/tasks/text-to-image.md +++ b/docs/inference-providers/tasks/text-to-image.md @@ -34,7 +34,7 @@ Explore all available models and find the one that suits you best [here](https:/ From 89a8eb600654cb0630f0e68cf6d1cf06919e00ea Mon Sep 17 00:00:00 2001 From: "Eliott C." Date: Fri, 26 Sep 2025 18:31:49 +0200 Subject: [PATCH 04/19] Revise rate limits for organization paid plans (#1958) Updated rate limits for Team, Enterprise, and Enterprise Plus organizations. --- docs/hub/rate-limits.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/hub/rate-limits.md b/docs/hub/rate-limits.md index b19c59fac2..f3fa24ab85 100644 --- a/docs/hub/rate-limits.md +++ b/docs/hub/rate-limits.md @@ -64,9 +64,9 @@ Here are the current rate limits (in September '25) based on your plan: | Anonymous user (per IP address) | 500 \* | 3,000 \* | 100 \* | | Free user | 1,000 \* | 5,000 \* | 200 \* | | PRO user | 2,500 | 12,000 | 400 | -| Team organization | 3,000 | 15,000 | 400 | -| Enterprise organization | 6,000 | 30,000 | 600 | -| Enterprise Plus organization | 10,000 | 50,000 | 1,000 | +| Team organization | 3,000 | 20,000 | 400 | +| Enterprise organization | 6,000 | 50,000 | 600 | +| Enterprise Plus organization | 10,000 | 100,000 | 1,000 | | Enterprise Plus organization
When Organization IP Ranges are defined | 100,000 | 500,000 | 10,000 | | Academia Hub organization | 2,500 | 12,000 | 400 | From 7bf1ee1946477137e75f4b3b620002b0a986306d Mon Sep 17 00:00:00 2001 From: Imad Saddik <79410781+ImadSaddik@users.noreply.github.com> Date: Sat, 27 Sep 2025 19:19:59 +0100 Subject: [PATCH 05/19] Fixed a few typos in gguf.md (#1961) --- docs/hub/gguf.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hub/gguf.md b/docs/hub/gguf.md index 37cf7987b7..564d14c074 100644 --- a/docs/hub/gguf.md +++ b/docs/hub/gguf.md @@ -82,8 +82,8 @@ Find more information [here](https://github.com/huggingface/huggingface.js/tree/ | Q4_0 | [GH](https://github.com/huggingface/huggingface.js/pull/615#discussion_r1557654249) | 4-bit round-to-nearest quantization (`q`). Each block has 32 weights. Weight formula: `w = q * block_scale`. Legacy quantization method (not used widely as of today). | | Q4_1 | [GH](https://github.com/huggingface/huggingface.js/pull/615#discussion_r1557682290) | 4-bit round-to-nearest quantization (`q`). Each block has 32 weights. Weight formula: `w = q * block_scale + block_minimum`. Legacy quantization method (not used widely as of today). | | Q4_K | [GH](https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305) | 4-bit quantization (`q`). Super-blocks with 8 blocks, each block has 32 weights. Weight formula: `w = q * block_scale(6-bit) + block_min(6-bit)`, resulting in 4.5 bits-per-weight. | -| Q3_K | [GH](https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305) | 3-bit quantization (`q`). Super-blocks with 16 blocks, each block has 16 weights. Weight formula: `w = q * block_scale(6-bit)`, resulting. 3.4375 bits-per-weight. | -| Q2_K | [GH](https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305) | 2-bit quantization (`q`). Super-blocks with 16 blocks, each block has 16 weight. Weight formula: `w = q * block_scale(4-bit) + block_min(4-bit)`, resulting in 2.625 bits-per-weight. | +| Q3_K | [GH](https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305) | 3-bit quantization (`q`). Super-blocks with 16 blocks, each block has 16 weights. Weight formula: `w = q * block_scale(6-bit)`, resulting in 3.4375 bits-per-weight. | +| Q2_K | [GH](https://github.com/ggerganov/llama.cpp/pull/1684#issue-1739619305) | 2-bit quantization (`q`). Super-blocks with 16 blocks, each block has 16 weights. Weight formula: `w = q * block_scale(4-bit) + block_min(4-bit)`, resulting in 2.625 bits-per-weight. | | IQ4_NL | [GH](https://github.com/ggerganov/llama.cpp/pull/5590) | 4-bit quantization (`q`). Super-blocks with 256 weights. Weight `w` is obtained using `super_block_scale` & `importance matrix`. | | IQ4_XS | [HF](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/README.md?code=true#L59-L70) | 4-bit quantization (`q`). Super-blocks with 256 weights. Weight `w` is obtained using `super_block_scale` & `importance matrix`, resulting in 4.25 bits-per-weight. | | IQ3_S | [HF](https://huggingface.co/CISCai/OpenCodeInterpreter-DS-6.7B-SOTA-GGUF/blob/main/README.md?code=true#L59-L70) | 3-bit quantization (`q`). Super-blocks with 256 weights. Weight `w` is obtained using `super_block_scale` & `importance matrix`, resulting in 3.44 bits-per-weight. | From 332db841f6dcc6301eb5aa5d089ad04c20ec4d50 Mon Sep 17 00:00:00 2001 From: HuggingFaceInfra <148469759+HuggingFaceInfra@users.noreply.github.com> Date: Mon, 29 Sep 2025 09:08:00 +0200 Subject: [PATCH 06/19] Update Inference Providers documentation (automated) (#1959) Co-authored-by: Wauplin <11801849+Wauplin@users.noreply.github.com> --- docs/inference-providers/providers/cohere.md | 2 +- docs/inference-providers/providers/fal-ai.md | 2 +- docs/inference-providers/providers/hf-inference.md | 2 +- docs/inference-providers/providers/replicate.md | 2 +- docs/inference-providers/tasks/chat-completion.md | 2 +- docs/inference-providers/tasks/image-segmentation.md | 2 +- docs/inference-providers/tasks/image-text-to-text.md | 2 +- docs/inference-providers/tasks/image-to-image.md | 2 +- docs/inference-providers/tasks/text-to-image.md | 2 +- scripts/inference-providers/package.json | 2 +- scripts/inference-providers/pnpm-lock.yaml | 10 +++++----- 11 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/inference-providers/providers/cohere.md b/docs/inference-providers/providers/cohere.md index ed9f1df06f..affbfcf1ea 100644 --- a/docs/inference-providers/providers/cohere.md +++ b/docs/inference-providers/providers/cohere.md @@ -59,6 +59,6 @@ Find out more about Chat Completion (VLM) [here](../tasks/chat-completion). diff --git a/docs/inference-providers/providers/fal-ai.md b/docs/inference-providers/providers/fal-ai.md index 07c03757ac..ab9a3b8057 100644 --- a/docs/inference-providers/providers/fal-ai.md +++ b/docs/inference-providers/providers/fal-ai.md @@ -67,7 +67,7 @@ Find out more about Text To Image [here](../tasks/text_to_image). diff --git a/docs/inference-providers/providers/hf-inference.md b/docs/inference-providers/providers/hf-inference.md index 29ac5d82e5..ca8163fd80 100644 --- a/docs/inference-providers/providers/hf-inference.md +++ b/docs/inference-providers/providers/hf-inference.md @@ -100,7 +100,7 @@ Find out more about Image Segmentation [here](../tasks/image_segmentation). diff --git a/docs/inference-providers/providers/replicate.md b/docs/inference-providers/providers/replicate.md index c3af798347..de9832ff4d 100644 --- a/docs/inference-providers/providers/replicate.md +++ b/docs/inference-providers/providers/replicate.md @@ -57,7 +57,7 @@ Find out more about Image To Image [here](../tasks/image_to_image). diff --git a/docs/inference-providers/tasks/chat-completion.md b/docs/inference-providers/tasks/chat-completion.md index 8da832fb2f..1cef1d601c 100644 --- a/docs/inference-providers/tasks/chat-completion.md +++ b/docs/inference-providers/tasks/chat-completion.md @@ -74,7 +74,7 @@ conversational /> diff --git a/docs/inference-providers/tasks/image-segmentation.md b/docs/inference-providers/tasks/image-segmentation.md index eeeab94cff..b06785d6de 100644 --- a/docs/inference-providers/tasks/image-segmentation.md +++ b/docs/inference-providers/tasks/image-segmentation.md @@ -31,7 +31,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/image-text-to-text.md b/docs/inference-providers/tasks/image-text-to-text.md index d31105008f..cd1f42f784 100644 --- a/docs/inference-providers/tasks/image-text-to-text.md +++ b/docs/inference-providers/tasks/image-text-to-text.md @@ -31,7 +31,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/image-to-image.md b/docs/inference-providers/tasks/image-to-image.md index ff27e4875c..a78612bbca 100644 --- a/docs/inference-providers/tasks/image-to-image.md +++ b/docs/inference-providers/tasks/image-to-image.md @@ -37,7 +37,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/text-to-image.md b/docs/inference-providers/tasks/text-to-image.md index 24c53bb34c..94b29b35bf 100644 --- a/docs/inference-providers/tasks/text-to-image.md +++ b/docs/inference-providers/tasks/text-to-image.md @@ -34,7 +34,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/scripts/inference-providers/package.json b/scripts/inference-providers/package.json index a11646d674..21212f7b7e 100644 --- a/scripts/inference-providers/package.json +++ b/scripts/inference-providers/package.json @@ -14,7 +14,7 @@ "author": "", "license": "ISC", "dependencies": { - "@huggingface/inference": "^4.8.0", + "@huggingface/inference": "^4.9.0", "@huggingface/tasks": "^0.19.48", "@types/node": "^22.5.0", "handlebars": "^4.7.8", diff --git a/scripts/inference-providers/pnpm-lock.yaml b/scripts/inference-providers/pnpm-lock.yaml index 3a101347b7..7a14f5712c 100644 --- a/scripts/inference-providers/pnpm-lock.yaml +++ b/scripts/inference-providers/pnpm-lock.yaml @@ -9,8 +9,8 @@ importers: .: dependencies: '@huggingface/inference': - specifier: ^4.8.0 - version: 4.8.0 + specifier: ^4.9.0 + version: 4.9.0 '@huggingface/tasks': specifier: ^0.19.48 version: 0.19.48 @@ -189,8 +189,8 @@ packages: cpu: [x64] os: [win32] - '@huggingface/inference@4.8.0': - resolution: {integrity: sha512-Eq98EAXqYn4rKMfrbEXuhc3IjKfaeIO6eXNOZk9xk6v5akrIWRtd6d1h0fjAWyX4zRbdUpXRh6MvsqXnzGvXCA==} + '@huggingface/inference@4.9.0': + resolution: {integrity: sha512-ajT+7fs2648pfCEeJOxBRFnXuAAQO8Tw3ArrQbCija7rUjOotK2wxu4tK2VzIsx3UwsxgEFSpnpYi+Cyu19IUg==} engines: {node: '>=18'} '@huggingface/jinja@0.5.1': @@ -415,7 +415,7 @@ snapshots: '@esbuild/win32-x64@0.23.1': optional: true - '@huggingface/inference@4.8.0': + '@huggingface/inference@4.9.0': dependencies: '@huggingface/jinja': 0.5.1 '@huggingface/tasks': 0.19.48 From aedd980c1f28a2854a62eb22b81b0be55cd6ef98 Mon Sep 17 00:00:00 2001 From: Tomsun28 Date: Mon, 29 Sep 2025 17:19:50 +0800 Subject: [PATCH 07/19] doc: add zai as inference provider doc (#1960) * doc: add zai as inference provider doc * update * generate z.ai * rerun zai docs * tip formatting --------- Co-authored-by: SBrandeis Co-authored-by: Wauplin --- docs/inference-providers/_toctree.yml | 2 + docs/inference-providers/index.md | 1 + docs/inference-providers/providers/zai-org.md | 62 +++++++++++++++++++ .../inference-providers/scripts/generate.ts | 2 + .../templates/providers/zai-org.handlebars | 21 +++++++ 5 files changed, 88 insertions(+) create mode 100644 docs/inference-providers/providers/zai-org.md create mode 100644 scripts/inference-providers/templates/providers/zai-org.handlebars diff --git a/docs/inference-providers/_toctree.yml b/docs/inference-providers/_toctree.yml index 27fe0a5286..d39278584b 100644 --- a/docs/inference-providers/_toctree.yml +++ b/docs/inference-providers/_toctree.yml @@ -107,6 +107,8 @@ title: Scaleway - local: providers/together title: Together + - local: providers/zai-org + title: Z.ai - local: hub-api title: Hub API diff --git a/docs/inference-providers/index.md b/docs/inference-providers/index.md index 6e4529de36..090d9ebad9 100644 --- a/docs/inference-providers/index.md +++ b/docs/inference-providers/index.md @@ -29,6 +29,7 @@ Our platform integrates with leading AI infrastructure providers, giving you acc | [SambaNova](./providers/sambanova) | ✅ | | ✅ | | | | | [Scaleway](./providers/scaleway) | ✅ | | ✅ | | | | | [Together](./providers/together) | ✅ | ✅ | | ✅ | | | +| [Z.ai](./providers/zai-org) | ✅ | ✅ | | | | | ## Why Choose Inference Providers? diff --git a/docs/inference-providers/providers/zai-org.md b/docs/inference-providers/providers/zai-org.md new file mode 100644 index 0000000000..3adfa30947 --- /dev/null +++ b/docs/inference-providers/providers/zai-org.md @@ -0,0 +1,62 @@ + + +# Z.ai + +> [!TIP] +> All supported Z.ai models can be found [here](https://huggingface.co/models?inference_provider=zai-org&sort=trending) + +
+ + + + +
+ +
+ + + + +
+ +Z.ai is an AI platform that provides cutting-edge large language models powered by GLM series. Their flagship models feature Mixture-of-Experts (MoE) architecture with advanced reasoning, coding, and agentic capabilities. + +For latest pricing, visit the [pricing page](https://docs.z.ai/guides/overview/pricing). + +## Resources + - **Website**: https://z.ai/ + - **Documentation**: https://docs.z.ai/ + - **API Documentation**: https://docs.z.ai/api-reference/introduction + - **GitHub**: https://github.com/zai-org + - **Hugging Face**: https://huggingface.co/zai-org + +## Supported tasks + + +### Chat Completion (LLM) + +Find out more about Chat Completion (LLM) [here](../tasks/chat-completion). + + + + diff --git a/scripts/inference-providers/scripts/generate.ts b/scripts/inference-providers/scripts/generate.ts index 46963ec5d8..c424200f7e 100644 --- a/scripts/inference-providers/scripts/generate.ts +++ b/scripts/inference-providers/scripts/generate.ts @@ -49,6 +49,7 @@ const PROVIDERS_HUB_ORGS: Record = { sambanova: "sambanovasystems", scaleway: "scaleway", together: "togethercomputer", + "zai-org": "zai-org" }; const PROVIDERS_URLS: Record = { @@ -68,6 +69,7 @@ const PROVIDERS_URLS: Record = { sambanova: "https://sambanova.ai/", scaleway: "https://www.scaleway.com", together: "https://together.xyz/", + "zai-org": "https://z.ai/" }; const INFERENCE_PROVIDERS = Object.keys(PROVIDERS_HUB_ORGS); diff --git a/scripts/inference-providers/templates/providers/zai-org.handlebars b/scripts/inference-providers/templates/providers/zai-org.handlebars new file mode 100644 index 0000000000..fa776ddc62 --- /dev/null +++ b/scripts/inference-providers/templates/providers/zai-org.handlebars @@ -0,0 +1,21 @@ +# Z.ai + +> [!TIP] +> All supported Z.ai models can be found [here](https://huggingface.co/models?inference_provider=zai-org&sort=trending) + +{{{logoSection}}} + +{{{followUsSection}}} + +Z.ai is an AI platform that provides cutting-edge large language models powered by GLM series. Their flagship models feature Mixture-of-Experts (MoE) architecture with advanced reasoning, coding, and agentic capabilities. + +For latest pricing, visit the [pricing page](https://docs.z.ai/guides/overview/pricing). + +## Resources + - **Website**: https://z.ai/ + - **Documentation**: https://docs.z.ai/ + - **API Documentation**: https://docs.z.ai/api-reference/introduction + - **GitHub**: https://github.com/zai-org + - **Hugging Face**: https://huggingface.co/zai-org + +{{{tasksSection}}} From 3301e278dc97d28c30d1e010cbde3102df4286fd Mon Sep 17 00:00:00 2001 From: HuggingFaceInfra <148469759+HuggingFaceInfra@users.noreply.github.com> Date: Tue, 30 Sep 2025 10:55:57 +0200 Subject: [PATCH 08/19] Update Inference Providers documentation (automated) (#1962) Co-authored-by: Wauplin <11801849+Wauplin@users.noreply.github.com> --- docs/inference-providers/providers/cohere.md | 2 +- docs/inference-providers/providers/fal-ai.md | 2 +- docs/inference-providers/providers/hf-inference.md | 2 +- docs/inference-providers/tasks/chat-completion.md | 4 ++-- docs/inference-providers/tasks/image-segmentation.md | 2 +- docs/inference-providers/tasks/image-text-to-text.md | 2 +- docs/inference-providers/tasks/text-to-image.md | 2 +- 7 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/inference-providers/providers/cohere.md b/docs/inference-providers/providers/cohere.md index affbfcf1ea..ed9f1df06f 100644 --- a/docs/inference-providers/providers/cohere.md +++ b/docs/inference-providers/providers/cohere.md @@ -59,6 +59,6 @@ Find out more about Chat Completion (VLM) [here](../tasks/chat-completion). diff --git a/docs/inference-providers/providers/fal-ai.md b/docs/inference-providers/providers/fal-ai.md index ab9a3b8057..b8ba7c51ed 100644 --- a/docs/inference-providers/providers/fal-ai.md +++ b/docs/inference-providers/providers/fal-ai.md @@ -67,7 +67,7 @@ Find out more about Text To Image [here](../tasks/text_to_image). diff --git a/docs/inference-providers/providers/hf-inference.md b/docs/inference-providers/providers/hf-inference.md index ca8163fd80..29ac5d82e5 100644 --- a/docs/inference-providers/providers/hf-inference.md +++ b/docs/inference-providers/providers/hf-inference.md @@ -100,7 +100,7 @@ Find out more about Image Segmentation [here](../tasks/image_segmentation). diff --git a/docs/inference-providers/tasks/chat-completion.md b/docs/inference-providers/tasks/chat-completion.md index 1cef1d601c..2e6b0584ed 100644 --- a/docs/inference-providers/tasks/chat-completion.md +++ b/docs/inference-providers/tasks/chat-completion.md @@ -64,7 +64,7 @@ The API supports: @@ -74,7 +74,7 @@ conversational /> diff --git a/docs/inference-providers/tasks/image-segmentation.md b/docs/inference-providers/tasks/image-segmentation.md index b06785d6de..eeeab94cff 100644 --- a/docs/inference-providers/tasks/image-segmentation.md +++ b/docs/inference-providers/tasks/image-segmentation.md @@ -31,7 +31,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/image-text-to-text.md b/docs/inference-providers/tasks/image-text-to-text.md index cd1f42f784..d31105008f 100644 --- a/docs/inference-providers/tasks/image-text-to-text.md +++ b/docs/inference-providers/tasks/image-text-to-text.md @@ -31,7 +31,7 @@ Explore all available models and find the one that suits you best [here](https:/ diff --git a/docs/inference-providers/tasks/text-to-image.md b/docs/inference-providers/tasks/text-to-image.md index 94b29b35bf..6fbb14bda7 100644 --- a/docs/inference-providers/tasks/text-to-image.md +++ b/docs/inference-providers/tasks/text-to-image.md @@ -34,7 +34,7 @@ Explore all available models and find the one that suits you best [here](https:/ From 9829252a840d53c257718701484b0c0f3fe41250 Mon Sep 17 00:00:00 2001 From: Assaf Vayner Date: Tue, 30 Sep 2025 12:01:30 -0700 Subject: [PATCH 09/19] xet protocol files (#1963) * xet protocol file and CI setup * xet-protocol -> xet --- .github/workflows/xet_build_documentation.yml | 20 + .../workflows/xet_build_pr_documentation.yml | 21 + .../workflows/xet_upload_pr_documentation.yml | 16 + docs/xet/_toctree.yml | 30 + docs/xet/api.md | 193 ++++++ docs/xet/auth.md | 151 +++++ docs/xet/chunking.md | 150 +++++ docs/xet/deduplication.md | 189 ++++++ docs/xet/download-protocol.md | 361 ++++++++++++ docs/xet/file-id.md | 33 ++ docs/xet/file-reconstruction.md | 133 +++++ docs/xet/hashing.md | 178 ++++++ docs/xet/index.md | 51 ++ docs/xet/shard.md | 549 ++++++++++++++++++ docs/xet/upload-protocol.md | 177 ++++++ docs/xet/xorb.md | 135 +++++ 16 files changed, 2387 insertions(+) create mode 100644 .github/workflows/xet_build_documentation.yml create mode 100644 .github/workflows/xet_build_pr_documentation.yml create mode 100644 .github/workflows/xet_upload_pr_documentation.yml create mode 100644 docs/xet/_toctree.yml create mode 100644 docs/xet/api.md create mode 100644 docs/xet/auth.md create mode 100644 docs/xet/chunking.md create mode 100644 docs/xet/deduplication.md create mode 100644 docs/xet/download-protocol.md create mode 100644 docs/xet/file-id.md create mode 100644 docs/xet/file-reconstruction.md create mode 100644 docs/xet/hashing.md create mode 100644 docs/xet/index.md create mode 100644 docs/xet/shard.md create mode 100644 docs/xet/upload-protocol.md create mode 100644 docs/xet/xorb.md diff --git a/.github/workflows/xet_build_documentation.yml b/.github/workflows/xet_build_documentation.yml new file mode 100644 index 0000000000..fc7fc51ad4 --- /dev/null +++ b/.github/workflows/xet_build_documentation.yml @@ -0,0 +1,20 @@ +name: Build Xet documentation + +on: + push: + paths: + - "docs/xet/**" + branches: + - main + +jobs: + build: + uses: huggingface/doc-builder/.github/workflows/build_main_documentation.yml@main + with: + commit_sha: ${{ github.sha }} + package: hub-docs + package_name: xet + path_to_docs: hub-docs/docs/xet/ + additional_args: --not_python_module + secrets: + hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }} diff --git a/.github/workflows/xet_build_pr_documentation.yml b/.github/workflows/xet_build_pr_documentation.yml new file mode 100644 index 0000000000..bc2d676ce5 --- /dev/null +++ b/.github/workflows/xet_build_pr_documentation.yml @@ -0,0 +1,21 @@ +name: Build Xet PR Documentation + +on: + pull_request: + paths: + - "docs/xet/**" + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + +jobs: + build: + uses: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml@main + with: + commit_sha: ${{ github.event.pull_request.head.sha }} + pr_number: ${{ github.event.number }} + package: hub-docs + package_name: xet + path_to_docs: hub-docs/docs/xet/ + additional_args: --not_python_module diff --git a/.github/workflows/xet_upload_pr_documentation.yml b/.github/workflows/xet_upload_pr_documentation.yml new file mode 100644 index 0000000000..0bdb1f02d3 --- /dev/null +++ b/.github/workflows/xet_upload_pr_documentation.yml @@ -0,0 +1,16 @@ +name: Upload Xet PR Documentation + +on: + workflow_run: + workflows: ["Build Xet PR Documentation"] + types: + - completed + +jobs: + build: + uses: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml@main + with: + package_name: xet + secrets: + hf_token: ${{ secrets.HF_DOC_BUILD_PUSH }} + comment_bot_token: ${{ secrets.COMMENT_BOT_TOKEN }} diff --git a/docs/xet/_toctree.yml b/docs/xet/_toctree.yml new file mode 100644 index 0000000000..2f723cf2c3 --- /dev/null +++ b/docs/xet/_toctree.yml @@ -0,0 +1,30 @@ +- local: index + title: Xet Protocol Specification + +- title: Building a client library for xet storage + sections: + - local: upload-protocol + title: Upload Protocol + - local: download-protocol + title: Download Protocol + - local: api + title: CAS API + - local: auth + title: Authentication and Authorization + - local: file-id + title: Hugging Face Hub Files Conversion to Xet File ID's + +- title: Overall Xet architecture + sections: + - local: chunking + title: Content-Defined Chunking + - local: hashing + title: Hashing Methods + - local: file-reconstruction + title: File Reconstruction + - local: xorb + title: Xorb Format + - local: shard + title: Shard Format + - local: deduplication + title: Deduplication diff --git a/docs/xet/api.md b/docs/xet/api.md new file mode 100644 index 0000000000..70c08a26b2 --- /dev/null +++ b/docs/xet/api.md @@ -0,0 +1,193 @@ +# CAS API Documentation + +This document describes the HTTP API endpoints used by the Content Addressable Storage (CAS) client to interact with the remote CAS server. + +## Authentication + +To authenticate, authorize, and obtain the API base URL, follow the instructions in [Authentication](./auth). + +## Converting Hashes to Strings + +Sometimes hashes are used in API paths as hexadecimal strings (reconstruction, xorb upload, global dedupe API). + +To convert a 32 hash to a 64 hexadecimal character string to be used as part of an API path there is a specific procedure, MUST NOT directly convert each byte. + +### Procedure + +For every 8 bytes in the hash (indices 0-7, 8-15, 16-23, 24-31) reverse the order of each byte in those regions then concatenate the regions back in order. + +Otherwise stated, consider each 8 byte part of a hash as a little endian 64 bit unsigned integer, then concatenate the hexadecimal representation of the 4 numbers in order (each padded with 0's to 16 characters). + +> [!NOTE] +> In all cases that a hash is represented as a string it is converted from a byte array to a string using this procedure. + +### Example + +Suppose a hash value is: +`[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]` + +Then before converting to a string it will first have its bytes reordered to: +`[7, 6, 5, 4, 3, 2, 1, 0, 15, 14, 13, 12, 11, 10, 9, 8, 23, 22, 21, 20, 19, 18, 17, 16, 31, 30, 29, 28, 27, 26, 25, 24]` + +So the string value of the the provided hash [0..32] is **NOT** `000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f`. +It is: `07060504030201000f0e0d0c0b0a0908171615141312111f1e1d1c1b1a1918`. + +## Endpoints + +### 1. Get File Reconstruction + +- **Description**: Retrieves reconstruction information for a specific file, includes byte range support when `Range` header is set. +- **Path**: `/v1/reconstructions/{file_id}` +- **Method**: `GET` +- **Parameters**: + - `file_id`: File hash in hex format (64 lowercase hexadecimal characters). +See [file hashes](./hashing#file-hashes) for computing the file hash and [converting hashes to strings](./api#converting-hashes-to-strings). +- **Headers**: + - `Range`: OPTIONAL. Format: `bytes={start}-{end}` (end is inclusive). +- **Minimum Token Scope**: `read` +- **Body**: None. +- **Response**: JSON (`QueryReconstructionResponse`) + + ```json + { + "offset_into_first_range": 0, + "terms": [...], + "fetch_info": {...} + } + ``` + +- **Error Responses**: See [Error Cases](./api#error-cases) + - `400 Bad Request`: Malformed `file_id` in the path. Fix the path before retrying. + - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header. + - `404 Not Found`: The file does not exist. Not retryable. + - `416 Range Not Satisfiable`: The requested byte range start exceeds the end of the file. Not retryable. + +```txt +GET /v1/reconstructions/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +-H "Authorization: Bearer " +OPTIONAL: -H Range: "bytes=0-100000" +``` + +### Example File Reconstruction Response Body + +See [QueryReconstructionResponse](./download-protocol#queryreconstructionresponse-structure) for more details in the download protocol specification. + +### 2. Query Chunk Deduplication (Global Deduplication) + +- **Description**: Checks if a chunk exists in the CAS for deduplication purposes. +- **Path**: `/v1/chunks/{prefix}/{hash}` +- **Method**: `GET` +- **Parameters**: + - `prefix`: The only acceptable prefix for the Global Deduplication API is `default-merkledb`. + - `hash`: Chunk hash in hex format (64 lowercase hexadecimal characters). +See [Chunk Hashes](./hashing#chunk-hashes) to compute the chunk hash and [converting hashes to strings](./api#converting-hashes-to-strings). +- **Minimum Token Scope**: `read` +- **Body**: None. +- **Response**: Shard format bytes (`application/octet-stream`), deserialize as a [shard](./shard#global-deduplication). +- **Error Responses**: See [Error Cases](./api#error-cases) + - `400 Bad Request`: Malformed hash in the path. Fix the path before retrying. + - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header. + - `404 Not Found`: Chunk not already tracked by global deduplication. Not retryable. + +```txt +GET /v1/chunks/default-merkledb/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +-H "Authorization: Bearer " +``` + +#### Example Shard Response Body + +An example shard response body can be found in [Xet reference files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.shard.dedupe). + +### 3. Upload Xorb + +- **Description**: Uploads a serialized Xorb to the server; uploading real data in serialized format. +- **Path**: `/v1/xorbs/{prefix}/{hash}` +- **Method**: `POST` +- **Parameters**: + - `prefix`: The only acceptable prefix for the Xorb upload API is `default`. + - `hash`: Xorb hash in hex format (64 lowercase hexadecimal characters). +See [Xorb Hashes](./hashing#xorb-hashes) to compute the hash, and [converting hashes to strings](./api#converting-hashes-to-strings). +- **Minimum Token Scope**: `write` +- **Body**: Serialized Xorb bytes (`application/octet-stream`). +See [xorb format serialization](./xorb). +- **Response**: JSON (`UploadXorbResponse`) + +```json +{ + "was_inserted": true +} +``` + +- Note: `was_inserted` is `false` if the Xorb already exists; this is not an error. + +- **Error Responses**: See [Error Cases](./api#error-cases) + - `400 Bad Request`: Malformed hash in the path, Xorb hash does not match the body, or body is incorrectly serialized. + - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header. + - `403 Forbidden`: Token provided but does not have a wide enough scope (for example, a `read` token was provided). Clients MUST retry with a `write` scope token. + +```txt +POST /v1/xorbs/default/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef +-H "Authorization: Bearer " +``` + +#### Example Xorb Request Body + +An example xorb request body can be found in [Xet reference files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb). + +### 4. Upload Shard + +- **Description**: Uploads a Shard to the CAS. +Uploads file reconstructions and new xorb listing, serialized into the shard format; marks the files as uploaded. +- **Path**: `/v1/shards` +- **Method**: `POST` +- **Minimum Token Scope**: `write` +- **Body**: Serialized Shard data as bytes (`application/octet-stream`). +See [Shard format guide](./shard#shard-upload). +- **Response**: JSON (`UploadShardResponse`) + +```json +{ + "result": 0 +} +``` + +- Where `result` is: + - `0`: The Shard already exists. + - `1`: `SyncPerformed` — the Shard was registered. + +The value of `result` does not carry any meaning, if the upload shard API returns a `200 OK` status code, the upload was successful and the files listed are considered uploaded. + +- **Error Responses**: See [Error Cases](./api#error-cases) + - `400 Bad Request`: Shard is incorrectly serialized or Shard contents failed verification. + - Can mean that a referenced Xorb doesn't exist or the shard is too large + - `401 Unauthorized`: Refresh the token to continue making requests, or provide a token in the `Authorization` header. + - `403 Forbidden`: Token provided but does not have a wide enough scope (for example, a `read` token was provided). + +```txt +POST /v1/shards +-H "Authorization: Bearer " +``` + +#### Example Shard Request Body + +An example shard request body can be found in [Xet reference files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.shard.verification-no-footer). + +## Error Cases + +### Non-Retryable Errors + +- **400 Bad Request**: Returned when the request parameters are invalid (for example, invalid Xorb/Shard on upload APIs). +- **401 Unauthorized**: Refresh the token to continue making requests, or provide a token in the `Authorization` header. +- **403 Forbidden**: Token provided but does not have a wide enough scope (for example, a `read` token was provided for an API requiring `write` scope). +- **404 Not Found**: Occurs on `GET` APIs where the resource (Xorb, file) does not exist. +- **416 Range Not Satisfiable**: Reconstruction API only; returned when byte range requests are invalid. Specifically, the requested start range is greater than or equal to the length of the file. + +### Retryable Errors + +- **Connection Errors**: Often caused by network issues. Retry if intermittent. +Clients SHOULD ensure no firewall blocks requests and SHOULD NOT use DNS overrides. +- **429 Rate Limiting**: Lower your request rate using a backoff strategy, then wait and retry. +Assume all APIs are rate limited. +- **500 Internal Server Error**: The server experienced an intermittent issue; clients SHOULD retry their requests. +- **503 Service Unavailable**: Service is temporarily unable to process requests; wait and retry. +- **504 Gateway Timeout**: Service took too long to respond; wait and retry. diff --git a/docs/xet/auth.md b/docs/xet/auth.md new file mode 100644 index 0000000000..735b4b9be4 --- /dev/null +++ b/docs/xet/auth.md @@ -0,0 +1,151 @@ +# Authentication and Authorization + +To invoke any API's mentioned in this specification a client MUST first acquire a token (and the URL) to authenticate against the server which serves these API's. + +The Xet protocol server uses bearer authentication via a token generated by the Hugging Face Hub (). + +The following section explains how to acquire such a token. + +## Token Request API Endpoints + +**URL Pattern:** + +```txt +https://huggingface.co/api/{repo_type}s/{repo_id}/xet-{token_type}-token/{revision} +``` + +**Parameters:** + +All parameters are required to form the URL. + +- `repo_type`: Type of repository - `model`, `dataset`, or `space` +- `repo_id`: Repository identifier in format `namespace/repo-name` +- `token_type`: Either `read` or `write`. +- `revision`: Git revision (branch, tag, or commit hash; default to using `main` if no specific ref is required) + +To understand the distinction for between `token_type` values read onwards in this document to [Token Scope](./auth#token-scope). + +**Example URLs:** + +```txt +https://huggingface.co/api/models/sentence-transformers/all-MiniLM-L6-v2/xet-read-token/main +https://huggingface.co/api/datasets/HuggingFaceM4/the_cauldron/xet-write-token/v1.1 +https://huggingface.co/api/spaces/jsulz/ready-xet-go/xet-read-token/main +``` + +**HTTP Method:** GET + +**Required Headers:** + +- `Authorization`: Bearer token for Hugging Face Hub authentication + +### Response Format + +A JSON encoded object with the following format: + +```typescript +{ + "accessToken": string, + "exp": number, + "casUrl": string, +} +``` + +- accessToken is the token to be used when invoking API's on the Xet CAS service (any Xet API denoted in this specification) +- exp is the unix timestamp of when this token expires +- casUrl is the API service endpoint URL + +Users MAY assume the "accessToken" and "casUrl" fields lengths have an upper limit of 64000 characters. + +#### Example Response Object + +```json +{ + "accessToken": "xet_xxxxxxxxxxx", + "exp": 1848535668, + "casUrl": "https://cas-server.xethub.hf.co" +} +``` + +### Error Handling + +#### HTTP Errors + +- **401 Unauthorized**: Invalid or missing Hub authentication token +- **403 Forbidden**: Insufficient permissions for the requested token type +- **404 Not Found**: Repository or revision does not exist + +### Implementation Example + +Here's a basic implementation flow: + +1. **Make the request:** + + ```http + GET /api/models/black-forest-labs/FLUX.1-dev/xet-read-token/main + Host: huggingface.co + Authorization: Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + ``` + +2. **Parse the response** + + ```python + endpoint = response_json["casUrl"] + access_token = response_json["accessToken"] + expiration = response_json["exp"] + ``` + +3. **Use the token with Xet service:** + Use Bearer authentication with the value for the `accessToken` key to authenticate with the Xet service at `endpoint` until `expiration` time. + +4. **Token refresh (when needed):** + Use the same API to generate a new token. + + > [!NOTE] + > In `xet-core` we SHOULD add 30 seconds of buffer time before the provided `expiration` time to refresh the token. + +## Token Scope + +Xet tokens can have either a `read` or a `write` scope. +`write` scope supersedes `read` scope and all `read` scope API's can be invoked when using a `write` scope token. +The type of token issued is determined on the `token_type` URI path component when requesting the token from the Hugging Face Hub (see above). + +Check API specification for what scope level is necessary to invoke each API (briefly, only `POST /shard` and `POST /xorb/*` API's require `write` scope). + +The scope of the Xet tokens is limited to the repository and ref for which they were issued. To upload or download from different repositories or refs (different branches) clients MUST be issued different tokens. + +## Token Scope Relative to Hugging Face Hub Authentication Token + +When requesting a Xet token from the Hugging Face Hub, you will only receive a Xet token matching the requested parameters if you actually have access to them, based on the access afforded to your Hub authentication token. + +If you require a `write` scope Xet token, then you MUST request it using a Hugging Face Hub token that has write access to the particular repository and ref that you want to access. + +If you request a `read` scope Xet token, then you MUST request it using a Hugging Face Hub token that has at least read access to the particular repository and ref you want to access. + +If you are using Fine-grained Hugging Face Hub Access Tokens, your tokens MUST have read or write access to the contents of repositories to be issues read or write Xet tokens respectively. + +## Security Considerations + +- Xet tokens are time-limited and SHOULD be refreshed/swapped before expiration +- Store tokens securely and SHOULD NOT log them (both Hub authentication tokens and Xet tokens) +- SHOULD use read tokens when possible; only request write tokens when necessary + +## Diagram + +```mermaid +sequenceDiagram + autonumber + actor C as Client + participant H as Hugging Face Hub (https://huggingface.co) + participant CAS as CAS API + + loop Repeat after token expiration + C->>H: https://huggingface.co/api/{repo_type}s/{repo_id}/xet-{token_type}-token/{revision} + H->>C: { casUrl, accessToken, exp } + + loop Client invoking CAS API's + C->>CAS: CAS API's using accessToken and casUrl
(Reconstruction, Xorb & Shard upload, Global Dedupe) + CAS->>C: API Responses, returns code 401 Unauthorized if token expired + end + end +``` diff --git a/docs/xet/chunking.md b/docs/xet/chunking.md new file mode 100644 index 0000000000..1d501e044f --- /dev/null +++ b/docs/xet/chunking.md @@ -0,0 +1,150 @@ +# Content-Defined Chunking Algorithm + +The goal in chunking is to convert file data into smaller variable length chunks, approximately 64 KiB in length. +Chunks boundaries MUST be computed in a deterministic way such that chunking the same data in 2 different places yields chunks that can be deduplicated. + +```txt + +---------+---------+---------+---------+---------+---------+---------+-------------- +File -> | chunk 0 | chunk 1 | chunk 2 | chunk 3 | chunk 4 | chunk 5 | chunk 6 | chunk 7 | ... + +---------+---------+---------+---------+---------+---------+---------+-------------- +``` + +## Step-by-step Algorithm (Gearhash-based CDC) + +### Constant Parameters + +- target_chunk_size: `64 KiB` +- MIN_CHUNK_SIZE: `8 KiB` (minimum chunk size) +- MAX_CHUNK_SIZE: `128 KiB` (maximum chunk size) +- MASK: `0xFFFF000000000000` (16 one-bits → boundary probability 1/2^16 per byte) +- TABLE[256]: table of 256 64-bit constants ([rust-gearhash-table]) + +### State + +- h: 64-bit hash, initialized to 0 +- start_offset: start offset of the current chunk, initialized to 0 + +### Per-byte Update Rule (Gearhash) + +For each input byte `b`, update the hash with 64-bit wrapping arithmetic: + +```text +h = (h << 1) + TABLE[b] +``` + +### Boundary Test and Size Constraints + +At each position after updating `h`, let `size = current_offset - start_offset + 1`. + +- If `size < MIN_CHUNK_SIZE`: skip testing `MASK`; continue +- Else if `size >= MAX_CHUNK_SIZE`: force a boundary +- Else if `(h & MASK) == 0`: boundary at this position + +When a boundary found or taken: + +- Emit the chunk `[start_offset, current_offset + 1)` +- Set `start_offset = current_offset + 1` +- Reset `h = 0` + +At end-of-file, if `start_offset < len(data)`, emit the final chunk `[start_offset, len(data))`. + +### Pseudocode + +```text +Inputs: (See above for constant parameters) + data: byte array + +State: + h = 0 + start_offset = 0 // start of the "current chunk" + +if len(data) < MIN_CHUNK_SIZE: + emit chunk [0, len(data)) + done + +for i in range(0, len(data)): + b = data[i] + h = (h << 1) + TABLE[b] // 64-bit wrapping + size = i + 1 - start_offset + + if size < MIN_CHUNK_SIZE: + continue + + if size >= MAX_CHUNK_SIZE or (h & MASK) == 0: + emit chunk [start_offset, i + 1) + start_offset = i + 1 + h = 0 + +if start_offset < len(data): + emit chunk [start_offset, len(data)) +``` + +### Boundary probability and mask selection + +Given that MASK has 16 one-bits, for a random 64-bit hash `h`, the chance that all those 16 bits are zero is 1 / 2^16. On average, that means you’ll see a match about once every 64 KiB. + +### Properties + +- Deterministic boundaries: same content → same chunks +- Locality: small edits only affect nearby boundaries +- Linear time and constant memory: single 64-bit state and counters + +### Intuition and Rationale + +- The table `TABLE[256]` injects pseudo-randomness per byte value so that the evolving hash `h` behaves like a random 64-bit value with respect to the mask test. This makes boundaries content-defined yet statistically evenly spaced. +- The left shift `(h << 1)` amplifies recent bytes, helping small changes affect nearby positions without globally shifting all boundaries. +- Resetting `h` to 0 at each boundary prevents long-range carryover and keeps boundary decisions for each chunk statistically independent. + +### Implementation Notes + +- Only reset `h` when you emit a boundary. This ensures chunking is stable even when streaming input in pieces. +- Apply the mask test only once `size >= MIN_CHUNK_SIZE`. This reduces the frequency of tiny chunks and stabilizes average chunk sizes. +- MUST force a boundary at `MAX_CHUNK_SIZE` even if `(h & MASK) != 0`. This guarantees bounded chunk sizes and prevents pathological long chunks when matches are rare. +- Use 64-bit wrapping arithmetic for `(h << 1) + TABLE[b]`. This is the behavior in the reference implementation [rust-gearhash]. + +### Edge Cases + +- Tiny files: if `len(data) < MIN_CHUNK_SIZE`, the entire `data` is emitted as a single chunk. +- Long runs without a match: if no position matches `(h & MASK) == 0` before `MAX_CHUNK_SIZE`, a boundary is forced at `MAX_CHUNK_SIZE` to cap chunk size. + +### Portability and Determinism + +- With a fixed `T[256]` table and mask, the algorithm is deterministic across platforms: same input → same chunk boundaries. +- Endianness does not affect behavior because updates are byte-wise and use scalar 64-bit operations. +- SIMD-accelerated implementations (when available) are optimizations only; they produce the same boundaries as the scalar path [rust-gearhash]. + +## Minimum-size Skip-ahead (Cut-point Skipping Optimization) + +Computing and testing the rolling hash at every byte is expensive for large data, and early tests inside the first few bytes of a chunk are disallowed by the `MIN_CHUNK_SIZE` constraint anyway. +We are able to intentionally skip testing some data with cut-point skipping to accelerate scanning without affecting correctness. + +The hash function by virtue of the use of 64 byte integer length and the bit shift (`(h << 1) + TABLE[b]`) causes the hash at any byte offset to only depend on the last 64 bytes. +With a Gear rolling hash window of 64 bytes, the first boundary test is deferred until at least `MIN_CHUNK_SIZE - 64 - 1` bytes into the chunk. +This ensures that, by the time the first boundary can be considered (at offset `MIN_CHUNK_SIZE`), at least one full hash window of bytes from the current chunk has influenced the hash state. + +- Effect: + - Distribution quality is preserved because the first admissible test uses a well-mixed hash (full window), avoiding bias from the earliest bytes. + - Performance improves by avoiding per-byte hashing/judgment in the prefix where boundaries cannot be taken. + - Correctness is preserved because boundaries MUST NOT be set before `MIN_CHUNK_SIZE` and the hash produced at a testable offset is the same as the hash computed had we not skipped any bytes. +- Notes: + - This is an optimization of the search procedure only; it does not change the boundary condition, mask, or emitted chunk set compared to a byte-by-byte implementation that simply refrains from taking boundaries before `MIN_CHUNK_SIZE`. + - In the reference code, this appears as advancing the scan pointer by up to `MIN_CHUNK_SIZE - 64 - 1` before invoking the mask test loop. + +## References + +- rust-gearhash: Fast, SIMD-accelerated GEAR hashing for CDC [rust-gearhash] +- FastCDC paper (background and design rationale of CDC) [fastcdc-paper] + +## Sample Reference + +The [xet-team/xet-spec-reference-files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files) repository contains the original file [Electric_Vehicle_Population_Data_20250917.csv](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv). + +In the same repository in file [Electric_Vehicle_Population_Data_20250917.csv.chunks](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.chunks) +the chunks produced out of [Electric_Vehicle_Population_Data_20250917.csv](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv) are listed. +Each line in the file is a 64 hexadecimal character string version of the hash of the chunk, followed by a space and then the number of bytes in that chunk. + +Implementors should use the chunk lengths to determine that they are producing the right chunk boundaries for this file with their chunking implementation. + +[rust-gearhash]: https://github.com/srijs/rust-gearhash +[rust-gearhash-table]: https://github.com/srijs/rust-gearhash/blob/adad44e7141cfd29d898cf6e0858f50b995db286/src/table.rs#L5 +[fastcdc-paper]: https://www.usenix.org/conference/atc16/technical-sessions/presentation/xia diff --git a/docs/xet/deduplication.md b/docs/xet/deduplication.md new file mode 100644 index 0000000000..1ff30209d2 --- /dev/null +++ b/docs/xet/deduplication.md @@ -0,0 +1,189 @@ +# Xet Chunk-Level Deduplication Specification + +## Overview + +Chunk-level deduplication is a fundamental optimization technique in the Xet system that eliminates redundant data by identifying and sharing identical content blocks (chunks) across files and repositories. +This specification details the procedures, algorithms, and security mechanisms that enable efficient storage and transfer while maintaining data integrity and access control. + +Deduplication in Xet operates at the chunk level rather than the file level, providing fine-grained deduplication capabilities that can identify shared content even when files differ significantly. +This approach is particularly effective for scenarios common in machine learning and data science workflows, such as: + +- Multiple versions of datasets with incremental changes +- Model checkpoints that share common layers or parameters +- Documentation and configuration files with similar content +- Large files where only portions have changed between versions + +## Core Concepts + +### Chunks + +A **chunk** is a variable-sized content block derived from files using Content-Defined Chunking (CDC) with a rolling hash function. Chunks are the fundamental unit of deduplication in Xet. + +- **Target size**: 64KB (configurable) +- **Size range**: 8KB to 128KB (minimum and maximum constraints) +- **Identification**: Each chunk is uniquely identified by its cryptographic hash (MerkleHash) + +[Detailed chunking description](./chunking) + +### Xorbs + +**Xorbs** are objects that aggregate multiple chunks for efficient storage and transfer: + +- **Maximum size**: 64MB +- **Maximum chunks**: 8,192 chunks per xorb +- **Purpose**: Batch multiple chunks together to reduce metadata and network overhead when uploading and downloading groups of chunks + +### Shards (Xorb Lists) + +**Shards** are objects that contain a list of xorbs that can be deduped against (for the context of deduplication, ignore the file info section of the shard format). + +- **Maximum size**: 64MB +- **Purpose**: Provide a format on a positive reply to a global deduplication request with information about xorbs that already exist in the CAS system. + +### CAS (Content Addressable Storage) + +The **CAS** system provides the underlying storage infrastructure: + +- **Content addressing**: All objects are stored and retrieved by their cryptographic hash +- **Immutability**: Once stored, file content cannot be modified +- **Deduplication**: Identical content is automatically deduplicated at the storage level + +## Deduplication Procedure + +### 1. File Processing and Chunking + +When a file is processed for upload, it undergoes the following steps: + +```mermaid +graph TD + A[File Input] --> B[Content-Defined Chunking] + B --> C[Hash Computation] + C --> D[Chunk Creation] + D --> E[Deduplication Query] +``` + +1. **Chunking**: Content-defined chunking using GearHash algorithm creates variable-sized chunks of file data +2. **Hash Computation**: Each chunk's content is hashed using a cryptographic hash function (Blake3-based MerkleHash) +3. **Chunk Object Creation**: Chunks are wrapped with metadata including hash, size, and data + +### 2. Multi-Level Deduplication Strategy + +Xet employs a three-tiered deduplication strategy to maximize efficiency while minimizing latency: + +#### Level 1: Local Session Deduplication + +**Scope**: Current upload session +**Mechanism**: In-memory hash lookup table +**Purpose**: Eliminate redundancy within the current file or session + +**Benefits**: + +- Fastest lookup (in-memory) +- Zero network overhead +- Immediate deduplication feedback + +#### Level 2: Cached Metadata Deduplication + +**Scope**: Previously uploaded files and sessions +**Mechanism**: Local shard file metadata cache +**Purpose**: Leverage deduplication against recently uploaded content + +**Benefits**: + +- Fast local disk access +- No network latency +- Persistent across sessions + +#### Level 3: Global Deduplication API + +**Scope**: Entire Xet system +**Mechanism**: Global deduplication service with HMAC protection +**Purpose**: Discover deduplication opportunities across all users and repositories + +### 3. Global Deduplication Process + +The global deduplication system provides deduplication capabilities across all data that is managed by the Xet system: + +#### Eligibility Criteria + +Not all chunks are eligible for global deduplication queries to manage system load: + +1. **First chunk**: The first chunk of every file is always eligible. +2. **Hash pattern matching**: Chunks are eligible if: the last 8 bytes of the hash interpreted as a little-endian 64 bit integer % 1024 == 0. + +**Recommendations:** +**Spacing constraints**: The global dedupe API is optimized to return information about nearby chunks when there is a match. Consider only issueing a request to an eligible chunk every ~4MB of data. + +#### Query Process + +1. **Background Query**: Global deduplication queries SHOULD run asynchronously to avoid blocking upload +2. **HMAC Protection**: Chunk hashes are protected using HMAC keys +3. **Shard Response**: When a match is found, the API returns a shard containing: + - **CAS Info Section**: Contains metadata about many xorbs that store chunks + - **HMAC Key**: Included in the shard metadata header used to encrypt chunk hashes +4. **Encrypted Chunk Matching**: All chunk hashes in the returned shard have been encrypted with the HMAC key +5. **Match Discovery Process**: To find matches, clients MUST: + - Encrypt their chunk hash using the provided HMAC key + - Search for the encrypted hash within the shard's chunk listings + - For subsequent chunks, repeat the encryption and search process + - Track the original (non-encrypted) chunk hash while noting which xorb contains that chunk +6. **Metadata Caching**: Client downloads and caches shard metadata for future deduplication + +#### HMAC Security Mechanism + +Global deduplication uses HMAC (Hash-based Message Authentication Code) to protect chunk hashes while enabling deduplication. + +**Security Properties**: + +Raw chunk hashes are never transmitted from servers to clients; a client has to encrypt their raw chunk hash and find a match to know a raw chunk hash exists in the system. +They MAY know this chunk hash because they own this data, the match has made them privy to know which xorb has this chunk hash and the position in the xorb, but has not revealed any other raw chunk hashes in that xorb or other xorbs. + +## Technical Implementation Details + +### Chunk Hash Computation + +Each chunk has its content hashed using a cryptographic hash function (Blake3-based MerkleHash) to create a unique identifier for content addressing. +[See section about hashing](./hashing#chunk-hashes). + +### Xorb Formation + +When new chunks need to be stored, they are aggregated into xorbs based on size and count limits. If adding a new chunk would exceed the maximum xorb size or chunk count, the current xorb is finalized and uploaded. [See section about xorb formation](./xorb) + +### File Reconstruction Information + +When chunks are deduplicated, the system creates file reconstruction information that includes: + +- Hash of the xorb containing the chunks +- Flags for the CAS block +- Total bytes in the segment +- Start and end indices within the xorb (start inclusive, end exclusive) + +This information allows the system to reconstruct files by: + +1. Identifying which xorbs contain the needed chunks +2. Extracting the specific chunk ranges from each xorb +3. Concatenating chunks in the correct order + +[See section about file reconstruction](./file-reconstruction). + +## Fragmentation Prevention + +While deduplication is valuable for saving space, doing it too aggressively can cause file fragmentation—meaning a file’s chunks end up scattered across many different xorbs. This can make reading files slower and less efficient. +To avoid this, in xet-core we aim (and encourage implementors) to keep long, continuous runs of chunks together in the same xorb whenever possible. Implementations SHOULD keep long, continuous runs together when feasible. +Instead of always deduplicating every possible chunk, the system sometimes chooses to reference a straight run of chunks in a single xorb, even if it means skipping deduplication for a few chunks. +This approach balances the benefits of deduplication with the need to keep files easy and fast to read. +Consider for example referencing a deduplicated chunks in a minimum run of chunks (e.g. at least 8 chunks) or targeting an average contiguous run of chunks totalling length >= 1MB. + +## Conclusion + +Xet's chunk-level deduplication system provides a comprehensive solution for efficient data storage and transfer in large-scale data workflows. +By combining local, cached, and global deduplication strategies with robust security mechanisms and fragmentation prevention, +the system achieves significant storage savings while maintaining performance and data integrity. + +The multi-tiered approach ensures that deduplication is both effective and efficient: + +- Local deduplication provides immediate benefits within sessions +- Cached deduplication leverages recent upload history +- Global deduplication enables cross-repository optimization while preserving security + +The system's design prioritizes both efficiency and safety, with comprehensive error handling, performance monitoring, and security measures that make it suitable for production use in data-intensive applications. diff --git a/docs/xet/download-protocol.md b/docs/xet/download-protocol.md new file mode 100644 index 0000000000..4b0f859033 --- /dev/null +++ b/docs/xet/download-protocol.md @@ -0,0 +1,361 @@ +# Download Protocol + +This document describes the complete process of downloading a single file from the Xet protocol using the Content Addressable Storage (CAS) reconstruction API. + +## Overview + +File download in the Xet protocol is a two-stage process: + +1. **Reconstruction Query**: Query the CAS API to get file reconstruction metadata +2. **Data Fetching**: Download and reassemble the file using the reconstruction metadata + +## Stage 1: Calling the Reconstruction API + +### Single File Reconstruction + +To download a file given a file hash, first call the reconstruction API to get the file reconstruction. Follow the steps in [api](./api#1-get-file-reconstruction). + +Note that you will need at least a `read` scope auth token, [auth reference](./auth). + +> [!TIP] +> For large files it is RECOMMENDED to request the reconstruction in batches i.e. the first 10GB, download all the data, then the next 10GB and so on. Clients can use the `Range` header to specify a range of file data. + +## Stage 2: Understanding the Reconstruction Response + +The reconstruction API returns a `QueryReconstructionResponse` object with three key components: + +### QueryReconstructionResponse Structure + +```json +{ + "offset_into_first_range": 0, + "terms": [ + { + "hash": "a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456", + "unpacked_length": 263873, + "range": { + "start": 0, + "end": 4 + } + }, + ... + ], + "fetch_info": { + "a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456": [ + { + "range": { + "start": 0, + "end": 4 + }, + "url": "https://transfer.xethub.hf.co/xorb/default/a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456", + "url_range": { + "start": 0, + "end": 131071 + } + }, + ... + ], + ... + } +} +``` + +### Fields + +#### offset_into_first_range + +- Type: `number` +- For a full file or when the specified range start is 0, then this is guaranteed to be `0` +- For range queries this is the byte offset into the first term (deserialized/chunks decompressed) to start to keep data from. + - since the requested range may start in the middle of a chunk, and data MUST be downloaded in full chunks (since they may need to be deserialized) then this offset tells a client how many bytes to skip in the first chunk (or possibly multiple chunks within the first term). + +#### terms + +- Type: `Array` +- Ordered list of reconstruction terms describing what chunks to download from which xorb +- Each `CASReconstructionTerm` contains: + - `hash`: The xorb hash (64-character lowercase hex string) + - `range`: Chunk index range`{ start: number, end: number }` within the xorb; end-exclusive `[start, end)` + - `unpacked_length`: Expected length after decompression (for validation) + +#### fetch_info + +- Type: `Map>` +- Maps xorb hashes to required information to download some of their chunks. + - The mapping is to an array of 1 or more `CASReconstructionFetchInfo` +- Each `CASReconstructionFetchInfo` contains: + - `url`: HTTP URL for downloading the xorb data, presigned URL containing authorization information + - `url_range` (bytes_start, bytes_end): Byte range `{ start: number, end: number }` for the Range header; end-inclusive `[start, end]` + - The `Range` header MUST be set as `Range: bytes=-` when downloading this chunk range + - `range` (index_start, index_end): Chunk index range `{ start: number, end: number }` that this URL provides; end-exclusive `[start, end)` + - This range indicates which range of chunk indices within this xorb that this fetch info term is describing + +## Stage 3: Downloading and Reconstructing the File + +### Process Overview + +1. Process each `CASReconstructionTerm` in order from the `terms` array +2. For each `CASReconstructionTerm`, find matching fetch info using the term's hash + + 1. get the list of fetch_info items under the xorb hash from the `CASReconstructionTerm`. The xorb hash is guaranteed to exist as a key in the fetch_info map. + 2. linearly iterate through the list of `CASReconstructionFetchInfo` and find one which refers to a chunk range that is equal or encompassing the term's chunk range. + - Such a fetch_info item is guaranteed to exist. If none exist the server is at fault. +3. Download the required data using HTTP `GET` request and MUST set the `Range` header +4. Deserialize the downloaded xorb data to extract chunks + + 1. This series of chunks contains chunks at indices specified by the `CASReconstructionFetchInfo`'s `range` field. Trim chunks at the beginning or end to match the chunks specified by the reconstruction term's `range` field. + 2. (for the first term only) skip `offset_into_first_range` bytes +5. Concatenate the results in term order to reconstruct the file + +### Detailed Download Process + +#### Download Reconstruction + +- use the reconstruction api to download the reconstruction object for a given file + +```python +file_id = "0123...abcdef" +api_endpoint, token = get_token() # follow auth instructions +url = api_endpoint + "/reconstructions/" + file_id +reconstruction = get(url, headers={"Authorization": "Bearer: " + token}) + +# break the reconstruction into components +terms = reconstruction["terms"] +fetch_info = reconstruction["fetch_info"] +offset_into_first_range = reconstruction["offset_into_first_range"] +``` + +#### Match Terms to Fetch Info + +For each `CASReconstructionTerm` in the `terms` array: + +- Look up the term's `hash` in the `fetch_info` map to get a list of `CASReconstructionFetchInfo` +- Find a `CASReconstructionFetchInfo` entry where the fetch info's `range` contains the term's `range` + - linearly search through the array of `CASReconstructionFetchInfo` and find the element where the range block (`{ "start": number, "end": number }`) of the `CASReconstructionFetchInfo` has start <= term's range start AND end >= term's range end. + - The server is meant to guarantee a match, if there isn't a match this download is considered failed and the server made an error. + +```python +for term in terms: + xorb_hash = term["hash"] + fetch_info_entries = fetch_info[xorb_hash] + fetch_info_entry = None + for entry in fetch_info_entries: + if entry["range"][start] <= term["range"]["start"] and entry["range"]["end"] >= term["range"]["end"]: + fetch_info_entry = entry + break + if fetch_info_entry is None: + # Error! +``` + +#### Step 2: Download Xorb Data + +For each matched fetch info: + +1. Make an HTTP GET request to the `url` in the fetch info entry +2. Include a `Range` header: `bytes={url_range.start}-{url_range.end}` + +```python +for term in terms: + ... + data_url = fetch_info_entry["url"] + range_header = "bytes=" + fetch_info_entry["url_range"]["start"] + "-" + fetch_info_entry["url_range"]["end"] + data = get(data_url, headers={"Range": range_header}) +``` + +#### Deserialize Downloaded Data + +The downloaded data is in xorb format and MUST be deserialized: + +1. **Parse xorb structure**: The data contains compressed chunks with headers +2. **Decompress chunks**: Each chunk has a header followed by compressed data +3. **Extract byte indices**: Track byte boundaries between chunks for range extraction +4. **Validate length**: Decompressed length MUST match `unpacked_length` from the term + +**Note**: The deserialization process depends on the [Xorb format](./xorb). + +```python +for term in terms: + ... + chunks = {} + for i in range(fetch_info_entry["range"]["start"], fetch_info_entry["range"]["end"]): + chunk = deserialize_chunk(data) # assume data is a reader that advances forwards + chunks[i] = chunk + # at this point data should be fully consumed +``` + +#### Step 4: Extract Term Data + +From the deserialized xorb data: + +1. Use the term's `range` to identify which chunks are needed +2. Extract only the chunks specified by `range.start` to `range.end-1` (end-exclusive) +3. Apply any range offsets if processing a partial file download + +```python +file_chunks = [] +for term in terms: + ... + for i in range(term["range"]["start"], term["range"]["end"]): + chunk = chunks[i] + # it is possible that the offset captures multiple chunks, so we may need to skip whole chunks + if offset_into_first_range > len(chunk): + offset_into_first_range -= len(chunk) + continue + if offset_info_first_range > 0: + chunk = chunk[offset_into_first_range:] + offset_info_first_range = 0 + + file_chunks.push(chunk) +``` + +#### Step 5: Stitch Results Together + +Write all of the chunks to the output file or buffer. + +If a range was specified then the total data will need to be truncated to the amount of bytes requested. +When a range is specified but the range does not end on a chunk boundary the last byte of the requested range will be in the middle of the last chunk. +A client knows the start of the data from `offset_into_first_range` and can then use the length of the specified range to know end end offset. + +```python +with open(file_path) as f: + for chunk in file_chunks: + f.write(chunk) +``` + +## Range Downloads + +For partial file downloads, the reconstruction API supports range queries: + +- Include `Range: bytes=start-end` header in reconstruction request +- The `offset_into_first_range` field indicates where your range starts within the first term +- The end of the content will need to be truncated to fit the requested range. + - Except if the requested range exceeds the total file length, then the returned content will be shorter and no truncation is necessary. + +When downloading individual term data: + +A client MUST include the `Range` header formed with the values from the `url_range` field to specify the exact range of data of a xorb that they are accessing. Not specifying this header will cause result in an authorization failure. + +Xet global deduplication requires that access to xorbs is only granted to authorized ranges. +Not specifying this header will result in an authorization failure. + +## Performance Considerations + +- **Range coalescing**: Multiple terms may share the same fetch info for efficiency, so a single fetch info may be larger than any 1 term and could be used to fulfil multiple terms. +Consider downloading such content only once and reusing the data. +- **Parallel downloads**: Terms can be downloaded in parallel, but MUST be assembled in order + - On file systems with fast seeking, it MAY be advantageous to open the output file in different threads and writing contents at different offsets +- **Caching**: Clients SHOULD consider caching downloaded xorb ranges to avoid redundant requests +- **Retry logic**: Implement exponential backoff for transient failures + +### Caching recommendations + +1. It can be ineffective to cache the reconstruction object + 1. The fetch_info section provides short-expiration pre-signed URL's hence Clients SHOULD NOT cache the urls beyond their short expiration + 2. To get those URL's to access the data you will need to call the reconstruction API again anyway +2. Cache chunks by range not just individually + 1. If you need a chunk from a xorb it is very likely that you will need another, so cache them close +3. Caching helps when downloading similar contents. May not be worth to cache data if you are always downloading different things + +## More complex QueryReconstruction Example + +Here's an example of a serialized `QueryReconstructionResponse` struct that shows how file reconstruction would work across multiple xorbs: + +```json +{ + "offset_into_first_range": 0, + "terms": [ + { + "hash": "a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456", + "unpacked_length": 263873, + "range": { + "start": 1, + "end": 4 + } + }, + { + "hash": "fedcba0987654321098765432109876543210fedcba098765432109876543", + "unpacked_length": 143890, + "range": { + "start": 0, + "end": 3 + } + }, + { + "hash": "a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456", + "unpacked_length": 3063572, + "range": { + "start": 3, + "end": 43 + } + }, + ], + "fetch_info": { + "a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456": [ + { + "range": { + "start": 1, + "end": 43 + }, + "url": "https://transfer.xethub.hf.co/xorb/default/a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130721%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d6796aa6097c82ba7e33b4725e8396f8a9638f7c3d4b5a6b7c8d9e0f1a2b3c4d", + "url_range": { + "start": 57980, + "end": 1433008 + } + } + ], + "fedcba0987654321098765432109876543210fedcba098765432109876543": [ + { + "range": { + "start": 0, + "end": 3 + }, + "url": "https://transfer.xethub.hf.co/xorb/default/fedcba0987654321098765432109876543210fedcba098765432109876543?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130721%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=d6796aa6097c82ba7e33b4725e8396f8a9638f7c3d4b5a6b7c8d9e0f1a2b3c4d", + "url_range": { + "start": 0, + "end": 65670 + } + } + ] + } +} +``` + +This example shows reconstruction of a file that requires: + +- Chunks `[1, 4)` from the first xorb (~264KB of unpacked data) +- Chunks `[0, 2)` from the second xorb (~144KB of unpacked data) +- Chunks `[3, 43)` from the same xorb from the first term (~3MB of unpacked data) + +The `fetch_info` provides the HTTP URLs and byte ranges needed to download the required chunk data from each xorb. The ranges provided within `fetch_info` and term sections are always end-exclusive i.e. `{ "start": 0, "end": 3 }` is a range of 3 chunks at indices 0, 1 and 2. +The ranges provided under a `fetch_info` items' `url_range` key are to be used to form the `Range` header when downloading the chunk range. +A `"url_range"` value of `{ "start": X, "end": Y }` creates a `Range` header value of `bytes=X-Y`. + +When downloading and deserializing the chunks from xorb `a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456` we will have the chunks at indices `[1, 43)`. +We will need to only use the chunks at `[1, 4)` to fulfill the first term and then chunks `[3, 43)` to fulfill the third term. +Note that in this example the chunk at index 3 is used twice! This is the benefit of deduplication; we only need to download the chunk content once. + +## Diagram + +```mermaid +sequenceDiagram + autonumber + actor client as Client + participant S as CAS API + participant Transfer as Transfer Service (Xet storage) + + client->>S: GET /reconstructions/{file_id}
Authorization: Bearer
Range: bytes=start-end (optional) + S-->>client: 200 OK
QueryReconstructionResponse {offset_into_first_range, terms[], fetch_info{}} + + loop For each term in terms (ordered) + client->>client: Find fetch_info by xorb hash, entry whose range contains term.range + client->>Transfer: GET {url}
Range: bytes=url_range.start-url_range.end + Transfer-->>client: 206 Partial Content
xorb byte range + client->>client: Deserialize xorb → chunks for fetch_info.range + client->>client: Trim to term.range, apply offset for first term + client->>client: Append chunks to output + end + + alt Range requested + client->>client: Truncate output to requested length + end +``` diff --git a/docs/xet/file-id.md b/docs/xet/file-id.md new file mode 100644 index 0000000000..9e4bae8afd --- /dev/null +++ b/docs/xet/file-id.md @@ -0,0 +1,33 @@ +# Getting a Xet File ID from the Hugging Face Hub + +This section explains the Xet file ID used in the reconstruction API to download a file from the Hugging Face Hub using the xet protocol. + +Given a particular namespace, repository and branch or commit hash and file path from the root of the repository, build the "resolve" URL for the file following this format: + +```js +Define: +namespace: username/organization, e.g. Qwen +repository: the repository name e.g. Qwen-Image-Edit +branch: any git branch or commit hash e.g. main +filepath: filepath in repository e.g. transformer/diffusion_pytorch_model-00001-of-00009.safetensors + +resolve URL: + +https://huggingface.co/{namespace}/{repository}/resolve/{branch}/{filepath} + + +Example: + +https://huggingface.co/Qwen/Qwen-Image-Edit/resolve/main/transformer/diffusion_pytorch_model-00001-of-00009.safetensors +``` + +Then make a `GET` request to the resolve URL using your standard Hugging Face Hub credentials/token. + +If the file is stored on the xet system then a successful response will have a `X-Xet-Hash` header. + +The string value of this header is the Xet file ID and SHOULD be used in the path of the reconstruction API URL. +This is the string representation of the hash and can be used directly in the file reconstruction API on download. + +> [!NOTE] +> The resolve URL will return a 302 redirect http status code, following the redirect will download the content via the old LFS compatible route rather than through the Xet protocol. +In order to use the Xet protocol you MUST NOT follow this redirect. diff --git a/docs/xet/file-reconstruction.md b/docs/xet/file-reconstruction.md new file mode 100644 index 0000000000..9e70f20f7e --- /dev/null +++ b/docs/xet/file-reconstruction.md @@ -0,0 +1,133 @@ +# File Reconstruction: Term-based Representation + +This document describes how a file can be represented and reconstructed from a compact, deduplicated form using a series of terms. Each term specifies where to source data (a content-addressed container called a xorb) and which chunk indices within that container are required. + +## Glossary + +- **Xorb**: A content-addressed container holding a sequence of chunks. It is identified by a cryptographic hash (the “xorb hash”). +- **Chunk**: A unit of data inside a xorb. Chunks are ordered and individually decompressible/decodable into raw bytes. +- **Chunk index**: The 0-based position of a chunk within a particular xorb. +- **Term**: A pair that identifies a xorb and a contiguous chunk index range within that xorb: `(xorb hash, chunk range [start, end))`. +- **Reconstruction**: A list of terms. + +## Core Idea + +After following the [chunking procedure](./chunking) a file can be represented as an ordering of chunks. +Those chunks are then packed into [xorbs](./xorb) and given the set of xorbs we convert the file representation to "reconstruction" made up of "terms". +When forming xorbs the ordering and grouping of chunks prioritizes contiguous runs of chunks that appear in a file such that when referencing a xorb we maximize the term range length. + +Any file’s raw bytes can be described as the concatenation of data produced by a sequence of terms. +Each term references a contiguous range of chunks within a particular xorb. +The file is reconstructed by retrieving those chunk ranges, decoding them to raw bytes, and concatenating in order. + +### Diagram + +A file with 4 terms. Each term is a pointer to chunk range within a xorb. + +```txt +File Reconstruction + + ┌----------------------------┬┬--------------------------┬┬---------------------------┬┬-------------------------┐ + ¦ X0 ¦¦ X1 ¦¦ X2 ¦¦ X3 ¦ + ¦ start: 0 ¦¦ start: 0 ¦¦ start: 300 ¦¦ start: 300 ¦ + ¦ end: 1024 ¦¦ end: 700 ¦¦ end: 1024 ¦¦ end: 700 ¦ + ├----------------------------++--------------------------++---------------------------++-------------------------┤ + ¦ /¦ /\ /\ / + ¦ / ¦ / \ ¦ \ / + ¦ / ¦ /---/ \----\ ¦ \----\ / + ¦ ¦ ¦ / \ ¦ \ / + ¦ ¦ ¦ / \ ¦ \ / + ┌-------------------------┐ ┌-------------------------┐ ┌-------------------------┐ ┌-------------------------┐ + ¦ X0 ¦ ¦ X1 ¦ ¦ X2 ¦ ¦ X3 ¦ + ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ + ¦0 1024¦ ¦0 1000¦ ¦0 1090¦ ¦0 870 ¦ + └-------------------------┘ └-------------------------┘ └-------------------------┘ └-------------------------┘ +``` + +## Term Format + +Each term consists of: + +- **Xorb hash**: A 32 byte hash value that is the key to the xorb in the CAS (Content Addressed Store). +- **Chunk range**: A half-open interval [start, end) of chunk indices within that xorb. The range includes the chunk at index `start` and excludes the chunk at index `end`. + +## Reconstruction Rules + +Given an ordered list of terms describing a file: + +1. For each term, fetch the specified chunk range from the identified xorb. +2. Decode/decompress the chunks into raw bytes, preserving the original order. +3. If reconstructing the entire file, concatenate the decoded outputs of all terms in their listed order. +4. If reconstructing a byte sub-range of the file, the first and last terms may be partially used: + - Skip a prefix of bytes within the first term’s decoded output so the file-level range starts at the requested offset. + - Truncate the tail of the last term’s decoded output so the file-level range ends at the requested position. + +### Ordering and Coverage + +- Terms are ordered according to the file’s byte order. Concatenating their decoded outputs yields the requested file region. +- Gaps MUST NOT be present. If gaps exist, the reconstruction would not produce a contiguous byte stream. + +### Multiple Terms per Xorb and Coalescing + +- A file may contain multiple terms that reference the same xorb, potentially with disjoint chunk ranges. This enables deduplication across distant parts of the file. +- When multiple terms target overlapping or adjacent chunk ranges within the same xorb, implementations SHOULD coalesce these into a single retrieval to reduce I/O and request overhead, while preserving the term-level reconstruction semantics. + +### Chunk and Byte Boundaries + +- Chunk ranges are specified in chunk index space, not byte offsets. The decoded size of each chunk is not required to be uniform. +- The byte length contributed by a term equals the sum of the decoded sizes of its referenced chunks, minus any initial skip or final truncation when reconstructing sub-ranges. +- Slicing within a term (for sub-range reconstruction) happens at byte granularity in the decoded output of the addressed chunk sequence. + +### Determinism and Integrity + +- The xorb hash binds the identity of the underlying chunk set to its content. If the xorb hash is correct and the specified chunk range is retrieved and decoded as defined, the resulting bytes are deterministic. +- A file-level identity (e.g., a cryptographic hash of the reconstructed bytes) can be validated by reconstructing and hashing the result. + +### Example (Conceptual) + +Assume a file is represented by the following ordered terms: + +| Term | Xorb hash (conceptual) | Chunk range | +|------|-------------------------|-------------| +| 1 | X1 | [0, 5) | +| 2 | X2 | [3, 8) | +| 3 | X1 | [9, 12) | + +Reconstruction proceeds by obtaining chunks 0,1,2,3,4 from xorb X1, chunks 3,4,5,6,7 from xorb X2, and chunks 9,10,11 from xorb X1, decoding each contiguous range, and concatenating in the term order 1 → 2 → 3. + +## Serialization and Deserialization + +This section summarizes how the term-based reconstruction is persisted and exchanged. + +### Serialization into shards (file info section) + +A file’s reconstruction can be serialized into a shard as part of its file info section. +Conceptually, this section encodes the complete set of terms that describe the file. +When stored this way, the representation is canonical and sufficient to reconstruct the full file solely from its referenced xorb ranges. + +Reference: [shard format file info](./shard#2-file-info-section) + +### Deserialization from the reconstruction API (JSON) + +A reconstruction API can return a JSON object that carries the full reconstruction. +This response is represented by a structure named “QueryReconstructionResponse”, where the `terms` key enumerates the ordered list of terms required to reconstruct the entire file. +The `terms` list contains, for each term, the xorb identifier and the contiguous chunk index range to retrieve. +Other fields may provide auxiliary details (such as offsets or fetch hints) that optimize retrieval without altering the meaning of the `terms` sequence. + +Reference: [api](./api), [download protocol](./download-protocol) + +## Fragmentation and Why Longer Ranges Matter + +Fragmentation refers to representing a file with many very short, scattered ranges across many xorbs. While this can maximize deduplication opportunities, it often harms read performance and increases overhead. + +- Meaning of fragmentation: The file’s byte stream is assembled from numerous small term ranges, potentially spanning many xorbs. This implies more lookups, more range fetches, and poorer locality. +- Costs of fragmentation: + - Larger reconstruction objects + - Potentially more network requests (overhead per request) +- Why prioritize longer ranges: + - Fewer, longer ranges reduce round-trips and enable more sequential reads + - Simpler scheduling and write patterns during reconstruction + +In practice there is a balance: longer ranges improve reconstruction performance, while finer granularity can increase deduplication savings. +Favoring longer contiguous chunk ranges within the same xorb, and coalescing adjacent or overlapping ranges when feasible, helps maintain good read performance without sacrificing correctness. +In `xet-core` we use a fragmentation prevention mechanism that targets that the average term contains 8 chunks. diff --git a/docs/xet/hashing.md b/docs/xet/hashing.md new file mode 100644 index 0000000000..6b2c3ab7c9 --- /dev/null +++ b/docs/xet/hashing.md @@ -0,0 +1,178 @@ +# Hashing + +- [Chunk hashes](#chunk-hashes) - compute for each chunk from chunk data. +- [Xorb Hashes](#xorb-hashes) - compute for each xorb from its chunk hashes. +- [File Hashes](#file-hashes) - compute for each file from its chunk hashes. +- [Term Verification Hashes](#term-verification-hashes) - compute for each term in a reconstruction when serializing a shard from the chunk hashes in the xorb that is used in that term. + +The Xet protocol utilizes a few different hashing types. + +All hashes referenced are 32 bytes (256 bits) long. + +## Chunk Hashes + +After cutting a chunk of data, the chunk hash is computed via a blake3 keyed hash with the following key (DATA_KEY): + +### DATA_KEY + +```json +[ + 102, 151, 245, 119, 91, 149, 80, 222, 49, 53, 203, 172, 165, 151, 24, 28, 157, 228, 33, 16, 155, 235, 43, 88, 180, 208, 176, 75, 147, 173, 242, 41 +] +``` + +[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/data_hash.rs#L308-L311) + +## Xorb Hashes + +Xorbs are composed of a series of chunks; given the series of chunks that make up a xorb, to compute the hash or xorb hash we will compute a MerkleHash using a [Merkle Tree](https://en.wikipedia.org/wiki/Merkle_tree) data structure with custom hashing functions. +**The xorb hash will be the root node hash of the MerkleTree.** + +The leaf node hashes are the chunk hashes as described in the previous section. + +The hash function used to compute internal node hashes is as follows: + +- concatenate the hashes together such that for each chunk there is a line in order formatted like `{chunk_hash:x} : {size}\n` + - the hash first in lowercase hex format (64 hex characters e.g. `a3f91d6e8b47c20ff9d84a1c77dcb8e5a91e6fbf2b2d483af6d3c1e90ac57843`) + - a space, a colon, a space (` : `) + - the chunk length number e.g. 64000 + - finally a newline `\n` character +- Then take the bytes from this string and compute a blake3 keyed hash with the following key (INTERNAL_NODE_KEY) + +[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/aggregated_hashes.rs#L103-L109) + +### INTERNAL_NODE_KEY + +```json +[ + 1, 126, 197, 199, 165, 71, 41, 150, 253, 148, 102, 102, 180, 138, 2, 230, 93, 221, 83, 111, 55, 199, 109, 210, 248, 99, 82, 230, 74, 83, 113, 63 +] +``` + +### Example of data for internal node + +Consider that a node were 4 chunks with the following pairs of hashes and lengths: + +```txt +hash,length (bytes) +1f6a2b8e9d3c4075a2e8c5fd4f0b763e6f3c1d7a9b2e6487de3f91ab7c6d5401,10000 +7c94fe2a38bdcf9b4d2a6f7e1e08ac35bc24a7903d6f5a0e7d1c2b93e5f748de,20000 +cfd18a92e0743bb09e56dbf76ea2c34d99b5a0cf271f8d429b6cd148203df061,25000 +e38d7c09a21b4cf8d0f92b3a85e6df19f7c20435e0b1c78a9d635f7b8c2e4da1,64000 +``` + +Then to form the buffer to compute the internal node hash we will create this string (note the `\n` newline at the end): + +```txt +"1f6a2b8e9d3c4075a2e8c5fd4f0b763e6f3c1d7a9b2e6487de3f91ab7c6d5401 : 10000 +7c94fe2a38bdcf9b4d2a6f7e1e08ac35bc24a7903d6f5a0e7d1c2b93e5f748de : 20000 +cfd18a92e0743bb09e56dbf76ea2c34d99b5a0cf271f8d429b6cd148203df061 : 25000 +e38d7c09a21b4cf8d0f92b3a85e6df19f7c20435e0b1c78a9d635f7b8c2e4da1 : 64000 +" +``` + +Then compute the blake3 keyed hash with INTERNAL_NODE_KEY to get the final hash. + +### Example Python code for the internal hash function + +```python +from blake3 import blake3 + +def internal_hash_function(node): + buffer = "" + for chunk in node: + size = len(chunk) + chunk_hash = compute_chunk_hash(chunk) + buffer += f"{chunk_hash:x} : {size}\n" + + blake3(bytes(buffer), key=INTERNAL_NODE_KEY) +``` + +## File Hashes + +After chunking a whole file, to compute the file hash, follow the same procedure used to compute the xorb hash and then take that final hash as data to compute a blake3 keyed hash with a key that is all 0's. + +This means create a MerkleTree using the same hashing functions described in the previous section. +Then take the root node's hash and compute a blake3 keyed hash with the key being 32 0-value bytes. + +[reference implementation](https://github.com/huggingface/xet-core/blob/main/merklehash/src/aggregated_hashes.rs#L123-L125) + +## Term Verification Hashes + +When uploading a shard, each term in each file info in the shard MUST have a matching FileVerificationEntry section that contains a hash. + +To generate this hash, take the chunk hashes for the specific range of chunks that make up the term and: + +1. **Concatenate the raw hash bytes**: Take all the chunk hashes in the range (from `chunk_index_start` to `chunk_index_end` in the xorb specified in the term) and concatenate their raw 32-byte representations together in order. + +2. **Apply keyed hash**: Compute a blake3 keyed hash of the concatenated bytes using the following verification key (VERIFICATION_KEY): + +### VERIFICATION_KEY + +```json +[ + 127, 24, 87, 214, 206, 86, 237, 102, 18, 127, 249, 19, 231, 165, 195, 243, 164, 205, 38, 213, 181, 219, 73, 230, 65, 36, 152, 127, 40, 251, 148, 195 +] +``` + +The result of the blake3 keyed hash is the verification hash that MUST be used in the FileVerificationEntry for the term. + +[reference implementation](https://github.com/huggingface/xet-core/blob/main/mdb_shard/src/chunk_verification.rs#L4-L16) + +### Example Python code for the verification hash + +```python +def verification_hash_function(term): + buffer = bytes() + # note chunk ranges are end exclusive + for chunk_hash in term.xorb.chunk_hashes[term.chunk_index_start : term.chunk_index_end]: + buffer.extend(bytes(chunk_hash)) + return blake3(buffer, key=VERIFICATION_KEY) +``` + +## Reference Files + +Reference files are provided in Hugging Face Dataset repository [xet-team/xet-spec-reference-files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files). + +In this repository there are a number of different samples implementors can use to verify hash computations. + +> Note that all hashes are represented as strings. +To get the raw value of these hashes you must invert the endianness of each byte octet in the hash string, reversing the procedure described in [api](./api#converting-hashes-to-strings). + +### Chunk Hashes Sample + +There are 3 chunks files, for each file name, the first 64 characters are the string format of the chunk hash of the data in the file: + +- [b10aa1dc71c61661de92280c41a188aabc47981739b785724a099945d8dc5ce4.chunk](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/b10aa1dc71c61661de92280c41a188aabc47981739b785724a099945d8dc5ce4.chunk) +- [26255591fa803b6baf25d88c315b8a6f5153d5bcfdf18ec5ef526264e0ccc907.chunk](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/26255591fa803b6baf25d88c315b8a6f5153d5bcfdf18ec5ef526264e0ccc907.chunk) +- [099cb228194fe640e36a6c7d274ee5ed3a714ccd557a0951d9b6b43a7292b5d1.chunk](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/099cb228194fe640e36a6c7d274ee5ed3a714ccd557a0951d9b6b43a7292b5d1.chunk) + +### File Hash Sample + +The [xet-team/xet-spec-reference-files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files) repository contains the original file +[Electric_Vehicle_Population_Data_20250917.csv](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv). + +When processed through the Xet upload protocol the chunks that are produced for this file are listed (formatted ` `) in the file +[Electric_Vehicle_Population_Data_20250917.csv.chunks](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.chunks). + +Using these chunks to compute a file hash of the entire file the result is the hash stored in the file +[Electric_Vehicle_Population_Data_20250917.csv.xet-file-hash](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.xet-file-hash) or the raw value `118a53328412787fee04011dcf82fdc4acf3a4a1eddec341c910d30a306aaf97`. + +### Xorb Hash Sample + +All of the chunks of [Electric_Vehicle_Population_Data_20250917.csv](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv) can fit into 1 single xorb. + +The xorb produced with all of the chunks in order for this file can be found serialized in file [eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb). + +The hash of this xorb is `eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632`, the value in [Electric_Vehicle_Population_Data_20250917.csv.xet-xorb-hash](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.xet-xorb-hash). + +The chunks that make up this xorb are listed in a file [eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb.chunks](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb.chunks); +note this file is equivalent to [Electric_Vehicle_Population_Data_20250917.csv.chunks](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.chunks). + +### Range Hash Sample + +In the reconstruction of [Electric_Vehicle_Population_Data_20250917.csv](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv) +with xorb [eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb) there is 1 range that contains all 796 chunks. + +The verification range hash for this range is the value in [eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb.range-hash](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb.range-hash) +which is `d81c11b1fc9bc2a25587108c675bbfe65ca2e5d350b0cd92c58329fcc8444178`. diff --git a/docs/xet/index.md b/docs/xet/index.md new file mode 100644 index 0000000000..ab27c53ad1 --- /dev/null +++ b/docs/xet/index.md @@ -0,0 +1,51 @@ +# Xet Protocol Specification + +> [!NOTE] +> Version 0.1.0 (1.0.0 on release) +> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119](https://www.ietf.org/rfc/rfc2119.txt) [RFC8174](https://www.ietf.org/rfc/rfc8174.txt) +when, and only when, they appear in all capitals, as shown here. + +This specification defines the end-to-end Xet protocol for content-addressed data: chunking and hashing rules, deduplication strategy, xorb and shard object formats, file reconstruction semantics, authentication, and the CAS APIs for upload and download. +Its goal is interoperability and determinism: independent implementations MUST produce the same hashes, objects, and API behavior so data written by one client can be read by another with integrity and performance. +Implementors can create their own clients, SDKs, and tools that speak the Xet protocol and interface with the CAS service, as long as they MUST adhere to the requirements defined here. + +## Building a Client Library for Xet Storage + +- [Upload Protocol](./upload-protocol): End-to-end top level description of the upload flow. +- [Download Protocol](./download-protocol): Instructions for the download procedure. +- [CAS API](./api): HTTP endpoints for reconstruction, global chunk dedupe, xorb upload, and shard upload, including error semantics. +- [Authentication and Authorization](./auth): How to obtain Xet tokens from the Hugging Face Hub, token scopes, and security considerations. +- [Converting Hugging Face Hub Files to Xet File ID's](./file-id): How to obtain a Xet file id from the Hugging Face Hub for a particular file in a model or dataset repository. + +## Overall Xet Architecture + +- [Content-Defined Chunking](./chunking): Gearhash-based CDC with parameters, boundary rules, and performance optimizations. +- [Hashing Methods](./hashing): Descriptions and definitions of the different hashing functions used for chunks, xorbs and term verification entries. +- [File Reconstruction](./file-reconstruction): Defining "term"-based representation of files using xorb hash + chunk ranges. +- [Xorb Format](./xorb): Explains grouping chunks into xorbs, 64 MiB limits, binary layout, and compression schemes. +- [Shard Format](./shard): Binary shard structure (header, file info, CAS info, footer), offsets, HMAC key usage, and bookends. +- [Deduplication](./deduplication): Explanation of chunk level dedupe including global system-wide chunk level dedupe. + +## Reference Implementation + +### xet-core: hf-xet + git-xet + +The primary reference implementation of the protocol written in Rust 🦀 lives in the [xet-core](https://github.com/huggingface/xet-core) repository under multiple crates: + +- [cas_types](https://github.com/huggingface/xet-core/tree/main/cas_types) - Common re-usable types for interacting with CAS API's +- [cas_client](https://github.com/huggingface/xet-core/tree/main/cas_client) - Client interface that calls CAS API's, including comprehensive implementation of download protocol. +- [mdb_shard](https://github.com/huggingface/xet-core/tree/main/mdb_shard) - Library for interacting with shards and the shard binary format. +- [deduplication](https://github.com/huggingface/xet-core/tree/main/deduplication) - Exposes interfaces to deduplicate chunks locally and using global deduplication + - [deduplication/src/chunking.rs](https://github.com/huggingface/xet-core/blob/main/deduplication/src/chunking.rs) - The reference implementation of the chunking algorithm. +- [merklehash](https://github.com/huggingface/xet-core/tree/main/merklehash) - Exports a `MerkleHash` type extensively used to represent hashes. Exports functions to compute the different hashes used to track chunks, xorbs and files. +- [data](https://github.com/huggingface/xet-core/tree/main/data) - Comprehensive package exposing interfaces to upload and download contents +- [hf_xet](https://github.com/huggingface/xet-core/tree/main/hf_xet) - Python bindings to use the Xet protocol for uploads and downloads with the Hugging Face Hub. +- [git-xet](ttps://github.com/huggingface/xet-core/tree/main/git-xet) - git lfs custom transfer agent that uploads files using the xet protocol to the Hugging Face Hub. + +### huggingface.js + +There is also a second reference implementation in Huggingface.js that can be used when downloading or uploading files with the `@huggingface/hub` library. + +- Download uses the `XetBlob` that can be found in [XetBlob.ts](https://github.com/huggingface/huggingface.js/blob/main/packages/hub/src/utils/XetBlob.ts). +- The upload implementation is more comprehensive but the root of it begins in [uploadShards](https://github.com/huggingface/huggingface.js/blob/main/packages/hub/src/utils/uploadShards.ts). + - The upload process uses xet-core constructs compiled from Rust to WebAssembly, particularly all functions exported from the [hf_xet_thin_wasm](https://github.com/huggingface/xet-core/tree/main/hf_xet_thin_wasm) crate. diff --git a/docs/xet/shard.md b/docs/xet/shard.md new file mode 100644 index 0000000000..ab62b370e1 --- /dev/null +++ b/docs/xet/shard.md @@ -0,0 +1,549 @@ +# MDB Shard File Format Specification + +A Shard is a serialized object containing file reconstruction information and xorb metadata for deduplication purposes. + +The Shard format is the vehicle for uploading the file reconstruction upload and communicating information about xorbs and chunks that clients can deduplicate their data against. + +## Overview + +The MDB (Merkle Database) shard file format is a binary format used to store file metadata and content-addressable storage (CAS) information for efficient deduplication and retrieval. +This document describes the binary layout and deserialization process for the shard format. +Implementors of the xet protocol MUST use the shard format when implementing the [upload protocol](./upload-protocol). +The shard format is used on the shard upload (record files) and global deduplication APIs. + +## Use As API Request and Response Bodies + +The shard format is used in the shard upload API as the request payload and in the global deduplication/chunk query API as the response payload. + +### Shard Upload + +The shard in this case is a serialization format that allows clients to denote the files that they are uploading. +Each file reconstruction maps to a File Info block in the File Info section. +Additionally, the listing of all new xorbs that the client created are mapped to items (CAS Info blocks) in the CAS Info section so that they may be deduplicated against in the future. + +When uploading a shard the footer section MUST be omitted. + +An example of a shard that can be used for file upload can be found in [Xet reference files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.shard.verification-no-footer). +A version of this shard that also contains the footer in [Xet reference files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.shard.verification) too, see the README for the reference files dataset for more context. + +### Global Deduplication + +Shards returned by the Global Deduplication API have an empty File Info Section, and only contain relevant information in the CAS Info section. +The CAS Info section returned by this API contains xorbs, where a xorb described in the CAS Info section contains the chunk that was queried. +Clients can deduplicate their content against any of the other xorbs described in any CAS Info block in the CAS Info section of the returned shard. +Other xorb descriptions returned in a shard are possibly more likely to reference content that the client has. + +An example of a shard that can be returned for a global deduplication query can be found in [Xet reference files](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv.shard.dedupe). + +## File Structure + +A shard file consists of the following sections in order: + +```txt +┌─────────────────────┐ +│ Header │ +├─────────────────────┤ +│ File Info Section │ +├─────────────────────┤ +│ CAS Info Section │ +├─────────────────────┤ +│ Footer │ +└─────────────────────┘ +``` + +## Overall File Layout with Byte Offsets + +```txt +Offset 0: +┌───────────────────────────────────────────────────────┐ +│ Header (48 bytes) │ ← Fixed size +└───────────────────────────────────────────────────────┘ + +Offset footer.file_info_offset: +┌───────────────────────────────────────────────────────┐ +│ │ +│ File Info Section │ ← Variable size +│ (Multiple file blocks + │ +│ bookend entry) │ +│ │ +└───────────────────────────────────────────────────────┘ + +Offset footer.cas_info_offset: +┌───────────────────────────────────────────────────────┐ +│ │ +│ CAS Info Section │ ← Variable size +│ (Multiple CAS blocks + │ +│ bookend entry) │ +│ │ +└───────────────────────────────────────────────────────┘ + +Offset footer.footer_offset: +┌───────────────────────────────────────────────────────┐ +│ Footer (200 bytes, sometimes omitted) │ ← Fixed size +└───────────────────────────────────────────────────────┘ +``` + +## Constants + +- `MDB_SHARD_HEADER_VERSION`: 2 +- `MDB_SHARD_FOOTER_VERSION`: 1 +- `MDB_FILE_INFO_ENTRY_SIZE`: 48 bytes (size of each file info structure) +- `MDB_CAS_INFO_ENTRY_SIZE`: 48 bytes (size of each CAS info structure) +- `MDB_SHARD_HEADER_TAG`: 32-byte magic identifier + +## Data Types + +All multi-byte integers are stored in little-endian format. + +- `u8`: 8-bit unsigned integer +- `u32`: 32-bit unsigned integer +- `u64`: 64-bit unsigned integer +- Byte Array types are denoted like in rust as `[u8; N]` where `N` is the number of bytes in the array. +- Hash: 32-byte hash value, a special `[u8; 32]` + +## 1. Header (MDBShardFileHeader) + +**Location**: Start of file (offset 0) +**Size**: 48 bytes + +```rust +struct MDBShardFileHeader { + tag: [u8; 32], // Magic number identifier + version: u64, // Header version (must be 2) + footer_size: u64, // Size of footer in bytes, set to 0 if footer is omitted +} +``` + +**Memory Layout**: + +```txt +┌────────────────────────────────────────────────────────────────┬───────────┬───────────┐ +│ tag (32 bytes) │ version │ footer_sz │ +│ Magic Number Identifier │ (8 bytes) │ (8 bytes) │ +└────────────────────────────────────────────────────────────────┴───────────┴───────────┘ +0 32 40 48 +``` + +**Deserialization steps**: + +1. Read 32 bytes for the magic tag +2. Verify tag matches `MDB_SHARD_HEADER_TAG` +3. Read 8 bytes for version (u64) +4. Verify version equals 2 +5. Read 8 bytes for footer_size (u64) + +> [!NOTE] +> When serializing, footer_size MUST be the number of bytes that make up the footer, or 0 if the footer is omitted. + +## 2. File Info Section + +**Location**: `footer.file_info_offset` to `footer.cas_info_offset` or directly after the header + +This section contains a sequence of 0 or more file information (File Info) blocks, each consisting at least a header and at least 1 data sequence entry, and OPTIONAL verification entries and metadata extension section. +The file info section ends when reaching the bookend entry. + +Each File Info block within the overall section is a serialization of a [file reconstruction](./file-reconstruction) into a binary format. +For each file, there is a `FileDataSequenceHeader` and for each term a `FileDataSequenceEntry` with OPTIONAL a matching `FileVerificationEntry` and also OPTIONAL at the end a `FileMetadataExt`. + +A shard File Info section can contain more than 1 File Info block in series, after completing reading all the content for 1 file description, the next one immediately begins. +If when reading the header of the next section a reader encounters the bookend entry that means the file info section is over; you have read the last file description in this shard. + +### File Info Section Layout + +**Without Optional Components**: + +```txt +┌─────────────────────┐ +│ FileDataSeqHeader │ ← File 1 +├─────────────────────┤ +│ FileDataSeqEntry │ +├─────────────────────┤ +│ FileDataSeqEntry │ +├─────────────────────┤ +│ ... │ +├─────────────────────┤ +│ FileDataSeqHeader │ ← File 2 +├─────────────────────┤ +│ FileDataSeqEntry │ +├─────────────────────┤ +│ ... │ +├─────────────────────┤ +│ Bookend Entry │ ← All 0xFF hash + zeros +└─────────────────────┘ +``` + +**With All Optional Components**: + +```txt +┌─────────────────────┐ +│ FileDataSeqHeader │ ← File 1 (flags indicate verification + metadata) +├─────────────────────┤ +│ FileDataSeqEntry │ +├─────────────────────┤ +│ FileDataSeqEntry │ +├─────────────────────┤ +│ ... │ +├─────────────────────┤ +│ FileVerifyEntry │ ← One per FileDataSeqEntry +├─────────────────────┤ +│ FileVerifyEntry │ +├─────────────────────┤ +│ ... │ +├─────────────────────┤ +│ FileMetadataExt │ ← One per file (if flag set) +├─────────────────────┤ +│ FileDataSeqHeader │ ← File 2 +├─────────────────────┤ +│ ... │ +├─────────────────────┤ +│ Bookend Entry │ ← All 0xFF hash + zeros +└─────────────────────┘ +``` + +### FileDataSequenceHeader + +```rust +struct FileDataSequenceHeader { + file_hash: Hash, // 32-byte file hash + file_flags: u32, // Flags indicating conditional sections that follow + num_entries: u32, // Number of FileDataSequenceEntry structures + _unused: [u8; 8], // Reserved space 8 bytes +} +``` + +**File Flags**: + +- `MDB_FILE_FLAG_WITH_VERIFICATION` (0x80000000 or 1 << 31): Has verification entries +- `MDB_FILE_FLAG_WITH_METADATA_EXT` (0x40000000 or 1 << 30): Has metadata extension + +Given the `file_data_sequence_header.file_flags & MASK` (bitwise AND) operations, if the result != 0 then the effect is true. + +**Memory Layout**: + +```txt +┌────────────────────────────────────────────────────────────────┬──────────┬───────────┬────────────┐ +│ file_hash (32 bytes) │file_flags│num_entries│ _unused │ +│ File Hash Value │(4 bytes) │(4 bytes) │ (8 bytes) │ +└────────────────────────────────────────────────────────────────┴──────────┴───────────┴────────────┘ +0 32 36 40 48 +``` + +### FileDataSequenceEntry + +Each `FileDataSequenceEntry` is 1 term is essentially the binary serialization of a [file reconstruction term](./file-reconstruction#term-format). + +```rust +struct FileDataSequenceEntry { + cas_hash: Hash, // 32-byte Xorb hash in the term + cas_flags: u32, // CAS flags (reserved for future, set to 0) + unpacked_segment_bytes: u32, // Term size when unpacked + chunk_index_start: u32, // Start chunk index within the Xorb for the term + chunk_index_end: u32, // End chunk index (exclusive) within the Xorb for the term +} +``` + +> [!NOTE] +> Note that when describing a chunk range in a `FileDataSequenceEntry` use ranges that are start-inclusive but end-exclusive i.e. `[chunk_index_start, chunk_index_end)` + +**Memory Layout**: + +```txt +┌────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┐ +│ cas_hash (32 bytes) │cas_flags│unpacked │chunk_idx│chunk_idx│ +│ CAS Block Hash │(4 bytes)│seg_bytes│start │end │ +│ │ │(4 bytes)│(4 bytes)│(4 bytes)│ +└────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┘ +0 32 36 40 44 48 +``` + +### FileVerificationEntry (OPTIONAL) + +Verification Entries MUST be set for shard uploads. + +To generate verification hashes for shard upload read the section about [Verification Hashes](./hashing#Term-Verification-Hashes). + +```rust +struct FileVerificationEntry { + range_hash: Hash, // 32-byte verification hash + _unused: [u8; 16], // Reserved (16 bytes) +} +``` + +**Memory Layout**: + +```txt +┌────────────────────────────────────────────────────────────────┬────────────────────────────────┐ +│ range_hash (32 bytes) │ _unused (16 bytes) │ +│ Verification Hash │ Reserved Space │ +└────────────────────────────────────────────────────────────────┴────────────────────────────────┘ +0 32 48 +``` + +When a shard has verification entries, all file info sections MUST have verification entries. +If only some subset of files in the shard have verification entries, the shard is considered invalid. +Every `FileDataSequenceEntry` will have a matching `FileVerificationEntry` in this case where the range_hash is computed with the chunk hashes for that range of chunks. + +For any file the nth `FileVerificationEntry` correlates to the nth `FileDataSequenceEntry`, and like `FileDataSequenceEntries` if there are verification entries there will be `file_data_sequence_header.num_entries` verification entries (following the num_entries data sequence entries). + +### FileMetadataExt (OPTIONAL) + +This section is REQUIRED per file for shards uploaded through the shard upload API. + +There is only 1 `FileMetadataExt` instance per file info block and it is the last component of that file info block when present. + +The sha256 field is the 32 byte SHA256 of the file contents of the file described. + +```rust +struct FileMetadataExt { + sha256: Hash, // 32-byte SHA256 hash + _unused: [u8; 16], // Reserved (16 bytes) +} +``` + +**Memory Layout**: + +```txt +┌────────────────────────────────────────────────────────────────┬────────────────────────────────┐ +│ sha256 (32 bytes) │ _unused (16 bytes) │ +│ SHA256 Hash │ Reserved Space │ +└────────────────────────────────────────────────────────────────┴────────────────────────────────┘ +0 32 48 +``` + +### File Info Bookend + +The end of the file info sections is marked by a bookend entry. + +The bookend entry is 48 bytes long where the first 32 bytes are all `0xFF`, followed by 16 bytes of all `0x00`. + +Suppose you were attempting to deserialize a `FileDataSequenceHeader` and it's file hash was all 1 bits then this entry is a bookend entry and the next bytes start the next section. + +Since the file info section immediately follows the header, a client MAY skip deserializing the footer to know where it starts deserializing this section. +The file info section begins right after the header and ends when the bookend is reached. + +**Deserialization steps**: + +1. Seek to `footer.file_info_offset` +2. Read `FileDataSequenceHeader` +3. Check if `file_hash` is all `0xFF` (bookend marker) - if so, stop +4. Read `file_data_sequence_header.num_entries` × `FileDataSequenceEntry` structures +5. If `file_flags & MDB_FILE_FLAG_WITH_VERIFICATION != 0`: read `file_data_sequence_header.num_entries` × `FileVerificationEntry` +6. If `file_flags & MDB_FILE_FLAG_WITH_METADATA_EXT != 0`: read 1 × `FileMetadataExt` +7. Repeat from step 2 until bookend found + +## 3. CAS Info Section + +**Location**: `footer.cas_info_offset` to `footer.footer_offset` or directly after the file info section bookend + +This section contains CAS (Content Addressable Storage) block information. Each CAS Info block represents a xorb by first having a `CASChunkSequenceHeader` which contains the number of `CASChunkSequenceEntries` to follow that make up this block. The CAS Info section ends when reaching the bookend entry. + +### CAS Info Section Layout + +```txt +┌─────────────────────┐ +│ CASChunkSeqHeader │ ← CAS Block 1 +├─────────────────────┤ +│ CASChunkSeqEntry │ +├─────────────────────┤ +│ CASChunkSeqEntry │ +├─────────────────────┤ +│ ... │ +├─────────────────────┤ +│ CASChunkSeqHeader │ ← CAS Block 2 +├─────────────────────┤ +│ CASChunkSeqEntry │ +├─────────────────────┤ +│ ... │ +├─────────────────────┤ +│ Bookend Entry │ ← All 0xFF hash + zeros +└─────────────────────┘ +``` + +**Deserialization steps**: + +1. Seek to `footer.cas_info_offset` +2. Read `CASChunkSequenceHeader` +3. Check if `cas_hash` is all 0xFF (bookend marker) - if so, stop +4. Read `cas_chunk_sequence_header.num_entries` × `CASChunkSequenceEntry` structures +5. Repeat from step 2 until bookend found + +### CASChunkSequenceHeader + +```rust +struct CASChunkSequenceHeader { + cas_hash: Hash, // 32-byte Xorb hash + cas_flags: u32, // CAS flags (reserved for later, set to 0) + num_entries: u32, // Number of chunks in this Xorb + num_bytes_in_cas: u32, // Total size of all raw chunk bytes in this Xorb + num_bytes_on_disk: u32, // Length of the xorb after serialized when uploaded +} +``` + +**Memory Layout**: + +```txt +┌────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┐ +│ cas_hash (32 bytes) │cas_flags│num_ │num_bytes│num_bytes│ +│ CAS Block Hash │(4 bytes)│entries │in_cas │on_disk │ +│ │ │(4 bytes)│(4 bytes)│(4 bytes)│ +└────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┘ +0 32 36 40 44 48 +``` + +### CASChunkSequenceEntry + +Every `CASChunkSequenceHeader` will have a `num_entries` number field. +This number is the number of `CASChunkSequenceEntry` items that should be deserialized that are associated with the xorb described by this CAS Info block. + +```rust +struct CASChunkSequenceEntry { + chunk_hash: Hash, // 32-byte chunk hash + chunk_byte_range_start: u32, // Start position in CAS block + unpacked_segment_bytes: u32, // Size when unpacked + _unused: [u8; 8], // Reserved space 8 bytes +} +``` + +**Memory Layout**: + +```txt +┌────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────────────┐ +│ chunk_hash (32 bytes) │chunk_ │unpacked │ _unused │ +│ Chunk Hash │byte_ │segment_ │ (8 bytes) │ +│ │range_ │bytes │ │ +│ │start │(4 bytes)│ │ +│ │(4 bytes)│ │ │ +└────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────────────┘ +0 32 36 40 48 +``` + +### CAS Info Bookend + +The end of the cas info sections is marked by a bookend entry. + +The bookend entry is 48 bytes long where the first 32 bytes are all `0xFF`, followed by 16 bytes of all `0x00`. + +Suppose you were attempting to deserialize a `CASChunkSequenceHeader` and it's hash was all 1 bits then this entry is a bookend entry and the next bytes start the next section. + +Since the cas info section immediately follows the file info section bookend, a client MAY skip deserializing the footer to know where the cas info section starts starts deserialize this section, it begins right after the file info section bookend and ends when the next bookend is reached. + +## 4. Footer (MDBShardFileFooter) + +> [!NOTE] +> MUST NOT include the footer when serializing the shard as the body for the shard upload API. + +**Location**: End of file minus footer_size +**Size**: 200 bytes + +```rust +struct MDBShardFileFooter { + version: u64, // Footer version (must be 1) + file_info_offset: u64, // Offset to file info section + cas_info_offset: u64, // Offset to CAS info section + _buffer: [u8; 48], // Reserved space (48 bytes) + chunk_hash_hmac_key: Hash, // HMAC key for chunk hashes (32 bytes) + shard_creation_timestamp: u64, // Creation time (seconds since epoch) + shard_key_expiry: u64, // Expiry time (seconds since epoch) + _buffer2: [u8; 72], // Reserved space (72 bytes) + footer_offset: u64, // Offset where footer starts +} +``` + +**Memory Layout**: + +> [!NOTE] +> Fields are not exactly to scale + +```txt +┌─────────┬─────────┬─────────┬─────────────────────────────────────────────────────────────┬─────────────────────────────────────┐ +│ version │file_info│cas_info │ _buffer (reserved) │ chunk_hash_hmac_key │ +│(8 bytes)│offset │offset │ (48 bytes) │ (32 bytes) │ +│ │(8 bytes)│(8 bytes)│ │ │ +└─────────┴─────────┴─────────┴─────────────────────────────────────────────────────────────┴─────────────────────────────────────┘ +0 8 16 24 72 104 + +┌─────────┬──────────┬─────────────────────────────────────────────────────────────────────────────┬─────────┐ +│creation │shard_ │ _buffer (reserved) │footer_ │ +│timestamp│key_expiry│ (72 bytes) │offset │ +│(8 bytes)│ (8 bytes)│ │(8 bytes)│ +└─────────┴──────────┴─────────────────────────────────────────────────────────────────────────────┴─────────┘ +104 112 120 192 200 +``` + +**Deserialization steps**: + +1. Seek to `file_size - footer_size` +2. Read all fields sequentially as u64 values +3. Verify version equals 1 + +### Use of Footer Fields + +#### file_info_offset and cas_info_offset + +These offsets allow you to seek into the shard data buffer to reach these sections without deserializing linearly. + +#### HMAC Key Protection + +If `footer.chunk_hash_hmac_key` is non-zero (as a response shard from the global dedupe API), chunk hashes in the CAS Info section are protected with [HMAC](https://en.wikipedia.org/wiki/HMAC): + +- The stored chunk hashes are `HMAC(original_hash, footer.chunk_hash_hmac_key)` +- To check if a chunk of data that you have matches a chunk listed in the shard, compute `HMAC(chunk_hash, footer.chunk_hash_hmac_key)` for your chunk hash and search through the shard results. +If you find a match (matched_chunk) then you know the original chunk hash of your chunk and the matched_chunk is the same and you can deduplicate your chunk by referencing the xorb that matched_chunk belongs to. + +#### Shard Key Expiry + +The shard key expiry is a 64 bit unix timestamp of when the shard received is to be considered expired (usually in the order of days or weeks after the shard was sent back). + +After this expiry time has passed clients SHOULD consider this shard expired and SHOULD NOT use it to deduplicate data. +Uploads that reference xorbs that were referenced by this shard can be rejected at the server's discretion. + +## Complete Deserialization Algorithm + +```text +// ** option 1, read linearly, streaming ** +// assume shard is a read-able file-like object and the reader position is at start of shard +// 1. Read and validate header +header = read_header(shard) + +// 2. Read file info section +file_info = read_file_info_section(shard) // read through file info bookend + +// 3. Read CAS info section +cas_info = read_cas_info_section(shard) // read through cas info bookend + +// 4. Read footer +footer = read_footer(shard) + +// shard reader should now be at EOF + + +// ** option 2, read footer and seek ** +// assume shard is a read-able seek-able file-like object +// 1. Read and validate header +seek(start of shard) +header = read_header(shard) + +// 2. Read and validate footer (needed for offsets) +seek(end of shard minus header.footer_size) +footer = read_footer(shard) + +// 3. Read file info section +seek(footer.file_info_offset) +file_info = read_file_info_section(shard) // until footer.cas_info_offset + +// 4. Read CAS info section +seek(footer.cas_info_offset) +cas_info = read_cas_info_section(shard) // until footer.footer_offset +``` + +## Version Compatibility + +- Header version 2: Current format +- Footer version 1: Current format +- Shards with different versions will be rejected + +## Error Handling + +- Always verify magic numbers and versions +- Check that offsets are within file bounds +- Verify that bookend markers are present where expected diff --git a/docs/xet/upload-protocol.md b/docs/xet/upload-protocol.md new file mode 100644 index 0000000000..f993eaeb1d --- /dev/null +++ b/docs/xet/upload-protocol.md @@ -0,0 +1,177 @@ +# Upload Protocol + +This document describes how files are uploaded in the Xet protocol to the Content Addressable Storage (CAS) service. +The flow converts input files into chunks, applies deduplication, groups chunks into xorbs, uploads xorbs, then forms and uploads shards that reference those xorbs. +The steps can be done all be done concurrently except that all xorbs MUST be uploaded before a shard referencing them is uploaded. +Content addressing uses hashes as stable keys for deduplication and integrity verification. + +## Xet Object Types + +### Chunk + +A chunk is a slice of data from a real file. + +A chunk has an associated hash computed through the [chunk hashing process](./hashing#chunk-hashes) and its data is determined by finding chunk boundaries following the chunking algorithm defined in [chunking](./chunking). + +A chunk is ~64KiB of data with a maximum of 128KiB and minimum of 8KiB. +However, the minimum chunk size limit is not enforced for the last chunk of a file or if the file is smaller than 8KiB. + +### Xorb + +A Xorb is composed of a sequence of chunks. + +Chunks in a xorb are not simply concatenated but instead compressed and appended after a header as described in [xorb](./xorb#xorb-format). +Chunks are collected in a xorb for more efficient upload and downloads of "ranges" of chunks. +Each chunk has an associated index (beginning at 0) and chunks may addressed from xorbs using through an end exclusive chunk index range i.e. [0, 100). + +Xorbs are created by grouping sequences of chunks from files and are referenced in file reconstructions to provide instructions to rebuild the file. + +Xorbs have an associated hash computed according to the instructions for the [xorb hashing process](./hashing#xorb-hashes). + +Xorbs are always less than or equal to 64MiB in length and on average contain 1024 chunks, but this number is variable. + +### File Reconstruction + +A file reconstruction is a "recipe" to recreate a file using data from xorbs. + +Each file reconstruction is made of a series of "terms" where each term contains a xorb hash and a chunk index range. +To reconstruct a file, a user needs will need the chunks at the specified range for each term, deserialized and decompressed and concatenated in term order. + +### Shards + +Shards are serialized representations of file information and xorb metadata. + +A shard may contain multiple file reconstructions or none. +A shard may also contain information about xorbs, particularly what chunks are in a particular xorb. + +Shards are used to communicate a "file upload" or registering the file in the CAS (Content Addressed Store) as well as registering groups of xorbs associated with the same upload. + +Shards are also used to communicate xorb metadata that can be used for deduplication using the Global Deduplication API. + +The shard format is specified in [shard](./shard). + +> [!NOTE] +> In xet-core the shard format is used to keep a local cache with fast lookup of known chunks for deduplication, other implementors of the xet protocol may choose to reuse the shard format for that purpose as well, however that is not a requirement of the protocol. + +## Steps + +### 1. Chunking + +Using the chunking algorithm described in [chunking](./chunking) first split the file into variable sized chunks. +Each unique chunk MUST have a unique hash computed as described in the [Chunk Hashing section](./hashing#chunk-hashes). +This chunk hash will be used to attempt to deduplicate any chunk against other known chunks. + +### 2. Deduplication + +Given a chunk hash, attempt to find if the chunk already exists in the Xet system. + +To deduplicate a chunk is to find if the current chunk hash already exists, either in the current upload process, in a local cache of known chunks or using the [Global Deduplication API](./api#2-query-chunk-deduplication-global-deduplication). + +When a chunk is deduplicated it SHOULD NOT be re-uploaded to the CAS (by being included in a xorb in the next step), but when rebuilding the file, the chunk needs to be included by referencing the xorb that includes it and the specific chunk index. + +> [!NOTE] +> Note that Deduplication is considered an optimization and is an OPTIONAL component of the upload process, however it provides potential resource saving. + +For more detail visit the [deduplication document](./deduplication) + +### 3. Xorb Formation and Hashing + +Contiguous runs of chunks are collected into xorbs (roughly 64 MiB total length per xorb), preserving order within each run. See formation rules: [xorb](./xorb#collecting-chunks). +The xorb's content-addressed key is computed using the chunks in the xorb. See: [hashing](./hashing#xorb-hashes). + +Given the xorb hash chunks in the xorb can be referred in file reconstructions. + +### 4. Xorb Serialization and Upload + +Each xorb is serialized into its binary representation as defined by the xorb format. See: [xorb](./xorb). +The client uploads each new xorb via the [Xorb upload API](./api#3-upload-xorb). + +The serialization and upload steps are separated from collecting chunks and hashing as these steps can be done independently while still referencing the xorb in creating file reconstructions. +However a xorb MUST be uploaded before a file reconstruction that references it is uploaded in a shard. + +### 5. Shard Formation, Collect Required Components + +Map each file to a reconstruction using available xorbs, the file reconstruction MUST point to ranges of chunks within xorbs that refer to each chunk in the file. +Terms for chunks that are deduplicated using results from the Global Dedupe API will use xorb hashes that already exist in CAS. + +Then for each file: + +- Compute the file hash using the [file hashing process](./hashing#file-hashes). +- For each xorb range (a "term") compute a [verification hash](./hashing#term-verification-hashes) in order to upload it. + - These hashes are used to ensure that the client uploading the file in the shard authoritatively has access to the actual file data. +- Compute the sha256 for the file contents + +With these components it is now possible to completely serialize a [file info block](./shard#2-file-info-section) in the shard format. + +In addition to the file info information, it is also necessary to collect all metadata for new xorbs that were created. +This metadata is the xorb hash, the hash and length of each chunk, the serialized length of the xorb and the sum of the chunk lengths for a xorb. +With these components it is now possible to serialize for each xorb a [CAS Info block](./shard#3-cas-info-section). + +### 6. Shard Serialization and Upload + +Given the information collected in the previous section, serialize a shard for a batch of files following the format specified in the [shard spec](./shard). + +The client uploads the shard via the [shard upload](./api#4-upload-shard) endpoint on the CAS server. +For this to succeed, all xorbs referenced by the shard MUST have already completed uploading. + +This API registers files as uploaded. + +> [!NOTE] +> For a large batch of files or a batch of large files if the serialized shard will be greater than 64 MiB you MUST break up the content into multiple shards. + +### Done + +After all xorbs and all shards are successfully uploaded, the full upload is considered complete. +Files can then be downloaded by any client using the [download protocol](./download-protocol). + +> [!NOTE] +> If this file is being uploaded to the Hugging Face Hub, users will need to commit a git lfs pointer file using the sha256 of the file contents. + +## Ordering and Concurrency + +There are some natural ordering requirements in the upload process, e.g. you MUST have determined a chunk boundary before computing the chunk hash, and you MUST have collected a sequence of chunks to create a xorb to compute the xorb hash etc. + +However there is one additional enforced requirement about ordering: **all xorbs referenced by a shard MUST be uploaded before that shard is uploaded**. +If any xorb referenced by a shard is not already uploaded when the shard upload API is called, the server will reject the request. +All xorbs whose hash is used as an entry in the cas info section and in data entries of the file info section are considered "referenced" by a shard. + +## Integrity and Idempotency + +- Hashing of chunks, xorbs, and shards ensures integrity and enables deduplication across local and global scopes. See: [hashing](./hashing). + - the same chunk data produces the same chunk hash + - the same set of chunks will produce the same xorb hash +- Consistent chunking algorithm yields that the same data will be split into the same chunks at the same boundaries, allowing those chunks to be matched to other data and deduplicated. +- Upload endpoints are idempotent with respect to content-addressed keys; re-sending an already-present xorb or shard is safe. + +## Diagram + +```mermaid +sequenceDiagram + autonumber + participant C as Client + participant S as CAS Server + + C->>C: Chunking: split file into chunks and compute chunk hashes + + Note right of C: 2) Local deduplication (OPTIONAL) + + loop For each chunk if chunk % 1024 == 0
(global dedupe eligible) + opt Global deduplication (OPTIONAL) + C->>S: GET /v1/chunks/default-merkledb/{chunk_hash} + S-->>C: 200 dedupe information or 404 not found + end + end + + C->>C: Xorb formation (group chunks ~64 MiB), hashing, serialization + + loop For each new Xorb + C->>S: POST /v1/xorbs/default/{xorb_hash} + S-->>C: 200 OK + end + + C->>C: Shard formation (files -> reconstructions) and serialization + C->>S: POST /v1/shards + S-->>C: 200 OK + + Note over C,S: All referenced Xorbs MUST be uploaded before Shard upload.
Endpoints are idempotent by content-addressed keys. +``` diff --git a/docs/xet/xorb.md b/docs/xet/xorb.md new file mode 100644 index 0000000000..36ef1c04a1 --- /dev/null +++ b/docs/xet/xorb.md @@ -0,0 +1,135 @@ +# Xorb Formation & Serialization Format + +A "Xorb" (Xet Orb, pronounced like "zorb") is a sequence of chunks and a serialization format for a series of chunks. + +## Collecting Chunks + +Using the chunking algorithm a file is mapped to a series of chunks, once those chunks are found, they need to be collected into collections of Xorbs. + +It is advantageous to collect series of chunks in Xorbs such that they can be referred to as a whole range of chunks. + +Suppose a file is chunked into chunks A, B, C, D in the order ABCD. Then create a Xorb X1 with chunks A, B, C, D in this order (starting at chunk index 0), let's say this Xorb's hash is X1. Then to reconstruct the file we ask for Xorb X1 chunk range `[0, 4)`. + +While there's no explicit limit on the number of chunks in a Xorb, there is a limit of 64MiB on the total size of the Xorb as serialized. +Since some chunks will get compressed, it is generally advised to collect chunks until their total uncompressed length is near 64 MiB then serialize the struct. +Namely, Xorbs point to roughly 64 MiB worth of data. +(Recall that the target chunk size is 64 KiB so expect roughly ~1024 chunks per Xorb). + +The CAS server will reject Xorb uploads that exceed the 64 MiB serialized size limit. + +It is RECOMMENDED to pack chunks from multiple files into a Xorb if the size requirements allow, i.e. file X and Y both produced 10 new chunks each totalling a total of ~128000 bytes, then all those chunks can fit in a new Xorb. + +## Xorb Format + +A Xorb is a series of "Chunks" that is serialized according to a specific format that enables accessing chunks of ranges and builds in chunk level compression. + +```txt +┌─────────┬─────────────────────────────────┬─────────┬─────────────────────────────────┬─────────┬─────────────────────────────────┬────────── +│ Chunk │ │ Chunk │ │ Chunk │ │ +│ Header │ Compressed Chunk Data │ Header │ Compressed Chunk Data │ Header │ Compressed Chunk Data │ ... +│ │ │ │ │ │ │ +└─────────┴─────────────────────────────────┴─────────┴─────────────────────────────────┴─────────┴─────────────────────────────────┴─────────── +│ Chunk 0 │ Chunk 1 │ Chunk 2 │ ... +``` + +### Chunk Addressing + +Each chunk has an index within the Xorb it is in, starting at 0. +Chunks can be addressed individually by their index but are usually addressed or fetched in range. +Chunk ranges are always specified start inclusive and end exclusive i.e. `[start, end)`. + +## Chunk Format + +A chunk consists of a header followed by compressed data. The header contains metadata about the chunk, particularly the compression scheme required to know how to deserialize the chunk. + +### Chunk Header Structure + +The chunk header is serialized as follows: + +- **Version** (1 byte): Protocol version, currently `0` +- **Compressed Size** (3 bytes): Size of data after compression as a 3 byte little-endian unsigned integer. +- **Compression Type** (1 byte): Algorithm used for compression (See mapping below) +- **Uncompressed Size** (3 bytes): Size of raw chunk data (before compression) as a 3 byte little-endian unsigned integer. + +Both Compressed and Uncompressed Size can fit in a 3 byte integer, given that that a raw uncompressed chunk can be 128KiB at most, +requiring 18 binary digits to represent. +If utilizing the intended compression scheme results in a larger compressed chunk then the chunk SHOULD be stored uncompressed with then +the uncompressed size also being at a maximum of 128KiB. + +#### Chunk Header Layout + +```txt +┌─────────┬─────────────────────────────────┬──────────────┬─────────────────────────────────┐ +│ Version │ Compressed Size │ Compression │ Uncompressed Size │ +│ 1 byte │ 3 bytes │ Type │ 3 bytes │ +│ │ (little-endian) │ 1 byte │ (little-endian) │ +└─────────┴─────────────────────────────────┴──────────────┴─────────────────────────────────┘ +0 1 4 5 8 +``` + +### Chunk Compression Schemes + +| Value | Name | Description | +|-------|------|-------------| +| `0` | `None` | No compression - data is stored as-is | +| `1` | `LZ4` | Standard LZ4 compression | +| `2` | `ByteGrouping4LZ4` | Byte grouping with 4-byte groups followed by LZ4 compression. Optimized for floating-point and other structured data where grouping bytes by position improves compression ratios | + +#### Byte Grouping LZ4 Compression + +Byte grouping LZ4 compression is an optimization technique that improves compression ratios for structured data like floating-point numbers, integers, and other data types where values have similar byte patterns at specific positions. + +1. **Byte Grouping Phase**: The input data is reorganized by grouping bytes by their position within each 4-byte groups: + Create 4 buffers, for each 4 bytes of the chunk data (B1, B2, B3, B4) append each byte to their respective group i.e. in order from 1 to 4. Then concatenate the groups in order (1, 2, 3, 4). + + Example: + + - Original data: `[A1, A2, A3, A4, B1, B2, B3, B4, C1, C2, C3, C4, ...]` + - Grouped data: `[A1, B1, C1, ..., A2, B2, C2, ..., A3, B3, C3, ..., A4, B4, C4, ...]` + + If the total number of bytes in the chunk is not a multiple of 4, append the remaining bytes following the pattern (1 byte to each group) to the first 1-3 groups until there are no more bytes left in the chunk. + +2. **LZ4 Compression**: The grouped data is then compressed using standard LZ4 compression. + +#### Chunk Data + +Following the header is the compressed data block, exactly `compressed_size` bytes long. + +### Picking a Compression Scheme + +Picking the chunk compression scheme for the Xorb is a task left to the client when uploading the Xorb. +The goal is to minimize the overall size of the Xorb for faster transmission at the cost of resources to decompress a chunk on the receiving end. + +When picking a compression scheme for the chunk there are a number of strategies and implementors MAY make their decisions as to how to pick a compression scheme. +Note that a Xorb MAY contain chunks that utilize different compression schemes. + +1. **Brute Force** + + Try all possible compression schemes, pick the best one. + The best one MAY be the one producing the smallest compressed chunk or the fastest to decompress. + +2. **Best Effort Prediction** + + In `xet-core`, to predict if BG4 will be useful we maximum KL divergence between the distribution of per-byte pop-counts on a sample of each of the 4 groups that would be formed. + You can read more about it in [bg4_prediction.rs](./cas_object/src/byte_grouping/bg4_prediction.rs) and accompanying scripts. + + If the predictor does not show that BG4 will be better, we use Lz4 and in either case we will store the chunk as the uncompressed version if the compression scheme used does not show any benefit. + +#### Example Chunk Serialization + +```python +VERSION = 0 +buffer = bytes() + +for chunk in xorb.chunks: + uncompressed_length = len(chunk) + compressed, compression_scheme = pick_compression_scheme_and_compress(chunk) + header = Header(VERSION, len(compressed), compression_scheme, uncompressed_length) + buffer.write(header) + buffer.write(compressed) +``` + +## Xorb Format Sample + +For a sample of a serialized xorb object see [eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632.xorb). +The hash of this xorb is `eea25d6ee393ccae385820daed127b96ef0ea034dfb7cf6da3a950ce334b7632` and it is composed of chunks from file [Electric_Vehicle_Population_Data_20250917.csv](https://huggingface.co/datasets/xet-team/xet-spec-reference-files/blob/main/Electric_Vehicle_Population_Data_20250917.csv). From 787066b0e676f6695204cf3b51e87a9f3c4d4f1c Mon Sep 17 00:00:00 2001 From: HuggingFaceInfra <148469759+HuggingFaceInfra@users.noreply.github.com> Date: Wed, 1 Oct 2025 09:24:59 +0200 Subject: [PATCH 10/19] Update Inference Providers documentation (automated) (#1964) Co-authored-by: Wauplin <11801849+Wauplin@users.noreply.github.com> --- docs/inference-providers/providers/zai-org.md | 2 +- docs/inference-providers/tasks/chat-completion.md | 2 +- scripts/inference-providers/package.json | 2 +- scripts/inference-providers/pnpm-lock.yaml | 12 ++++++------ 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/inference-providers/providers/zai-org.md b/docs/inference-providers/providers/zai-org.md index 3adfa30947..5d54c5e771 100644 --- a/docs/inference-providers/providers/zai-org.md +++ b/docs/inference-providers/providers/zai-org.md @@ -56,7 +56,7 @@ Find out more about Chat Completion (LLM) [here](../tasks/chat-completion). diff --git a/docs/inference-providers/tasks/chat-completion.md b/docs/inference-providers/tasks/chat-completion.md index 2e6b0584ed..8fba8af8cb 100644 --- a/docs/inference-providers/tasks/chat-completion.md +++ b/docs/inference-providers/tasks/chat-completion.md @@ -64,7 +64,7 @@ The API supports: diff --git a/scripts/inference-providers/package.json b/scripts/inference-providers/package.json index 21212f7b7e..597b457bfe 100644 --- a/scripts/inference-providers/package.json +++ b/scripts/inference-providers/package.json @@ -15,7 +15,7 @@ "license": "ISC", "dependencies": { "@huggingface/inference": "^4.9.0", - "@huggingface/tasks": "^0.19.48", + "@huggingface/tasks": "^0.19.49", "@types/node": "^22.5.0", "handlebars": "^4.7.8", "node": "^20.17.0", diff --git a/scripts/inference-providers/pnpm-lock.yaml b/scripts/inference-providers/pnpm-lock.yaml index 7a14f5712c..f306b97031 100644 --- a/scripts/inference-providers/pnpm-lock.yaml +++ b/scripts/inference-providers/pnpm-lock.yaml @@ -12,8 +12,8 @@ importers: specifier: ^4.9.0 version: 4.9.0 '@huggingface/tasks': - specifier: ^0.19.48 - version: 0.19.48 + specifier: ^0.19.49 + version: 0.19.49 '@types/node': specifier: ^22.5.0 version: 22.5.0 @@ -197,8 +197,8 @@ packages: resolution: {integrity: sha512-yUZLld4lrM9iFxHCwFQ7D1HW2MWMwSbeB7WzWqFYDWK+rEb+WldkLdAJxUPOmgICMHZLzZGVcVjFh3w/YGubng==} engines: {node: '>=18'} - '@huggingface/tasks@0.19.48': - resolution: {integrity: sha512-9OIMhl6lSvuim0EisMPbwO4fUmMlbAXh5/GIMVAQALoByouockaaDxMuzJ9IffHRsl3qjS+QSD/mPe65kuPeyQ==} + '@huggingface/tasks@0.19.49': + resolution: {integrity: sha512-TBBs2klFqo0f9BgLBeG5sNH/oXt3nDxp8wEjwcR1/AepHb8VJAFf9oAW1SS9jmiWRrW8ZjXQJ3THOxsonHVBJA==} '@jridgewell/resolve-uri@3.1.2': resolution: {integrity: sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==} @@ -418,11 +418,11 @@ snapshots: '@huggingface/inference@4.9.0': dependencies: '@huggingface/jinja': 0.5.1 - '@huggingface/tasks': 0.19.48 + '@huggingface/tasks': 0.19.49 '@huggingface/jinja@0.5.1': {} - '@huggingface/tasks@0.19.48': {} + '@huggingface/tasks@0.19.49': {} '@jridgewell/resolve-uri@3.1.2': {} From ff8b5fab87fc64065602a3a03e6e19e6efcd270b Mon Sep 17 00:00:00 2001 From: HuggingFaceInfra <148469759+HuggingFaceInfra@users.noreply.github.com> Date: Thu, 2 Oct 2025 10:16:09 +0200 Subject: [PATCH 11/19] Update Inference Providers documentation (automated) (#1967) Co-authored-by: Wauplin <11801849+Wauplin@users.noreply.github.com> --- docs/inference-providers/tasks/image-classification.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/inference-providers/tasks/image-classification.md b/docs/inference-providers/tasks/image-classification.md index 8fa784a7d6..2f2b72e30d 100644 --- a/docs/inference-providers/tasks/image-classification.md +++ b/docs/inference-providers/tasks/image-classification.md @@ -24,6 +24,7 @@ Image classification is the task of assigning a label or class to an entire imag - [google/vit-base-patch16-224](https://huggingface.co/google/vit-base-patch16-224): A strong image classification model. - [facebook/deit-base-distilled-patch16-224](https://huggingface.co/facebook/deit-base-distilled-patch16-224): A robust image classification model. +- [facebook/convnext-large-224](https://huggingface.co/facebook/convnext-large-224): A strong image classification model. Explore all available models and find the one that suits you best [here](https://huggingface.co/models?inference=warm&pipeline_tag=image-classification&sort=trending). From 849b2a3f59fa237f8b11b3204e0c5a3a3d475d1b Mon Sep 17 00:00:00 2001 From: Rajat Arya Date: Thu, 25 Sep 2025 17:13:54 -0700 Subject: [PATCH 12/19] Reorganized Xet docs, hopefully this renders ok --- docs/hub/index.md | 6 +- docs/hub/repositories.md | 2 +- docs/hub/storage-backends.md | 116 ++------------- docs/hub/xet/legacy-git-lfs.md | 17 +++ docs/hub/xet/overview.md | 25 ++++ docs/hub/xet/protocol/deduplication.md | 189 +++++++++++++++++++++++++ docs/hub/xet/security.md | 3 + docs/hub/xet/using-xet-storage.md | 49 +++++++ 8 files changed, 297 insertions(+), 110 deletions(-) create mode 100644 docs/hub/xet/legacy-git-lfs.md create mode 100644 docs/hub/xet/overview.md create mode 100644 docs/hub/xet/protocol/deduplication.md create mode 100644 docs/hub/xet/security.md create mode 100644 docs/hub/xet/using-xet-storage.md diff --git a/docs/hub/index.md b/docs/hub/index.md index ebed7f8d57..14e5a3d46c 100644 --- a/docs/hub/index.md +++ b/docs/hub/index.md @@ -27,7 +27,7 @@ The Hugging Face Hub is a platform with over 1.7M models, 400k datasets, and 600 Getting Started Repository Settings Storage Limits -Storage Backends +Storage Backend - Xet Pull requests and Discussions Notifications Collections @@ -105,7 +105,7 @@ The Hugging Face Hub is a platform with over 1.7M models, 400k datasets, and 600 We are helping the community work together towards the goal of advancing Machine Learning 🔥. -The Hugging Face Hub is a platform with over 1.7M models, 400k datasets, and 600k demos in which people can easily collaborate in their ML workflows. The Hub works as a central place where anyone can share, explore, discover, and experiment with open-source Machine Learning. +The Hugging Face Hub is a platform with over 2M models, 500k datasets, and 600k demos in which people can easily collaborate in their ML workflows. The Hub works as a central place where anyone can share, explore, discover, and experiment with open-source Machine Learning. No single company, including the Tech Titans, will be able to “solve AI” by themselves – the only way we'll achieve this is by sharing knowledge and resources in a community-centric approach. We are building the largest open-source collection of models, datasets, and demos on the Hugging Face Hub to democratize and advance ML for everyone 🚀. @@ -122,7 +122,7 @@ On it, you'll be able to upload and discover... - Spaces: _interactive apps for demonstrating ML models directly in your browser_ The Hub offers **versioning, commit history, diffs, branches, and over a dozen library integrations**! -All repositories build on [Xet](https://huggingface.co/join/xet), a new technology to efficiently store Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. +All repositories build on [Xet](./storage-backends), a new technology to efficiently store Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. You can learn more about the features that all repositories share in the [**Repositories documentation**](./repositories). diff --git a/docs/hub/repositories.md b/docs/hub/repositories.md index 8a9e7dbfdf..33e01e1ca9 100644 --- a/docs/hub/repositories.md +++ b/docs/hub/repositories.md @@ -17,7 +17,7 @@ In these pages, you will go over the basics of getting started with Git and Xet - [Getting Started with Repositories](./repositories-getting-started) - [Settings](./repositories-settings) - [Storage Limits](./storage-limits) -- [Storage Backends](./storage-backends) +- [Storage Backend - Xet](./storage-backends) - [Pull Requests & Discussions](./repositories-pull-requests-discussions) - [Pull Requests advanced usage](./repositories-pull-requests-discussions#pull-requests-advanced-usage) - [Collections](./collections) diff --git a/docs/hub/storage-backends.md b/docs/hub/storage-backends.md index cf09be009f..13123694fb 100644 --- a/docs/hub/storage-backends.md +++ b/docs/hub/storage-backends.md @@ -1,4 +1,4 @@ -# Storage +# Xet Storage Repositories on the Hugging Face Hub are different from those on software development platforms. They contain files that are: @@ -11,112 +11,16 @@ Storing these files directly in a pure Git repository is impractical. Not only a Instead, on the Hub, these large files are tracked using "pointer files" and identified through a `.gitattributes` file (both discussed in more detail below), which remain in the Git repository while the actual data is stored in remote storage (like [Amazon S3](https://aws.amazon.com/s3/)). As a result, the repository stays small and typical Git workflows remain efficient. -Historically, Hub repositories have relied on [Git LFS](https://git-lfs.com/) for this mechanism. While Git LFS remains supported (see the [Legacy section below](#legacy-storage-git-lfs)), the Hub has adopted Xet, a modern custom storage system built specifically for AI/ML development. It enables chunk-level deduplication, smaller uploads, and faster downloads than Git LFS. +Historically, Hub repositories have relied on [Git LFS](https://git-lfs.com/) for this mechanism. While Git LFS remains supported (see [Legacy Storage using Git LFS](#legacy-storage-git-lfs)), the Hub has adopted Xet, a modern custom storage system built specifically for AI/ML development. It enables chunk-level deduplication, smaller uploads, and faster downloads than Git LFS. -## Xet +In these pages you will get started in using Xet Storage, including the open-source Xet protocol. -[In August 2024 Hugging Face acquired XetHub](https://huggingface.co/blog/xethub-joins-hf), a [seed-stage startup based in Seattle](https://www.geekwire.com/2023/ex-apple-engineers-raise-7-5m-for-new-seattle-data-storage-startup/), to replace Git LFS on the Hub. +## Contents -Like Git LFS, a Xet-backed repository utilizes S3 as the remote storage with a `.gitattributes` file at the repository root helping identify what files should be stored remotely. +- [Xet History & Overview](./xet/overview) +- [Using Xet Storage](./xet/using-xet-storage) +- [Deduplication](./xet/deduplication) +- [Open Source Xet Protocol](./xet/protocol/index) +- [Backwards Compatibility & Legacy](./xet/legacy-git-lfs) +- [Security](./xet/security) -
- - -
- -A Git LFS pointer file provides metadata to locate the actual file contents in remote storage: - -- **SHA256**: Provides a unique identifier for the actual large file. This identifier is generated by computing the SHA-256 hash of the file’s contents. -- **Pointer size**: The size of the pointer file stored in the Git repository. -- **Size of the remote file**: Indicates the size of the actual large file in bytes. This metadata is useful for both verification purposes and for managing storage and transfer operations. - -A Xet pointer includes all of this information by design. Refer to the section on [backwards compatibility with Git LFS](#backward-compatibility-with-lfs) with the addition of a `Xet backed hash` field for referencing the file in Xet storage. - -
- - -
- -Unlike Git LFS, which deduplicates at the file level, Xet-enabled repositories deduplicate at the level of bytes. When a file backed by Xet storage is updated, only the modified data is uploaded to remote storage, significantly saving on network transfers. For many workflows, like incremental updates to model checkpoints or appending/inserting new data into a dataset, this improves iteration speed for yourself and your collaborators. To learn more about deduplication in Xet storage, refer to the [Deduplication](#deduplication) section below. - -### Using Xet Storage - -To start using Xet Storage, you need a Xet-enabled repository and a Xet-aware version of the [huggingface_hub](https://huggingface.co/docs/huggingface_hub) Python library. As of May 23rd, 2025, Xet-enabled repositories are the default [for all new users and organizations on the Hub](https://huggingface.co/changelog/xet-default-for-new-users). - -> [!TIP] -> For user and organization profiles created before May 23rd, 2025, you can make Xet the default for all your repositories by [signing up here](https://huggingface.co/join/xet). You can apply for yourself or your entire organization (requires [admin permissions](https://huggingface.co/docs/hub/organizations-security)). Once approved, all existing repositories will be automatically migrated to Xet and future repositories will be Xet-enabled by default. -> -> PRO users and Team or Enterprise organizations will be fast-tracked for access. - -To access a Xet-aware version of the `huggingface_hub`, simply install the latest version: - -```bash -pip install -U huggingface_hub -``` - -As of `huggingface_hub` 0.32.0, this will also install `hf_xet`. The `hf_xet` package integrates `huggingface_hub` with [`xet-core`](https://github.com/huggingface/xet-core), the Rust client for the Xet backend. - -If you use the `transformers` or `datasets` libraries, it's already using `huggingface_hub`. So long as the version of `huggingface_hub` >= 0.32.0, no further action needs to be taken. - -Where versions of `huggingface_hub` >= 0.30.0 and < 0.32.0 are installed, `hf_xet` must be installed explicitly: - -```bash -pip install -U hf-xet -``` - -And that's it! You now get the benefits of Xet deduplication for both uploads and downloads. Team members using a version of `huggingface_hub` < 0.30.0 will still be able to upload and download repositories through the [backwards compatibility provided by the LFS bridge](#backward-compatibility-with-lfs). - -To see more detailed usage docs, refer to the `huggingface_hub` docs for: - -- [Upload](https://huggingface.co/docs/huggingface_hub/guides/upload#faster-uploads) -- [Download](https://huggingface.co/docs/huggingface_hub/guides/download#hfxet) -- [Managing the `hf_xet` cache](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#chunk-based-caching-xet) - -#### Recommendations - -Xet integrates seamlessly with the Hub's current Python-based workflows. However, there are a few steps you may consider to get the most benefits from Xet storage: - -- **Use `hf_xet`**: While Xet remains backward compatible with legacy clients optimized for Git LFS, the `hf_xet` integration with `huggingface_hub` delivers optimal chunk-based performance and faster iteration on large files. -- **Utilize `hf_xet` environment variables**: The default installation of `hf_xet` is designed to support the broadest range of hardware. To take advantage of setups with more network bandwidth or processing power read up on `hf_xet`'s [environment variables](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#xet) to further speed up downloads and uploads. -- **Leverage frequent, incremental commits**: Xet's chunk-level deduplication means you can safely make incremental updates to models or datasets. Only changed chunks are uploaded, so frequent commits are both fast and storage-efficient. -- **Be Specific in .gitattributes**: When defining patterns for Xet or LFS, use precise file extensions (e.g., `*.safetensors`, `*.bin`) to avoid unnecessarily routing smaller files through large-file storage. -- **Prioritize community access**: Xet substantially increases the efficiency and scale of large file transfers. Instead of structuring your repository to reduce its total size (or the size of individual files), organize it for collaborators and community users so they may easily navigate and retrieve the content they need. - -#### Current Limitations - -While Xet brings fine-grained deduplication and enhanced performance to Git-based storage, some features and platform compatibilities are still in development. As a result, keep the following constraints in mind when working with a Xet-enabled repository: - -- **64-bit systems only**: The `hf_xet` client currently requires a 64-bit architecture; 32-bit systems are not supported. -- **Git client integration (git-xet)**: Planned but remains under development. - -### Deduplication - -Xet-enabled repositories utilize [content-defined chunking (CDC)](https://huggingface.co/blog/from-files-to-chunks) to deduplicate on the level of bytes (~64KB of data, also referred to as a "chunk"). Each chunk is identified by a rolling hash that determines chunk boundaries based on the actual file contents, making it resilient to insertions or deletions anywhere in the file. When a file is uploaded to a Xet-backed repository using a Xet-aware client, its contents are broken down into these variable-sized chunks. Only new chunks not already present in Xet storage are kept after chunking, everything else is discarded. - -To avoid the overhead of communicating and managing at the level of chunks, new chunks are grouped together in [64MB blocks](https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation) and uploaded. Each block is stored once in a content-addressed store (CAS), keyed by its hash. - -The Hub's [current recommendation](https://huggingface.co/docs/hub/storage-limits#recommendations) is to limit files to 20GB. At a 64KB chunk size, a 20GB file has 312,500 chunks, many of which go unchanged from version to version. Git LFS is designed to notice only that a file has changed and store the entirety of that revision. By deduplicating at the level of chunks, the Xet backend enables storing only the modified content in a file (which might only be a few KB or MB) and securely deduplicates shared blocks across repositories. For the large binary files found in Model and Dataset repositories, this provides significant improvements to file transfer times. - -For more details, refer to the [From Files to Chunks](https://huggingface.co/blog/from-files-to-chunks) and [From Chunks to Blocks](https://huggingface.co/blog/from-chunks-to-blocks) blog posts, or the [Git is for Data](https://www.cidrdb.org/cidr2023/papers/p43-low.pdf) paper by Low et al. that served as the launch point for XetHub prior to being acquired by Hugging Face. - -### Backward Compatibility with LFS - -Xet storage provides a seamless transition for existing Hub repositories. It isn't necessary to know if the Xet backend is involved at all. Xet-backed repositories continue to use the Git LFS pointer file format; the addition of the `Xet backed hash` is only added to the web interface as a convenience. Practically, this means existing repos and newly created repos will not look any different if you do a `bare clone` of them. Each of the large files (or binary files) will continue to have a pointer file that matches the Git LFS pointer file specification. - -This symmetry allows non-Xet-aware clients (e.g., older versions of the `huggingface_hub`) to interact with Xet-backed repositories without concern. In fact, within a repository a mixture of Git LFS and Xet backed files are supported. The Xet backend indicates whether a file is in Git LFS or Xet storage, allowing downstream services to request the proper URL(s) from S3, regardless of which storage system holds the content. - -Within the Xet architecture, backward compatibility for downloads is achieved by the Git LFS bridge. While a Xet-aware client will receive file reconstruction information from CAS to download the Xet-backed file, a legacy client will get a single URL from the bridge which does the work of reconstructing the request file and returning the URL to the resource. This allows downloading files through a URL so that you can continue to use the Hub's web interface or `curl`. - -Meanwhile, uploads from non‑Xet‑aware clients still follow the standard Git LFS path, even if the file is already Xet-backed. Once the file is uploaded to LFS, a background process automatically migrates the content, turning it into a Xet-backed revision. Coupled with the Git LFS bridge, this lets repository maintainers and the rest of the Hub adopt Xet at their own pace without disruption. - -### Security Model - -Xet storage provides data deduplication over all chunks stored in Hugging Face. This is done via cryptographic hashing in a privacy sensitive way. The contents of chunks are protected and are associated with repository permissions. i.e. you can only read chunks which are required to reproduce files you have access to, and no more. See [xet-core](https://github.com/huggingface/xet-core) for details. - -## Legacy Storage: Git LFS - -The legacy storage system on the Hub, Git LFS utilizes many of the same conventions as Xet-backed repositories. The Hub's Git LFS backend is [Amazon Simple Storage Service (S3)](https://aws.amazon.com/s3/). When Git LFS is invoked, it stores the file contents in S3 using the SHA hash to name the file for future access. This storage architecture is relatively simple and has allowed Hub to store millions of models, datasets, and spaces repositories' files (45PB total as of this writing). - -The primary limitation of Git LFS is its file-centric approach to deduplication. Any change to a file, irrespective of how large of small that change is, means the entire file is versioned - incurring significant overheads in file transfers as the entire file is uploaded (if committing to a repository) or downloaded (if pulling the latest version to your machine). - -This leads to a worse developer experience along with a proliferation of additional storage. diff --git a/docs/hub/xet/legacy-git-lfs.md b/docs/hub/xet/legacy-git-lfs.md new file mode 100644 index 0000000000..d9a2a761e2 --- /dev/null +++ b/docs/hub/xet/legacy-git-lfs.md @@ -0,0 +1,17 @@ +## Backward Compatibility with LFS + +Xet storage provides a seamless transition for existing Hub repositories. It isn't necessary to know if the Xet backend is involved at all. Xet-backed repositories continue to use the Git LFS pointer file format; the addition of the `Xet backed hash` is only added to the web interface as a convenience. Practically, this means existing repos and newly created repos will not look any different if you do a `bare clone` of them. Each of the large files (or binary files) will continue to have a pointer file that matches the Git LFS pointer file specification. + +This symmetry allows non-Xet-aware clients (e.g., older versions of the `huggingface_hub`) to interact with Xet-backed repositories without concern. In fact, within a repository a mixture of Git LFS and Xet backed files are supported. The Xet backend indicates whether a file is in Git LFS or Xet storage, allowing downstream services to request the proper URL(s) from S3, regardless of which storage system holds the content. + +Within the Xet architecture, backward compatibility for downloads is achieved by the Git LFS bridge. While a Xet-aware client will receive file reconstruction information from CAS to download the Xet-backed file, a legacy client will get a single URL from the bridge which does the work of reconstructing the request file and returning the URL to the resource. This allows downloading files through a URL so that you can continue to use the Hub's web interface or `curl`. + +Meanwhile, uploads from non‑Xet‑aware clients still follow the standard Git LFS path, even if the file is already Xet-backed. Once the file is uploaded to LFS, a background process automatically migrates the content, turning it into a Xet-backed revision. Coupled with the Git LFS bridge, this lets repository maintainers and the rest of the Hub adopt Xet at their own pace without disruption. + +## Legacy Storage: Git LFS + +The legacy storage system on the Hub, Git LFS utilizes many of the same conventions as Xet-backed repositories. The Hub's Git LFS backend is [Amazon Simple Storage Service (S3)](https://aws.amazon.com/s3/). When Git LFS is invoked, it stores the file contents in S3 using the SHA hash to name the file for future access. This storage architecture is relatively simple and has allowed Hub to store millions of models, datasets, and spaces repositories' files (45PB total as of this writing). + +The primary limitation of Git LFS is its file-centric approach to deduplication. Any change to a file, irrespective of how large of small that change is, means the entire file is versioned - incurring significant overheads in file transfers as the entire file is uploaded (if committing to a repository) or downloaded (if pulling the latest version to your machine). + +This leads to a worse developer experience along with a proliferation of additional storage. diff --git a/docs/hub/xet/overview.md b/docs/hub/xet/overview.md new file mode 100644 index 0000000000..a390bbc527 --- /dev/null +++ b/docs/hub/xet/overview.md @@ -0,0 +1,25 @@ +## Xet History & Overview + +[In August 2024 Hugging Face acquired XetHub](https://huggingface.co/blog/xethub-joins-hf), a [seed-stage startup based in Seattle](https://www.geekwire.com/2023/ex-apple-engineers-raise-7-5m-for-new-seattle-data-storage-startup/), to replace Git LFS on the Hub. + +Like Git LFS, a Xet-backed repository utilizes S3 as the remote storage with a `.gitattributes` file at the repository root helping identify what files should be stored remotely. + +
+ + +
+ +A Git LFS pointer file provides metadata to locate the actual file contents in remote storage: + +- **SHA256**: Provides a unique identifier for the actual large file. This identifier is generated by computing the SHA-256 hash of the file’s contents. +- **Pointer size**: The size of the pointer file stored in the Git repository. +- **Size of the remote file**: Indicates the size of the actual large file in bytes. This metadata is useful for both verification purposes and for managing storage and transfer operations. + +A Xet pointer includes all of this information by design. Refer to the section on [backwards compatibility with Git LFS](#backward-compatibility-with-lfs) with the addition of a `Xet backed hash` field for referencing the file in Xet storage. + +
+ + +
+ +Unlike Git LFS, which deduplicates at the file level, Xet-enabled repositories deduplicate at the level of bytes. When a file backed by Xet storage is updated, only the modified data is uploaded to remote storage, significantly saving on network transfers. For many workflows, like incremental updates to model checkpoints or appending/inserting new data into a dataset, this improves iteration speed for yourself and your collaborators. To learn more about deduplication in Xet storage, refer to the [Deduplication](#deduplication) section below. diff --git a/docs/hub/xet/protocol/deduplication.md b/docs/hub/xet/protocol/deduplication.md new file mode 100644 index 0000000000..abb882f9e7 --- /dev/null +++ b/docs/hub/xet/protocol/deduplication.md @@ -0,0 +1,189 @@ +# Xet Chunk-Level Deduplication Specification + +## Overview + +Chunk-level deduplication is a fundamental optimization technique in the Xet system that eliminates redundant data by identifying and sharing identical content blocks (chunks) across files and repositories. +This specification details the procedures, algorithms, and security mechanisms that enable efficient storage and transfer while maintaining data integrity and access control. + +Deduplication in Xet operates at the chunk level rather than the file level, providing fine-grained deduplication capabilities that can identify shared content even when files differ significantly. +This approach is particularly effective for scenarios common in machine learning and data science workflows, such as: + +- Multiple versions of datasets with incremental changes +- Model checkpoints that share common layers or parameters +- Documentation and configuration files with similar content +- Large files where only portions have changed between versions + +## Core Concepts + +### Chunks + +A **chunk** is a variable-sized content block derived from files using Content-Defined Chunking (CDC) with a rolling hash function. Chunks are the fundamental unit of deduplication in Xet. + +- **Target size**: 64KB (configurable) +- **Size range**: 8KB to 128KB (minimum and maximum constraints) +- **Identification**: Each chunk is uniquely identified by its cryptographic hash (MerkleHash) + +[Detailed chunking description](./chunking) + +### Xorbs + +**Xorbs** are objects that aggregate multiple chunks for efficient storage and transfer: + +- **Maximum size**: 64MB +- **Maximum chunks**: 8,192 chunks per xorb +- **Purpose**: Batch multiple chunks together to reduce metadata and network overhead when uploading and downloading groups of chunks + +### Shards (Xorb Lists) + +**Shards** are objects that contain a list of xorbs that can be deduped against (for the context of deduplication, ignore the file info section of the shard format). + +- **Maximum size**: 64MB +- **Purpose**: Provide a format on a positive reply to a global deduplication request with information about xorbs that already exist in the CAS system. + +### CAS (Content Addressable Storage) + +The **CAS** system provides the underlying storage infrastructure: + +- **Content addressing**: All objects are stored and retrieved by their cryptographic hash +- **Immutability**: Once stored, file content cannot be modified +- **Deduplication**: Identical content is automatically deduplicated at the storage level + +## Deduplication Procedure + +### 1. File Processing and Chunking + +When a file is processed for upload, it undergoes the following steps: + +```mermaid +graph TD + A[File Input] --> B[Content-Defined Chunking] + B --> C[Hash Computation] + C --> D[Chunk Creation] + D --> E[Deduplication Query] +``` + +1. **Chunking**: Content-defined chunking using GearHash algorithm creates variable-sized chunks of file data +2. **Hash Computation**: Each chunk's content is hashed using a cryptographic hash function (Blake3-based MerkleHash) +3. **Chunk Object Creation**: Chunks are wrapped with metadata including hash, size, and data + +### 2. Multi-Level Deduplication Strategy + +Xet employs a three-tiered deduplication strategy to maximize efficiency while minimizing latency: + +#### Level 1: Local Session Deduplication + +**Scope**: Current upload session +**Mechanism**: In-memory hash lookup table +**Purpose**: Eliminate redundancy within the current file or session + +**Benefits**: + +- Fastest lookup (in-memory) +- Zero network overhead +- Immediate deduplication feedback + +#### Level 2: Cached Metadata Deduplication + +**Scope**: Previously uploaded files and sessions +**Mechanism**: Local shard file metadata cache +**Purpose**: Leverage deduplication against recently uploaded content + +**Benefits**: + +- Fast local disk access +- No network latency +- Persistent across sessions + +#### Level 3: Global Deduplication API + +**Scope**: Entire Xet ecosystem +**Mechanism**: Global deduplication service with HMAC protection +**Purpose**: Discover deduplication opportunities across all users and repositories + +### 3. Global Deduplication Process + +The global deduplication system provides deduplication capabilities across all data that is managed by the Xet system: + +#### Eligibility Criteria + +Not all chunks are eligible for global deduplication queries to manage system load: + +1. **First chunk**: The first chunk of every file is always eligible. +2. **Hash pattern matching**: Chunks are eligible if: the last 8 bytes of the hash interpreted as a little-endian 64 bit integer % 1024 == 0. + +**Recommendations:** +**Spacing constraints**: The global dedupe API is optimized to return information about nearby chunks when there is a match. Consider only issueing a request to an eligible chunk every ~4MB of data. + +#### Query Process + +1. **Background Query**: Global deduplication queries SHOULD run asynchronously to avoid blocking upload +2. **HMAC Protection**: Chunk hashes are protected using HMAC keys +3. **Shard Response**: When a match is found, the API returns a shard containing: + - **CAS Info Section**: Contains metadata about many xorbs that store chunks + - **HMAC Key**: Included in the shard metadata header used to encrypt chunk hashes +4. **Encrypted Chunk Matching**: All chunk hashes in the returned shard have been encrypted with the HMAC key +5. **Match Discovery Process**: To find matches, clients MUST: + - Encrypt their chunk hash using the provided HMAC key + - Search for the encrypted hash within the shard's chunk listings + - For subsequent chunks, repeat the encryption and search process + - Track the original (non-encrypted) chunk hash while noting which xorb contains that chunk +6. **Metadata Caching**: Client downloads and caches shard metadata for future deduplication + +#### HMAC Security Mechanism + +Global deduplication uses HMAC (Hash-based Message Authentication Code) to protect chunk hashes while enabling deduplication. + +**Security Properties**: + +Raw chunk hashes are never transmitted from servers to clients; a client has to encrypt their raw chunk hash and find a match to know a raw chunk hash exists in the system. +They MAY know this chunk hash because they own this data, the match has made them privy to know which xorb has this chunk hash and the position in the xorb, but has not revealed any other raw chunk hashes in that xorb or other xorbs. + +## Technical Implementation Details + +### Chunk Hash Computation + +Each chunk has its content hashed using a cryptographic hash function (Blake3-based MerkleHash) to create a unique identifier for content addressing. +[See section about hashing](./hashing#chunk-hashes). + +### Xorb Formation + +When new chunks need to be stored, they are aggregated into xorbs based on size and count limits. If adding a new chunk would exceed the maximum xorb size or chunk count, the current xorb is finalized and uploaded. [See section about xorb formation](./xorb) + +### File Reconstruction Information + +When chunks are deduplicated, the system creates file reconstruction information that includes: + +- Hash of the xorb containing the chunks +- Flags for the CAS block +- Total bytes in the segment +- Start and end indices within the xorb (start inclusive, end exclusive) + +This information allows the system to reconstruct files by: + +1. Identifying which xorbs contain the needed chunks +2. Extracting the specific chunk ranges from each xorb +3. Concatenating chunks in the correct order + +[See section about file reconstruction](./file-reconstruction). + +## Fragmentation Prevention + +While deduplication is valuable for saving space, doing it too aggressively can cause file fragmentation—meaning a file’s chunks end up scattered across many different xorbs. This can make reading files slower and less efficient. +To avoid this, in xet-core we aim (and encourage implementors) to keep long, continuous runs of chunks together in the same xorb whenever possible. Implementations SHOULD keep long, continuous runs together when feasible. +Instead of always deduplicating every possible chunk, the system sometimes chooses to reference a straight run of chunks in a single xorb, even if it means skipping deduplication for a few chunks. +This approach balances the benefits of deduplication with the need to keep files easy and fast to read. +Consider for example referencing a deduplicated chunks in a minimum run of chunks (e.g. at least 8 chunks) or targeting an average contiguous run of chunks totalling length >= 1MB. + +## Conclusion + +Xet's chunk-level deduplication system provides a comprehensive solution for efficient data storage and transfer in large-scale data workflows. +By combining local, cached, and global deduplication strategies with robust security mechanisms and fragmentation prevention, +the system achieves significant storage savings while maintaining performance and data integrity. + +The multi-tiered approach ensures that deduplication is both effective and efficient: + +- Local deduplication provides immediate benefits within sessions +- Cached deduplication leverages recent upload history +- Global deduplication enables cross-repository optimization while preserving security + +The system's design prioritizes both efficiency and safety, with comprehensive error handling, performance monitoring, and security measures that make it suitable for production use in data-intensive applications. diff --git a/docs/hub/xet/security.md b/docs/hub/xet/security.md new file mode 100644 index 0000000000..ce8f0c5c54 --- /dev/null +++ b/docs/hub/xet/security.md @@ -0,0 +1,3 @@ +## Security Model + +Xet storage provides data deduplication over all chunks stored in Hugging Face. This is done via cryptographic hashing in a privacy sensitive way. The contents of chunks are protected and are associated with repository permissions. i.e. you can only read chunks which are required to reproduce files you have access to, and no more. See [xet-core](https://github.com/huggingface/xet-core) for details. diff --git a/docs/hub/xet/using-xet-storage.md b/docs/hub/xet/using-xet-storage.md new file mode 100644 index 0000000000..2e4cb6bcd6 --- /dev/null +++ b/docs/hub/xet/using-xet-storage.md @@ -0,0 +1,49 @@ +## Using Xet Storage + +To start using Xet Storage, you need a Xet-enabled repository and a Xet-aware version of the [huggingface_hub](https://huggingface.co/docs/huggingface_hub) Python library. As of May 23rd, 2025, Xet-enabled repositories are the default [for all new users and organizations on the Hub](https://huggingface.co/changelog/xet-default-for-new-users). + +> [!TIP] +> For user and organization profiles created before May 23rd, 2025, you can make Xet the default for all your repositories by [signing up here](https://huggingface.co/join/xet). You can apply for yourself or your entire organization (requires [admin permissions](https://huggingface.co/docs/hub/organizations-security)). Once approved, all existing repositories will be automatically migrated to Xet and future repositories will be Xet-enabled by default. +> +> PRO users and Team or Enterprise organizations will be fast-tracked for access. + +To access a Xet-aware version of the `huggingface_hub`, simply install the latest version: + +```bash +pip install -U huggingface_hub +``` + +As of `huggingface_hub` 0.32.0, this will also install `hf_xet`. The `hf_xet` package integrates `huggingface_hub` with [`xet-core`](https://github.com/huggingface/xet-core), the Rust client for the Xet backend. + +If you use the `transformers` or `datasets` libraries, it's already using `huggingface_hub`. So long as the version of `huggingface_hub` >= 0.32.0, no further action needs to be taken. + +Where versions of `huggingface_hub` >= 0.30.0 and < 0.32.0 are installed, `hf_xet` must be installed explicitly: + +```bash +pip install -U hf-xet +``` + +And that's it! You now get the benefits of Xet deduplication for both uploads and downloads. Team members using a version of `huggingface_hub` < 0.30.0 will still be able to upload and download repositories through the [backwards compatibility provided by the LFS bridge](#backward-compatibility-with-lfs). + +To see more detailed usage docs, refer to the `huggingface_hub` docs for: + +- [Upload](https://huggingface.co/docs/huggingface_hub/guides/upload#faster-uploads) +- [Download](https://huggingface.co/docs/huggingface_hub/guides/download#hfxet) +- [Managing the `hf_xet` cache](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#chunk-based-caching-xet) + +### Recommendations + +Xet integrates seamlessly with the Hub's current Python-based workflows. However, there are a few steps you may consider to get the most benefits from Xet storage: + +- **Use `hf_xet`**: While Xet remains backward compatible with legacy clients optimized for Git LFS, the `hf_xet` integration with `huggingface_hub` delivers optimal chunk-based performance and faster iteration on large files. +- **Utilize `hf_xet` environment variables**: The default installation of `hf_xet` is designed to support the broadest range of hardware. To take advantage of setups with more network bandwidth or processing power read up on `hf_xet`'s [environment variables](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#xet) to further speed up downloads and uploads. +- **Leverage frequent, incremental commits**: Xet's chunk-level deduplication means you can safely make incremental updates to models or datasets. Only changed chunks are uploaded, so frequent commits are both fast and storage-efficient. +- **Be Specific in .gitattributes**: When defining patterns for Xet or LFS, use precise file extensions (e.g., `*.safetensors`, `*.bin`) to avoid unnecessarily routing smaller files through large-file storage. +- **Prioritize community access**: Xet substantially increases the efficiency and scale of large file transfers. Instead of structuring your repository to reduce its total size (or the size of individual files), organize it for collaborators and community users so they may easily navigate and retrieve the content they need. + +### Current Limitations + +While Xet brings fine-grained deduplication and enhanced performance to Git-based storage, some features and platform compatibilities are still in development. As a result, keep the following constraints in mind when working with a Xet-enabled repository: + +- **64-bit systems only**: The `hf_xet` client currently requires a 64-bit architecture; 32-bit systems are not supported. +- **Git client integration (git-xet)**: Under active development - coming soon, stay tuned! From bc59deedaa646623f7be6c4a64fdb9199b0e0973 Mon Sep 17 00:00:00 2001 From: Rajat Arya Date: Thu, 25 Sep 2025 17:49:45 -0700 Subject: [PATCH 13/19] Add xet protocol subsection pages, finish toctree --- docs/hub/xet/protocol/architecture.md | 15 +++++++++++++++ docs/hub/xet/protocol/building-client.md | 12 ++++++++++++ 2 files changed, 27 insertions(+) create mode 100644 docs/hub/xet/protocol/architecture.md create mode 100644 docs/hub/xet/protocol/building-client.md diff --git a/docs/hub/xet/protocol/architecture.md b/docs/hub/xet/protocol/architecture.md new file mode 100644 index 0000000000..f1e75dc8e4 --- /dev/null +++ b/docs/hub/xet/protocol/architecture.md @@ -0,0 +1,15 @@ +## Overall Xet Architecture + +(TODO: Add diagram here) + +These pages will describe the overall Xet architecture, including how to perform chunking & hashing of files for upload, how to process file reconstruction information when downloading, the specfics of the Xorb and Shard formats, and how deduplication is achieved by a client library. + +## Contents + +- [Content-Defined Chunking](./xet/protocol/chunking) +- [Hashing Methods](./xet/protocol/hashing) +- [File Reconstruction](./xet/protocol/file-reconstruction) +- [Xorb Format](./xet/protocol/xorb) +- [Shard Format](./xet/protocol/shard) +- [Deduplication](./xet/protocol/deduplication) + diff --git a/docs/hub/xet/protocol/building-client.md b/docs/hub/xet/protocol/building-client.md new file mode 100644 index 0000000000..e161cc30e0 --- /dev/null +++ b/docs/hub/xet/protocol/building-client.md @@ -0,0 +1,12 @@ +## Building a client library for Xet storage + +These pages explain how you can build a client library for Xet storage. Included in these pages are the steps involved in uploading & downloading files, the web service APIs involved, and how authentication and authorization works. + +## Contents + +- [Upload Protocol](./xet/protocol/upload-protocol) +- [Download Protocol](./xet/protocol/download-protocol) +- [CAS API](./xet/protocol/api) +- [Authentication and Authorization](./xet/protocol/auth) +- [Hub file conversion to Xet File ID](./xet/protocol/file-id) + From 0a4ca4c811c8465ab306b312defe911e749a6a34 Mon Sep 17 00:00:00 2001 From: Rajat Arya Date: Thu, 25 Sep 2025 18:33:39 -0700 Subject: [PATCH 14/19] Fix links in xet proto contents pages --- docs/hub/xet/protocol/architecture.md | 12 ++++++------ docs/hub/xet/protocol/building-client.md | 10 +++++----- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/hub/xet/protocol/architecture.md b/docs/hub/xet/protocol/architecture.md index f1e75dc8e4..a306d3f089 100644 --- a/docs/hub/xet/protocol/architecture.md +++ b/docs/hub/xet/protocol/architecture.md @@ -6,10 +6,10 @@ These pages will describe the overall Xet architecture, including how to perform ## Contents -- [Content-Defined Chunking](./xet/protocol/chunking) -- [Hashing Methods](./xet/protocol/hashing) -- [File Reconstruction](./xet/protocol/file-reconstruction) -- [Xorb Format](./xet/protocol/xorb) -- [Shard Format](./xet/protocol/shard) -- [Deduplication](./xet/protocol/deduplication) +- [Content-Defined Chunking](./chunking) +- [Hashing Methods](./hashing) +- [File Reconstruction](./file-reconstruction) +- [Xorb Format](./xorb) +- [Shard Format](./shard) +- [Deduplication](./deduplication) diff --git a/docs/hub/xet/protocol/building-client.md b/docs/hub/xet/protocol/building-client.md index e161cc30e0..fb4ddc1820 100644 --- a/docs/hub/xet/protocol/building-client.md +++ b/docs/hub/xet/protocol/building-client.md @@ -4,9 +4,9 @@ These pages explain how you can build a client library for Xet storage. Included ## Contents -- [Upload Protocol](./xet/protocol/upload-protocol) -- [Download Protocol](./xet/protocol/download-protocol) -- [CAS API](./xet/protocol/api) -- [Authentication and Authorization](./xet/protocol/auth) -- [Hub file conversion to Xet File ID](./xet/protocol/file-id) +- [Upload Protocol](./upload-protocol) +- [Download Protocol](./download-protocol) +- [CAS API](./api) +- [Authentication and Authorization](./auth) +- [Hub file conversion to Xet File ID](./file-id) From c832190170bed1b9cf3f802518fd0a64d9b1155e Mon Sep 17 00:00:00 2001 From: Jared Sulzdorf Date: Thu, 25 Sep 2025 19:02:49 -0700 Subject: [PATCH 15/19] quick fixes of some links --- docs/hub/storage-backends.md | 2 +- docs/hub/xet/overview.md | 4 ++-- docs/hub/xet/using-xet-storage.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/hub/storage-backends.md b/docs/hub/storage-backends.md index 13123694fb..b53145aed8 100644 --- a/docs/hub/storage-backends.md +++ b/docs/hub/storage-backends.md @@ -11,7 +11,7 @@ Storing these files directly in a pure Git repository is impractical. Not only a Instead, on the Hub, these large files are tracked using "pointer files" and identified through a `.gitattributes` file (both discussed in more detail below), which remain in the Git repository while the actual data is stored in remote storage (like [Amazon S3](https://aws.amazon.com/s3/)). As a result, the repository stays small and typical Git workflows remain efficient. -Historically, Hub repositories have relied on [Git LFS](https://git-lfs.com/) for this mechanism. While Git LFS remains supported (see [Legacy Storage using Git LFS](#legacy-storage-git-lfs)), the Hub has adopted Xet, a modern custom storage system built specifically for AI/ML development. It enables chunk-level deduplication, smaller uploads, and faster downloads than Git LFS. +Historically, Hub repositories have relied on [Git LFS](https://git-lfs.com/) for this mechanism. While Git LFS remains supported (see [Backwards Compatibility & Legacy](./xet/legacy-git-lfs)), the Hub has adopted Xet, a modern custom storage system built specifically for AI/ML development. It enables chunk-level deduplication, smaller uploads, and faster downloads than Git LFS. In these pages you will get started in using Xet Storage, including the open-source Xet protocol. diff --git a/docs/hub/xet/overview.md b/docs/hub/xet/overview.md index a390bbc527..e00b759f7b 100644 --- a/docs/hub/xet/overview.md +++ b/docs/hub/xet/overview.md @@ -15,11 +15,11 @@ A Git LFS pointer file provides metadata to locate the actual file contents in r - **Pointer size**: The size of the pointer file stored in the Git repository. - **Size of the remote file**: Indicates the size of the actual large file in bytes. This metadata is useful for both verification purposes and for managing storage and transfer operations. -A Xet pointer includes all of this information by design. Refer to the section on [backwards compatibility with Git LFS](#backward-compatibility-with-lfs) with the addition of a `Xet backed hash` field for referencing the file in Xet storage. +A Xet pointer includes all of this information by design. Refer to the section on [backwards compatibility with Git LFS](legacy-git-lfs#backward-compatibility-with-lfs) with the addition of a `Xet backed hash` field for referencing the file in Xet storage.
-Unlike Git LFS, which deduplicates at the file level, Xet-enabled repositories deduplicate at the level of bytes. When a file backed by Xet storage is updated, only the modified data is uploaded to remote storage, significantly saving on network transfers. For many workflows, like incremental updates to model checkpoints or appending/inserting new data into a dataset, this improves iteration speed for yourself and your collaborators. To learn more about deduplication in Xet storage, refer to the [Deduplication](#deduplication) section below. +Unlike Git LFS, which deduplicates at the file level, Xet-enabled repositories deduplicate at the level of bytes. When a file backed by Xet storage is updated, only the modified data is uploaded to remote storage, significantly saving on network transfers. For many workflows, like incremental updates to model checkpoints or appending/inserting new data into a dataset, this improves iteration speed for yourself and your collaborators. To learn more about deduplication in Xet storage, refer to [Deduplication](deduplication). diff --git a/docs/hub/xet/using-xet-storage.md b/docs/hub/xet/using-xet-storage.md index 2e4cb6bcd6..0b10567544 100644 --- a/docs/hub/xet/using-xet-storage.md +++ b/docs/hub/xet/using-xet-storage.md @@ -23,7 +23,7 @@ Where versions of `huggingface_hub` >= 0.30.0 and < 0.32.0 are installed, `hf_xe pip install -U hf-xet ``` -And that's it! You now get the benefits of Xet deduplication for both uploads and downloads. Team members using a version of `huggingface_hub` < 0.30.0 will still be able to upload and download repositories through the [backwards compatibility provided by the LFS bridge](#backward-compatibility-with-lfs). +And that's it! You now get the benefits of Xet deduplication for both uploads and downloads. Team members using a version of `huggingface_hub` < 0.30.0 will still be able to upload and download repositories through the [backwards compatibility provided by the LFS bridge](legacy-git-lfs#backward-compatibility-with-lfs). To see more detailed usage docs, refer to the `huggingface_hub` docs for: From 93aeefd2f151ebdc774bd706a3db7cb456a07334 Mon Sep 17 00:00:00 2001 From: Rajat Arya Date: Thu, 2 Oct 2025 15:02:39 -0700 Subject: [PATCH 16/19] Reorganize Storage Backend docs --- docs/hub/_toctree.yml | 15 +- docs/hub/repositories.md | 2 +- .../hub/{storage-backends.md => xet/index.md} | 5 +- docs/hub/xet/protocol/architecture.md | 15 -- docs/hub/xet/protocol/building-client.md | 12 -- docs/hub/xet/protocol/deduplication.md | 189 ------------------ 6 files changed, 16 insertions(+), 222 deletions(-) rename docs/hub/{storage-backends.md => xet/index.md} (92%) delete mode 100644 docs/hub/xet/protocol/architecture.md delete mode 100644 docs/hub/xet/protocol/building-client.md delete mode 100644 docs/hub/xet/protocol/deduplication.md diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index 515f7f35f2..5a0224e0aa 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -47,8 +47,19 @@ title: Repository Settings - local: storage-limits title: Storage Limits - - local: storage-backends - title: Storage Backends + - local: xet/index + title: Storage Backend (Xet) + sections: + - local: xet/overview + title: Xet History & Overview + - local: xet/using-xet-storage + title: Using Xet Storage + - local: xet/deduplication + title: Deduplication + - local: xet/legacy-git-lfs + title: Backwards Compatibility & Legacy + - local: xet/security + title: Security - local: repositories-pull-requests-discussions title: Pull Requests & Discussions - local: notifications diff --git a/docs/hub/repositories.md b/docs/hub/repositories.md index 33e01e1ca9..6daf32ded0 100644 --- a/docs/hub/repositories.md +++ b/docs/hub/repositories.md @@ -17,7 +17,7 @@ In these pages, you will go over the basics of getting started with Git and Xet - [Getting Started with Repositories](./repositories-getting-started) - [Settings](./repositories-settings) - [Storage Limits](./storage-limits) -- [Storage Backend - Xet](./storage-backends) +- [Storage Backend (Xet)](/.xet) - [Pull Requests & Discussions](./repositories-pull-requests-discussions) - [Pull Requests advanced usage](./repositories-pull-requests-discussions#pull-requests-advanced-usage) - [Collections](./collections) diff --git a/docs/hub/storage-backends.md b/docs/hub/xet/index.md similarity index 92% rename from docs/hub/storage-backends.md rename to docs/hub/xet/index.md index b53145aed8..045c5b468e 100644 --- a/docs/hub/storage-backends.md +++ b/docs/hub/xet/index.md @@ -1,4 +1,4 @@ -# Xet Storage +# Xet: our Storage Backend Repositories on the Hugging Face Hub are different from those on software development platforms. They contain files that are: @@ -13,14 +13,13 @@ Instead, on the Hub, these large files are tracked using "pointer files" and ide Historically, Hub repositories have relied on [Git LFS](https://git-lfs.com/) for this mechanism. While Git LFS remains supported (see [Backwards Compatibility & Legacy](./xet/legacy-git-lfs)), the Hub has adopted Xet, a modern custom storage system built specifically for AI/ML development. It enables chunk-level deduplication, smaller uploads, and faster downloads than Git LFS. -In these pages you will get started in using Xet Storage, including the open-source Xet protocol. +In these pages you will get started in using Xet Storage. ## Contents - [Xet History & Overview](./xet/overview) - [Using Xet Storage](./xet/using-xet-storage) - [Deduplication](./xet/deduplication) -- [Open Source Xet Protocol](./xet/protocol/index) - [Backwards Compatibility & Legacy](./xet/legacy-git-lfs) - [Security](./xet/security) diff --git a/docs/hub/xet/protocol/architecture.md b/docs/hub/xet/protocol/architecture.md deleted file mode 100644 index a306d3f089..0000000000 --- a/docs/hub/xet/protocol/architecture.md +++ /dev/null @@ -1,15 +0,0 @@ -## Overall Xet Architecture - -(TODO: Add diagram here) - -These pages will describe the overall Xet architecture, including how to perform chunking & hashing of files for upload, how to process file reconstruction information when downloading, the specfics of the Xorb and Shard formats, and how deduplication is achieved by a client library. - -## Contents - -- [Content-Defined Chunking](./chunking) -- [Hashing Methods](./hashing) -- [File Reconstruction](./file-reconstruction) -- [Xorb Format](./xorb) -- [Shard Format](./shard) -- [Deduplication](./deduplication) - diff --git a/docs/hub/xet/protocol/building-client.md b/docs/hub/xet/protocol/building-client.md deleted file mode 100644 index fb4ddc1820..0000000000 --- a/docs/hub/xet/protocol/building-client.md +++ /dev/null @@ -1,12 +0,0 @@ -## Building a client library for Xet storage - -These pages explain how you can build a client library for Xet storage. Included in these pages are the steps involved in uploading & downloading files, the web service APIs involved, and how authentication and authorization works. - -## Contents - -- [Upload Protocol](./upload-protocol) -- [Download Protocol](./download-protocol) -- [CAS API](./api) -- [Authentication and Authorization](./auth) -- [Hub file conversion to Xet File ID](./file-id) - diff --git a/docs/hub/xet/protocol/deduplication.md b/docs/hub/xet/protocol/deduplication.md deleted file mode 100644 index abb882f9e7..0000000000 --- a/docs/hub/xet/protocol/deduplication.md +++ /dev/null @@ -1,189 +0,0 @@ -# Xet Chunk-Level Deduplication Specification - -## Overview - -Chunk-level deduplication is a fundamental optimization technique in the Xet system that eliminates redundant data by identifying and sharing identical content blocks (chunks) across files and repositories. -This specification details the procedures, algorithms, and security mechanisms that enable efficient storage and transfer while maintaining data integrity and access control. - -Deduplication in Xet operates at the chunk level rather than the file level, providing fine-grained deduplication capabilities that can identify shared content even when files differ significantly. -This approach is particularly effective for scenarios common in machine learning and data science workflows, such as: - -- Multiple versions of datasets with incremental changes -- Model checkpoints that share common layers or parameters -- Documentation and configuration files with similar content -- Large files where only portions have changed between versions - -## Core Concepts - -### Chunks - -A **chunk** is a variable-sized content block derived from files using Content-Defined Chunking (CDC) with a rolling hash function. Chunks are the fundamental unit of deduplication in Xet. - -- **Target size**: 64KB (configurable) -- **Size range**: 8KB to 128KB (minimum and maximum constraints) -- **Identification**: Each chunk is uniquely identified by its cryptographic hash (MerkleHash) - -[Detailed chunking description](./chunking) - -### Xorbs - -**Xorbs** are objects that aggregate multiple chunks for efficient storage and transfer: - -- **Maximum size**: 64MB -- **Maximum chunks**: 8,192 chunks per xorb -- **Purpose**: Batch multiple chunks together to reduce metadata and network overhead when uploading and downloading groups of chunks - -### Shards (Xorb Lists) - -**Shards** are objects that contain a list of xorbs that can be deduped against (for the context of deduplication, ignore the file info section of the shard format). - -- **Maximum size**: 64MB -- **Purpose**: Provide a format on a positive reply to a global deduplication request with information about xorbs that already exist in the CAS system. - -### CAS (Content Addressable Storage) - -The **CAS** system provides the underlying storage infrastructure: - -- **Content addressing**: All objects are stored and retrieved by their cryptographic hash -- **Immutability**: Once stored, file content cannot be modified -- **Deduplication**: Identical content is automatically deduplicated at the storage level - -## Deduplication Procedure - -### 1. File Processing and Chunking - -When a file is processed for upload, it undergoes the following steps: - -```mermaid -graph TD - A[File Input] --> B[Content-Defined Chunking] - B --> C[Hash Computation] - C --> D[Chunk Creation] - D --> E[Deduplication Query] -``` - -1. **Chunking**: Content-defined chunking using GearHash algorithm creates variable-sized chunks of file data -2. **Hash Computation**: Each chunk's content is hashed using a cryptographic hash function (Blake3-based MerkleHash) -3. **Chunk Object Creation**: Chunks are wrapped with metadata including hash, size, and data - -### 2. Multi-Level Deduplication Strategy - -Xet employs a three-tiered deduplication strategy to maximize efficiency while minimizing latency: - -#### Level 1: Local Session Deduplication - -**Scope**: Current upload session -**Mechanism**: In-memory hash lookup table -**Purpose**: Eliminate redundancy within the current file or session - -**Benefits**: - -- Fastest lookup (in-memory) -- Zero network overhead -- Immediate deduplication feedback - -#### Level 2: Cached Metadata Deduplication - -**Scope**: Previously uploaded files and sessions -**Mechanism**: Local shard file metadata cache -**Purpose**: Leverage deduplication against recently uploaded content - -**Benefits**: - -- Fast local disk access -- No network latency -- Persistent across sessions - -#### Level 3: Global Deduplication API - -**Scope**: Entire Xet ecosystem -**Mechanism**: Global deduplication service with HMAC protection -**Purpose**: Discover deduplication opportunities across all users and repositories - -### 3. Global Deduplication Process - -The global deduplication system provides deduplication capabilities across all data that is managed by the Xet system: - -#### Eligibility Criteria - -Not all chunks are eligible for global deduplication queries to manage system load: - -1. **First chunk**: The first chunk of every file is always eligible. -2. **Hash pattern matching**: Chunks are eligible if: the last 8 bytes of the hash interpreted as a little-endian 64 bit integer % 1024 == 0. - -**Recommendations:** -**Spacing constraints**: The global dedupe API is optimized to return information about nearby chunks when there is a match. Consider only issueing a request to an eligible chunk every ~4MB of data. - -#### Query Process - -1. **Background Query**: Global deduplication queries SHOULD run asynchronously to avoid blocking upload -2. **HMAC Protection**: Chunk hashes are protected using HMAC keys -3. **Shard Response**: When a match is found, the API returns a shard containing: - - **CAS Info Section**: Contains metadata about many xorbs that store chunks - - **HMAC Key**: Included in the shard metadata header used to encrypt chunk hashes -4. **Encrypted Chunk Matching**: All chunk hashes in the returned shard have been encrypted with the HMAC key -5. **Match Discovery Process**: To find matches, clients MUST: - - Encrypt their chunk hash using the provided HMAC key - - Search for the encrypted hash within the shard's chunk listings - - For subsequent chunks, repeat the encryption and search process - - Track the original (non-encrypted) chunk hash while noting which xorb contains that chunk -6. **Metadata Caching**: Client downloads and caches shard metadata for future deduplication - -#### HMAC Security Mechanism - -Global deduplication uses HMAC (Hash-based Message Authentication Code) to protect chunk hashes while enabling deduplication. - -**Security Properties**: - -Raw chunk hashes are never transmitted from servers to clients; a client has to encrypt their raw chunk hash and find a match to know a raw chunk hash exists in the system. -They MAY know this chunk hash because they own this data, the match has made them privy to know which xorb has this chunk hash and the position in the xorb, but has not revealed any other raw chunk hashes in that xorb or other xorbs. - -## Technical Implementation Details - -### Chunk Hash Computation - -Each chunk has its content hashed using a cryptographic hash function (Blake3-based MerkleHash) to create a unique identifier for content addressing. -[See section about hashing](./hashing#chunk-hashes). - -### Xorb Formation - -When new chunks need to be stored, they are aggregated into xorbs based on size and count limits. If adding a new chunk would exceed the maximum xorb size or chunk count, the current xorb is finalized and uploaded. [See section about xorb formation](./xorb) - -### File Reconstruction Information - -When chunks are deduplicated, the system creates file reconstruction information that includes: - -- Hash of the xorb containing the chunks -- Flags for the CAS block -- Total bytes in the segment -- Start and end indices within the xorb (start inclusive, end exclusive) - -This information allows the system to reconstruct files by: - -1. Identifying which xorbs contain the needed chunks -2. Extracting the specific chunk ranges from each xorb -3. Concatenating chunks in the correct order - -[See section about file reconstruction](./file-reconstruction). - -## Fragmentation Prevention - -While deduplication is valuable for saving space, doing it too aggressively can cause file fragmentation—meaning a file’s chunks end up scattered across many different xorbs. This can make reading files slower and less efficient. -To avoid this, in xet-core we aim (and encourage implementors) to keep long, continuous runs of chunks together in the same xorb whenever possible. Implementations SHOULD keep long, continuous runs together when feasible. -Instead of always deduplicating every possible chunk, the system sometimes chooses to reference a straight run of chunks in a single xorb, even if it means skipping deduplication for a few chunks. -This approach balances the benefits of deduplication with the need to keep files easy and fast to read. -Consider for example referencing a deduplicated chunks in a minimum run of chunks (e.g. at least 8 chunks) or targeting an average contiguous run of chunks totalling length >= 1MB. - -## Conclusion - -Xet's chunk-level deduplication system provides a comprehensive solution for efficient data storage and transfer in large-scale data workflows. -By combining local, cached, and global deduplication strategies with robust security mechanisms and fragmentation prevention, -the system achieves significant storage savings while maintaining performance and data integrity. - -The multi-tiered approach ensures that deduplication is both effective and efficient: - -- Local deduplication provides immediate benefits within sessions -- Cached deduplication leverages recent upload history -- Global deduplication enables cross-repository optimization while preserving security - -The system's design prioritizes both efficiency and safety, with comprehensive error handling, performance monitoring, and security measures that make it suitable for production use in data-intensive applications. From fadd760a4d0765fa8f3137dfdf211e8b9598a43a Mon Sep 17 00:00:00 2001 From: Rajat Arya Date: Thu, 2 Oct 2025 15:16:09 -0700 Subject: [PATCH 17/19] Fix TOC and link to xet protocol --- docs/hub/repositories.md | 2 +- docs/hub/xet/index.md | 16 ++++++++++------ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/hub/repositories.md b/docs/hub/repositories.md index 6daf32ded0..2fbf0986d8 100644 --- a/docs/hub/repositories.md +++ b/docs/hub/repositories.md @@ -17,7 +17,7 @@ In these pages, you will go over the basics of getting started with Git and Xet - [Getting Started with Repositories](./repositories-getting-started) - [Settings](./repositories-settings) - [Storage Limits](./storage-limits) -- [Storage Backend (Xet)](/.xet) +- [Storage Backend (Xet)](./xet/index) - [Pull Requests & Discussions](./repositories-pull-requests-discussions) - [Pull Requests advanced usage](./repositories-pull-requests-discussions#pull-requests-advanced-usage) - [Collections](./collections) diff --git a/docs/hub/xet/index.md b/docs/hub/xet/index.md index 045c5b468e..b295218140 100644 --- a/docs/hub/xet/index.md +++ b/docs/hub/xet/index.md @@ -11,15 +11,19 @@ Storing these files directly in a pure Git repository is impractical. Not only a Instead, on the Hub, these large files are tracked using "pointer files" and identified through a `.gitattributes` file (both discussed in more detail below), which remain in the Git repository while the actual data is stored in remote storage (like [Amazon S3](https://aws.amazon.com/s3/)). As a result, the repository stays small and typical Git workflows remain efficient. -Historically, Hub repositories have relied on [Git LFS](https://git-lfs.com/) for this mechanism. While Git LFS remains supported (see [Backwards Compatibility & Legacy](./xet/legacy-git-lfs)), the Hub has adopted Xet, a modern custom storage system built specifically for AI/ML development. It enables chunk-level deduplication, smaller uploads, and faster downloads than Git LFS. +Historically, Hub repositories have relied on [Git LFS](https://git-lfs.com/) for this mechanism. While Git LFS remains supported (see [Backwards Compatibility & Legacy](./legacy-git-lfs)), the Hub has adopted Xet, a modern custom storage system built specifically for AI/ML development. It enables chunk-level deduplication, smaller uploads, and faster downloads than Git LFS. + +### Open Source Xet Protocol + +If you are looking to understand the underlying Xet protocol or are looking to build a new client library to access Xet Storage, check out the [Xet Protocol Specification](https://huggingface.co/docs/xet/index). In these pages you will get started in using Xet Storage. ## Contents -- [Xet History & Overview](./xet/overview) -- [Using Xet Storage](./xet/using-xet-storage) -- [Deduplication](./xet/deduplication) -- [Backwards Compatibility & Legacy](./xet/legacy-git-lfs) -- [Security](./xet/security) +- [Xet History & Overview](./overview) +- [Using Xet Storage](./using-xet-storage) +- [Deduplication](./deduplication) +- [Backwards Compatibility & Legacy](./legacy-git-lfs) +- [Security](./security) From ce9ce6a208bbba2e131b0a4439212575a1792c2d Mon Sep 17 00:00:00 2001 From: Rajat Arya Date: Thu, 2 Oct 2025 15:34:06 -0700 Subject: [PATCH 18/19] Fix storage-backends rename --- docs/hub/_redirects.yml | 2 +- docs/hub/index.md | 4 ++-- docs/hub/repositories.md | 2 +- docs/hub/xet/deduplication.md | 10 ++++++++++ 4 files changed, 14 insertions(+), 4 deletions(-) create mode 100644 docs/hub/xet/deduplication.md diff --git a/docs/hub/_redirects.yml b/docs/hub/_redirects.yml index 0831a66db8..6437a556db 100644 --- a/docs/hub/_redirects.yml +++ b/docs/hub/_redirects.yml @@ -18,4 +18,4 @@ api-webhook: webhooks adapter-transformers: adapters security-two-fa: security-2fa repositories-recommendations: storage-limits -xet: storage-backends#xet +xet: xet/index diff --git a/docs/hub/index.md b/docs/hub/index.md index 14e5a3d46c..999dffc4c4 100644 --- a/docs/hub/index.md +++ b/docs/hub/index.md @@ -27,7 +27,7 @@ The Hugging Face Hub is a platform with over 1.7M models, 400k datasets, and 600 Getting Started Repository Settings Storage Limits -Storage Backend - Xet +Storage Backend (Xet) Pull requests and Discussions Notifications Collections @@ -122,7 +122,7 @@ On it, you'll be able to upload and discover... - Spaces: _interactive apps for demonstrating ML models directly in your browser_ The Hub offers **versioning, commit history, diffs, branches, and over a dozen library integrations**! -All repositories build on [Xet](./storage-backends), a new technology to efficiently store Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. +All repositories build on [Xet](./xet/index), a new technology to efficiently store Large Files inside Git, intelligently splitting files into unique chunks and accelerating uploads and downloads. You can learn more about the features that all repositories share in the [**Repositories documentation**](./repositories). diff --git a/docs/hub/repositories.md b/docs/hub/repositories.md index 2fbf0986d8..0134925578 100644 --- a/docs/hub/repositories.md +++ b/docs/hub/repositories.md @@ -3,7 +3,7 @@ Models, Spaces, and Datasets are hosted on the Hugging Face Hub as [Git repositories](https://git-scm.com/about), which means that version control and collaboration are core elements of the Hub. In a nutshell, a repository (also known as a **repo**) is a place where code and assets can be stored to back up your work, share it with the community, and work in a team. Unlike other collaboration platforms, our Git repositories are optimized for Machine Learning and AI files – large binary files, usually in specific file formats like Parquet and Safetensors, and up to [Terabyte-scale sizes](https://huggingface.co/blog/from-files-to-chunks)! -To achieve this, we built [Xet](./storage-backends), a modern custom storage system built specifically for AI/ML development, enabling chunk-level deduplication, smaller uploads, and faster downloads. +To achieve this, we built [Xet](./xet/index), a modern custom storage system built specifically for AI/ML development, enabling chunk-level deduplication, smaller uploads, and faster downloads.
diff --git a/docs/hub/xet/deduplication.md b/docs/hub/xet/deduplication.md new file mode 100644 index 0000000000..0d2855e5e5 --- /dev/null +++ b/docs/hub/xet/deduplication.md @@ -0,0 +1,10 @@ +## Deduplication + +Xet-enabled repositories utilize [content-defined chunking (CDC)](https://huggingface.co/blog/from-files-to-chunks) to deduplicate on the level of bytes (~64KB of data, also referred to as a "chunk"). Each chunk is identified by a rolling hash that determines chunk boundaries based on the actual file contents, making it resilient to insertions or deletions anywhere in the file. When a file is uploaded to a Xet-backed repository using a Xet-aware client, its contents are broken down into these variable-sized chunks. Only new chunks not already present in Xet storage are kept after chunking, everything else is discarded. + +To avoid the overhead of communicating and managing at the level of chunks, new chunks are grouped together in [64MB blocks](https://huggingface.co/blog/from-chunks-to-blocks#scaling-deduplication-with-aggregation) and uploaded. Each block is stored once in a content-addressed store (CAS), keyed by its hash. + +The Hub's [current recommendation](https://huggingface.co/docs/hub/storage-limits#recommendations) is to limit files to 20GB. At a 64KB chunk size, a 20GB file has 312,500 chunks, many of which go unchanged from version to version. Git LFS is designed to notice only that a file has changed and store the entirety of that revision. By deduplicating at the level of chunks, the Xet backend enables storing only the modified content in a file (which might only be a few KB or MB) and securely deduplicates shared blocks across repositories. For the large binary files found in Model and Dataset repositories, this provides significant improvements to file transfer times. + +For more details, refer to the [From Files to Chunks](https://huggingface.co/blog/from-files-to-chunks) and [From Chunks to Blocks](https://huggingface.co/blog/from-chunks-to-blocks) blog posts, or the [Git is for Data](https://www.cidrdb.org/cidr2023/papers/p43-low.pdf) paper by Low et al. that served as the launch point for XetHub prior to being acquired by Hugging Face. + From 51eab755ae8bbd0680d5b33c4101a527b8ac9fd1 Mon Sep 17 00:00:00 2001 From: Rajat Arya Date: Thu, 2 Oct 2025 15:35:38 -0700 Subject: [PATCH 19/19] Update xet protocol to 1.0.0 --- docs/xet/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/xet/index.md b/docs/xet/index.md index ab27c53ad1..20bd70fc79 100644 --- a/docs/xet/index.md +++ b/docs/xet/index.md @@ -1,7 +1,7 @@ # Xet Protocol Specification > [!NOTE] -> Version 0.1.0 (1.0.0 on release) +> Version 1.0.0 > The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119](https://www.ietf.org/rfc/rfc2119.txt) [RFC8174](https://www.ietf.org/rfc/rfc8174.txt) when, and only when, they appear in all capitals, as shown here.