From fe88e1da25920ca6f292f1a8285a63e828da9585 Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 09:13:52 +0200 Subject: [PATCH 01/10] update page to inference providers only --- docs/hub/models-inference.md | 73 ++++++++++++++++++++++++++++-------- 1 file changed, 57 insertions(+), 16 deletions(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index 4a8f81465d..b7bbcd117b 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -1,30 +1,71 @@ # Inference Providers -Please refer to the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers) for detailed information. +Hugging Face's model pages have free inference for thousands of models, so you can try them all out right in the browser. It's also powered by Inference Providers. -## What is HF-Inference API? +Inference Providers give developers streamlined, unified access to hundreds of machine learning models, powered by the best serverless inference partners. 👉 **For complete documentation, visit the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers)**. -HF-Inference API is one of the many providers available on the Hugging Face Hub. -It is deployed by Hugging Face ourselves, using text-generation-inference for LLMs for instance. This service used to be called “Inference API (serverless)” prior to Inference Providers. +## Inference Providers on the Hub -For more details about the HF-Inference API, check out its [dedicated page](https://huggingface.co/docs/inference-providers/providers/hf-inference). +Inference Providers is deeply integrated with the Hugging Face Hub, and you can use it in a few different ways: -## What technology do you use to power the HF-Inference API? +- **Interactive Widgets** - Test models directly on model pages with interactive widgets that use Inference Providers under the hood. Check out the [DeepSeek-R1-0528 model page](https://huggingface.co/models/deepseek-ai/DeepSeek-R1-0528) for an example. +- **Inference Playground** - Easily test and compare chat completion models with your prompts. Check out the [Inference Playground](https://huggingface.co/playground) to get started. +- **Search** - Filter models by inference provider on the [models page](https://huggingface.co/models?inference_provider=all) to find models available through specific providers. -The HF-Inference API is powered by [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index) under the hood. +## Build with Inference Providers -## Why don't I see an inference widget, or why can't I use the API? +You can integrate Inference Providers into your own applications using our SDKs or HTTP clients. Here's a quick start with Python and JavaScript, for more details, check out the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers). -For some tasks, there might not be support by any Inference Provider, and hence, there is no widget. + -## How can I see my usage? + -To check usage across all providers, check out your [billing page](https://huggingface.co/settings/billing). +```python +from huggingface_hub import InferenceClient -To check your HF-Inference usage specifically, check out the [Inference Dashboard](https://ui.endpoints.huggingface.co/endpoints). The dashboard shows both your serverless and dedicated endpoints usage. +client = InferenceClient(provider="auto") # Automatically selects best provider -## Is there programmatic access to Inference Providers? +# Chat completion +completion = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V3-0324", + messages=[{"role": "user", "content": "Hello!"}] +) +``` -Yes! We provide client wrappers in both JS and Python: -- [JS (`@huggingface/inference`)](https://huggingface.co/docs/huggingface.js/inference/classes/InferenceClient) -- [Python (`huggingface_hub`)](https://huggingface.co/docs/huggingface_hub/guides/inference) + + + + +```javascript +import { InferenceClient } from "@huggingface/inference"; + +const client = new InferenceClient(process.env.HF_TOKEN); + +const chatCompletion = await client.chatCompletion({ + provider: "auto", // Automatically selects best provider + model: "deepseek-ai/DeepSeek-V3-0324", + messages: [{ role: "user", content: "Hello!" }] +}); +``` + + + + + +You'll need a Hugging Face token with inference permissions. Create one at [Settings > Tokens](https://huggingface.co/settings/tokens/new?ownUserPermissions=inference.serverless.write&tokenType=fineGrained). + +### How Inference Providers works + +Hugging Face’s Inference Providers give developers unified access to hundreds of machine learning models, powered by our serverless inference partners. This new approach builds on our previous Serverless Inference API, offering more models, improved performance, and greater reliability thanks to world-class providers. + +To dive deeper into Inference Providers, check out the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers). Here are some key resources: + +- **[Quick Start](https://huggingface.co/docs/inference-providers)** +- **[Pricing & Billing Guide](https://huggingface.co/docs/inference-providers/pricing)** +- **[Hub Integration Details](https://huggingface.co/docs/inference-providers/hub-integration)** + +### What was the HF-Inference API? + +HF-Inference API is one of the providers available through Inference Providers. It was previously called "Inference API (serverless)" and is powered by [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index) under the hood. + +For more details about the HF-Inference provider specifically, check out its [dedicated page](https://huggingface.co/docs/inference-providers/providers/hf-inference). From 94a0018efbafa0d17150cdc3d8978a73085e8bdf Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 09:53:04 +0200 Subject: [PATCH 02/10] update ToC --- docs/hub/_toctree.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index 16fcf1ad6c..5ba9ead00e 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -127,7 +127,7 @@ - local: models-widgets-examples title: Widget Examples - local: models-inference - title: Inference API docs + title: Model Inference - local: models-download-stats title: Models Download Stats - local: models-faq From 111aec5dd8abfb7f6ec816fa685cb9b9a794f5ba Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 10:32:40 +0200 Subject: [PATCH 03/10] Update docs/hub/models-inference.md Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com> --- docs/hub/models-inference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index b7bbcd117b..231d0aaf3b 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -8,7 +8,7 @@ Inference Providers give developers streamlined, unified access to hundreds of m Inference Providers is deeply integrated with the Hugging Face Hub, and you can use it in a few different ways: -- **Interactive Widgets** - Test models directly on model pages with interactive widgets that use Inference Providers under the hood. Check out the [DeepSeek-R1-0528 model page](https://huggingface.co/models/deepseek-ai/DeepSeek-R1-0528) for an example. +- **Interactive Widgets** - Test models directly on model pages with interactive widgets that use Inference Providers under the hood. Check out the [DeepSeek-R1-0528 model page](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) for an example. - **Inference Playground** - Easily test and compare chat completion models with your prompts. Check out the [Inference Playground](https://huggingface.co/playground) to get started. - **Search** - Filter models by inference provider on the [models page](https://huggingface.co/models?inference_provider=all) to find models available through specific providers. From bef1e90d77d53a2384c5f6495bb677233e3478b9 Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 10:33:02 +0200 Subject: [PATCH 04/10] Update docs/hub/models-inference.md Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com> --- docs/hub/models-inference.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index 231d0aaf3b..5b82668880 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -23,7 +23,12 @@ You can integrate Inference Providers into your own applications using our SDKs ```python from huggingface_hub import InferenceClient -client = InferenceClient(provider="auto") # Automatically selects best provider +import os + +client = InferenceClient( + api_key=os.environ["HF_TOKEN"], + provider="auto", # Automatically selects best provider +) # Chat completion completion = client.chat.completions.create( From 391920ede4f62aabd723d92efb4b4c1ebdb8ffae Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 11:30:06 +0200 Subject: [PATCH 05/10] Update docs/hub/models-inference.md Co-authored-by: Lucain --- docs/hub/models-inference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index 5b82668880..986fd217e5 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -1,6 +1,6 @@ # Inference Providers -Hugging Face's model pages have free inference for thousands of models, so you can try them all out right in the browser. It's also powered by Inference Providers. +Hugging Face's model pages have pay-as-you-go inference for thousands of models, so you can try them all out right in the browser. Service is powered by Inference Providers and includes a free-tier. Inference Providers give developers streamlined, unified access to hundreds of machine learning models, powered by the best serverless inference partners. 👉 **For complete documentation, visit the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers)**. From ff74e98cb22a70bea0fb9c889ae6bc67bbaf33b5 Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 11:36:39 +0200 Subject: [PATCH 06/10] add datastudio --- docs/hub/models-inference.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index b7bbcd117b..1f26e37aa8 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -11,6 +11,7 @@ Inference Providers is deeply integrated with the Hugging Face Hub, and you can - **Interactive Widgets** - Test models directly on model pages with interactive widgets that use Inference Providers under the hood. Check out the [DeepSeek-R1-0528 model page](https://huggingface.co/models/deepseek-ai/DeepSeek-R1-0528) for an example. - **Inference Playground** - Easily test and compare chat completion models with your prompts. Check out the [Inference Playground](https://huggingface.co/playground) to get started. - **Search** - Filter models by inference provider on the [models page](https://huggingface.co/models?inference_provider=all) to find models available through specific providers. +- **Data Studio** - Use AI to explore datasets on the Hub. Check out [Data Studio](https://huggingface.co/datasets/fka/awesome-chatgpt-prompts/viewer?views%5B%5D=train) on your favorite dataset. ## Build with Inference Providers From 0ce3f92f19bad2194514fad0ee9f4e14fb80ee3d Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 11:50:30 +0200 Subject: [PATCH 07/10] add openai and image generation --- docs/hub/models-inference.md | 63 +++++++++++++++++++++++++++++++++++- 1 file changed, 62 insertions(+), 1 deletion(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index b9df2f5572..498d200d92 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -21,6 +21,8 @@ You can integrate Inference Providers into your own applications using our SDKs +You can use our Python SDK to interact with Inference Providers. + ```python from huggingface_hub import InferenceClient @@ -34,14 +36,47 @@ client = InferenceClient( # Chat completion completion = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3-0324", - messages=[{"role": "user", "content": "Hello!"}] + messages=[{"role": "user", "content": "A story about hiking in the mountains"}] +) + +# Image generation +image = client.text_to_image( + prompt="A serene lake surrounded by mountains at sunset, photorealistic style", + model="black-forest-labs/FLUX.1-dev" +) + +``` + +Or, you can just use the OpenAI API compatible client. + +```python +import os +from huggingface_hub import InferenceClient + +client = InferenceClient( + api_key=os.environ["HF_TOKEN"], +) + +completion = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V3-0324", + messages=[ + { + "role": "user", + "content": "A story about hiking in the mountains" + } + ], ) ``` +> [!NOTE] +> The OpenAI API compatible client is not supported for image generation. + +You can use our JavaScript SDK to interact with Inference Providers. + ```javascript import { InferenceClient } from "@huggingface/inference"; @@ -52,8 +87,34 @@ const chatCompletion = await client.chatCompletion({ model: "deepseek-ai/DeepSeek-V3-0324", messages: [{ role: "user", content: "Hello!" }] }); + +const imageBlob = await client.textToImage({ + model: "black-forest-labs/FLUX.1-dev", + inputs: + "A serene lake surrounded by mountains at sunset, photorealistic style", +}); +``` + +Or, you can just use the OpenAI API compatible client. + +```javascript +import { OpenAI } from "openai"; + +const client = new OpenAI({ + baseURL: "https://router.huggingface.co/v1", + apiKey: process.env.HF_TOKEN, +}); + +const completion = await client.chat.completions.create({ + model: "meta-llama/Llama-3.1-8B-Instruct", + messages: [{ role: "user", content: "A story about hiking in the mountains" }], +}); + ``` +> [!NOTE] +> The OpenAI API compatible client is not supported for image generation. + From 0596118e9bae0a4a771a6d834c46f3fdf42fab74 Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 11:51:56 +0200 Subject: [PATCH 08/10] drop intro paragraph --- docs/hub/models-inference.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index 498d200d92..8eb12b7e77 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -123,8 +123,6 @@ You'll need a Hugging Face token with inference permissions. Create one at [Sett ### How Inference Providers works -Hugging Face’s Inference Providers give developers unified access to hundreds of machine learning models, powered by our serverless inference partners. This new approach builds on our previous Serverless Inference API, offering more models, improved performance, and greater reliability thanks to world-class providers. - To dive deeper into Inference Providers, check out the [Inference Providers Documentation](https://huggingface.co/docs/inference-providers). Here are some key resources: - **[Quick Start](https://huggingface.co/docs/inference-providers)** From 5c7625f0e8500f7a9a708c97c3e5dd45a3a55d2b Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 12:03:01 +0200 Subject: [PATCH 09/10] fix tip notes with mdx --- docs/hub/models-inference.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index 8eb12b7e77..e5a58dc1ca 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -68,8 +68,11 @@ completion = client.chat.completions.create( ) ``` -> [!NOTE] -> The OpenAI API compatible client is not supported for image generation. + + +The OpenAI API compatible client is not supported for image generation. + + @@ -112,8 +115,11 @@ const completion = await client.chat.completions.create({ ``` -> [!NOTE] -> The OpenAI API compatible client is not supported for image generation. + + +The OpenAI API compatible client is not supported for image generation. + + From 798f45613b68ff915a4c06993a200687ad33570b Mon Sep 17 00:00:00 2001 From: burtenshaw Date: Thu, 3 Jul 2025 15:52:11 +0200 Subject: [PATCH 10/10] use openai client --- docs/hub/models-inference.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/hub/models-inference.md b/docs/hub/models-inference.md index e5a58dc1ca..a6d85060ec 100644 --- a/docs/hub/models-inference.md +++ b/docs/hub/models-inference.md @@ -51,9 +51,10 @@ Or, you can just use the OpenAI API compatible client. ```python import os -from huggingface_hub import InferenceClient +from openai import OpenAI -client = InferenceClient( +client = OpenAI( + base_url="https://router.huggingface.co/v1", api_key=os.environ["HF_TOKEN"], )