Clarify default 'auto' provider policy for chat completions#2654
Clarify default 'auto' provider policy for chat completions#2654Wauplin wants to merge 2 commits into
Conversation
The default 'auto' policy for chat/conversational requests is no longer strictly equivalent to ':fastest'. It now prefers providers that support the model's advanced capabilities (tool calling and structured outputs), using throughput as a tiebreaker. Update the provider-selection docs to describe 'auto' accurately and document ':fastest' as an explicit policy suffix for users who always want the highest-throughput provider.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Co-authored-by: Julien Chaumond <julien@huggingface.co>
| <hfoption id="python"> | ||
|
|
||
| Next, we'll use a powerful language model like `deepseek-ai/DeepSeek-R1-0528` from DeepSeek via an Inference Provider. The `:fastest` policy is the default for chat completions, automatically selecting the best performing provider for this model. | ||
| Next, we'll use a powerful language model like `deepseek-ai/DeepSeek-R1-0528` from DeepSeek via an Inference Provider. The `auto` policy is the default for chat completions, automatically selecting the best provider for this model. |
There was a problem hiding this comment.
The example below can be a bit confusing imo. We specify the provider when building the client, but we also specify :fastest as a suffix to the model name when calling the chat completion task. IIUC, :fastest or other model-suffix specifications are only compatible with the CONVERSATIONAL_AUTO_ROUTER (provider="auto", or not specified), but would fail with any other provider (i.e., "groq"). Including :fastest in this way would also ignore, I think, the "best provider" selection we are referring to. For example, :cheapest would actually be routed to the cheapest provider and the auto selection specified in the client would no longer refer to the "best" provider (again, if I'm understanding this correctly).
The model-suffix style would also fail for non-conversational tasks.
| <hfoption id="javascript"> | ||
|
|
||
| Next, we'll use a powerful language model like `deepseek-ai/DeepSeek-R1-0528` from DeepSeek via an Inference Provider. The `:fastest` policy is the default for chat completions, automatically selecting the best performing provider for this model. | ||
| Next, we'll use a powerful language model like `deepseek-ai/DeepSeek-R1-0528` from DeepSeek via an Inference Provider. The `auto` policy is the default for chat completions, automatically selecting a suitable provider for this model. |
There was a problem hiding this comment.
In this version we don't even include provider="auto" (which is the default), but we also specify :fastest later.
|
|
||
| > [!TIP] | ||
| > If you plan to use a specific provider, append it to the model id as `<repo>:<provider>` (for example `moonshotai/Kimi-K2-Instruct-0905:groq`). Otherwise, omit the suffix and the fastest available provider will be selected by default (equivalent to the `:fastest` policy). You can also use `:cheapest` or `:preferred` (follows your [Inference Provider settings](https://hf.co/settings/inference-providers) order). | ||
| > If you plan to use a specific provider, append it to the model id as `<repo>:<provider>` (for example `moonshotai/Kimi-K2-Instruct-0905:groq`). Otherwise, omit the suffix and a suitable provider will be selected by default (the `auto` policy, which prioritizes providers supporting the model's advanced capabilities such as tool calling and structured outputs). You can also use `:fastest`, `:cheapest`, or `:preferred` (follows your [Inference Provider settings](https://hf.co/settings/inference-providers) order). |
There was a problem hiding this comment.
This sentence, together with the example below, seems to imply that :auto would be accepted as a suffix specification, but it's not.
| By default, our system automatically selects a suitable provider for the specified model (the `auto` policy). For chat models, `auto` prioritizes providers that support the model's advanced capabilities, such as tool calling and structured outputs, using throughput (tokens per second) as a tiebreaker. | ||
|
|
||
| You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`. | ||
| You can change the provider selection policy by appending a policy suffix to the model id: `:fastest` for the highest-throughput provider (tokens per second), `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`. |
There was a problem hiding this comment.
Similar to the previous comment: we use the term "policy" to refer both to the "auto" provider selection and to the model suffixes, but they are not interchangeable.
(remaining changes not reviewed, it looks like they are similar)
4 participants
The default 'auto' policy for chat/conversational requests is no longer strictly equivalent to ':fastest'. It now prefers providers that support the model's advanced capabilities (tool calling and structured outputs), using throughput as a tiebreaker. Update the provider-selection docs to describe 'auto' accurately and document ':fastest' as an explicit policy suffix for users who always want the highest-throughput provider.
from https://github.com/huggingface-internal/moon-landing/pull/19060
Note
Low Risk
Documentation-only wording changes; no runtime, API, or authentication behavior is modified in this diff.
Overview
Updates Inference Providers documentation so the default
autopolicy is no longer described as equivalent to:fastest.Across the main index, Responses API guide, first-app tutorial, and Inspect eval guide, prose now states that
autopicks a suitable provider—for chat models, favoring backends that support advanced features (tool calling, structured outputs) and using throughput only as a tiebreaker.:fastestis documented explicitly as the suffix for highest-throughput routing when users want the old behavior.The Provider Selection and OpenAI-compatible endpoint sections gain the same distinction, including listing
:fastestalongside:cheapestand:preferredwhere policy suffixes are explained.Reviewed by Cursor Bugbot for commit af5ca46. Bugbot is set up for automated code reviews on this repo. Configure here.