You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/inference-providers/guides/evaluation-inspect-ai.md
+1-5Lines changed: 1 addition & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -100,7 +100,7 @@ inspect view
100
100
## Example: Comparing several inference providers for a task
101
101
In this section, we will evaluate the same model across different providers. Inference Providers gives us access to many providers for the same model. Performance might vary across providers, so this is a useful factor, in addition to speed and cost, to choose the most appropriate inference provider for your task.
102
102
103
-
If we don't specify a provider, like we did in our previous examples, the system automatically selects the fastest available provider for the specified model (equivalent to the `:fastest` policy). But we can also select the provider by appending the provider name to the model id (e.g. `openai/gpt-oss-120b:sambanova`).
103
+
If we don't specify a provider, like we did in our previous examples, the system automatically selects the fastest available provider for the specified model (equivalent to the `:fastest` policy). But we can also select the provider by appending the provider name to the model id (e.g. `openai/gpt-oss-120b:groq`).
104
104
105
105
Let's run the evaluations for `gpt-oss-120b` across several providers. Please note that this time we are using the `eval_set` function directly in Python for extra flexibility (e.g., changing the list of providers):
106
106
@@ -109,12 +109,10 @@ from inspect_ai import eval_set
109
109
110
110
target_providers = [
111
111
"together",
112
-
"sambanova",
113
112
"groq",
114
113
"novita",
115
114
"cerebras",
116
115
"nscale",
117
-
"hyperbolic",
118
116
"fireworks-ai",
119
117
"scaleway"
120
118
]
@@ -139,15 +137,13 @@ Launching the viewer and sorting by score, we can compare the performance across
139
137
140
138
| Model | Provider | Score |
141
139
|-------|----------|-------|
142
-
| openai/gpt-oss-120b | hyperbolic | 0.84 |
143
140
| openai/gpt-oss-120b | fireworks-ai | 0.82 |
144
141
| openai/gpt-oss-120b | nscale | 0.82 |
145
142
| openai/gpt-oss-120b | scaleway | 0.81 |
146
143
| openai/gpt-oss-120b | together | 0.81 |
147
144
| openai/gpt-oss-120b | cerebras | 0.8 |
148
145
| openai/gpt-oss-120b | novita | 0.8 |
149
146
| openai/gpt-oss-120b | groq | 0.8 |
150
-
| openai/gpt-oss-120b | sambanova | 0.8 |
151
147
152
148
> [!TIP]
153
149
> **Why performance varies across providers**: As seen above, the same model can produce different results when served by different inference providers due to several factors: variations in inference implementations, differences in hardware (GPU generations, optimizations), and non-determinism introduced by load balancing and batching strategies. Performance can vary across the matrix of provider-model combinations and may change with updates to inference stacks, GPU generations, and model versions. Evaluating across multiple providers helps identify the best-performing combinations for your specific use case.
-**🎯 All-in-One API**: A single API for text generation, image generation, document embeddings, NER, summarization, image classification, and more.
85
-
-**🔀 Multi-Provider Support**: Easily run models from top-tier providers like fal, Replicate, Sambanova, Together AI, and others.
83
+
-**🔀 Multi-Provider Support**: Easily run models from top-tier providers like fal, Replicate, Together AI, and others.
86
84
-**🚀 Scalable & Reliable**: Built for high availability and low-latency performance in production environments.
87
85
-**🔧 Developer-Friendly**: Simple requests, fast responses, and a consistent developer experience across Python and JavaScript clients.
88
86
-**👷 Easy to integrate**: Drop-in replacement for the OpenAI chat completions API.
@@ -135,7 +133,7 @@ By default, our system automatically selects the fastest available provider for
135
133
136
134
You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`.
137
135
138
-
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:sambanova"`).
136
+
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:groq"`).
139
137
140
138
```python
141
139
import os
@@ -166,7 +164,7 @@ By default, our system automatically selects the fastest available provider for
166
164
167
165
You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`.
168
166
169
-
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:sambanova"`).
167
+
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:groq"`).
170
168
171
169
```python
172
170
import os
@@ -198,7 +196,7 @@ By default, our system automatically selects the fastest available provider for
198
196
199
197
You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`.
200
198
201
-
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:sambanova"`).
199
+
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:groq"`).
202
200
203
201
```python
204
202
import os
@@ -246,7 +244,7 @@ By default, our system automatically selects the fastest available provider for
246
244
247
245
You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`.
248
246
249
-
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:sambanova"`).
247
+
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:groq"`).
@@ -276,7 +274,7 @@ By default, our system automatically selects the fastest available provider for
276
274
277
275
You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`.
278
276
279
-
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:sambanova"`).
277
+
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:groq"`).
280
278
281
279
```javascript
282
280
importOpenAIfrom"openai";
@@ -309,7 +307,7 @@ By default, our system automatically selects the fastest available provider for
309
307
310
308
You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`.
311
309
312
-
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:sambanova"`).
310
+
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:groq"`).
313
311
314
312
```js
315
313
importfetchfrom"node-fetch";
@@ -348,7 +346,7 @@ By default, our system automatically selects the fastest available provider for
348
346
349
347
You can change the provider selection policy by appending a policy suffix to the model id: `:cheapest` for the most cost-efficient provider (lowest price per output token), or `:preferred` to follow your preference order in [Inference Provider settings](https://hf.co/settings/inference-providers). For example, `openai/gpt-oss-120b:cheapest`.
350
348
351
-
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:sambanova"`).
349
+
You can also select the provider of your choice by appending the provider name to the model id (e.g. `"openai/gpt-oss-120b:groq"`).
0 commit comments