Skip to content

Commit d635c22

Browse files
mythzmythz
committed
Add check results
1 parent a7b4b54 commit d635c22

1 file changed

Lines changed: 311 additions & 0 deletions

File tree

‎docs/checks/2025-10-27.txt‎

Lines changed: 311 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,311 @@
1+
LLM Provider Test Results - https://github.com/ServiceStack/llms/actions/runs/18846268606
2+
=========================
3+
Date: 2025-10-27 15:22:42 UTC
4+
llms-py version: 2.0.19
5+
6+
Testing provider: openrouter_free
7+
-----------------------------------
8+
Status: PASS
9+
10+
Checking 27 models for provider 'openrouter_free':
11+
12+
✓ qwen2.5vl (1461ms)
13+
✓ llama4:109b (792ms)
14+
✓ llama4:400b (1384ms)
15+
✓ deepseek-v3.1:671b (2352ms)
16+
✗ deepseek-r1:671b 429 Provider returned error - deepseek/deepseek-r1-0528:free is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations
17+
✗ gemini-2.0-flash 429 Provider returned error - google/gemini-2.0-flash-exp:free is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations
18+
✓ glm-4.5-air (3527ms)
19+
✓ mai-ds-r1 (2972ms)
20+
✓ llama3.3:70b (794ms)
21+
✓ nemotron-nano:9b (2433ms)
22+
✓ deepseek-r1-distill-llama:70b (2096ms)
23+
✓ gpt-oss:20b (95416ms)
24+
✗ mistral-small3.2:24b 503 Provider returned error - {"detail":"No instances available (yet) for chute_id='8cac9bf4-d7e2-55eb-b4bb-1416b7f57131'"}
25+
✓ mistral-nemo:12b (2503ms)
26+
✓ devstral-small (985ms)
27+
✗ venice-uncensored:24b 429 Provider returned error - cognitivecomputations/dolphin-mistral-24b-venice-edition:free is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations
28+
✓ llama3.3:8b (594ms)
29+
✓ kimi-dev:72b (671ms)
30+
✓ gemma3:27b (935ms)
31+
✗ qwen3-coder 429 Provider returned error - qwen/qwen3-coder:free is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations
32+
✗ qwen3:235b 429 Provider returned error - qwen/qwen3-235b-a22b:free is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations
33+
✓ qwen3:30b (1444ms)
34+
✓ qwen3:8b (4905ms)
35+
✗ qwen3:4b 429 Provider returned error - qwen/qwen3-4b:free is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations
36+
✓ qwen2.5vl:32b (1958ms)
37+
✓ qwen-2.5:72b (1364ms)
38+
✓ minimax-m2 (1025ms)
39+
40+
41+
Testing provider: groq
42+
-----------------------------------
43+
Status: PASS
44+
45+
Checking 14 models for provider 'groq':
46+
47+
✓ allam-2-7b (471ms)
48+
✓ compound (236ms)
49+
✓ compound-mini (412ms)
50+
✓ llama3.1:8b (141ms)
51+
✓ llama3.3:70b (146ms)
52+
✓ llama4:109b (91ms)
53+
✓ llama4:400b (104ms)
54+
✓ llama-guard-4:12b (172ms)
55+
✓ llama-prompt-guard2:22m (147ms)
56+
✓ llama-prompt-guard2:86m (160ms)
57+
✓ kimi-k2 (157ms)
58+
✓ gpt-oss:120b (125ms)
59+
✓ gpt-oss:20b (81ms)
60+
✓ qwen3:32b (116ms)
61+
62+
63+
Testing provider: codestral
64+
-----------------------------------
65+
Status: PASS
66+
67+
Checking 1 model for provider 'codestral':
68+
69+
✓ codestral:22b (408ms)
70+
71+
72+
Testing provider: openrouter
73+
-----------------------------------
74+
Status: PASS
75+
76+
Checking 54 models for provider 'openrouter':
77+
78+
✓ llama3.1:70b (466ms)
79+
✓ llama3.3:70b (450ms)
80+
✓ phi-4:14b (2930ms)
81+
✓ codestral:22b (500ms)
82+
✓ mistral-nemo:12b (594ms)
83+
✓ mixtral:8x22b (1822ms)
84+
✓ mistral-small3.2:24b (658ms)
85+
✓ nova-micro (721ms)
86+
✓ nova-lite (714ms)
87+
✓ nova-pro (681ms)
88+
✓ claude-opus-4-1 (3292ms)
89+
✓ claude-sonnet-4-5 (5626ms)
90+
✓ claude-sonnet-4-0 (2119ms)
91+
✓ gpt-5 (3758ms)
92+
✓ gpt-5-chat (605ms)
93+
✓ gpt-5-mini (7159ms)
94+
✓ gpt-5-nano (1977ms)
95+
✓ gpt-5-codex (1798ms)
96+
✓ gpt-oss:120b (634ms)
97+
✓ gpt-oss:20b (666ms)
98+
✓ gpt-4o (623ms)
99+
✓ gpt-4o-mini (661ms)
100+
✓ gpt-4.1 (1158ms)
101+
✓ gpt-4.1-mini (1810ms)
102+
✓ gpt-4.1-nano (904ms)
103+
✓ o4-mini (2316ms)
104+
✓ o4-mini-high (3053ms)
105+
✓ grok-4 (3763ms)
106+
✓ grok-4-fast (666ms)
107+
✓ grok-code-fast-1 (12287ms)
108+
✓ glm-4.6 (957ms)
109+
✓ glm-4.5v (579ms)
110+
✓ glm-4.5 (1457ms)
111+
✓ glm-4.5-air (1034ms)
112+
✓ kimi-k2 (955ms)
113+
✓ deepseek-v3.1:671b (676ms)
114+
✓ deepseek-v3.2-exp (1878ms)
115+
✓ deepseek-chat-v3.1:671b (1800ms)
116+
✓ deepseek-r1:671b (25181ms)
117+
✓ deepseek-v3.1-terminus (898ms)
118+
✓ qwen3:8b (918ms)
119+
✓ qwen3:30b (834ms)
120+
✓ qwen3:32b (992ms)
121+
✓ qwen3:235b (1797ms)
122+
✓ qwen3-coder (5309ms)
123+
✓ qwen3-coder-flash (1329ms)
124+
✓ qwen3-coder-plus (1840ms)
125+
✓ qwen3-coder:30b (1085ms)
126+
✓ qwen3-max (1233ms)
127+
✓ qwen3-vl:235b (3813ms)
128+
✓ qwen3-vl-thinking:235b (1768ms)
129+
✓ ling-1t (9809ms)
130+
✓ llama4:109b (977ms)
131+
✓ llama4:400b (449ms)
132+
133+
134+
Testing provider: google
135+
-----------------------------------
136+
Status: PASS
137+
138+
Checking 5 models for provider 'google':
139+
140+
✓ gemini-flash-latest (2310ms)
141+
✓ gemini-flash-lite-latest (1452ms)
142+
✗ gemini-2.5-pro 503 The model is overloaded. Please try again later.
143+
✓ gemini-2.5-flash (2680ms)
144+
✓ gemini-2.5-flash-lite (1484ms)
145+
146+
147+
Testing provider: anthropic
148+
-----------------------------------
149+
Status: PASS
150+
151+
Checking 6 models for provider 'anthropic':
152+
153+
✓ claude-sonnet-4-5 (2368ms)
154+
✓ claude-sonnet-4-0 (2435ms)
155+
✓ claude-3-7-sonnet (689ms)
156+
✓ claude-haiku-4-5 (1785ms)
157+
✓ claude-3-5-haiku (1019ms)
158+
✓ claude-3-haiku (542ms)
159+
160+
161+
Testing provider: openai
162+
-----------------------------------
163+
Status: PASS
164+
165+
Checking 17 models for provider 'openai':
166+
167+
✓ gpt-5-nano (2130ms)
168+
✓ gpt-5-mini (2101ms)
169+
✓ gpt-5 (1213ms)
170+
✓ gpt-5-chat-latest (722ms)
171+
✓ gpt-4.1-nano (472ms)
172+
✓ gpt-4.1-mini (431ms)
173+
✓ gpt-4.1 (458ms)
174+
✓ o4-mini (946ms)
175+
✓ o3 (1311ms)
176+
✓ gpt-4o-mini-search-preview (1211ms)
177+
✓ gpt-4o-search-preview (2286ms)
178+
✓ o3-mini (1817ms)
179+
✓ o1-mini (1200ms)
180+
✓ chatgpt-4o-latest (763ms)
181+
✓ gpt-4o-mini (1386ms)
182+
✓ gpt-4o (836ms)
183+
✓ gpt-3.5-turbo (421ms)
184+
185+
186+
Testing provider: grok
187+
-----------------------------------
188+
Status: PASS
189+
190+
Checking 7 models for provider 'grok':
191+
192+
✓ grok-4 (5181ms)
193+
✓ grok-code-fast-1 (2298ms)
194+
✓ grok-4-fast-reasoning (520ms)
195+
✓ grok-4-fast-non-reasoning (381ms)
196+
✓ grok-4-0709 (4032ms)
197+
✓ grok-3-mini (3286ms)
198+
✓ grok-3 (512ms)
199+
200+
201+
Testing provider: qwen
202+
-----------------------------------
203+
Status: PASS
204+
205+
Checking 26 models for provider 'qwen':
206+
207+
✓ qwen3-max (2114ms)
208+
✓ qwen-max (1499ms)
209+
✓ qwen-plus (1621ms)
210+
✓ qwen-flash (1541ms)
211+
✓ qwq-plus (7567ms)
212+
✓ qwen-turbo (1504ms)
213+
✓ qwen-omni-turbo (1188ms)
214+
✓ qwen3-omni-flash (1658ms)
215+
✓ qwen3-coder-plus (1597ms)
216+
✓ qwen3-coder-flash (1770ms)
217+
✓ qwen3-next:80b (1005ms)
218+
✓ qwen3:235b (1344ms)
219+
✓ qwen3:32b (1495ms)
220+
✓ qwen3:30b (1673ms)
221+
✓ qwen3:14b (1545ms)
222+
✓ qwen3:8b (1308ms)
223+
✓ qwen3:4b (1500ms)
224+
✓ qwen3:1.7b (1217ms)
225+
✓ qwen3:0.6b (1227ms)
226+
✓ qwen3-coder (1411ms)
227+
✓ qwen3-coder:30b (1541ms)
228+
✓ qwen3-vl-plus (3576ms)
229+
✓ qwen3-vl-thinking:235b (6104ms)
230+
✓ qwen3-vl:235b (2027ms)
231+
✓ qwen3-vl:30b (1562ms)
232+
✓ qwen2.5-omni:7b (1557ms)
233+
234+
235+
Testing provider: z.ai
236+
-----------------------------------
237+
Status: PASS
238+
239+
Checking 7 models for provider 'z.ai':
240+
241+
✓ glm-4.6 (30598ms)
242+
✓ glm-4.5 (13676ms)
243+
✓ glm-4.5-air (10571ms)
244+
✓ glm-4.5-x (15580ms)
245+
✓ glm-4.5-airx (4563ms)
246+
✓ glm-4.5-flash (16433ms)
247+
✓ glm-4:32b (1233ms)
248+
249+
250+
Testing provider: qwen
251+
-----------------------------------
252+
Status: PASS
253+
254+
Checking 26 models for provider 'qwen':
255+
256+
✓ qwen3-max (1764ms)
257+
✓ qwen-max (1446ms)
258+
✓ qwen-plus (1690ms)
259+
✓ qwen-flash (1493ms)
260+
✓ qwq-plus (4742ms)
261+
✓ qwen-turbo (1867ms)
262+
✓ qwen-omni-turbo (1369ms)
263+
✓ qwen3-omni-flash (1811ms)
264+
✓ qwen3-coder-plus (1539ms)
265+
✓ qwen3-coder-flash (1582ms)
266+
✓ qwen3-next:80b (1107ms)
267+
✓ qwen3:235b (1285ms)
268+
✓ qwen3:32b (1308ms)
269+
✓ qwen3:30b (1487ms)
270+
✓ qwen3:14b (1109ms)
271+
✓ qwen3:8b (1140ms)
272+
✓ qwen3:4b (1042ms)
273+
✓ qwen3:1.7b (1046ms)
274+
✓ qwen3:0.6b (1007ms)
275+
✓ qwen3-coder (1458ms)
276+
✓ qwen3-coder:30b (1640ms)
277+
✓ qwen3-vl-plus (3729ms)
278+
✓ qwen3-vl-thinking:235b (4998ms)
279+
✓ qwen3-vl:235b (1927ms)
280+
✓ qwen3-vl:30b (1129ms)
281+
✓ qwen2.5-omni:7b (1024ms)
282+
283+
284+
Testing provider: mistral
285+
-----------------------------------
286+
Status: PASS
287+
288+
Checking 18 models for provider 'mistral':
289+
290+
✓ mistral-medium (566ms)
291+
✓ magistral-medium (2067ms)
292+
✓ devstral-medium (714ms)
293+
✓ codestral:22b (542ms)
294+
✗ mistral-small3.2:24b Service tier capacity exceeded for this model.
295+
✓ magistral-small (871ms)
296+
✓ devstral-small (408ms)
297+
✓ voxtral-small (349ms)
298+
✓ voxtral-mini (283ms)
299+
✓ mistral-large:123b (384ms)
300+
✓ pixtral-large:124b (1228ms)
301+
✓ pixtral:12b (383ms)
302+
✓ mistral-nemo:12b (373ms)
303+
✓ mistral:7b (321ms)
304+
✓ mixtral:8x7b (498ms)
305+
✓ mixtral:8x22b (953ms)
306+
✓ ministral:8b (242ms)
307+
✓ ministral:3b (244ms)
308+
309+
310+
=========================
311+
Test run completed

0 commit comments

Comments
 (0)