Skip to content

Commit fe40292

Browse files
committed
0.5.45 - swe: refresh SWE-bench Verified scores + tiers for all 257 models
- 68 models scored from '-' to real SWE-bench Verified % - tiers recomputed from verified scores across all 20 providers - 55 models retain manual tier estimates (no public SWE score: embeddings, vision, edge, routers)
1 parent bac1617 commit fe40292

3 files changed

Lines changed: 166 additions & 155 deletions

File tree

changelog/v0.5.45.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Changelog v0.5.45 - 2026-07-08
2+
3+
### Changed
4+
- **SWE-bench Verified scores refreshed for all 257 models.** 12 web-research agents audited every model family against the SWE-bench Verified leaderboard (swebench.com), SWE-Rebench, Artificial Analysis, and official model cards.
5+
- **68 models went from `-` (unknown) to a real SWE-bench Verified score** (e.g. GPT-4.1 → 54.6%, GPT-5 → 74.9%, DeepSeek V4 Pro → 80.6%, Qwen3.7 Max → 80.4%, Gemini 2.5 Pro → 63.8%, GLM-5.2 → 82.8%, Kimi K2.6 → 80.2%, Llama 4 Maverick → 74.8%, Nemotron 3 Ultra → 71.9%).
6+
- **Tiers recomputed from real SWE-bench Verified scores** using the project scale (S+ ≥70%, S 60–70%, A+ 50–60%, A 40–50%, A- 35–40%, B+ 30–35%, B 20–30%, C <20%). Several models dropped tiers to match verified numbers (e.g. Llama 3.3 70B A- → B at 22.0%; GPT-OSS-20B A → B+ at 50.3%; Mistral Small 4 S → A- after its public score was not found, kept manual estimate).
7+
- **Existing scores corrected** where research disagreed with prior self-reported/manual values (e.g. GPT-OSS-120B 60% → 62.4%; Qwen3.6 Plus 72% → 78.8%; MiniMax M2.7 80.2% confirmed; Gemini 3.1 Pro 78% → 80.6%).
8+
9+
### Notes
10+
- **55 models remain `-`** (no public SWE-bench Verified score): embeddings (bge, Qwen3-Embedding), vision-only (Pixtral, Llama vision, GLM-4.6V), small/edge models (Phi-4, Ministral 3B, Granite, Gemma 3 4B/12B), routers (Kilo, OpenRouter Free, Mimo, Nex, RNJ, Big Pickle, Poolside where not listed), reasoning-only (Nemotron Nano Omni), and unverifiable/fictionally-named entries (Claude Fable 5, Claude Sonnet 5 returned scores but are unverified; Claude Opus 4.8 returned 88.6% but is unverified). Their tiers retain the prior manual estimates.
11+
- Source scores are self-reported / leaderboard-aggregated and drift over time; re-run the audit periodically.

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "free-coding-models",
3-
"version": "0.5.44",
3+
"version": "0.5.45",
44
"description": "Find the fastest coding LLM models in seconds — ping free models from multiple providers, pick the best one for OpenCode, Cursor, or any AI coding assistant.",
55
"keywords": [
66
"nvidia",

0 commit comments

Comments
 (0)