Skip to content

Commit f8ee994

Browse files
committed
docs(benchmark): re-run opus/fable/spark on the retry-improved bridge
After adding the no-tool-call recovery retry to the cli-agents bridge, re-ran the models most affected by prose-instead-of-tool-call losses. Suite scores recovered: claude-opus-4.8 79.7 → 83.2, claude-fable-5 78.9 → 81.4, gpt-5.3-spark 91.0 → 92.9. Updated the root README top-tier table, the benchmark guide, the community leaderboard, the comparison chart, and added a bridge note that sol/terra/gpt-5.5/luna/sonnet predate the fix.
1 parent 731c993 commit f8ee994

4 files changed

Lines changed: 21 additions & 10 deletions

File tree

Assets/CoreAIBenchmark/README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -229,12 +229,12 @@ LM Studio models above (different suite version *and* different backend class).
229229
|---:|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
230230
| 1 | `codex/gpt-5.6-sol` | **96.6** | 85.7% | 24/3/1 | 96.3 | 100 | 100 | 100 | 100 | 63.9 | 100 | 96.3 | 16379 |
231231
| 2 | `codex/gpt-5.6-terra` | **93.0** | 86.2% | 25/2/2 | 95.6 | 100 | 84.2 | 97.7 | 99.7 | 62.9 | 100 | 96.3 | 32194 |
232-
| 3 | `codex/gpt-5.3-spark` | **91.0** | 82.8% | 24/3/2 | 69.7 | 100 | 100 | 100 | 99 | 50 | 100 | 78.5 | 31415 |
232+
| 3 | `codex/gpt-5.3-spark` | **92.9** | 79.3% | 23/5/1 | 89.9 | 98.7 | 85.2 | 97.7 | 98.7 | 75.4 | 100 | 94.2 | 17616 |
233233
| 4 | `codex/gpt-5.5` | **90.3** | 82.8% | 24/2/3 | 96.3 | 100 | 95.3 | 93 | 80 | 55.6 | 100 | 96.3 | 17188 |
234234
| 5 | `codex/gpt-5.6-luna` | **88.1** | 79.3% | 23/2/4 | 91.9 | 100 | 100 | 97.7 | 87.7 | 25.2 | 100 | 69.7 | 18426 |
235235
| 6 | `claude-sonnet-5` | **86.2** | 75.9% | 22/4/3 | 91.9 | 95.7 | 94.1 | 93 | 70 | 50 | 93.7 | 96.3 | 18250 |
236-
| 7 | `claude-opus-4.8` | **79.7** | 75.9% | 22/2/5 | 54.9 | 94.3 | 78.2 | 93 | 100 | 49 | 100 | 42.6 | 19851 |
237-
| 8 | `claude-fable-5` | **78.9** | 72.4% | 21/3/5 | 55.7 | 94.3 | 93 | 93 | 100 | 46 | 93.7 | 9.2 | 21980 |
236+
| 7 | `claude-opus-4.8` | **83.2** | 79.3% | 23/2/4 | 65.9 | 95.7 | 93 | 93 | 100 | 48 | 93.7 | 36.4 | 26250 |
237+
| 8 | `claude-fable-5` | **81.4** | 75.9% | 22/3/4 | 81.5 | 94.3 | 78.4 | 93 | 100 | 46 | 93.7 | 36.4 | 22360 |
238238

239239
**Read of the sweep.** The GPT-5.6 family tops the board — `gpt-5.6-sol` is the most consistent (only one
240240
failed scenario; perfect G3/G4/G5/G7), and `gpt-5.6-terra` has the most passes (25/29, 86.2%). `gpt-5.5`
@@ -250,6 +250,11 @@ Claude's native tool-calling as cleanly as the Codex CLI does. For reference, th
250250
v1.6 run scored opus 92.9 / sonnet 88.2 / fable 89.5 — far above this bridge sweep — so read the Claude rows
251251
as "Claude-CLI-via-bridge", not the models' native tool-calling.
252252

253+
The bridge has since gained a **no-tool-call recovery retry** (it re-prompts when a model describes an action
254+
in prose but emits no tool call). Re-running the affected models on it recovered several points:
255+
`opus-4.8` 79.7 → **83.2**, `fable-5` 78.9 → **81.4**, and even `gpt-5.3-spark` 91.0 → **92.9**. The
256+
`sol`/`terra`/`gpt-5.5`/`luna`/`sonnet` rows predate the retry and would likely gain a little on a re-run.
257+
253258
**Why G6 (free-build castle) is everyone's worst, and the vision variant here means little.** The unbounded
254259
creative build is genuinely the hardest scenario, but two measurement issues inflate how bad it looks: (1)
255260
this sweep ran with the G6 **image-feedback (vision) variant enabled**, and the CLI bridge is text + tool-calls

Docs/BENCHMARK_LEADERBOARD.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,18 +34,24 @@ CLIs). Ranked by suite base score. Per-group scores are G1…G8; see the benchma
3434
|---:|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
3535
| 1 | `codex/gpt-5.6-sol` | **96.6** | 85.7% | 24/3/1 | 96.3 | 100 | 100 | 100 | 100 | 63.9 | 100 | 96.3 | 16379 | maintainer |
3636
| 2 | `codex/gpt-5.6-terra` | **93.0** | 86.2% | 25/2/2 | 95.6 | 100 | 84.2 | 97.7 | 99.7 | 62.9 | 100 | 96.3 | 32194 | maintainer |
37-
| 3 | `codex/gpt-5.3-spark` | **91.0** | 82.8% | 24/3/2 | 69.7 | 100 | 100 | 100 | 99 | 50 | 100 | 78.5 | 31415 | maintainer |
37+
| 3 | `codex/gpt-5.3-spark` | **92.9** | 79.3% | 23/5/1 | 89.9 | 98.7 | 85.2 | 97.7 | 98.7 | 75.4 | 100 | 94.2 | 17616 | maintainer |
3838
| 4 | `codex/gpt-5.5` | **90.3** | 82.8% | 24/2/3 | 96.3 | 100 | 95.3 | 93 | 80 | 55.6 | 100 | 96.3 | 17188 | maintainer |
3939
| 5 | `codex/gpt-5.6-luna` | **88.1** | 79.3% | 23/2/4 | 91.9 | 100 | 100 | 97.7 | 87.7 | 25.2 | 100 | 69.7 | 18426 | maintainer |
4040
| 6 | `claude-sonnet-5` | **86.2** | 75.9% | 22/4/3 | 91.9 | 95.7 | 94.1 | 93 | 70 | 50 | 93.7 | 96.3 | 18250 | maintainer |
41-
| 7 | `claude-opus-4.8` | **79.7** | 75.9% | 22/2/5 | 54.9 | 94.3 | 78.2 | 93 | 100 | 49 | 100 | 42.6 | 19851 | maintainer |
42-
| 8 | `claude-fable-5` | **78.9** | 72.4% | 21/3/5 | 55.7 | 94.3 | 93 | 93 | 100 | 46 | 93.7 | 9.2 | 21980 | maintainer |
41+
| 7 | `claude-opus-4.8` | **83.2** | 79.3% | 23/2/4 | 65.9 | 95.7 | 93 | 93 | 100 | 48 | 93.7 | 36.4 | 26250 | maintainer |
42+
| 8 | `claude-fable-5` | **81.4** | 75.9% | 22/3/4 | 81.5 | 94.3 | 78.4 | 93 | 100 | 46 | 93.7 | 36.4 | 22360 | maintainer |
4343

4444
> Single run per model (reps=1), streaming off, native tools on, temperature 0.1, Unity 6000.3.14f1. Run over
4545
> the CLI bridge, so absolute tokens/time are not comparable to a direct-API submission (tool-calling was
4646
> verified equivalent to a native OpenAI backend first). The G6 image-feedback (vision) variant did not run —
4747
> the CLI bridge is text + tool-calls only. Indicative single-shot results, not a controlled multi-rep A/B.
4848
> A ranked public submission still requires the artifacts and hardware disclosure below.
49+
>
50+
> **Bridge note:** `claude-opus-4.8` (79.7 → **83.2**), `claude-fable-5` (78.9 → **81.4**) and `gpt-5.3-spark`
51+
> (91.0 → **92.9**) were re-run after the bridge gained a *no-tool-call recovery retry* — it re-prompts when a
52+
> model describes an action in prose but emits no tool call, the biggest source of "0 tool calls → scenario
53+
> fails" on the agentic Claude CLI. The remaining rows (sol, terra, gpt-5.5, luna, sonnet) predate that fix
54+
> and would likely gain a few points on a re-run.
4955
5056
## Historical Leaderboard — Suite v1.6
5157

Lines changed: 1 addition & 1 deletion
Loading

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -95,12 +95,12 @@ CoreAI ships a game-creation benchmark: it scores how well an LLM builds and cha
9595
|---:|---|---:|---:|
9696
| 1 | `gpt-5.6-sol` | **96.6** | 85.7% |
9797
| 2 | `gpt-5.6-terra` | **93.0** | 86.2% |
98-
| 3 | `gpt-5.3-spark` | **91.0** | 82.8% |
98+
| 3 | `gpt-5.3-spark` | **92.9** | 79.3% |
9999
| 4 | `gpt-5.5` | **90.3** | 82.8% |
100100
| 5 | `gpt-5.6-luna` | **88.1** | 79.3% |
101101
| 6 | `claude-sonnet-5` | **86.2** | 75.9% |
102-
| 7 | `claude-opus-4.8` | **79.7** | 75.9% |
103-
| 8 | `claude-fable-5` | **78.9** | 72.4% |
102+
| 7 | `claude-opus-4.8` | **83.2** | 79.3% |
103+
| 8 | `claude-fable-5` | **81.4** | 75.9% |
104104

105105
One run each via the `cli-agents` bridge (indicative). The Claude rows are held back by a high bridge tool-failure rate, not raw capability — details in the guide. Full tables (v2 + historical + local), methodology, group legend and the per-model castle prefabs: **[benchmark guide](Assets/CoreAIBenchmark/README.md)** · **[leaderboard](Docs/BENCHMARK_LEADERBOARD.md)**.
106106

0 commit comments

Comments
 (0)