Skip to content

Commit 4af62fb

Browse files
committed
benchmark: native vertical comparison chart, Claude-understated note, drop G7 screenshot
- Regenerate the frontier chart from the benchmark's own Build Model Comparison Report (vertical bar chart, 8 models ranked best-first) instead of a hand-drawn stand-in; sync tool-err numbers to the authoritative per-model run JSON (spark 35.5, opus 23.9, fable 36.1). - Note in README + leaderboard + caption that Claude scores are a lower bound: the CLI bridge gave them a high tool-failure rate (well-formed calls that fail to land), understating results. - G7 (comprehensive integration) no longer captures a scene screenshot — the Player/Gate/Key composite was unreadable and added nothing to the score. G6 stays the only hero image.
1 parent 087e94e commit 4af62fb

6 files changed

Lines changed: 30 additions & 7 deletions

File tree

Assets/CoreAIBenchmark/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ The benchmark answers the practical production question: is this model usable fo
2121
| G7 - Comprehensive integration | 5/5 (hardest) | World-building and Lua logic staying cross-consistent in one session | A key-and-gate puzzle: spawn Player/Gate/Key to an exact plan, install a `key_found` proximity slot, and keep the spawned world and the logic describing it in agreement end to end. |
2222
| G8 - Described-state selection | 4/5 | Reasoning over a GIVEN textual world state, not live scene sensing | The prompt describes an already-populated scene; the model selects named objects, clears only junk, raises only undersized towers, and encodes the described wave-scaling rule as Lua. It is a single-turn conditional-selection test, not sustained multi-turn recovery. |
2323

24-
Groups are implemented as real PlayMode benchmark scenarios under `Assets/CoreAIBenchmark/Tests/PlayMode/Benchmarks`. G6 (hero image) and G7 capture scene screenshots because the built scene is part of the result; the other groups are primarily graded through logic execution and traces.
24+
Groups are implemented as real PlayMode benchmark scenarios under `Assets/CoreAIBenchmark/Tests/PlayMode/Benchmarks`. Only G6 (the free-build castle) captures a hero scene screenshot, because the built scene is itself the result; all other groups — including G7 — are graded purely through world state, logic execution and tool-call traces, so no screenshot is taken for them.
2525

2626
## Scoring
2727

Assets/CoreAIBenchmark/Tests/PlayMode/Benchmarks/GameIntegrationScenarioG7.cs

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,11 @@ private abstract class G7Scenario : GameBenchmarkScenario
3535
{
3636
public sealed override string Group => "G7";
3737
public override int Difficulty => 9;
38-
public override bool CaptureScene => true;
38+
// No scene shot for G7: the Player/Gate/Key puzzle photographs as a cluttered composite of
39+
// overlapping primitives and floating world-space labels that is unreadable and adds nothing
40+
// to the score (user-reported). Grading reads world state + Lua consistency, not the image.
41+
// G6 stays the only hero image; scoring is unaffected.
42+
public override bool CaptureScene => false;
3943
public override int? RepsOverride => 1;
4044
public override int TokenBudget => 2500;
4145
public override double TimeBudgetMs => 40000;

Docs/BENCHMARK_LEADERBOARD.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,11 @@ CLIs). Ranked by suite base score. Per-group scores are G1…G8; see the benchma
5252
> model describes an action in prose but emits no tool call, the biggest source of "0 tool calls → scenario
5353
> fails" on the agentic Claude CLI. The remaining rows (sol, terra, gpt-5.5, luna, sonnet) predate that fix
5454
> and would likely gain a few points on a re-run.
55+
>
56+
> ⚠️ **Claude scores are understated.** The Claude models ran over the CLI bridge, not a native API, and
57+
> behaved unstably there — a high tool-failure rate (opus ~24%, sonnet ~23%, fable ~36%, where well-formed
58+
> tool calls still fail to land) that cost points on tool correctness and task completion. Treat the Claude
59+
> rows as a **lower bound** on their real capability; a native-API run would score higher.
5560
5661
## Historical Leaderboard — Suite v1.6
5762

Docs/Images/benchmark_v2_frontier.md

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,21 @@
88
|---:|---|---:|---:|---|---:|---:|
99
| 1 | `gpt-5.6-sol` | **96.6** | 85.7% | 24/3/1 | 2.8% | 16379 |
1010
| 2 | `gpt-5.6-terra` | **93.0** | 86.2% | 25/2/2 | 1.5% | 32194 |
11-
| 3 | `gpt-5.3-spark` | **92.9** | 79.3% | 23/5/1 | 7.1% | 17616 |
11+
| 3 | `gpt-5.3-spark` | **92.9** | 79.3% | 23/5/1 | 35.5% | 17616 |
1212
| 4 | `gpt-5.5` | **90.3** | 82.8% | 24/2/3 | 13.5% | 17188 |
1313
| 5 | `gpt-5.6-luna` | **88.1** | 79.3% | 23/2/4 | 9.2% | 18426 |
1414
| 6 | `claude-sonnet-5` | **86.2** | 75.9% | 22/4/3 | 23.2% | 18250 |
15-
| 7 | `claude-opus-4.8` | **83.2** | 79.3% | 23/2/4 | 22.0% | 26250 |
16-
| 8 | `claude-fable-5` | **81.4** | 75.9% | 22/3/4 | 32.1% | 22360 |
15+
| 7 | `claude-opus-4.8` | **83.2** | 79.3% | 23/2/4 | 23.9% | 26250 |
16+
| 8 | `claude-fable-5` | **81.4** | 75.9% | 22/3/4 | 36.1% | 22360 |
1717

18+
> Generated by `CoreAI/Benchmarks/Build Model Comparison Report` from the newest per-model run JSON.
1819
> `opus-4.8`, `fable-5` and `gpt-5.3-spark` were re-run after the `cli-agents` bridge gained a
1920
> no-tool-call recovery retry. Single run per model over the CLI bridge; speed is reported, never scored.
20-
> See [BENCHMARK_LEADERBOARD.md](../BENCHMARK_LEADERBOARD.md) for the full per-group (G1…G8) table.
21+
>
22+
> ⚠️ **Claude scores are understated.** The Claude rows were driven over the CLI bridge, where a high
23+
> tool-failure rate (opus ~24%, sonnet ~23%, fable ~36% — well-formed calls that still fail to land)
24+
> cost points on tool correctness and task completion. Read them as a lower bound; a native-API run
25+
> would score higher. See [BENCHMARK_LEADERBOARD.md](../BENCHMARK_LEADERBOARD.md) for the full per-group (G1…G8) table.
2126
2227
```mermaid
2328
xychart-beta
1.43 KB
Loading

README.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,16 @@ CoreAI ships a game-creation benchmark: it scores how well an LLM builds and cha
102102
| 7 | `claude-opus-4.8` | **83.2** | 79.3% |
103103
| 8 | `claude-fable-5` | **81.4** | 75.9% |
104104

105-
One run each via the `cli-agents` bridge (indicative). The Claude rows are held back by a high bridge tool-failure rate, not raw capability — details in the guide. Full tables (v2 + historical + local), methodology, group legend and the per-model castle prefabs: **[benchmark guide](Assets/CoreAIBenchmark/README.md)** · **[leaderboard](Docs/BENCHMARK_LEADERBOARD.md)**.
105+
One run each via the `cli-agents` bridge (indicative).
106+
107+
> ⚠️ **Claude scores are understated.** The Claude models (`sonnet-5`, `opus-4.8`, `fable-5`) were driven
108+
> over the `cli-agents` CLI bridge, not a native API. On that bridge they behaved unstably — a high
109+
> tool-failure rate (opus ~24%, sonnet ~23%, fable ~36% vs 1–13% for the Codex models) where
110+
> well-formed tool calls still failed to land. Those failures cost points on tool correctness and task
111+
> completion, so the Claude rows sit a few points below their true capability. Read them as a **lower
112+
> bound**, not a ceiling — a native-API run would score higher.
113+
114+
Full tables (v2 + historical + local), methodology, group legend and the per-model castle prefabs: **[benchmark guide](Assets/CoreAIBenchmark/README.md)** · **[leaderboard](Docs/BENCHMARK_LEADERBOARD.md)**.
106115

107116
---
108117

0 commit comments

Comments
 (0)