Skip to content

Commit 45524ce

Browse files
committed
docs(benchmark): simplify Claude-understated note to 'non-native unstable API'
Drop the bridge/CLI wording from the understated caveat in README, leaderboard and chart caption — it now just says the Claude models ran through a non-native, unstable API with a high tool-failure rate, so their scores are a lower bound. Chart subtitle updated to match.
1 parent 263fbfc commit 45524ce

4 files changed

Lines changed: 13 additions & 14 deletions

File tree

Docs/BENCHMARK_LEADERBOARD.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,10 @@ CLIs). Ranked by suite base score. Per-group scores are G1…G8; see the benchma
5353
> fails" on the agentic Claude CLI. The remaining rows (sol, terra, gpt-5.5, luna, sonnet) predate that fix
5454
> and would likely gain a few points on a re-run.
5555
>
56-
> ⚠️ **Claude scores are understated.** The Claude models ran over the CLI bridge, not a native API, and
57-
> behaved unstably there — a high tool-failure rate (opus ~24%, sonnet ~23%, fable ~36%, where well-formed
58-
> tool calls still fail to land) that cost points on tool correctness and task completion. Treat the Claude
59-
> rows as a **lower bound** on their real capability; a native-API run would score higher.
56+
> ⚠️ **Claude scores are understated.** The Claude models ran through a non-native, unstable API that gave
57+
> them a high tool-failure rate (opus ~24%, sonnet ~23%, fable ~36%, where well-formed tool calls still fail
58+
> to land) — costing points on tool correctness and task completion. Treat the Claude rows as a **lower
59+
> bound** on their real capability; a native API run would score higher.
6060
6161
## Historical Leaderboard — Suite v1.6
6262

Docs/Images/benchmark_v2_frontier.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,10 @@
1919
> `opus-4.8`, `fable-5` and `gpt-5.3-spark` were re-run after the `cli-agents` bridge gained a
2020
> no-tool-call recovery retry. Single run per model over the CLI bridge; speed is reported, never scored.
2121
>
22-
> ⚠️ **Claude scores are understated.** The Claude rows were driven over the CLI bridge, where a high
23-
> tool-failure rate (opus ~24%, sonnet ~23%, fable ~36% — well-formed calls that still fail to land)
24-
> cost points on tool correctness and task completion. Read them as a lower bound; a native-API run
25-
> would score higher. See [BENCHMARK_LEADERBOARD.md](../BENCHMARK_LEADERBOARD.md) for the full per-group (G1…G8) table.
22+
> ⚠️ **Claude scores are understated.** The Claude rows ran through a non-native, unstable API with a high
23+
> tool-failure rate (opus ~24%, sonnet ~23%, fable ~36% — well-formed calls that still fail to land) that
24+
> cost points on tool correctness and task completion. Read them as a lower bound; a native API run would
25+
> score higher. See [BENCHMARK_LEADERBOARD.md](../BENCHMARK_LEADERBOARD.md) for the full per-group (G1…G8) table.
2626
2727
```mermaid
2828
xychart-beta
-293 Bytes
Loading

README.md

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -104,12 +104,11 @@ CoreAI ships a game-creation benchmark: it scores how well an LLM builds and cha
104104

105105
One run each via the `cli-agents` bridge (indicative).
106106

107-
> ⚠️ **Claude scores are understated.** The Claude models (`sonnet-5`, `opus-4.8`, `fable-5`) were driven
108-
> over the `cli-agents` CLI bridge, not a native API. On that bridge they behaved unstably — a high
109-
> tool-failure rate (opus ~24%, sonnet ~23%, fable ~36% vs 1–13% for the Codex models) where
110-
> well-formed tool calls still failed to land. Those failures cost points on tool correctness and task
111-
> completion, so the Claude rows sit a few points below their true capability. Read them as a **lower
112-
> bound**, not a ceiling — a native-API run would score higher.
107+
> ⚠️ **Claude scores are understated.** The Claude models (`sonnet-5`, `opus-4.8`, `fable-5`) ran through
108+
> a non-native, unstable API that gave them a high tool-failure rate (opus ~24%, sonnet ~23%, fable ~36%
109+
> vs 1–13% for the Codex models) — well-formed tool calls that still failed to land. Those failures cost
110+
> points on tool correctness and task completion, so the Claude rows sit a few points below their true
111+
> capability. Read them as a **lower bound**, not a ceiling — a native API run would score higher.
113112
114113
Full tables (v2 + historical + local), methodology, group legend and the per-model castle prefabs: **[benchmark guide](Assets/CoreAIBenchmark/README.md)** · **[leaderboard](Docs/BENCHMARK_LEADERBOARD.md)**.
115114

0 commit comments

Comments
 (0)