Skip to content

Commit 087e94e

Browse files
committed
docs: fix accuracy drift found in documentation audit
- benchmark README: package graph 5.4.0 -> 5.5.0 (matches package.json lockstep) - BENCHMARK_DESIGN.md: bonus is correctness-only; speed reported never scored (was still describing a token/time efficiency bonus removed from GoalScore) - Docs/Images/benchmark_v2_frontier.md: stale 7-model pre-fix table + broken COMPARISON.svg link -> 8-model post-fix table referencing benchmark_v2_frontier.png - README: label suite as v1.7 (benchmark v2 prompts), not 'suite v2'; LLM pipeline diagram now shows Timeout/Logging/RetryingStreaming decorator chain - lua-migration.md + coreai-mods-and-hub-plan.md: mark the MoonSharp->LuaCs VM swap as completed in 5.4.0 (historical), so present-tense wording no longer misleads
1 parent 4f3f548 commit 087e94e

6 files changed

Lines changed: 46 additions & 31 deletions

File tree

Assets/CoreAIBenchmark/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# CoreAI Game-Creation Benchmark
22

3-
Package graph: `com.neoxider.coreaibenchmark` 5.4.0 depends on `com.neoxider.coreai`,
3+
Package graph: `com.neoxider.coreaibenchmark` 5.5.0 depends on `com.neoxider.coreai`,
44
`com.neoxider.coreaiunity`, and `com.neoxider.coreaimods` at the same version. The Mods dependency is
55
required because the scenarios instantiate and execute the real Lua tool/runtime.
66

Assets/CoreAIBenchmark/Tests/PlayMode/Benchmarks/BENCHMARK_DESIGN.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A scored benchmark that measures how well a model **builds a game** with CoreAI — driving the
44
real `execute_lua` and `world_command` tools — rather than how well it chats. Each scenario is
5-
graded **0..100** (with a bounded efficiency bonus that can push the displayed total above 100),
5+
graded **0..100** (with a bounded correctness bonus that can push the displayed total above 100),
66
so runs are comparable across models and over time.
77

88
## Scoring model (portable core)
@@ -16,10 +16,10 @@ under dotnet (`GoalScoreEditModeTests`). Summary:
1616
(harness-level), plus scenario-specific penalties (over-building, disallowed actions).
1717
- **Hard caps**: incomplete run (timeout/fault) or final-state failure → cap 60; tool never fired
1818
(prose only) → cap 40.
19-
- **Bonus** 0..20, **gated on base ≥ 90** (only solvers earn it), composed of:
20-
- *correctness/robustness* (the scenario's requested bonus), and
21-
- *efficiency* **fewer tokens** (up to 6) and **less time** (up to 6) than the scenario's budget;
22-
the further under budget, the more points. Total bonus is capped at 20.
19+
- **Bonus** 0..20, **gated on base ≥ 90** (only solvers earn it): the scenario's requested
20+
*correctness/robustness* bonus — being **more right** than the pass bar. **Correctness only.**
21+
Token/time efficiency is **reported as a tok/s metric, never scored**, so a merely-faster model
22+
cannot look smarter than an equally-correct but slower one. Total bonus is capped at 20.
2323
- `Total = Base + Bonus`; suites are compared on **Base** only.
2424
- Verdict: **PASS** (base ≥ 90 and all mandatory checkpoints) / **PARTIAL** (50–89) / **FAIL** (< 50).
2525
- Non-finite inputs are sanitized so a single NaN can never poison a score.
@@ -54,13 +54,13 @@ cannot call tools reads clearly as 'Not suitable for agentic roles'. Shown in th
5454

5555
The Markdown report is structured for quick reading and reuse:
5656
**embedded SVG results card****scorecard** (model, suite score, grade, verdict, generation tok/s,
57-
speed/efficiency bonus) → **dimension bars + a Mermaid chart****tool-call statistics** (counts /
57+
speed reported not scored) → **dimension bars + a Mermaid chart****tool-call statistics** (counts /
5858
failures / invalid, before the session) → **scenario scores****failed checkpoints****full model
5959
session** (the complete per-turn transcript, at the end).
6060

6161
Speed is reported honestly as **generation tokens/sec** (completion tokens ÷ wall-clock); prompt tokens
62-
are excluded because they are prefilled far faster than generated. The efficiency bonus is split into a
63-
**token** part and a **time** part so the reward for being fast and cheap is explicit.
62+
are excluded because they are prefilled far faster than generated. Speed is **reported, never scored**
63+
a token/time figure in the report, not points in the bonus — so a faster model never looks smarter.
6464

6565
Reports are machine-readable too: the JSON carries the summary, per-dimension scores, tool stats, and
6666
per-checkpoint dimensions, so **`Build Model Comparison Report`** can parse many runs and emit

Assets/dev/docs/coreai-mods-and-hub-plan.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,13 @@
11
# CoreAIMods extraction + CoreAI Hub (pages) — implementation plan
22

3-
Status: active. This document is the repo-side plan for (1) extracting the whole Lua layer into a
4-
separate `com.neoxider.coreaimods` package, (2) the extensible **CoreAI Hub** page system, and
3+
Status: active (planning). This document is the repo-side plan for (1) extracting the whole Lua layer
4+
into a separate `com.neoxider.coreaimods` package, (2) the extensible **CoreAI Hub** page system, and
55
(3) mod features (bundled/updatable mods, categories). Companion spec: `Docs/coreai-mod-system.md`.
66

7+
> Note (2026-07): the MoonSharp → Lua-CSharp VM swap referenced below is **already complete** (shipped
8+
> in 5.4.0; MoonSharp fully removed, LuaCs is the only VM). Any present-tense "MoonSharp" wording in
9+
> this plan is historical — only the Hub/mod-feature items remain forward-looking.
10+
711
## Goals
812

913
1. **Dependency inversion.** All Lua (sandbox, `execute_lua`/formulas, mod runtime, stores, bindings,

Docs/CoreAIMods/lua-migration.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,14 @@
11
# CoreAIMods VM migration (MoonSharp → Lua-CSharp) + mod-feature polish — working plan
22

3-
> Status: DRAFT v2 (repo-state snapshot refreshed 2026-07-10). This is a migration plan, not a claim
4-
> that Lua-CSharp is already the production VM.
5-
> All code, docs, commit messages in English. MoonSharp stays live until Phase 2 is green.
3+
> **ARCHIVED / HISTORICAL (completed).** This migration shipped in **5.4.0**: MoonSharp was fully
4+
> removed and Lua-CSharp (LuaCs) is now the only, production Lua VM. This document is kept for
5+
> historical context only — its present-tense wording ("MoonSharp stays live", "Prod stays on
6+
> MoonSharp until…") describes the plan as it stood *before* the swap and no longer reflects the
7+
> current codebase. Do not treat it as a description of current state.
8+
>
9+
> Status (historical): DRAFT v2 (repo-state snapshot refreshed 2026-07-10). This was a migration plan,
10+
> not a claim that Lua-CSharp was already the production VM at the time of writing.
11+
> All code, docs, commit messages in English.
612
> Module name decision: **keep `com.neoxider.coreaimods` / `CoreAI.Mods`** (no rename). A rename to
713
> CoreAILua, if ever wanted, is a separate GUID-preserving PR — never bundled with the VM swap.
814

Docs/Images/benchmark_v2_frontier.md

Lines changed: 19 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,28 @@
11
# 🏆 CoreAI Benchmark — CoreAI Game-Creation Benchmark v2 — frontier sweep
22

3-
7 model(s), ranked by suite score.
3+
8 model(s), ranked by suite base score (suite v1.7, benchmark v2 prompts, G1–G8).
44

5-
![Game-Creation Benchmark](COMPARISON.svg)
5+
![Game-Creation Benchmark](benchmark_v2_frontier.png)
66

7-
| # | Model | Suite | Pass-rate | P/PA/F | Eff | Tool-err | Tokens | Run |
8-
|---:|---|---:|---:|---|---:|---:|---:|---|
9-
| 1 | `gpt-5.6-sol` | **96.6** | 85.7% | 24/3/1 | 0 | 2.8% | 16379 | `gpt-5.6-sol` |
10-
| 2 | `gpt-5.6-terra` | **93** | 86.2% | 25/2/2 | 0 | 1.5% | 32194 | `gpt-5.6-terra` |
11-
| 3 | `gpt-5.5` | **90.3** | 82.8% | 24/2/3 | 0 | 13.5% | 17188 | `gpt-5.5` |
12-
| 4 | `gpt-5.6-luna` | **88.1** | 79.3% | 23/2/4 | 0 | 9.2% | 18426 | `gpt-5.6-luna` |
13-
| 5 | `claude-sonnet-5` | **86.2** | 75.9% | 22/4/3 | 0 | 23.2% | 18250 | `claude-sonnet-5` |
14-
| 6 | `claude-opus-4.8` | **79.7** | 75.9% | 22/2/5 | 0 | 22% | 19851 | `claude-opus-4.8` |
15-
| 7 | `claude-fable-5` | **78.9** | 72.4% | 21/3/5 | 0 | 32.1% | 21980 | `claude-fable-5` |
7+
| # | Model | Suite | Pass-rate | P/PA/F | Tool-err | Tokens |
8+
|---:|---|---:|---:|---|---:|---:|
9+
| 1 | `gpt-5.6-sol` | **96.6** | 85.7% | 24/3/1 | 2.8% | 16379 |
10+
| 2 | `gpt-5.6-terra` | **93.0** | 86.2% | 25/2/2 | 1.5% | 32194 |
11+
| 3 | `gpt-5.3-spark` | **92.9** | 79.3% | 23/5/1 | 7.1% | 17616 |
12+
| 4 | `gpt-5.5` | **90.3** | 82.8% | 24/2/3 | 13.5% | 17188 |
13+
| 5 | `gpt-5.6-luna` | **88.1** | 79.3% | 23/2/4 | 9.2% | 18426 |
14+
| 6 | `claude-sonnet-5` | **86.2** | 75.9% | 22/4/3 | 23.2% | 18250 |
15+
| 7 | `claude-opus-4.8` | **83.2** | 79.3% | 23/2/4 | 22.0% | 26250 |
16+
| 8 | `claude-fable-5` | **81.4** | 75.9% | 22/3/4 | 32.1% | 22360 |
17+
18+
> `opus-4.8`, `fable-5` and `gpt-5.3-spark` were re-run after the `cli-agents` bridge gained a
19+
> no-tool-call recovery retry. Single run per model over the CLI bridge; speed is reported, never scored.
20+
> See [BENCHMARK_LEADERBOARD.md](../BENCHMARK_LEADERBOARD.md) for the full per-group (G1…G8) table.
1621
1722
```mermaid
1823
xychart-beta
19-
title "Suite score by model"
20-
x-axis ["gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.5", "gpt-5.6-luna", "claude-sonnet…", "claude-opus-4…", "claude-fable-5"]
24+
title "Suite base score by model"
25+
x-axis ["gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.3-spark", "gpt-5.5", "gpt-5.6-luna", "claude-sonnet…", "claude-opus-4…", "claude-fable-5"]
2126
y-axis "Score" 0 --> 100
22-
bar [96.6, 93, 90.3, 88.1, 86.2, 79.7, 78.9]
27+
bar [96.6, 93.0, 92.9, 90.3, 88.1, 86.2, 83.2, 81.4]
2328
```
24-

README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@ Raw API calls get you text; CoreAI gives you the production layer around that te
8585

8686
## Game-Creation Benchmark
8787

88-
CoreAI ships a game-creation benchmark: it scores how well an LLM builds and changes a game by driving real `execute_lua` and `world_command` tools — 0-100 across eight scenario groups (**suite v2**, G1-G8), against any OpenAI-compatible endpoint.
88+
CoreAI ships a game-creation benchmark: it scores how well an LLM builds and changes a game by driving real `execute_lua` and `world_command` tools — 0-100 across eight scenario groups (**suite v1.7**, benchmark v2 prompts, G1-G8), against any OpenAI-compatible endpoint.
8989

9090
**Top-tier models — v2 frontier sweep (2026-07-11):**
9191

@@ -450,7 +450,8 @@ See [INSTALL.md](INSTALL.md) for step-by-step instructions per profile.
450450
└──────────────────────┬──────────────────────────────────────┘
451451
452452
┌─────────────────────────────────────────────────────────────┐
453-
│ LoggingLlmClientDecorator │
453+
│ LLM pipeline: Timeout( Logging( RetryingStreaming( … ) ) ) │
454+
│ • Request timeout • Logging • Pre-commit stream retry │
454455
│ • HTTP retry (429/5xx) • Retry-After • Exp. backoff │
455456
└──────────────────────┬──────────────────────────────────────┘
456457

0 commit comments

Comments
 (0)