|
1 | 1 | # TODO |
2 | 2 |
|
3 | | -> Updated 2026-07-10. Tracks open work by priority. Shipped work is in `CHANGELOG.md` (both packages); |
| 3 | +> Updated 2026-07-11. Tracks open work by priority. Shipped work is in `CHANGELOG.md` (both packages); |
4 | 4 | > non-blocking future work in `Assets/CoreAiUnity/Docs/BACKLOG.md`. |
5 | | -> Release candidate: 5.3.0. Verified 2026-07-10: EditMode 1,613 total / 1,609 passed / |
6 | | -> 0 failed / 4 optional third-party ignored; deterministic G8 PlayMode 3/3; full local |
7 | | -> `qwen3.5-4b-mtp` G1-G8 benchmark 88.1/100 (G8 3/3); PlayMode `FastNoLlm` 67/67. |
8 | | -
|
9 | | -## [R0] IMMEDIATE — verification gate on next editor start (blocks release) |
10 | | - |
11 | | -> The previously uncompiled audit wave is now compiled and verified in Unity 6000.3.14f1. |
12 | | -- [x] Compile clean (0 `error CS`); Hub integration compiled in the monorepo via `COREAI_HAS_HUB`. |
13 | | -- [x] Full EditMode suite green: 1,613 total / 1,609 passed / 4 optional third-party ignored / 0 failed. |
14 | | -- [x] Final post-5.3.0 PlayMode `FastNoLlm`: 67/67 passed. |
15 | | -- [x] Local 4B benchmark gate: suite v1.7 / benchmark v2, 88.1/100; G8 3/3. |
16 | | -- [x] Audit docs removed (2026-07-10). Per the owner's rule ("no audits should live in the project"), |
17 | | - all 2026-07-10 audit reports were deleted after their still-open code findings were transferred to |
18 | | - live TODO items: **CI** (F-12, below), **package isolation** (F-22, below), **durability** and |
19 | | - **full-tier Lua query** (in the R0.6 residual list below). Product/idea bets from the idea audit |
20 | | - moved to `Assets/CoreAiUnity/Docs/BACKLOG.md` ("Idea / positioning bets"). |
| 5 | +> Released: 5.6.1 (2026-07-11, all five packages in lockstep). Last full verification 2026-07-10: |
| 6 | +> EditMode 1,613 total / 1,609 passed / 0 failed / 4 optional third-party ignored; deterministic G8 |
| 7 | +> PlayMode 3/3; full local `qwen3.5-4b-mtp` G1-G8 benchmark 88.1/100 (G8 3/3); PlayMode `FastNoLlm` 67/67. |
21 | 8 |
|
22 | 9 | ## [R0.5] Demo pass (owner request: "показательны и корректны") |
23 | 10 |
|
|
30 | 17 | `qwen3.5-4b-mtp`. |
31 | 18 | - [ ] Verify the AI writes mods through the Hub chat in each kept Hub-enabled demo with local 4B/9B/27B |
32 | 19 | (LM Studio) and Opus 4.8 via the bundled preset (`Assets/Resources/CoreAIPresets/`, |
33 | | - bridge: `agent.sh openai-server -e claude -m opus`). |
34 | | -- [ ] Benchmark package (`com.neoxider.coreaibenchmark`): run the suite through the bridge with |
35 | | - `-m spark` (unlimited) as the API model; fix scenario breakage from the 5.1.0 wave if any. |
| 20 | + bridge: `agent.sh openai-server -e claude -m opus`). *(API models are already proven through the |
| 21 | + same bridge in the benchmark v2 sweep — remaining gap is specifically Hub-chat mod-writing per demo.)* |
| 22 | +- [x] Benchmark package (`com.neoxider.coreaibenchmark`): suite ran through the bridge with `-m spark` — |
| 23 | + full v1.7 G1-G8 run on the leaderboard (92.9, row 3); no scenario breakage from the 5.1.0 wave. |
36 | 24 |
|
37 | 25 | ## [R0.6] Release-engineering residuals (from the two 2026-07-10 repository audits) |
38 | 26 |
|
|
99 | 87 | - [ ] Pass `response_format` / `json_schema` to OpenAI-compatible providers; GBNF grammar for local models |
100 | 88 | where supported; keep post-validation as the fallback. (Decide whether to build.) |
101 | 89 |
|
| 90 | +### [R7.5] Multi-API — per-agent LLM provider configuration (owner request 2026-07-11) |
| 91 | + |
| 92 | +> Per-ROLE provider routing already ships: `LlmRoutingManifest` (role pattern → profile → |
| 93 | +> `OpenAiHttpLlmSettings` with its own URL/key/model) resolved per request by `RoutingLlmClient` via |
| 94 | +> `ILlmClientRegistry`. What's missing is the code-first/dev-facing layer (est. M, ~5 files). |
| 95 | +- [ ] `AgentBuilder.WithLlmProfile(profileId)` / `WithLlmBackend(OpenAiHttpOptions)` → `AgentConfig` → |
| 96 | + policy (`SetLlmProfileForRole`, pattern of existing `SetTemperature`). |
| 97 | +- [ ] Explicit profile on the request: make `LlmCompletionRequest.RoutingProfileId` an input hint |
| 98 | + (today write-only diagnostics); `RoutingLlmClient.Prepare` prefers it over role-pattern match. |
| 99 | +- [ ] Runtime profile registration: `RegisterProfile(profileId, OpenAiHttpOptions)` on the registry |
| 100 | + (build `OpenAiChatLlmClient` without an SO asset — `IOpenAiHttpSettings` ctor already exists). |
| 101 | +- [ ] Per-profile fallback + limits: `FallbackLlmClientDecorator` and timeout/retry settings currently |
| 102 | + apply only to the legacy-fallback client / global settings — decide per-profile story. |
| 103 | +- [ ] Hot-swap consistency: `CoreAiBackend.SetApiKey/SetApiBaseUrl` rebuilds only the legacy fallback; |
| 104 | + profile clients need re-`ApplyManifest`/`ApplyRouteTable` on change. |
| 105 | +- [ ] Key hygiene: N providers = N plaintext `apiKey` fields in `.asset` files; provide an out-of-asset |
| 106 | + key source (env var / `IServerManagedAuthProvider`-style) before promoting multi-API. |
| 107 | +- [ ] Docs: "per-agent providers" recipe (inspector-only path needs zero code: N × `OpenAiHttpLlmSettings` |
| 108 | + + manifest on `CoreAILifetimeScope`). |
| 109 | + |
102 | 110 | ### [R8] Vision — finish (feature already shipped) |
103 | 111 |
|
104 | 112 | - [ ] PlayMode round-trip `[Explicit]` test against a real vision-capable model (capture → model → assert). |
|
131 | 139 | 2026-07-01, found via a live G6 benchmark report contradiction (`0 tool-calls` / `1 spawns`); the same |
132 | 140 | root cause explained every "tool ran but stats say 0" symptom (benchmark `ToolCalls`/`FailedToolCalls` |
133 | 141 | undercounts, `ToolErrorRate` misreporting, "used tool" checkpoints failing despite executor state |
134 | | - proving a tool ran). See `Docs/BENCHMARK_STATS_AUDIT_2026-07-01.md` (Codex audit) for the full trace. |
| 142 | + proving a tool ran). *(The Codex audit doc with the full trace was removed per the "no audits in |
| 143 | + the project" rule; the trace survives in git history.)* |
135 | 144 | - [ ] Benchmark harness: `RecordingWorldExecutor.InvalidCommandCount` is tracked separately from `ToolCalls` |
136 | 145 | (invalid/malformed world commands are invisible in the "Tool calls" column). Defensible as a distinct |
137 | 146 | metric, but worth an explicit decision — either document the split or fold invalid attempts into |
|
176 | 185 |
|
177 | 186 | ## Shipped (recent) |
178 | 187 |
|
| 188 | +- 5.6.x — build-time policy registration; simpler agent API (`AgentBuilder.Build()` auto-applies to the |
| 189 | + global policy); solution-wide code-style pass (WHY/TODO/HACK comment rules); benchmark castle |
| 190 | + comparison scene + Stop-with-partial-report; native-API free models in README. |
| 191 | +- 5.5.0 — [R6] resilience wave: `TimeoutLlmClientDecorator` + `RetryingStreamingLlmClientDecorator` |
| 192 | + wired into the pipeline, `CircuitBreakerLlmClientDecorator` primitive; benchmark v2 tooling; CI |
| 193 | + merge-queue gate + package-graph job (F-12 partial). |
| 194 | +- 5.4.0 — MoonSharp removed; Lua-CSharp is the only VM. |
| 195 | +- 5.3.0 — benchmark v2; resilience primitives (`FallbackLlmClientDecorator` covered by tests). |
| 196 | +- 5.1.0-5.2.0 — audit remediation: safe mutation pipeline, bounded queues/stores; stability gate and |
| 197 | + extension APIs; streaming mutating-call deferral; `allowedLuaScenes` contract pinned. |
| 198 | +- 5.0.x — on-demand skills for built-in roles ("Lua Modding" skill); benchmark package extracted to |
| 199 | + `com.neoxider.coreaibenchmark`; version lockstep across five packages. |
179 | 200 | - 4.17.0 — tool-call history unlimited by default (`MaxToolCallHistoryMessages = 0`); per-agent / per-call |
180 | 201 | `MaxToolCallRoundtrips` override (`0` = unlimited, Programmer/Creator default unlimited), default cap raised |
181 | 202 | 10 → 20; clearer cap-reached stop message; honest provider-call tok/s labeling; `BenchmarkInfo.GroupDifficulty10` |
|
0 commit comments