Skip to content

README.md

Latest commit

 

History

History
118 lines (95 loc) · 8.23 KB

README.md

File metadata and controls

118 lines (95 loc) · 8.23 KB

Platform test & benchmark archive

This directory is the historical archive of platform-test and benchmark artifacts collected during Kakeya v0.1 → v0.3 development. Reports here are referenced from ADRs (docs/adr/) and from release notes; do not edit or re-run files in place once they have been committed — they are evidence, not working data.

When a new run is needed, write to a fresh, timestamp-suffixed filename and add an entry below.

File-naming conventions

Prefix Meaning
mac-mlx-1{a,b,c}-… Phase-1 MLX backend bring-up tests (v0.1 era).
mac-phase-b-… Phase-B sparse-logits proposer tests.
mac-streaming-… E2 server streaming integration tests.
mlx-… First-pass MLX platform test on Mac M4 24 GB.
bench_mac_kvcache_… Sink+window verifier KV peak comparison (vs baseline).
bench_mac_m4_… Mac M4 micro-bench (token throughput, single prompt).
bench_mlx_speculative_… MLX speculative decoder bench (CPU/CPU vs MLX/MLX).
bench_mlx_verifier_… MLX verifier-only forward bench.
bench_param_sweep_… Hyperparameter sweep (block size × num_diffusion_steps × proposer K).
bench_sparse_vs_dense_… Proposer LM-head sparse vs dense logits comparison.
bench_long_session_mac_… Long-session memory-stability run (the v0.3 §2.3.a / §2.3.b evidence).
…junit.xml / …coverage.xml Companion test-runner artifacts for the matching .json.
*.partial.json Live checkpoint written every N turns by bench_long_session.py.
*.aborted.json Annotated abort note when a long run was terminated for triage.

v0.3 long-session archive (the ADR 0006 → ADR 0007 → ADR 0008 evidence chain)

These five runs were the empirical chain that drove the architecture from "OpenAI-compatible HTTP server with stateless turns" toward the session-bound runtime described in ADR 0008. Each run was originally pushed on its own AgentMemory/bench-*-8e7f branch; consolidated here so ADRs and release notes can reference stable main paths.

Index (chronological)

# UTC date File (in this dir) Wall time Successful turns Errors Notes
1 2026-05-30 08:42 bench_long_session_mac_1780130542.aborted.json 12 412 s aborted 58 0* First 4 h attempt; aborted at ~3.4 h. Triage notes only.
bench_long_session_mac_1780130542.partial.json Last live checkpoint from run #1 before abort.
2 2026-05-30 ~16:* bench_long_session_mac_short_1780146230.json 1 800 s (30 min) 57 0 First clean 30 min after orphan-session fix.
bench_long_session_mac_short_1780146230.partial.json
3 2026-05-31 09:* bench_long_session_mac_short2_1780196477.json 1 800 s (30 min) 58 0 Adds in-flight metrics poller (metrics_poll_interval_s=0.25).
bench_long_session_mac_short2_1780196477.partial.json
4 2026-05-31 13:* bench_long_session_mac_short3_1780208693.json 1 800 s (30 min) 58 0 KV gauge gated to active sessions; KV peak = 7.4 MiB.
bench_long_session_mac_short3_1780208693.partial.json
5 2026-05-31 14:* bench_long_session_mac_4h_1780211323.json 14 400 s (4 h) 58 182 Memory bounded; throughput collapses to ~0 after turn 58.
bench_long_session_mac_4h_1780211323.partial.json

*The aborted.json records 0 errors only because every later request was rejected with HTTP 429 by the scheduler before the bench client even sent it; the client did not classify those as turn errors. The server log showed sustained 429s — that is the bug the orphan-session fix addressed.

What the chain tells us

  1. Memory is bounded. Runs #2-#5 all hold KV peak ≈ 0 / 0 / 7.4 MiB (depending on whether the gauge was wired to the engine yet) with KV drift +0.00 MiB over 10-min buckets. The 4 h run (#5) holds the same bound as the 30-min run (#4). This is the evidence behind ADR 0006 §2.3.a (memory-bounded claim — VERIFIED).

  2. Latency is not bounded. Every run shows positive latency_drift_p50 in the +38 s … +41 s range, and per-bucket p50 grows monotonically:

    bucket 0 (0-10 min):   ~15 s p50
bucket 1 (10-20 min):  ~38 s p50
bucket 2 (20-30 min):  ~55 s p50

    In run #5 (the only run long enough to expose this), p95 keeps rising until turns hit the 120 s client timeout and start to error — 182 such timeouts in the 3.5 h tail. This is the evidence behind ADR 0006 §2.3.b (latency-bounded claim — NOT achieved in v0.3).

  3. The cause is full-history prefill on every turn. The bench appends each prior assistant reply to the next prompt. With sink+window KV forced to reset at every request, prefill grows linearly with turn count. This is what made cross-request KV reuse a v0.3 hard requirement rather than a v0.4 nice-to-have, and is what motivated ADR 0007 (automatic prefix matching). When the Qwen3 chat template was found to inject generation-time-only placeholders that break token-id-level prefix matching, the design pivoted again, to the explicit session-bound protocol described in ADR 0008.

Source branches (audit trail)

File Source branch
bench_long_session_mac_1780130542.{aborted,partial}.json AgentMemory/bench-long-session-mac-results-8e7f
bench_long_session_mac_short_1780146230.{,partial.}json AgentMemory/bench-short-test-results-8e7f
bench_long_session_mac_short2_1780196477.{,partial.}json AgentMemory/bench-short-test-results-2-8e7f
bench_long_session_mac_short3_1780208693.{,partial.}json AgentMemory/bench-short-test-results-3-8e7f
bench_long_session_mac_4h_1780211323.{,partial.}json AgentMemory/bench-long-4h-mac-results-8e7f

These branches remain on origin/ for the original commit-hash audit trail; the JSON snapshots are reproduced verbatim here.

Earlier (v0.1 / v0.2) artifacts

The mlx-…, mac-mlx-1{a,b,c}-…, mac-phase-b-…, mac-streaming-…, bench_mac_*, bench_mlx_*, bench_param_sweep_…, and bench_sparse_vs_dense_… files predate the long-session work and were landed in earlier PRs (Phase-1 bring-up, MLX-1a/1b/1c probes, sparse-logits A/B, MLX speculative decoder bring-up). They are kept here unchanged for historical reproducibility and for ADR 0001 / ADR 0002 / ADR 0003 cross- references.