[codex] Clarify hardware test coverage (#24)

* Clarify hardware test coverage

* Fix Tower valuation and add repo navigation

---------

Co-authored-by: Codex <codex@openai.com>

Author: Lightheartdevs

.gitattributes

@@ -26,9 +26,11 @@
 # JSONL: data files where byte-identity matters. We pin as `binary` (rather
 # than `text eol=lf`) so SHAs match across platforms without depending on
 # git's autocrlf settings being correct on the reproducer's side.
+.gitattributes text eol=lf
 *.jsonl      binary
 
 # Generated/raw artifacts — also pin as binary.
 *.log        binary
 power.csv    binary
 thermals.csv binary
+*.gz         binary

README.md

@@ -12,21 +12,24 @@ but I'm making it public so that other people can use it too.
 | Every claim in this repo with a status tag | [`claims.yaml`](claims.yaml) — strong / provisional / held / retracted matrix |
 | What we **didn't** measure (and where PRs are welcome) | [`NOT-HERE-YET.md`](NOT-HERE-YET.md) |
 | What this evidence can and can't support | [`KNOWN-LIMITATIONS.md`](KNOWN-LIMITATIONS.md) — caveats on what we did measure |
+| Where the benchmark folders start | [`benchmarks/README.md`](benchmarks/README.md) — agent-task benchmark landing page |
 | **"Coder-Next or 27B (or 27B-no-think) for my task?"** | [`COMPARISON.md`](COMPARISON.md) — head-to-head decision doc |
 | The full single-table comparison across all entries | [`SCORECARD.md`](SCORECARD.md) |
+| How repo size is managed | [`REPO-SPACE.md`](REPO-SPACE.md) — storage hotspots and artifact policy |
 | How to benchmark a new local model | [`tooling/ADDING-A-MODEL.md`](tooling/ADDING-A-MODEL.md) |
 | How to replay a specific past run | [`tooling/REPRODUCING.md`](tooling/REPRODUCING.md) |
 
 ## Operating point (read before quoting)
 
-Agent-task benchmarks under [`benchmarks/`](benchmarks/) all use **Cyankiwi 4-bit AWQ** quants on **2× RTX PRO 6000 Blackwell at 500 W cap**. Other quants (official FP8, Unsloth UD4 GGUF, BF16), other VRAM tiers (24 GB / 48 GB), other hardware classes (Mac M-series unified memory), and languages other than Python are **not characterized** by those entries. See [`COMPARISON.md` § What this benchmark doesn't characterize](COMPARISON.md#what-this-benchmark-doesnt-characterize) for the full validity-boundary list, and [`ROADMAP.md`](ROADMAP.md) for what's queued to fill those gaps.
+Agent-task benchmarks under [`benchmarks/`](benchmarks/) all use **Cyankiwi 4-bit AWQ** quants on **2x RTX PRO 6000 Blackwell at 500 W cap** unless an entry README says otherwise. Other quants, other VRAM tiers, other hardware classes, and languages other than Python are **not characterized** by those entries. See [`COMPARISON.md` section "What this benchmark doesn't characterize"](COMPARISON.md#what-this-benchmark-doesnt-characterize) for the model-benchmark validity boundaries, and [`ROADMAP.md`](ROADMAP.md) for what's queued to fill those gaps.
 
-Rig-characterisation studies under [`hardware-tests/`](hardware-tests/) have their own operating-point scope — see each study's README. In particular, [`hardware-tests/qwen3.6-q8-fleet-2026-05-17/`](hardware-tests/qwen3.6-q8-fleet-2026-05-17/) ranks four hardware classes on **Q8_0 GGUF** under llama.cpp (with a Tower2 vLLM-FP8 appendix row for the model the llama.cpp/CUDA path crashes on).
+Rig-characterisation studies under [`hardware-tests/`](hardware-tests/) have their own operating-point scope. Start with [`hardware-tests/README.md`](hardware-tests/README.md) before quoting hardware claims. In particular, [`hardware-tests/qwen3.6-q8-fleet-2026-05-17/`](hardware-tests/qwen3.6-q8-fleet-2026-05-17/) ranks four hardware classes on **Q8_0 GGUF** dense and MoE workloads under llama.cpp, with a Tower2 vLLM-FP8 appendix row for the MoE model the llama.cpp/CUDA path crashes on.
 
 ## Layout
 
 ```text
 benchmarks/
+  README.md                    agent-task benchmark landing page and navigation map
   dreamserver-75-pr-audit/
     GPT-5.5/                   cloud, full audit
     Opus-4.7/                  cloud, full audit
@@ -44,7 +47,10 @@ benchmarks/
     Qwen3-Coder-Next-AWQ/      local, full memo repo (DOCU BUY, 1 of 3 runs shipped — verdict reliability caveat in README)
     Qwen3.6-35B-A3B-AWQ/       local, no usable deliverable (0 of 3 runs shipped, kept as failure-mode entry)
 hardware-tests/
+  README.md                       hardware-test landing page: coverage matrix, settled vs held questions
   vllm-power-sweep-2026-04-29/ rig characterisation: vLLM throughput vs GPU power cap, 28-cell sweep with raw CSVs + audit notes
+  qwen3.6-q8-fleet-2026-05-17/ cross-platform dense + MoE Q8 hardware comparison
+  local-ai-hardware-valuation-2026-05-17/ recomputable buyer valuation worksheet
 ```
 
 ## Benchmarks
@@ -57,9 +63,12 @@ hardware-tests/
 | [`microbench-2026-04-28`](benchmarks/microbench-2026-04-28/) | 12 smaller-scope task families (5-30 min deliverables) split across 3 phases — coding (Phase 1), structured business tasks (Phase 2), unbounded business/writing (Phase 3). Designed to surface task-class-specific differences between local 30B-class quantizations. N=3 per cell. Three highest-signal task families published as full per-model entries: adversarial-hallucination, market-research, doc-synthesis. | `Qwen3.6-27B-AWQ`, `Qwen3-Coder-Next-AWQ` |
 | [`microbench-phase-b-2026-05-02`](benchmarks/microbench-phase-b-2026-05-02/) | Bumps the four highest-signal cells of `microbench-2026-04-28` from N=3 → N=10 to bound the headline failure rates with proper Wilson CIs, and adds **27B-no-think** as a third arm across the **full 12-family grid** (~240 runs total). Settles the `p3_doc` 27B word-trim loop as a stable ~40% failure shape, and bounds Coder-Next's `p3_market` 0/3 STRUCTURAL_FAIL as 0/10 at N=10 (Wilson 95% [0%, 27.8%]). | `Qwen3.6-27B-AWQ` (thinking), `Qwen3.6-27B-AWQ` (no-think), `Qwen3-Coder-Next-AWQ` |
 
+For the benchmark landing page and per-folder navigation map, start with
+[`benchmarks/README.md`](benchmarks/README.md).
+
 ## Hardware tests
 
-`hardware-tests/` holds rig characterisation runs — power, throughput, and thermal sweeps on the lab hardware itself, separate from agent-task benchmarks. They support the same evidence base (e.g. validating the operating power cap a model run was conducted under), but they live in their own tree because they answer hardware questions, not model questions.
+`hardware-tests/` holds rig characterisation runs — power, throughput, and thermal sweeps on the lab hardware itself, separate from agent-task benchmarks. Start with [`hardware-tests/README.md`](hardware-tests/README.md): it makes the dense-vs-MoE coverage, backend exceptions, and "settled vs held" boundaries explicit.
 
 | Test | Shape | What it measures |
 |---|---|---|

REPO-SPACE.md

@@ -0,0 +1,71 @@
+# Repo Space And Data Layout
+
+This repo stores benchmark evidence, not just prose. That makes it useful, but
+it also means raw artifacts can grow quickly. The goal is to keep enough data
+for claims to be auditable while avoiding duplicate or low-value bulk.
+
+## Snapshot
+
+Measured on the working tree on 2026-05-17:
+
+| Area | Size |
+|---|---:|
+| `hardware-tests/` | 107.66 MiB |
+| `hardware-tests/qwen3.6-q8-fleet-2026-05-17/` | 105.41 MiB |
+| `benchmarks/` | 92.62 MiB |
+| `tooling/` | 0.39 MiB |
+
+Largest file families:
+
+| Extension | Files | Raw size | Gzip size at level 6 | Potential saving |
+|---|---:|---:|---:|---:|
+| `.csv` | 909 | 77.04 MiB | 13.53 MiB | 82.4% |
+| `.jsonl` | 753 | 30.56 MiB | 2.28 MiB | 92.5% |
+| `.patch` | 150 | 22.72 MiB | 5.00 MiB | 78.0% |
+| `.htm` | 33 | 26.88 MiB | 2.21 MiB | 91.8% |
+| `.html` | 54 | 17.37 MiB | 2.17 MiB | 87.5% |
+
+The only exact duplicate over 1 MiB found in this snapshot is the Qwen fleet
+prompt corpus:
+
+- `hardware-tests/qwen3.6-q8-fleet-2026-05-17/workloads/prompts.jsonl`
+- `hardware-tests/qwen3.6-q8-fleet-2026-05-17/harness/workloads/prompts.jsonl`
+
+Those files are byte-identical and cost about 6.07 MiB each. They were left in
+place in this PR because the harness copy helps the vendored harness remain
+standalone. A future cleanup can remove one copy if the harness default paths
+are changed and verified.
+
+## Keep In Git By Default
+
+- `README.md`, findings, audit notes, manifests, and source-of-truth claim docs.
+- Small structured receipts: `cell.json`, `grade.json`, `label.json`,
+  `summary.json`, `cost.json`, and compact aggregate CSVs.
+- Canonical headline tables used by published prose.
+- Scripts needed to regenerate derived outputs.
+
+## Compress Or Externalize By Default
+
+- Long sampler time series such as `power.csv` and `thermals.csv`.
+- Full `inferences.jsonl` and transcript JSONL files when a compact receipt is
+  enough for the claim.
+- Raw SEC filings, PDFs, HTML pages, and large patches that can be recovered
+  from a source URL plus hash.
+- Generated visual artifacts when the source data and render script are present.
+
+If a large raw artifact is needed for auditability, prefer `*.gz` plus a
+sidecar SHA256 file. If an artifact is needed only for archival completeness,
+prefer a GitHub release asset or another immutable object store and keep the
+URL, SHA256, byte count, and retrieval date in the repo.
+
+## Suggested Next Cleanup
+
+1. Decide whether published bundles must be runnable standalone, or only
+   auditable inside the repo. That determines whether duplicate prompt corpora
+   should stay.
+2. Convert raw sampler CSVs and long JSONL traces to `*.gz`, then update readers
+   and docs to treat compressed text as the canonical raw format.
+3. Add a CI check that fails on exact duplicate files over 1 MiB unless the path
+   is allowlisted in a manifest.
+4. For new benchmark entries, require a README at every public landing folder
+   and a manifest line for every large raw artifact.

benchmarks/README.md

@@ -0,0 +1,52 @@
+# Benchmarks
+
+Start here if your question is about model behavior on messy real-world tasks.
+This tree is separate from `hardware-tests/`: these entries ask what a model
+can actually produce when it has to research, code, audit, write, or build a
+deliverable. Hardware throughput and buyer-value questions live in
+[`../hardware-tests/README.md`](../hardware-tests/README.md).
+
+## Reading Order
+
+| If you want | Read first |
+|---|---|
+| Current model-selection synthesis | [`../COMPARISON.md`](../COMPARISON.md) |
+| Single-table benchmark summary | [`../SCORECARD.md`](../SCORECARD.md) |
+| Cross-cutting findings by date | [`findings-index.md`](findings-index.md) |
+| How to replay or add a run | [`../tooling/README.md`](../tooling/README.md) |
+
+## What Each Benchmark Answers
+
+| Folder | Primary question | Best first file |
+|---|---|---|
+| [`dreamserver-75-pr-audit`](dreamserver-75-pr-audit/) | Can the model complete a long-horizon 75-PR maintainer audit at all? | [`dreamserver-75-pr-audit/README.md`](dreamserver-75-pr-audit/README.md) |
+| [`dreamserver-1-pr-audit`](dreamserver-1-pr-audit/) | What is the floor task for local PR-audit competence? | [`dreamserver-1-pr-audit/README.md`](dreamserver-1-pr-audit/README.md) |
+| [`wallstreet-intern-test`](wallstreet-intern-test/) | Can the model build a traceable investment memo and supporting artifacts? | [`wallstreet-intern-test/README.md`](wallstreet-intern-test/README.md) |
+| [`microbench-2026-04-28`](microbench-2026-04-28/) | How do local models behave on 12 smaller task families at N=3? | [`microbench-2026-04-28/README.md`](microbench-2026-04-28/README.md) |
+| [`microbench-phase-b-2026-05-02`](microbench-phase-b-2026-05-02/) | Which early microbench signals survive an N=10 expansion and 27B no-think arm? | [`microbench-phase-b-2026-05-02/README.md`](microbench-phase-b-2026-05-02/README.md) |
+
+## Model Entry Shape
+
+Most model-entry folders use some subset of these artifacts:
+
+| File or folder | Meaning |
+|---|---|
+| `README.md` | Human-readable entry summary, caveats, and read order. |
+| `grade.json` | Programmatic verdict and grader dimensions where a grader exists. |
+| `label.json` | Failure-mode label from the repo taxonomy. |
+| `receipt.json` | Model, harness, launch, token, and environment receipt. |
+| `cost.json` | Wall time, token throughput, and rough energy/cost fields. |
+| `summary.json` | Run-level finish reason and iteration/token totals. |
+| `transcript.jsonl` | Full model/tool loop when the entry publishes it. |
+| `deliverable/`, `report/`, `memo/`, `model/`, `prs/` | The actual artifacts the model produced. |
+
+Read the benchmark README first, then the model-entry README, then the structured
+JSON receipts before diving into transcripts or deliverables.
+
+## Storage Notes
+
+Some benchmark folders intentionally keep raw source documents, patches,
+transcripts, and deliverables so claims stay auditable. Other entries are lean
+on purpose and publish only receipts, grades, labels, and summaries. For the
+repo-wide storage policy and current size hotspots, see
+[`../REPO-SPACE.md`](../REPO-SPACE.md).

hardware-tests/README.md

@@ -0,0 +1,60 @@
+# Hardware Tests
+
+Start here if your question is about machines rather than model behavior. This
+tree answers hardware questions: throughput, power, thermals, backend maturity,
+and buyer-value ratios. It is separate from the agent-task benchmark tree.
+
+## Coverage Matrix
+
+The cross-platform fleet study is not just a 27B dense run. It covers both a
+dense model and a MoE model across the fleet, with one explicit Blackwell
+engine/quant exception for the MoE row.
+
+| Host | Dense 27B Q8 | MoE 35B-A3B | Backend status | Use for |
+|---|---|---|---|---|
+| Blackwell 6000 Tower | llama.cpp CUDA, canonical | native llama.cpp Q8 retracted; vLLM FP8 defended row | CUDA Q8 MoE hits SOFT_MAX on sm_120; vLLM works | discrete-GPU speed, long-context engine sensitivity |
+| M5 Max MacBook Pro | llama.cpp Metal, canonical | llama.cpp Metal, canonical | clean Metal path | unified-memory Mac appliance behavior |
+| DGX Spark | llama.cpp CUDA aarch64, canonical | llama.cpp CUDA aarch64, canonical | clean CUDA aarch64 path | NVIDIA unified-memory desktop behavior |
+| EVO X2 / Strix Halo | llama.cpp Vulkan, canonical partial | llama.cpp Vulkan, canonical partial | Vulkan works; ROCm retry still pending | AMD APU / small-chassis behavior |
+
+Canonical rows live in
+[`qwen3.6-q8-fleet-2026-05-17/aggregate/canonical-headline.csv`](qwen3.6-q8-fleet-2026-05-17/aggregate/canonical-headline.csv).
+Appendix and engine-comparison rows live in
+[`qwen3.6-q8-fleet-2026-05-17/aggregate/appendix-headline.csv`](qwen3.6-q8-fleet-2026-05-17/aggregate/appendix-headline.csv).
+Do not mix those two tables in a cross-host ranking.
+
+## What Each Bundle Answers
+
+| Bundle | Primary question | Main caution |
+|---|---|---|
+| [`qwen3.6-q8-fleet-2026-05-17`](qwen3.6-q8-fleet-2026-05-17/) | How do four local-AI hardware classes handle the same dense and MoE Qwen3.6 workloads? | Multi-user serving is held; Tower2 MoE uses a defended vLLM FP8 exception because native llama.cpp Q8 crashes. |
+| [`local-ai-hardware-valuation-2026-05-17`](local-ai-hardware-valuation-2026-05-17/) | What are buyers paying per usable memory GB, bandwidth, and measured 27B Q8 tok/s? | Prices and wall-power assumptions are time-bound inputs. |
+| [`vllm-power-sweep-2026-04-29`](vllm-power-sweep-2026-04-29/) | Where is the RTX PRO 6000 Blackwell LLM-serving power-cap plateau? | Tower2-only, vLLM-only, AWQ-INT4 models. |
+| [`ltx23-power-sweep-2026-05-05`](ltx23-power-sweep-2026-05-05/) | Does the same GPU power-cap curve apply to diffusion/video generation? | Workload-specific to the LTX-2.3 workflow tested. |
+| [`cpu-fullpower-2026-05-05`](cpu-fullpower-2026-05-05/) | Can the Tower2 Threadripper PRO CPU sustain rated 350 W on its cooling stack? | One rig, one cooling design, CPU-side validation only. |
+
+## What The Cross-Platform Study Can Settle
+
+- Single-user prefill, decode, and TTFT for the tested dense and MoE model
+  rows.
+- Backend maturity and failure modes under the tested source SHA.
+- Chassis and thermal behavior under sustained local-inference operation.
+- Narrow buyer-value statements tied to the measured rows and dated prices.
+
+## What It Does Not Settle Yet
+
+- Best hardware for every quantization, model size, or engine.
+- Best multi-user serving hardware under proper batching engines.
+- Task quality of Q8 outputs.
+- Cross-day variance or unit-to-unit variance.
+- Total cost of ownership without plug-metered wall power on every host.
+
+For the active gap list, read [`../NOT-HERE-YET.md`](../NOT-HERE-YET.md).
+
+## Storage Posture
+
+The hardware-test tree is the current repo-size hotspot because it preserves
+raw sampler CSVs, JSONL inference traces, and audit bundles. That is useful for
+trust, but it should stay intentional. See [`../REPO-SPACE.md`](../REPO-SPACE.md)
+for the measured size breakdown and the recommended path for compressing or
+externalizing future raw artifacts.

hardware-tests/local-ai-hardware-valuation-2026-05-17/README.md

@@ -61,7 +61,7 @@ Current generated output, using the bundled price/spec assumptions:
 | System | Price | Usable AI GB | GB/s | `$/GB` | `$/GB/s` | 27B Q8 decode | `$/decode tok/s` |
 |---|---:|---:|---:|---:|---:|---:|---:|
 | Blackwell 6000 Tower, single-RTX-6000 anchor | $12,000 | 96 | 1792 | $125.00 | $6.70 | 49.78 | $241.04 |
-| Blackwell 6000 Tower, as-configured server | $33,000 | 96 | 1792 | $343.75 | $18.42 | 49.78 | $662.86 |
+| Blackwell 6000 Tower, as-configured server | $33,000 | 190 | 3584 | $173.68 | $9.21 | 49.78 | $662.86 |
 | M5 Max MacBook Pro 16 | $4,850 | 128 | 614 | $37.89 | $7.90 | 16.78 | $288.97 |
 | NVIDIA DGX Spark | $4,699 | 121 | 273 | $38.83 | $17.21 | 7.60 | $618.21 |
 | EVO X2 / Strix Halo | $3,000 | 124 | 256 | $24.19 | $11.72 | 7.82 | $383.59 |
@@ -91,8 +91,12 @@ is ecosystem, not raw memory value: Metal/macOS is not CUDA/Linux, and that
 matters for some serving stacks.
 
 The Blackwell 6000 Tower is the opposite profile: expensive capacity, excellent
-bandwidth, excellent measured speed. It is a speed purchase, not the cheapest
-way to buy 100 GB-class memory.
+bandwidth, excellent measured speed. The as-configured server row now values
+the actual two-GPU memory asset: two 96 GB RTX PRO 6000 Blackwell cards,
+modeled as 190 GB conservative usable aggregate VRAM and 3584 GB/s aggregate
+bandwidth. Its measured `$/decode tok/s` still uses the one-GPU canonical
+MMBT row, so do not quote the aggregate-memory row as aggregate-speed evidence
+until a measured dual-GPU or sharded row is available.
 
 ## What this can and cannot settle