Regrade historical phase-1 cells (27B/Coder) — flat-vs-nested metric bug (fixed in #28)

[Lightheartdevs]

Summary

tooling/graders/phase1_grade.py read flat metric keys (coverage_pct, ruff_issues, benchmark_s) while tooling/graders/code_task_grader.py writes nested ones (coverage.line_coverage_pct, ruff.issue_count, benchmark.elapsed_s). The mismatched .get()s silently returned defaults, so:

• bugfix: ruff_no_regression and bench_no_major_regression were always True → PASS could mask a ruff/perf regression.
• testwrite: coverage_improved was always False (0 - 0 > 1.0) → guaranteed FAIL regardless of the actual tests written.

pytest keys were read correctly; refactor uses pytest + structural checks and is unaffected.

Fixed in PR #28 (commit bdb7b0d, reads the nested keys). This issue tracks repairing the already-published historical numbers graded before that fix — a separate provenance repair, intentionally not folded into #28.

Evidence the historical cells are affected

benchmarks/microbench-2026-04-28/bug-fixing/Qwen3.6-27B-AWQ/grade.json carries the bug's fingerprint: before: {cov: None, ruff: 0, bench_s: None} with criteria.{ruff_no_regression,bench_no_major_regression}: True — i.e. graded by the same flat-key logic, with coverage/ruff/bench reading their defaults. On a corrected 397B regrade, think-mode testwrite went from a spurious cov=0 FAIL to 99% coverage / 153 passing tests (still FAIL, but on the real logalyzer_unchanged rule) — strong evidence the historical 0/N testwrite numbers are artifacts, not capability.

Scope — cells to regrade

• microbench-2026-04-28: Qwen3.6-27B-AWQ + Qwen3-Coder-Next-AWQ — bug-fixing and test-writing.
• microbench-phase-b-2026-05-02: the -no-think-v1 27B + Coder bug-fixing/test-writing cells.
• (refactoring cells: unaffected by this bug — skip unless re-verifying.)

Regrade vs re-run (key dependency)

Regrade needs the original agent workspace to re-run code_task_grader.py.

• bug-fixing: old workspaces still exist locally (agent-pilot/workspace/{27b,coder}_code_task_v1) → regradable without re-running the model.
• test-writing / refactoring: workspaces not found locally and no workspace tarball is stored in the published cells (only cost/grade/label/README/receipt/summary). These likely need re-running on the original models (27B-AWQ / Coder-Next-AWQ via vLLM) — confirm before assuming a pure regrade is possible.

Affected scripts / inputs

• tooling/graders/phase1_grade.py (fixed in Qwen3.5-397B-A17B microbench (N=1) + GGUF/thinking tooling support #28).
• Requires tooling/inputs/code-task-starter/ baseline + bench-sandbox:latest (both restored in Qwen3.5-397B-A17B microbench (N=1) + GGUF/thinking tooling support #28).

Published docs/claims that may need edits after regrade

• benchmarks/microbench-2026-04-28/findings.md (incl. the "27B reliable" framing and the testwrite/refactor "task-design" KNOWN-LIMITATIONS footnote — at least partly a grader artifact).
• benchmarks/microbench-phase-b-2026-05-02/findings.md + the per-cell bug-fixing/ and test-writing/ grade.json/README.md.
• Any SCORECARD / aggregate pass-rate claims that include phase-1 bug-fixing or test-writing.

Acceptance

• Bug-fixing cells regraded from existing workspaces with the fixed grader; verdicts + criteria updated.
• Test-writing/refactoring: confirm workspace availability; regrade or re-run as needed.
• Affected findings docs + per-cell artifacts corrected, with a note on what changed.
• Re-examine the testwrite/refactor "task-design" footnote in light of corrected metrics.