Re-freeze the manuscript on the 24-model corrected-reference board

[MaxGhenis]

Prepares the paper for SSRN/arXiv submission by advancing the frozen manuscript snapshot from the June 13-model board to the current 24-model v1.1 board.

Why the June snapshot could not ship

• Its references were generated with policyengine-us 1.723.0 and contain the four defects the audit program later found and fixed (verified: scenario_056 still carried the incorrect NJ state tax in the frozen data).
• Its abstract claimed "zero PolicyEngine reference defects," which the post-freeze triage falsified for exactly those references.
• Its GPT-5.5 wave ran with a low-effort pin later traced to a harness token-budget artifact.

What the re-freeze does

• Snapshot: 24 models, references regenerated with policyengine-us 1.755.4, every model at unconfigured provider defaults, decisive-diagnosis audit annotations (6,031 rows: 5,466 llm_error + 565 deterministic parse_contract_failure; zero unresolved, zero reference-suspect). The frozen us/data.json recombines byte-exactly to the published dashboard-data-20260710 artifact, and the manifest records the multi-wave response window (2026-06-12 to 2026-07-10).
• Abstract: the audit sentence now states the real record — 35 reference-suspect flags across three waves, 31 dissolved against primary sources and engine traces, 4 genuine defects fixed upstream before this snapshot's references were generated, and parse failures (1.2% of cells) scored as misses rather than dropped.
• Reference-output credibility: rewritten around the adversarial review program, with the audit-tooling hardening reported (trace-grounded derivation narratives with mechanism validators; a judge contract requiring per-model diagnoses, with hedged verdicts mechanically rejected and reference doubt routed to structured flags for manual adjudication).
• Limitations: discloses the scenario_121 encoding mismatch and the two input-semantics conventions slated for the next scenario refresh.
• Parse coverage: stated truthfully everywhere (565 unparsable cells, concentrated in Kimi K2.6 and GLM-5.2, stand as scored misses); the prior "zero missing values" claims are gone, including one in Failure modes that contradicted the appendix.
• Tooling: freeze_snapshot.py emits the live_dashboard_artifact block, freezes the trace-grounded narratives alongside annotations, and parse_contract_failure joins llm_error as a final annotation class in the validator.

Verification

• All 16 snapshot-artifact tests pass, including byte-exact recombination of the frozen export to the published artifact sha.
• Full Python suite: 481 passed; Ruff clean.
• Paper renders clean to HTML and PDF; rendered hashes re-pinned in the manifest. PDF spot-checks assert the corrected claims ("GPT-5.6 Sol at 88.7", "35 reference-suspect flags", "policyengine-us 1.755.4", "565 cells (1.2%)") and the absence of the stale zero-missing claim.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policybench-site	Ready	Preview, Comment	Jul 11, 2026 2:09pm