Republish with unified 20-model failure-audit annotations (20260707c) (#113)

* Republish with unified 20-model failure-audit annotations (20260707c)

One coherent Codex audit of every wrong cell across all 20 models
against the corrected narratives: 889 cases, 5,351 row annotations.
Covers GPT-5.5's default-effort rerun for the first time and retires
annotations that quoted the pre-correction narratives. Scores,
predictions, and weights byte-identical to 20260707b.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Gate publishes on judge-annotation coverage for wrong answers

validate_country_payload gains require_failure_annotations: every
scenarioPredictions leaf that is wrong by the failure audit's own
definition (thresholdScore < 100) must carry a judged failureSource.
publish-dashboard and the validate-dashboard CLI enforce it; plain
exports stay structural because fold-board staging legitimately
precedes the audit.

The 20260707b payload shipped GPT-5.5 wrong answers with no judge
annotation because nothing asserted coverage. publish_dashboard now
refuses that payload with 46 errors while the unified-audit payload
(20260707c) passes clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: MaxGhenis

annotations/us_full_run_20260612_policyengine_4_16_1_populace/us_audit_row_annotations.csv

@@ -1,9 +1,9 @@
 {
   "version": 1,
   "repo": "PolicyEngine/policybench",
-  "tag": "dashboard-data-20260707b",
+  "tag": "dashboard-data-20260707c",
   "asset": "dashboard-data.json",
-  "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260707b/dashboard-data.json",
-  "sha256": "a9b8bee0802ed9446e8c6a1d954afbe0ddbe478803b9041da8cfa2f0f382f43d",
-  "bytes": 57597347
+  "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260707c/dashboard-data.json",
+  "sha256": "0860f8ebab3853441ee91c7283dc465434e52e9c2266975af7d58baa712d7f0a",
+  "bytes": 57737111
 }

app/src/data.artifact.json

@@ -20,9 +20,9 @@
   ],
   "live_dashboard_artifact": {
     "asset": "dashboard-data.json",
-    "bytes": 57597347,
-    "derivation": "PolicyBench dataset v1.1, 20-model board (dashboard-data-20260707) with corrected reference-derivation narratives only. 177 Medicaid narratives are regenerated with the engine's eligibility category injected as authoritative grounding (29 had described 65+ people via the ACA adult expansion), and 3 narratives that still described pre-v1.1 reference values (scenario_008/scenario_100 Head Start, scenario_056 NJ state tax) are regenerated against the current engine. Every score, prediction leaf, modelStats entry, and globalWeights value is byte-identical to dashboard-data-20260707; the only changed payload fields are referenceExplanation (180 cells x 20 model leaves). Generator grounding mechanism and regenerated annotations landed in PR #110; the #107 audit adjudicated all 14 reference-suspect flags with zero real reference bugs.",
-    "notes": "Narrative-correction republish (same pattern as dashboard-data-20260702b): dataset version unchanged (v1.1), no selector changes. claude-fable-5 usage fields remain artifact-level values.",
+    "bytes": 57737111,
+    "derivation": "PolicyBench dataset v1.1, 20-model board with the unified failure-audit annotations. Every wrong cell across all 20 models is Codex-judged in one coherent pass against the corrected reference-derivation narratives (889 cases, 636 substantive verdicts, 5,351 row annotations) \u2014 replacing the patchwork of June and per-batch audits, covering GPT-5.5's default-effort rerun for the first time, and retiring annotations that quoted the pre-correction narratives. Every score, prediction leaf, modelStats entry, and globalWeights value is byte-identical to dashboard-data-20260707b; the only changed payload fields are the annotation classes (annotation, failureSource, failureSubtype, caseAnnotation, caseFailureSources, caseFailureSubtypes).",
+    "notes": "Annotation republish (pattern of 20260702b/20260707b): dataset version unchanged (v1.1), no selector changes. claude-fable-5 usage fields remain artifact-level values. Fresh audit flags 10 reference-suspect cases; 5 were previously adjudicated reference-correct (#107), 5 new are under adjudication.",
     "population_weights": {
       "sha256": "0b3b96b8247477631510371b9c3b021a18a535e8ebbf8c73a8aa67cff3110cd4",
       "source": "committed June artifact policybench/population_weights.json (unchanged)"
@@ -32,9 +32,9 @@
       "policyengine_us_version": "1.755.4",
       "policyengine_version": "4.16.1"
     },
-    "sha256": "a9b8bee0802ed9446e8c6a1d954afbe0ddbe478803b9041da8cfa2f0f382f43d",
-    "tag": "dashboard-data-20260707b",
-    "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260707b/dashboard-data.json"
+    "sha256": "0860f8ebab3853441ee91c7283dc465434e52e9c2266975af7d58baa712d7f0a",
+    "tag": "dashboard-data-20260707c",
+    "url": "https://github.com/PolicyEngine/policybench/releases/download/dashboard-data-20260707c/dashboard-data.json"
   },
   "live_dashboard_note": "Two artifacts are tracked. published_dashboard_artifact pins the manuscript's frozen record: it equals the combined export of the source run data.json files listed under source_run_artifacts, byte for byte. live_dashboard_artifact is the release asset the committed pointer app/src/data.artifact.json references; it starts from the frozen export and may gain additive updates (injected metrics, newly benchmarked models) without changing the frozen models' scores. Every publish-dashboard run must update live_dashboard_artifact (tag, sha256, bytes, derivation) in the same commit as the pointer.",
   "model_response_date": "2026-06-12 to 2026-06-13",

paper/snapshot/20260501/manifest.json

@@ -1414,7 +1414,9 @@ def main():
             payload = json_module.loads(payload_path.read_text(encoding="utf-8"))
         except json_module.JSONDecodeError as exc:
             raise SystemExit(f"{payload_path} is not valid JSON: {exc}") from exc
-        errors = validate_dashboard_payload(payload)
+        # The CLI check is the pre-publish gate, so it matches
+        # publish-dashboard strictness: wrong answers need judge annotations.
+        errors = validate_dashboard_payload(payload, require_failure_annotations=True)
         if errors:
             for error in errors:
                 print(f"ERROR: {error}")

policybench/cli.py

@@ -53,8 +53,18 @@ def _is_finite_number(value: Any) -> bool:
     return _is_number(value) and math.isfinite(value)
 
 
-def validate_country_payload(bench: Any, *, country: str | None = None) -> list[str]:
-    """Validate one country's bench payload. Returns a list of errors."""
+def validate_country_payload(
+    bench: Any,
+    *,
+    country: str | None = None,
+    require_failure_annotations: bool = False,
+) -> list[str]:
+    """Validate one country's bench payload. Returns a list of errors.
+
+    ``require_failure_annotations`` additionally requires every wrong answer
+    (thresholdScore < 100) to carry a judged ``failureSource``. Publishing
+    enforces it; staged exports that precede the failure audit do not.
+    """
     if not isinstance(bench, dict):
         return [f"bench payload must be an object, got {type(bench).__name__}"]
 
@@ -171,6 +181,24 @@ def validate_country_payload(bench: Any, *, country: str | None = None) -> list[
                             f"{variable}.{model}: groundTruth must be a finite "
                             f"number, got {record['groundTruth']!r}"
                         )
+                    # Every wrong answer must carry a judged failure
+                    # annotation. GPT-5.5's default-effort rerun shipped
+                    # unannotated because nothing asserted coverage — the
+                    # publish gate now fails instead of shipping unexplained
+                    # misses. "Wrong" matches the failure audit's own
+                    # definition: thresholdScore below 100.
+                    threshold = record.get("thresholdScore")
+                    if (
+                        require_failure_annotations
+                        and _is_finite_number(threshold)
+                        and float(threshold) < 100
+                        and not record.get("failureSource")
+                    ):
+                        errors.append(
+                            f"{prefix}.scenarioPredictions.{scenario_id}."
+                            f"{variable}.{model}: wrong answer has no judged "
+                            f"failureSource annotation"
+                        )
 
     failure_modes = bench.get("failureModes")
     if not isinstance(failure_modes, dict) or not {
@@ -184,7 +212,9 @@ def validate_country_payload(bench: Any, *, country: str | None = None) -> list[
     return errors
 
 
-def validate_dashboard_payload(payload: Any) -> list[str]:
+def validate_dashboard_payload(
+    payload: Any, *, require_failure_annotations: bool = False
+) -> list[str]:
     """Validate the combined app payload. Returns a list of errors."""
     if not isinstance(payload, dict):
         return [f"payload must be an object, got {type(payload).__name__}"]
@@ -213,13 +243,26 @@ def validate_dashboard_payload(payload: Any) -> list[str]:
 
     for country, bench in countries.items():
         if country in COUNTRY_CODES:
-            errors.extend(validate_country_payload(bench, country=country))
+            errors.extend(
+                validate_country_payload(
+                    bench,
+                    country=country,
+                    require_failure_annotations=require_failure_annotations,
+                )
+            )
 
     return errors
 
 
-def assert_valid_dashboard_payload(payload: Any, *, source: str = "payload") -> None:
-    errors = validate_dashboard_payload(payload)
+def assert_valid_dashboard_payload(
+    payload: Any,
+    *,
+    source: str = "payload",
+    require_failure_annotations: bool = False,
+) -> None:
+    errors = validate_dashboard_payload(
+        payload, require_failure_annotations=require_failure_annotations
+    )
     if errors:
         raise DashboardValidationError(source, errors)
 

policybench/dashboard_schema.py

@@ -84,7 +84,9 @@ def publish_dashboard(
 
     raw = source_path.read_bytes()
     payload = json.loads(raw)
-    assert_valid_dashboard_payload(payload, source=str(source_path))
+    assert_valid_dashboard_payload(
+        payload, source=str(source_path), require_failure_annotations=True
+    )
 
     digest = hashlib.sha256(raw).hexdigest()
     pointer = build_pointer(

policybench/publish_dashboard.py

@@ -106,6 +106,71 @@ def test_no_tools_condition_required():
     assert any("no_tools" in error for error in errors)
 
 
+def make_wrong_leaf(**extra) -> dict:
+    return {"prediction": 100.0, "groundTruth": 90.0, "thresholdScore": 0.0, **extra}
+
+
+def test_wrong_answer_without_failure_annotation_fails_publish_gate():
+    # The gap class shipped in 20260707b: GPT-5.5's default-effort rerun had
+    # wrong answers with no judged failureSource. Publishing must fail.
+    bench = make_bench()
+    bench["scenarioPredictions"]["scenario_001"]["snap"]["model-a"] = make_wrong_leaf()
+    errors = validate_country_payload(
+        bench, country="us", require_failure_annotations=True
+    )
+    assert errors == [
+        "countries.us.scenarioPredictions.scenario_001.snap.model-a: "
+        "wrong answer has no judged failureSource annotation"
+    ]
+
+
+def test_wrong_answer_without_failure_annotation_allowed_in_staged_export():
+    # Fold-board staging exports precede the failure audit, so the default
+    # (non-publish) validation stays structural.
+    bench = make_bench()
+    bench["scenarioPredictions"]["scenario_001"]["snap"]["model-a"] = make_wrong_leaf()
+    assert validate_country_payload(bench, country="us") == []
+
+
+def test_wrong_answer_with_failure_annotation_passes_publish_gate():
+    bench = make_bench()
+    bench["scenarioPredictions"]["scenario_001"]["snap"]["model-a"] = make_wrong_leaf(
+        failureSource="model",
+        failureSubtype="calculation_error",
+        annotation="The model omitted the standard deduction.",
+    )
+    errors = validate_country_payload(
+        bench, country="us", require_failure_annotations=True
+    )
+    assert errors == []
+
+
+def test_correct_answer_needs_no_failure_annotation():
+    bench = make_bench()
+    bench["scenarioPredictions"]["scenario_001"]["snap"]["model-a"] = {
+        "prediction": 90.0,
+        "groundTruth": 90.0,
+        "thresholdScore": 100.0,
+    }
+    errors = validate_country_payload(
+        bench, country="us", require_failure_annotations=True
+    )
+    assert errors == []
+
+
+def test_dashboard_payload_threads_annotation_requirement():
+    payload = make_payload()
+    payload["countries"]["us"]["scenarioPredictions"]["scenario_001"]["snap"][
+        "model-a"
+    ] = make_wrong_leaf()
+    assert validate_dashboard_payload(payload) == []
+    errors = validate_dashboard_payload(payload, require_failure_annotations=True)
+    assert errors == [
+        "countries.us.scenarioPredictions.scenario_001.snap.model-a: "
+        "wrong answer has no judged failureSource annotation"
+    ]
+
+
 def test_nan_score_reported():
     bench = make_bench()
     bench["modelStats"][0]["score"] = float("nan")