Wire up the paper-reported benchmark suite: tier manifests, orchestrator, fixes

This commit lands the production scaffolding called for by paper-l0/BENCHMARK_PLAN.md
so the L0 / GREG / IPF benchmark can be launched end-to-end against the saved
calibration package without further code changes.

Tier manifests (paper-l0/benchmarking/manifests/tier*.json)
  All ten paper-reported manifests, each reading from one shared
  calibration_package.pkl and differing only in target_filters:

    tier1_mixed.json            L0 + GREG on the full filtered slice (count +
                                dollar combined) over a five-state, ten-district
                                geography subset plus national targets.
    tier1_ipf.json              L0 + IPF on the same slice; IPF retains the
                                authored closed subset and the matched runs use
                                --train-on ipf_retained_authored.
    tier2_scaling_250 ...
    tier2_scaling_largest.json  Scaling ladder (250 / 500 / 1k / 2.5k / 5k / 10k
                                / largest coherent pre-production subset) over
                                the same package, growing geography coverage to
                                grow the target set instead of the unit universe.
    tier3_production.json       Least-filtered view; failures are reportable
                                results (status=failed with notes) rather than
                                missing rows.

  All tier manifests opt into method_options.ipf.return_na = true so non-
  convergence surfaces NaN and the runner converts that into a visible runtime
  error instead of writing NaN-laden weights to disk.

  Geographic ID format is the convention the saved package actually uses: state
  FIPS without leading zeros for single-digit states (e.g. CA = "6", CA-01 =
  "601"). The two pre-existing example manifests are updated to match.

Orchestrator (paper-l0/benchmarking/run_benchmark_suite.py)
  Exports each manifest, runs every method declared in it, and (when IPF is in
  the manifest) schedules matched-input rows for the other methods via
  --train-on ipf_retained_authored --score-on ipf_retained_authored. Aggregates
  per-manifest summaries into one tier_<n>_summary.csv per tier plus a unified
  suite_summary.csv. Failures (export-time IPFConversionError, runner non-zero
  exits, missing summary files) are recorded as status=failed rows with the
  captured reason in notes; the orchestrator never aborts the suite, which is
  load-bearing for Tier 3's "completed-or-failed" reporting story.

  Tier 2 rungs are discovered and ordered by target_filters.max_targets
  (smallest first; uncapped rungs sort last) so the summary reads top-to-bottom
  in increasing target count regardless of filename sort order.

  Both run_benchmark_suite.py and benchmark_cli.py prepend the in-tree repo
  root to sys.path before importing policyengine_us_data. This avoids being
  shadowed by an editable install of the same package name pointing at a
  sibling repo, which previously caused fit_l0_weights() to lose the seed
  parameter at script-invocation time only (not when imported from another
  module).

IPF runner: returnNA=FALSE design task closed
  paper-l0/benchmarking/runners/ipf_runner.R now accepts an optional 11th
  positional argument return_na (default TRUE for backwards compatibility).
  When return_na is TRUE and surveysd::ipf returns any NaN weights -- the
  silent-non-convergence path called out in the BENCHMARK_PLAN's "Immediate
  next design tasks" -- the runner exits with a clear error rather than
  writing NaN weights. The Python CLI plumbs method_options.ipf.return_na
  through to the runner so manifests control the behavior declaratively.

Modal-artifact ingest (paper-l0/benchmarking/patch_package_paths.py)
  The Modal-built calibration_package.pkl records absolute paths from the
  build container (/pipeline/artifacts/<run_id>/...) for metadata.dataset_path
  and metadata.db_path. The new helper rewrites those paths in-place to point
  at local copies of the dataset H5 and policy_data.db so the rest of the
  benchmark scaffold (and especially benchmark_export.build_ipf_inputs, which
  has hard existence checks on both paths) runs unchanged on a local checkout.

Documentation (paper-l0/benchmarking/README.md)
  New "Paper-reported tiers" section: which manifests belong to which tier,
  what each method does at each rung, the run_benchmark_suite.py entry point
  with example invocations, and the per-tier summary CSV layout.

Verification done before committing
  Three end-to-end smoke runs were exercised against the production
  calibration package (17,736 targets x 5,159,570 cloned units, n_clones=430)
  to verify the full pipeline:

    1. count-only smoke (29 targets, 100 epochs L0): all five rows green --
       L0 + GREG full-info, IPF on retained-authored subset, plus matched L0
       and GREG runs trained and scored on the IPF subset.
    2. mixed-target smoke (count + dollar combined): L0 and GREG both
       completed end-to-end; this exercises the Tier 1 / Tier 2 mixed-target
       path that count_like_only=true cannot reach.
    3. forced non-convergence (max_iter=5, epsP=1e-20): surveysd::ipf returned
       NaN weights, the runner stopped with the new "did not converge"
       message, and the orchestrator recorded status=failed with the full R
       traceback in notes.

  The smoke manifests, scratch run dirs, and download scratch were removed
  from the repo before committing. The plans (paper-l0/BENCHMARK_PLAN.md and
  outline.md) are intentionally not included in this commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: juaristi22

paper-l0/benchmarking/README.md

@@ -347,3 +347,65 @@ make benchmarking-export MANIFEST=paper-l0/benchmarking/manifests/greg_demo_smal
 make benchmarking-run-greg RUN_DIR=paper-l0/benchmarking/runs/greg_demo_small
 make benchmarking-run-l0 RUN_DIR=paper-l0/benchmarking/runs/greg_demo_small
 ```
+
+## Paper-reported tiers
+
+The manifests under `manifests/tier*.json` are the paper-reported benchmark
+configurations from `paper-l0/BENCHMARK_PLAN.md`. They all read from the same
+saved calibration package and differ only in `target_filters` — the unit
+universe, clone count, source dataset, and initial calibration package are
+fixed.
+
+| File | Tier | Methods | Scope |
+| --- | --- | --- | --- |
+| `tier1_mixed.json` | 1 | L0, GREG | Full filtered slice (count + dollar) over a 5-state, 10-district subset plus national targets |
+| `tier1_ipf.json` | 1 | L0, IPF | Same slice; IPF retains the authored closed subset |
+| `tier2_scaling_250.json` … `tier2_scaling_10000.json` | 2 | L0, GREG, IPF | Scaling ladder by `max_targets`, expanding geography coverage to grow the target set |
+| `tier2_scaling_largest.json` | 2 | L0, GREG, IPF | Largest coherent pre-production subset (no `max_targets` cap) |
+| `tier3_production.json` | 3 | L0, GREG, IPF | Least-filtered view; failures are reportable results |
+
+All Tier 2 / Tier 3 manifests set `method_options.ipf.return_na = true` so
+non-convergence surfaces NaN weights, which `ipf_runner.R` converts into a
+visible runtime error rather than a silent fitted-weight column. A bounded
+GREG variant is intentionally out of scope for the current benchmark.
+
+### One-shot orchestration
+
+`run_benchmark_suite.py` exports each manifest, runs every method declared in
+it, schedules matched IPF / L0 / GREG comparisons (`--train-on
+ipf_retained_authored --score-on ipf_retained_authored`) when IPF is in play,
+and aggregates per-tier summary tables.
+
+```bash
+# All three tiers end-to-end (requires built calibration_package.pkl).
+python paper-l0/benchmarking/run_benchmark_suite.py \
+  --runs-dir paper-l0/benchmarking/runs
+
+# A single tier.
+python paper-l0/benchmarking/run_benchmark_suite.py \
+  --tier tier_1 \
+  --runs-dir paper-l0/benchmarking/runs
+
+# A single rung (re-run after a CI failure).
+python paper-l0/benchmarking/run_benchmark_suite.py \
+  --manifest paper-l0/benchmarking/manifests/tier2_scaling_2500.json \
+  --runs-dir paper-l0/benchmarking/runs
+```
+
+Outputs in `--runs-dir`:
+
+- `tier_1_summary.csv`, `tier_2_summary.csv`, `tier_3_summary.csv` — one row
+  per method per manifest, with status (`completed` / `failed`), runtime,
+  target / unit counts, and the standard error metrics from
+  `compute_common_metrics`. Matched IPF / L0 / GREG rows are tagged with
+  `training_target_set = ipf_retained_authored`.
+- `suite_summary.csv` — concatenated view across all tiers.
+- `<manifest>/inputs/`, `<manifest>/outputs/` — the per-manifest bundle that
+  `benchmark_cli.py export` and `run` produce, including
+  `ipf_conversion_diagnostics.json` whenever IPF was in scope.
+
+Failures (export-time `IPFConversionError`, runner non-zero exit, missing
+output files) appear as `status = failed` rows with the captured reason in
+`notes`. The orchestrator never aborts the suite — Tier 3 explicitly relies
+on this so a GREG out-of-memory or IPF non-convergence is a reportable result
+rather than a missing row.

paper-l0/benchmarking/benchmark_cli.py

@@ -7,6 +7,15 @@
 import time
 from pathlib import Path
 
+# This script can be invoked directly (`python paper-l0/benchmarking/benchmark_cli.py
+# ...`), in which case Python sets sys.path[0] to the script's directory and a sibling
+# editable-installed `policyengine_us_data` package can shadow the in-tree copy. Pin
+# the in-tree repo root ahead of site-packages so `fit_l0_weights` resolves to the
+# version in this repo, not whichever editable install pip found first.
+_REPO_ROOT = Path(__file__).resolve().parents[2]
+if str(_REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(_REPO_ROOT))
+
 import numpy as np
 import pandas as pd
 
@@ -216,6 +225,7 @@ def _run_ipf(run_dir: Path):
     temp_unit_csv = outputs / "_ipf_unit_metadata.csv"
     unit_with_weights.to_csv(temp_unit_csv, index=False)
 
+    return_na_flag = "true" if bool(options.get("return_na", False)) else "false"
     cmd = [
         "Rscript",
         str(RUNNERS_DIR / "ipf_runner.R"),
@@ -229,6 +239,7 @@ def _run_ipf(run_dir: Path):
         str(float(options.get("epsH", 1e-2))),
         household_id_col,
         weight_col,
+        return_na_flag,
     ]
     proc, elapsed = _run_subprocess(cmd)
     if proc.returncode != 0:

paper-l0/benchmarking/manifests/greg_demo_small.example.json

@@ -1,55 +1,55 @@
 {
-  "name": "greg_demo_small",
-  "tier": "tier_a",
   "description": "Example GREG benchmark manifest for a reduced package and a coherent geography subset.",
-  "package_path": "policyengine_us_data/storage/calibration/calibration_package.pkl",
+  "external_inputs": {},
+  "method_options": {
+    "greg": {
+      "epsilon": 1e-07,
+      "maxit": 200
+    },
+    "l0": {
+      "beta": 0.65,
+      "device": "cpu",
+      "epochs": 1000,
+      "lambda_l0": 1e-08,
+      "lambda_l2": 1e-12,
+      "learning_rate": 0.15,
+      "seed": 42
+    }
+  },
   "methods": [
     "l0",
     "greg"
   ],
+  "name": "greg_demo_small",
+  "package_path": "policyengine_us_data/storage/calibration/calibration_package.pkl",
   "target_filters": {
+    "count_like_only": false,
+    "district_ids": [
+      "601",
+      "602",
+      "1201",
+      "1202",
+      "3601",
+      "3602",
+      "4801",
+      "4802",
+      "5301",
+      "5302"
+    ],
     "include_geo_levels": [
       "national",
       "state",
       "district"
     ],
     "include_national": true,
+    "max_targets": 1000,
     "state_ids": [
-      "06",
+      "6",
       "12",
       "36",
       "48",
       "53"
-    ],
-    "district_ids": [
-      "0601",
-      "0602",
-      "1201",
-      "1202",
-      "3601",
-      "3602",
-      "4801",
-      "4802",
-      "5301",
-      "5302"
-    ],
-    "count_like_only": false,
-    "max_targets": 1000
+    ]
   },
-  "external_inputs": {},
-  "method_options": {
-    "l0": {
-      "lambda_l0": 1e-08,
-      "epochs": 1000,
-      "device": "cpu",
-      "beta": 0.65,
-      "lambda_l2": 1e-12,
-      "learning_rate": 0.15,
-      "seed": 42
-    },
-    "greg": {
-      "maxit": 200,
-      "epsilon": 1e-07
-    }
-  }
-}
+  "tier": "tier_a"
+}

paper-l0/benchmarking/manifests/ipf_demo_small.example.json

@@ -1,43 +1,43 @@
 {
-  "name": "ipf_demo_small",
-  "tier": "tier_1",
   "description": "Example IPF benchmark. Uses the same target_filters path as GREG and L0; the IPF converter internally keeps only count-style targets whose constraints resolve through the declared bucket schemas and form a closed categorical margin system. Binary subset families survive only when authored parent totals exist on the exact reduced key; open subset families are dropped.",
-  "package_path": "policyengine_us_data/storage/calibration/calibration_package.pkl",
+  "external_inputs": {},
+  "method_options": {
+    "ipf": {
+      "bound": 10.0,
+      "epsH": 0.01,
+      "epsP": 0.0001,
+      "household_id_col": "household_id",
+      "max_iter": 5000,
+      "weight_col": "base_weight"
+    },
+    "l0": {
+      "beta": 0.65,
+      "device": "cpu",
+      "epochs": 1000,
+      "lambda_l0": 1e-08,
+      "lambda_l2": 1e-12,
+      "learning_rate": 0.15,
+      "seed": 42
+    }
+  },
   "methods": [
     "l0",
     "ipf"
   ],
+  "name": "ipf_demo_small",
+  "package_path": "policyengine_us_data/storage/calibration/calibration_package.pkl",
   "target_filters": {
+    "district_ids": [
+      "601",
+      "602",
+      "603",
+      "604",
+      "605"
+    ],
     "include_geo_levels": [
       "district"
     ],
-    "include_national": false,
-    "district_ids": [
-      "0601",
-      "0602",
-      "0603",
-      "0604",
-      "0605"
-    ]
+    "include_national": false
   },
-  "external_inputs": {},
-  "method_options": {
-    "l0": {
-      "lambda_l0": 1e-08,
-      "epochs": 1000,
-      "device": "cpu",
-      "beta": 0.65,
-      "lambda_l2": 1e-12,
-      "learning_rate": 0.15,
-      "seed": 42
-    },
-    "ipf": {
-      "max_iter": 5000,
-      "bound": 10.0,
-      "epsP": 1e-04,
-      "epsH": 1e-02,
-      "household_id_col": "household_id",
-      "weight_col": "base_weight"
-    }
-  }
-}
+  "tier": "tier_1"
+}

paper-l0/benchmarking/manifests/tier1_ipf.json

@@ -0,0 +1,59 @@
+{
+  "description": "Tier 1 IPF tractable comparison. Same five-state, ten-district geography slice as tier1_mixed.json. The IPF converter retains the authored subset that forms one coherent closed categorical system; the rest is dropped with explicit diagnostics. L0 and GREG can be matched against IPF's retained subset via --train-on ipf_retained_authored --score-on ipf_retained_authored.",
+  "external_inputs": {},
+  "method_options": {
+    "ipf": {
+      "bound": 10.0,
+      "epsH": 0.01,
+      "epsP": 0.0001,
+      "household_id_col": "household_id",
+      "max_iter": 5000,
+      "return_na": true,
+      "weight_col": "base_weight"
+    },
+    "l0": {
+      "beta": 0.65,
+      "device": "cpu",
+      "epochs": 5000,
+      "lambda_l0": 1e-08,
+      "lambda_l2": 1e-12,
+      "learning_rate": 0.15,
+      "seed": 42
+    }
+  },
+  "methods": [
+    "l0",
+    "ipf"
+  ],
+  "name": "tier1_ipf",
+  "package_path": "policyengine_us_data/storage/calibration/calibration_package.pkl",
+  "target_filters": {
+    "count_like_only": false,
+    "district_ids": [
+      "601",
+      "602",
+      "1201",
+      "1202",
+      "3601",
+      "3602",
+      "4801",
+      "4802",
+      "5301",
+      "5302"
+    ],
+    "include_geo_levels": [
+      "national",
+      "state",
+      "district"
+    ],
+    "include_national": true,
+    "state_ids": [
+      "6",
+      "12",
+      "36",
+      "48",
+      "53"
+    ]
+  },
+  "tier": "tier_1"
+}

paper-l0/benchmarking/manifests/tier1_mixed.json

@@ -0,0 +1,54 @@
+{
+  "description": "Tier 1 mixed-target tractable comparison. L0 and GREG fit the full filtered slice (count-like and dollar targets combined) for a coherent five-state, ten-district geography subset plus national targets. Same filtered slice is the input to the matched run reported in tier1_ipf.json.",
+  "external_inputs": {},
+  "method_options": {
+    "greg": {
+      "epsilon": 1e-07,
+      "maxit": 200
+    },
+    "l0": {
+      "beta": 0.65,
+      "device": "cpu",
+      "epochs": 5000,
+      "lambda_l0": 1e-08,
+      "lambda_l2": 1e-12,
+      "learning_rate": 0.15,
+      "seed": 42
+    }
+  },
+  "methods": [
+    "l0",
+    "greg"
+  ],
+  "name": "tier1_mixed",
+  "package_path": "policyengine_us_data/storage/calibration/calibration_package.pkl",
+  "target_filters": {
+    "count_like_only": false,
+    "district_ids": [
+      "601",
+      "602",
+      "1201",
+      "1202",
+      "3601",
+      "3602",
+      "4801",
+      "4802",
+      "5301",
+      "5302"
+    ],
+    "include_geo_levels": [
+      "national",
+      "state",
+      "district"
+    ],
+    "include_national": true,
+    "state_ids": [
+      "6",
+      "12",
+      "36",
+      "48",
+      "53"
+    ]
+  },
+  "tier": "tier_1"
+}