Merge pull request #531 from PolicyEngine/census-block-calibration

Add census-block-first calibration pipeline (from PR #516)

Author: baogorek

changelog_entry.yaml

@@ -0,0 +1,22 @@
+- bump: minor
+  changes:
+    added:
+    - Census-block-first calibration pipeline (calibration/ package) ported from PR #516
+    - Clone-and-assign module for population-weighted census block sampling
+    - Unified matrix builder with clone-by-clone simulation, COO caching, and target_overview-based querying
+    - Unified calibration CLI with L0 optimization and seeded takeup re-randomization
+    - 28 new tests for the calibration pipeline
+    - Integration test for build_matrix geographic masking (national/state/CD)
+    - Tests for drop_target_groups utility
+    - voluntary_filing.yaml takeup rate parameter
+    changed:
+    - Rewrote local_area_calibration_setup.ipynb for clone-based pipeline
+    - Renamed _get_geo_level to get_geo_level (now cross-module public API)
+    fixed:
+    - Fix Jupyter import error in unified_calibration.py (OutStream.reconfigure moved to main)
+    - Fix modal_app/remote_calibration_runner.py referencing deleted fit_calibration_weights.py
+    - Fix _coo_parts stale state bug on build_matrix re-call after failure
+    - Remove hardcoded voluntary_filing rate in favor of YAML parameter
+    removed:
+    - SparseMatrixBuilder, MatrixTracer, and fit_calibration_weights (replaced by unified pipeline)
+    - 8 old SparseMatrixBuilder-dependent tests (replaced by new test_calibration suite)

docs/calibration_matrix.ipynb

@@ -51,20 +51,16 @@
     "import pandas as pd\n",
     "\n",
     "from policyengine_us_data.storage import STORAGE_FOLDER\n",
-    "from policyengine_us_data.datasets.cps.local_area_calibration.sparse_matrix_builder import (\n",
-    "    SparseMatrixBuilder,\n",
+    "from policyengine_us_data.calibration.unified_matrix_builder import (\n",
+    "    UnifiedMatrixBuilder,\n",
     ")\n",
     "from policyengine_us_data.datasets.cps.local_area_calibration.calibration_utils import (\n",
-    "    get_all_cds_from_database,\n",
     "    STATE_CODES,\n",
     ")\n",
     "\n",
     "db_path = STORAGE_FOLDER / \"calibration\" / \"policy_data.db\"\n",
     "db_uri = f\"sqlite:///{db_path}\"\n",
-    "cds = get_all_cds_from_database(db_uri)\n",
-    "builder = SparseMatrixBuilder(\n",
-    "    db_uri, time_period=2024, cds_to_calibrate=cds\n",
-    ")"
+    "builder = UnifiedMatrixBuilder(db_uri, time_period=2024)"
    ]
   },
   {