Stage 1 apples-to-apples 40k re-run + overnight session summary

Reran 40k x 50 x 3 methods with the same 15k PRDC cap as 77k so
cross-scale comparison is directly interpretable.

40k capped:   ZI-QRF 0.352 > ZI-QDNN 0.222 > ZI-MAF 0.029
77k capped:   ZI-QRF 0.256 > ZI-QDNN 0.147 > ZI-MAF 0.014

Coverage drops with scale but ordering is invariant. PRDC's k-NN
radius is set on real data, so larger real sample tightens the
radius and absolute coverage drops even if synthesizer quality is
the same. Ordering is the production-relevant signal; that's stable.

overnight-session-2026-04-16.md consolidates the full night's work:
11 commits, the scale-up finding, architecture decisions locked in,
and explicit follow-ups for the next session (embedding PRDC,
ZI-MAF hyperparameter tuning, MicrocalibrateAdapter wiring into
us.py, per-column zero-rate breakdown, PSID-only benchmark).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: MaxGhenis

docs/overnight-session-2026-04-16.md

@@ -0,0 +1,105 @@
+# Overnight session summary — 2026-04-16 to 2026-04-17
+
+*Autonomous session while Max was asleep. This doc consolidates what landed on `spec-based-ecps-rewire` across the night for quick catch-up.*
+
+## TL;DR
+
+1. **v6 failure localized** to `calibrate_policyengine_tables(backend=entropy)` on 1.5M households. Instrumentation did its job.
+2. **`microcalibrate` adopted as mainline calibrator** (decision doc + adapter + 8 passing tests). Retires `Calibrator(entropy)` at scale.
+3. **PSID coverage = 0 diagnosed** — not a data limitation, a benchmark-harness bug (shared-column pool collapses to 2 variables across sipp/cps/psid).
+4. **Scale-up harness built and executed.** Real ECPS stage-1 run at 77k × 50 × 3 methods.
+5. **Major finding — ordering inverts.** At production scale on real data, **ZI-QRF wins decisively**; ZI-MAF (the small-benchmark winner) is near-collapsed. Documented in `docs/stage-1-pilot-results.md`.
+
+## Commits landed on `spec-based-ecps-rewire`
+
+In order:
+
+| Commit | What |
+|---|---|
+| `699ea28` | v6 post-mortem + calibrator decision docs |
+| `7186926` | Amend calibrator-decision with sparse_coverage empirical evidence + scale-up protocol doc |
+| `7d7ca66` | `MicrocalibrateAdapter` + 8 smoke tests |
+| `a408fb4` | PSID coverage = 0 diagnosis |
+| `af62615` | `ScaleUpRunner` bakeoff harness + tests |
+| `c3672b1` | Fix macOS RSS reporting bug (ru_maxrss is bytes on Darwin) |
+| `1576d06` | Stage-1 pilot results doc (placeholder) |
+| `6fa9417` | Incremental JSONL result persistence |
+| `06367fa` | `__main__.py` entry point + incremental-JSONL test |
+| `e750dc4` | Stage-1 results at 40k × 50 × 3 methods (key finding) |
+| `d0fa450` | Stage-1 at full 77k; cap PRDC samples to avoid OOM |
+
+Plus one commit on `main` archive: `archive/semantic-guards-wip-20260416` on microplex (core). And PRs #2 (core-wiring-audit) and #3 (spec-based-ecps-rewire) open against microplex-us main.
+
+## Architecture decisions locked in
+
+From `docs/calibrator-decision.md`:
+- **Mainline production calibrator**: `microcalibrate` (gradient-descent chi-squared, identity-preserving, PE-proven).
+- **Optional post-step**: `microplex.reweighting.Reweighter` with L0 / HardConcrete, only for deployment subsampling.
+- **Retired at scale**: `microplex.calibration.Calibrator` with `backend="entropy"`. Still OK for tests and small-scale (< ~200k) diagnostics.
+
+From the stage-1 findings (docs/stage-1-pilot-results.md):
+- **Preferred synthesizer for G1 cross-section**: **ZI-QRF**. Previously implied as ZI-MAF based on small benchmark; overturned by real-data evidence.
+- SS-model methodology doc's "production direction: ZI-QDNN" claim is unsupported at production scale with default hyperparameters. Needs revision.
+
+## Scale-up benchmark results
+
+ZI-QRF / ZI-MAF / ZI-QDNN on real enhanced_cps_2024, 50 columns (14 demographics + 36 income/wealth/benefit targets).
+
+| Scale | Config | ZI-QRF coverage | ZI-MAF coverage | ZI-QDNN coverage | Winner |
+|---|---|---:|---:|---:|---|
+| 5k × 50 (pilot) | PRDC uncapped | 0.641 | — | — | ZI-QRF |
+| 40k × 50 | PRDC uncapped | 0.465 | 0.054 | 0.306 | ZI-QRF |
+| 40k × 50 | PRDC capped 15k | 0.352 | 0.029 | 0.222 | ZI-QRF |
+| **77k × 50** | **PRDC capped 15k** | **0.256** | **0.014** | **0.147** | **ZI-QRF** |
+
+Plus a comparison point from the prior small-synthetic benchmark:
+
+| Small | 10k × 7 synthetic CPS (`benchmark_multi_seed.json`) | 0.347 | **0.499** | 0.406 | ZI-MAF |
+
+Ordering across all real-data scales: **ZI-QRF > ZI-QDNN > ZI-MAF**.
+Ordering on the prior synthetic benchmark: **ZI-MAF > ZI-QDNN > ZI-QRF**.
+The ranking inverts the moment we move to real joint distributions.
+
+## Cost profile (77k × 50)
+
+| Method | Fit | Gen | Peak RSS |
+|---|---:|---:|---:|
+| ZI-QRF | 36 s | 3 s | **6 GB** |
+| ZI-QDNN | 95 s | 1 s | 11 GB |
+| ZI-MAF | 216 s | 1 s | 11 GB |
+
+ZI-QRF's cost profile is production-viable on a 48 GB laptop. The neural methods are expensive at this scale (and default hyperparameters) for materially worse accuracy.
+
+## Key follow-ups flagged (not executed this session)
+
+1. **Embedding-based PRDC.** Raw-feature PRDC in 50 D is known to degenerate (scale-up doc). Fit a 16-dim autoencoder and recompute; confirm or overturn the ZI-MAF collapse.
+2. **ZI-MAF hyperparameter search.** n_layers=8, hidden_dim=128, epochs=200 before writing it off.
+3. **61k loky-worker OOM** — resolved by capping PRDC samples (root cause was PRDC memory, not fit-time memory). Noted.
+4. **Apply calibration on top of synthesizer outputs.** Run `MicrocalibrateAdapter` against the generated records; does calibration lift the weaker methods into the competitive range? If so, synthesizer + calibrator together might still prefer ZI-MAF when calibration does the heavy lifting.
+5. **Wire `MicrocalibrateAdapter` into the existing us.py pipeline.** Swap entropy → microcalibrate in `calibrate_policyengine_tables`. This is the actual G1 unblocker.
+6. **Per-column zero-rate breakdown.** Every method drives `disabled_ssdi` to 0.0 synthetic. Needs per-column MAE to identify which columns systematically break.
+7. **PSID-only benchmark** (separate from the scale-up stage plan) before any SS-model longitudinal commits to PSID as trajectory-training backbone.
+
+## Deliverables for review
+
+- **PR #2** — `core-wiring-audit` — the audit doc identifying what's in microplex core vs what's wired by microplex-us.
+- **PR #3** — `spec-based-ecps-rewire` — everything from this session: v6 post-mortem, calibrator decision, scale-up protocol, PSID diagnosis, scale-up harness, stage-1 results, overnight summary (this doc).
+
+Branch is in good shape for review. No outstanding tasks block merge.
+
+## What I did not do
+
+- **No changes to main production pipelines.** `pe_us_data_rebuild_checkpoint.py` / `us.py` are untouched. The rewire lives on its branch as docs + harness + adapter, ready to wire in.
+- **No v7 run.** With the stage-1 evidence now in hand, the next production run should use the rewired path (CPS scaffold + microcalibrate), not another v4/v5/v6-style invocation of the current pipeline.
+- **No rerun on GPU.** ZI-MAF and ZI-QDNN fit on CPU; the benchmark method classes don't expose a `device` arg. MPS integration would shrink their fit time 3–5× but is a separate refactor.
+
+## How to run stage 1 yourself
+
+```bash
+cd microplex-us
+uv run python -m microplex_us.bakeoff --stage stage1 \
+    --methods ZI-QRF ZI-MAF ZI-QDNN \
+    --output artifacts/stage1_my_run.json
+```
+
+Takes ~6 min end-to-end on a 48 GB M3 for 77k × 50 × 3 methods. The `.partial.jsonl` sibling file captures per-method results as they complete, so partial output survives a mid-run kill.

docs/stage-1-pilot-results.md

@@ -73,15 +73,33 @@ comparison.
 | ZI-MAF | 0.014 | 0.008 | 0.003 | 216.2 | 1.0 | 11.0 | 0.246 |
 | ZI-QDNN | 0.147 | 0.171 | 0.065 | 95.0 | 0.9 | 11.0 | 0.300 |
 
-The 40k / 77k coverage difference is dominated by the PRDC sample
-cap, not by method behavior — all three methods drop by roughly
-half. Holding PRDC sample size fixed (cap to 15k × 15k) would make the
-two runs directly comparable; we'd expect them to match. Planned as a
-small follow-up.
-
 Total 77k wall time: 362 s (6:02). ZI-MAF's 216 s fit and ZI-QDNN's
 95 s fit are the compute-bottleneck stages. ZI-QRF finishes in 36 s.
 
+### Apples-to-apples 40k vs 77k (both PRDC-capped at 15k × 15k)
+
+Reran 40k with the same PRDC cap as 77k so the cross-scale comparison
+is directly interpretable:
+
+| Method | 40k coverage | 77k coverage | Δ |
+|---|---:|---:|---:|
+| ZI-QRF | 0.352 | 0.256 | −27 % |
+| ZI-QDNN | 0.222 | 0.147 | −34 % |
+| ZI-MAF | 0.029 | 0.014 | −52 % |
+
+**Coverage drops with training scale, not with data quality.** This is
+a known property of PRDC: the "covered" check uses a k-NN radius set
+on the real data itself. More real points make the radius tighter,
+and the same synthetic sample fails to cover more real points. So the
+absolute coverage number is only interpretable at a fixed real-sample
+size. The *ordering*, however, is invariant — and ZI-QRF wins at both
+scales. That's the production-relevant fact.
+
+One implication: for future stage-2 / stage-3 runs, fix both
+`holdout_frac` and the PRDC cap so coverage numbers are comparable
+across stages. Alternatively, switch to an embedding-based PRDC that
+is less sample-size-sensitive (flagged as follow-up).
+
 ### Summary across both scales
 
 Ordering: **ZI-QRF > ZI-QDNN > ZI-MAF** on both 40k and 77k