Align UK model/data pipeline with US patterns: reported-anchoring, second-stage imputation, and downstream bugs

[MaxGhenis]

Summary

Surveying the UK model and enhanced FRS pipeline against what's been done in policyengine-us / policyengine-us-data over the last few months, several architectural gaps and specific bugs surface. Filing this as a tracking issue so fixes can land as separate, scoped PRs.

Architectural gaps vs US

1. _reported at runtime in would_claim_* formulas

US pattern (confirmed by reading policyengine_us_data/utils/takeup.py::assign_takeup_with_reported_anchors and call sites in cps.py): _reported columns are only consulted once, in the data-build pipeline, to anchor stochastic takeup flags (reported recipient → takeup=True with certainty; rest filled probabilistically to a target rate). At runtime, policyengine-us formulas only read the pre-computed takes_up_X_if_eligible flag.

UK pattern: some would_claim_* variables already match US — would_claim_uc, would_claim_pc are input-only with default_value = True, populated stochastically in policyengine-uk-data/datasets/frs.py. But several still compute from reported at runtime:

• would_claim_housing_benefit: claims_all_entitled_benefits | (housing_benefit_reported > 0)
• would_claim_IS: same shape
• would_claim_WTC: same shape
• would_claim_CTC: same shape

Reform-friendliness is broken as a result. A reform expanding housing-benefit eligibility only reaches existing housing_benefit_reported > 0 claimants — non-claimants who newly qualify get nothing unless the user manually sets claims_all_entitled_benefits = True everywhere.

Fix: convert each to input-only default_value = True (matching would_claim_uc), populate stochastically in uk-data/datasets/frs.py with the existing takeup-rate pattern.

2. Takeup assignment ignores reported data

In policyengine_uk_data/datasets/frs.py:833-868, all takeup flags are set by pure random draw:

pe_benunit["would_claim_uc"] = generator.random(len(pe_benunit)) < universal_credit_rate

This ignores the fact that respondents who reported receiving UC clearly did take it up. That makes the distribution of simulated claimants less accurate than it could be — a respondent with universal_credit_reported > 0 should have would_claim_uc = True with certainty, with the remaining draws calibrated to hit the overall target rate.

Fix: port policyengine_us_data/utils/takeup.py::assign_takeup_with_reported_anchors into uk-data, apply it to each takeup variable where a reported column exists.

3. SPI-donor imputation only replaces 6 income vars; rest stay as FRS donor's

policyengine_uk_data/datasets/imputations/income.py::impute_income creates a zero-weight copy of 10k FRS rows and runs a QRF on it with:

PREDICTORS = ["age", "gender", "region"]
IMPUTATIONS = [
    "employment_income",
    "self_employment_income",
    "savings_interest_income",
    "dividend_income",
    "private_pension_income",
    "property_income",
]

It replaces only these six variables on the SPI donor side. Everything else (rent, wealth, pension contributions, consumption, gift_aid, tax-reported characteristics, benefits-reported flags) stays as whatever middle-income FRS donor happened to be sampled. So a synthetic SPI donor ending up with £2M imputed self-employment income still carries a typical-FRS-donor's savings balance, pension contribution, and zero gift_aid.

This is exactly the failure mode PolicyEngine/policyengine-us-data#589 ("QRF-impute CPS-only variables for PUF clone half", merged 2026-03-14) fixed in the US pipeline. The US equivalent now trains a second-stage QRF on CPS where predictors = demographics + newly-imputed income vars and outputs = the ~60 CPS-only variables; for PUF-clone prediction it substitutes the already-imputed PUF incomes as predictors so CPS-only variables come out consistent with imputed income.

Fix: add the analogous second-stage QRF in policyengine_uk_data/datasets/imputations/, training on FRS data with predictors = demographics + 6 income vars, outputs = the consequential FRS-only variables (see sublist under bug #6 below).

Specific bugs this gap has produced

4. Degenerate income composition above £1M total income

Current weighted bracket analysis on the enhanced FRS (1.50.3) at 2025:

Bracket	Rows	employment_income share nonzero	self_employment_income share nonzero	dividend_income	private_pension_income	property_income
£200k+	558	85%	19%	37%	13%	32%
£500k+	30	40%	73%	33%	33%	20%
£1M+	10	0%	100%	0%	0%	0%

Every £1M+ row has 100% of its imputed income as self-employment and zero everywhere else. That's the QRF memorising a narrow "sole trader with massive profits and blank everything else" pattern in the SPI tail donor set.

gift_aid is zero across the entire dataset — the variable is in SPI_RENAMES but not in IMPUTATIONS, so it never gets populated on SPI donors, and FRS doesn't capture it. Missing ~£1-1.5bn/yr of Gift Aid tax relief.

5. Benefit-aggregate misses against OBR (2025)

Quick scan of model vs external reference points on the fresh 1.50.3 build:

variable	model	target	gap	notes
income_support	£1.07bn	~£0.2bn	+£0.9bn	Legacy, near-fully migrated. Likely same reported > 0 pattern as tax credits.
esa_contrib	£5.56bn	~£2.5bn	+£3.1bn	Large. Possibly reported-based takeup + no phase-out for migrated cases.
attendance_allowance	£8.76bn	~£7.0bn	+£1.8bn	Moderate. Likely cap-rate or takeup issue.
tax_free_childcare	£0.41bn	~£0.9bn	-£0.5bn	Understated by about half.
state_pension	£127.5bn (after #1618)	~£140bn	-£12bn	Remaining gap after fixing state_pension_type; likely Additional State Pension / SERPS coverage.

6. Cosmetic / small

• policyengine_uk/parameters/gov/dwp/tax_credits/min_benefit.yaml:10 has unit: currency-USD. Should be currency-GBP.
• benunit_weekly_hours has label "Average weekly hours worked by adults in the benefit unit" but the formula is adds = ["weekly_hours"] (sum, not average). Used once in one test.
• Some ~bool_expression paths trigger DeprecationWarning: Bitwise inversion '~' on bool that becomes an error in Python 3.16. Scanning gov_balance's dependency tree for offending Python-bool scalars would get ahead of that.

Candidate PR sequence (if we want to chew through this)

1. Convert remaining would_claim_* formulas to input-only. Two-repo PR pair: one in policyengine-uk (remove the formulas, keep default_value = True), one in policyengine-uk-data/datasets/frs.py (add stochastic assignment for each). Low risk; matches the pattern already applied to UC/PC.
2. Port assign_takeup_with_reported_anchors to uk-data and apply it to each takeup variable. Pure data-pipeline change. Catches reported=True → takeup=True for free and tightens per-variable calibration.
3. Add second-stage QRF for FRS-only variables on SPI donors. The biggest ticket but directly addresses the "£1M earners have zero-elsewhere" failure. Train QRF on FRS with predictors = demographics + 6 income vars, outputs = rent, mortgage_interest_repayment, gift_aid, covenanted_payments, charitable_investment_gifts, other_deductions, employee_pension_contributions*, employer_pension_contributions, total_wealth components, capital_allowances, deficiency_relief, and the full list of *_reported benefits (so high-income SPI donors don't carry a middle-income donor's HB/UC/WTC reports).
4. Add gift_aid to IMPUTATIONS in policyengine_uk_data/datasets/imputations/income.py so the SPI donor side at least gets this one ~£1-1.5bn deduction stream. Independent of (3); can land immediately as a one-line change plus regression bound.
5. Residual-benefit-aggregate follow-ups. Separate small PRs for IS phase-out (analogous to the TC PR Stop paying Tax Credits after DWP managed migration completed #1619), ESA contrib investigation, AA calibration, TFC understatement, Additional State Pension coverage. Each is ~20-100 lines plus a YAML test.

Happy to pick up whichever of these the team wants to see first.