Pre-circulation review: four blockers, majors, and a fix checklist

[MaxGhenis]

Two independent reviews of the repo at 3d2aabf (Claude Fable 5 and GPT-5.6 codex, both at max reasoning effort, run at Max's request; findings cross-verified against the JR16 PDF, the policyengine-uk 2.89.2 source, policyengine-uk-data, and every committed results file). Overall: the quantitative core is internally consistent — every paper number we checked matches the committed results, and the JR16 mechanics are mostly faithfully replicated — but there are four blockers to fix before this circulates, plus a tail of smaller items.

A companion PR fixes the mechanical subset (packaging, pins, citations, doc corrections — nothing that changes a number). The items below need either your local files or your judgment.

Blockers

1. The exposure crosswalk was never committed — nothing runs from a fresh clone

.gitignore:1 is the unanchored pattern data/, which also matches uk_ai_study/data/. git log --all -- uk_ai_study/data/ is empty: uk_soc2020_major_group_ai_exposure.csv has never been in the repo, even though exposure.py loads it and pyproject.toml force-includes it in wheels. Every entry point dies at load_major_group_exposure(). This also makes the intro's claim ("every result in this report can be regenerated from public code") false, and the exposure/θ/alternative-index values unauditable.

Fix: push your local uk_ai_study/data/ (the companion PR anchors the gitignore so it will stay committed), ideally with a build script or provenance note tying it to populace #325. Sanity check: its c_aioe column should reproduce the nine values in results/appendix/job_loss_by_major_group.csv (0.619, 0.604, 0.472, 0.744, −0.333, −0.140, 0.269, −0.539, −0.729).

2. Headline inequality results use the wrong income concept

In policyengine-uk 2.89.2, household_net_income adds household_benefits — which includes nhs_spending, dfe_education_spending, dft_subsidy_spending, childcare entitlements — and subtracts household_tax, which includes VAT, fuel duty, SDLT, TV licence, capital gains tax and student loan repayments. So the Gini, deciles, and income-change results are computed on a broad in-kind-and-indirect-tax concept, while in_poverty_bhc uses equiv_hbai_household_net_income (the actual HBAI cash concept). Two different income definitions across headline metrics, both called "equivalised household disposable income" in the paper. The baseline Gini of 0.3087 (HBAI BHC is ~0.34–0.35) is the tell.

Fix: use hbai_household_net_income / equiv_hbai_household_net_income consistently for deciles, changes, decomposition and Gini; regenerate all headline results. Whether "Gini rises in all 66 scenarios" survives on HBAI income is an empirical question the re-run answers.

3. The fig 4.4 decomposition and appendix B.3/B.5 mix incompatible weighting bases

_decile_frame (replicate_jr16.py) and appendix.py call sim.calculate(var, map_to="person") on household-level variables, which broadcasts the household total to every member; person-weight-summing then counts each household once per member. Committed evidence:

• fig4_4_decomposition.csv, decile 7: market −1.88% + benefits +0.71% + tax +0.33% = −0.84%, but disposable = −4.50%. JR16 defines these bars as additive; the gaps are −0.46 to −3.66pp across deciles.
• baseline_distributions.csv: "aggregate" benefits £683bn (actual ≈ £310–320bn incl. state pension), disposable income £3,259bn (actual ≈ £1.7tn) — while the person-level columns in the same table are right (IT+NI £312bn).

As printed, fig 4.4 shows the tax-benefit system amplifying the shock while the text describes cushioning. (Part of the residual gap is also perimeter: household_net_income includes items — in-kind benefits, indirect taxes, pension contributions, student loans — outside the four components shown.)

Fix: build the decomposition at household level (aggregate person components to household, weight by household_weight once, one shared denominator, explicit residual), and assert additivity per decile before writing output.

4. Displaced workers are left in a contradictory labour state

The pipeline zeroes employment_income but leaves hours_worked untouched, and in_work in policyengine-uk is hours>0 | earnings>0 — so displaced workers remain "in work" and keep the UC childcare element, tax-free childcare and extended childcare entitlements (all three condition on in_work; all three sit inside household_benefits, i.e. inside the income measure). Their unchanged pension-contribution inputs also keep being deducted from now-zero earnings. Separately, employment_status is set only in runner.py (inside a try/except whose status_applied flag is dead code) — the fig-4.4, grid, appendix and paper-scenario pipelines never set it, and in 2.89.2 it isn't consumed by ordinary tax-benefit formulas anyway.

Fix: one shared transition constructor used by every script: zero hours, decide pension/salary-sacrifice/statutory-pay treatment explicitly, and document what "displaced" means in model terms. Test a synthetic displaced worker end-to-end.

Major

5. Wage shock matches neither JR16 eq 3.5 nor the paper's eq (3). JR16 normalises by the employment-weighted mean complementarity (verified in the PDF, §3.3.2); shocks.py uses an earnings-weighted mean over the full pre-displacement wage bill (its comment says "as in JR16" — it isn't), and the paper prints JR16's equation while claiming the property only the earnings-weighted version has. Observable: in fig 4.2 no decile exceeds 2.66% while JR16's own fig 4.2 reaches 2.8%. Pick the estimand (JR16-literal vs fixed survivor pool), implement it, describe it accurately, and add a per-seed conservation test.

6. Survey weights are used twice in the displacement draw, making a represented person's risk depend on their record's grossing weight: within a group, ordering probability ∝ weight and the quota is consumed in weight units. Toy check (weights {1, 9}, quota 5): inclusion probabilities are 10.0% vs 54.4% where the intended risk is 50% for both. Uniform ordering keys within group (exposure is constant within 1-digit groups) give equal inclusion in expectation and preserve the quota.

7. The displacement universe is SOC-matched employees only (22.95m), and the asymmetry is undisclosed: employees without a SOC code are excluded from displacement but included in the wage uplift with mean-imputed θ. The methodology text ("the scenario displacement rate multiplied by (weighted) employee employment") doesn't match the code. Report match coverage, reconcile the universe to an external employee total, and treat unmatched workers consistently.

8. Committed artifacts the committed code cannot produce: all seven results/jr16/*.png main-text figures, decomposition_ci.png, b9/b11 PNGs, and results/appendix/gender_incidence.json (the whole §5.6 gender section) have no generating code; grid.csv carries two columns (net_revenue_change_bn, net_revenue_change_pct_of_receipts — the ones the paper quotes) that run_grid doesn't write, while the column it does write divides by a negative base and ships sign-flipped nonsense (u10w0 reads +186.6%); exposure_sensitivity.json is hand-rounded relative to the writer. Please push the actual scripts used, and make the grid resume logic fingerprint-aware (it currently skips all 66 cells on any rerun).

9. Fiscal perimeter switches between outputs with no bridge: presets use broad gov_balance; the grid uses IT+NI−benefits (re-expressed against gross receipts). Pick one primary perimeter and publish a component bridge.

10. Monte Carlo treatment is inconsistent: figs 4.1/4.2 average 50 draws; fig 4.4, the 66 grid cells, B.7/B.8, the preset JSONs and paper-anchored scenarios are single seed-0 draws; the robustness summary uses 20. Seed 0 vs 20-draw mean differs materially (Gini +1.32 vs +1.15pp). Also the abstract's "±£1.7bn" is one SD while other intervals are MC standard errors — label them distinctly.

11. Scenario-anchor attributions overstate their sources. (a) The "low = 1% per Acemoglu (2025)" anchor: his 0.93–1.16% is ten-year GDP growth, not an employment-displacement rate (a misreading inherited from JR16's own fn.8 — worth saying so). (b) Briggs & Kodnani's 7% is a task-exposure classification with re-employment assumed, and 2.6% is a productivity-growth median — call these inherited JR16 calibration conventions, and state the task→job, productivity→wage, full-year, no-re-employment pass-throughs. (c) The three "paper-anchored" scenarios in Table 3 are transports, not direct implementations — most seriously, Klein Teeselink's 4.5% appears to be the effect at maximal firm exposure vs zero (average-exposure effect ≈0.24%), applied here economy-wide; H&L's "junior" is a seniority classifier, not age<30, and the 16% scale factor is a US adopter share; Canaries' 16% is a relative decline vs the least-exposed quintile. The footnotes gesture at this; "anchored directly to headline estimates" (§5.7) overstates it. Please verify each against the source and either build an explicit transport model or relabel as author-designed stress tests. Related version inconsistency: literature.tex says 13%, Table 3's footnote and paper_scenarios.py use 16%, and PRESETS["high"] is 13% attributed to the same paper.

12. The OBR "fan" sentence (§6): the July 2025 Fiscal risks and sustainability report discusses AI as a productivity risk but does not publish an AI fiscal-scenario fan a static one-year £18.6bn can be placed "within". Verify against the exact OBR document intended, or delete.

13. "Monotone"/"rising steadily"/"no statistically meaningful reversal" claims are contradicted by the committed CSVs: transitions fall d2→d3 (1.199→1.174%) and d5→d6 (2.496→2.260%, non-overlapping CIs); the 20-draw disposable series reverses d5→d6 (−2.41 vs −1.87, diff ≈ 2.4 SE by an unpaired approximation). The reversal is structural, not draw noise. Describe it, or test paired draw-level differences before claiming insignificance.

Worth deciding / disclosing (medium)

14. Transition-share denominators: fig 4.1's shares divide by all persons in the decile (children included — the caption is right); the summary and results text alternate between "workers", "adults" and "people". One label, used everywhere; report conditional worker risk separately if wanted.
15. The capital-only cell shows the always-on capital shock drives the low scenario's disposable-income story (u1w0: −0.009% with capital vs −0.586% without). Good news from your own committed data: on the no-capital grid, Gini still rises in all 65 non-null cells (max +1.87pp) — the universal-Gini headline doesn't depend on the capital assumption and the paper could say so, citing B.11. The "employment-only" label on low is fixed in the companion PR.
16. Paired-runs sentence in §3.4 ("sampling variation in the survey cancels") overstates — paired designs remove between-sample noise, not sampling uncertainty about the population effect; only MC identity-draw error is quantified. The mechanism attributions in §5.5–5.7 (savings thinness, couple pooling) are hypotheses without saved decompositions — label them as such or produce the decompositions.
17. Crosswalk provenance (populace #325): reconstructed θ with a documented level offset, chained SOC2018→2010→ISCO→SOC2020 with unweighted many-to-many means, ASHE weights for 340/412 unit groups. One limitations sentence covering this would inoculate the exposure section.
18. "One of Europe's most redistributive tax-benefit systems" (intro/summary) — contestable; cite a cross-country redistribution measure or soften.
19. The conclusion calls the shock "regressive" after §5 establishes progressive incidence — rewrite once the corrected fig 4.4 exists (fixing it now would encode numbers that are about to change).
20. Negative incomes are bottom-coded to zero before every Gini — fine, but disclose and show sensitivity.

(Citation metadata — JR16 author initials, Brynjolfsson 2025a/b disambiguation, bibliography ordering, the two mis-attached Williamson citations, equivalisation-scale disclosure, the self-employed-capital sentence, README fn.3→§3.2, and assorted doc nits — are handled in the companion PR. Two names to verify against sources when convenient: whether "Hosseini" should be "Hosseini Maasoum", and the Rockall et al. IMF title.)

What checked out

So it's not all grim: preset table, MC summary (18.60 ± 1.66, range 15.58–21.57; +1.99 ± 0.23; +1.15 ± 0.11), uniform comparator, exposure sensitivity, grid corners and Gini range, figs 4.1–4.3 values, the job-loss table, b9/b11 claims, gender and age numbers, and the KT/H&L/Canaries table all reproduce from the committed files exactly as quoted. Eq 3.4's quota mechanics (incl. the min-zero shift and the stochastic-rounding quota fill — nice touch), the capital factor and rent exclusion, survivors-only wage scope, fixed baseline deciles, person-weighted poverty on a correctly-disclosed absolute line, the weighted Gini algebra, and the person_id = SERNUM*1000+PERSON join (verified against policyengine-uk-data) are all right. The limitations section is more honest than most working papers in this genre.

Decision checklist (the three calls that are yours)

1. Wage rule: JR16-literal (employment-weighted θ̄, sector % changes) or fixed survivor pool (earnings-weighted θ̄)? Either is defensible; paper and code must be the same one.
2. Transition contract: what does "displaced" set, beyond earnings=0 and hours=0 — pension contributions, salary sacrifice, statutory pay?
3. Narrative after the HBAI re-run: the cushioning story, the universal-Gini claim, and the monotonicity language get rewritten around whatever the corrected numbers say.

Push your local uk_ai_study/data/ + figure/gender scripts and answer 1–2 (or say "your call"), and we're happy to implement the fixes and run the full regeneration from an empty results/.

cc @vahid-ahmadi

---

Reviews run by Claude (Fable 5) and Codex (GPT-5.6, ultra reasoning); each reviewer's findings were verified by the other against the JR16 PDF, policyengine-uk 2.89.2 source, and the committed results before inclusion here.

[MaxGhenis]

Finding 0 (framing — precedes everything above): the headline incidence result is an assumption echo, and it takes a side in a live debate

This one came out of discussion after the review posted, and it's arguably the most important item here for whether the paper should circulate under the PolicyEngine name in its current shape.

The problem. The title, abstract and executive summary present "displacement risk is concentrated at the top of the income distribution" as the paper's central and robust finding. It isn't a finding — it's the allocation rule restated. Fig 4.1's gradient follows near-mechanically from two inputs chosen upstream of any simulation: the C-AIOE index, and where high-C-AIOE occupations sit in the household income distribution. The microsimulation contributes nothing to that claim; what it genuinely contributes is the mediation — what the tax-benefit system and the Exchequer do conditional on an incidence.

And the antecedent is hotly contested. The exposure→displacement mapping this study inherits from JR16 is one school in an open debate, and the other schools predict different incidence shapes:

• Expertise/compression (Autor). Autor's argument (e.g. Applied AI as an opportunity to rebuild middle-class jobs, NBER 2024) is that AI lets mid-skill workers perform expert tasks — threatening elite expertise rents (a price effect) while raising mid-skill capability. That is a compression story, not a top-decile displacement story.
• Adaptability (Manning and the adjustment literature). Displacement given exposure should be lowest exactly where this model puts the most displacement: high-skill workers have the greatest task breadth, retraining capacity and occupational mobility. The Humlum–Vestergaard population-scale null (already cited in §2) is consistent with adaptation absorbing exposure.
• Decision-making complementarity. The prediction-vs-judgment framing (Agrawal–Gans–Goldfarb) and the decision-making-premium literature (Deming) imply managers and high-agency professionals — SOC group 1, displaced at 9.6% in this model — may be the economy's strongest net complements to AI. The θ adjustment is the framework's gesture at this, but θ is a coarse occupation-level O*NET reconstruction and displacement still scales with exposure.
• The realized evidence — including the three papers this study itself implements — points at juniors and mid-skill functions (support, writing, translation), not top-decile households. §5.7 already concedes that exposure frameworks have "a structural blind spot" in the age dimension; the same concession applies to the income dimension, and the paper's own Table 3 scenarios produce income incidence nothing like fig 4.1.

Why this matters for PolicyEngine specifically. PE's positioning is that it computes how the system responds to inputs; it doesn't adjudicate contested empirical priors. As titled and framed, the paper adjudicates one.

The fix (restructure, don't shelve). Make incidence a first-class scenario axis, exactly as shock size already is. Nearly all the machinery exists in the repo — exposure-proportional, uniform, youth-tilted, and three paper-anchored allocations are already implemented. Concretely:

1. Scenario families: (a) exposure-proportional (JR16/IMF school — current central), (b) junior-concentrated (Canaries/H&L school), (c) expertise-compression — a middle-gains/top-rent-erosion variant in the Autor spirit (the one genuinely new build), (d) uniform (already done).
2. Lead with the result the committed numbers already show: incidence determines which failure mode the UK gets. Top-loaded shocks are fiscally expensive but poverty-light; flatter or junior-loaded shocks are fiscally cheaper but poverty-heavier (committed results: central £18.6bn / +2.10pp BHC vs uniform £14.2bn / +2.31pp vs youth-tilted £17.5bn / +2.32pp). That is a statement about the system, robust across the incidence debate rather than a side of it — and it's the claim only a full tax-benefit model can make.
3. Rebalance §2 to cite the other schools (Autor 2024; the adaptability argument; decision-making complementarity), not only the exposure school.
4. Retitle. "When the shock hits the top" can name one scenario family; it shouldn't be the paper's identity. Something in the direction of "Who bears the AI shock determines what it costs: incidence scenarios through the UK tax-benefit system."

This composes cleanly with the blockers above: the HBAI re-run (finding 2) already forces a rewrite of the results narrative, so restructuring around incidence scenarios is the same rewrite done once, properly.