Skip to content

Align itemized deduction targets with SOI itemizers and JCT tax expenditures#639

Merged
MaxGhenis merged 3 commits into
mainfrom
fix/itemizer-stratum-constraint
Mar 27, 2026
Merged

Align itemized deduction targets with SOI itemizers and JCT tax expenditures#639
MaxGhenis merged 3 commits into
mainfrom
fix/itemizer-stratum-constraint

Conversation

@baogorek
Copy link
Copy Markdown
Collaborator

@baogorek baogorek commented Mar 26, 2026

Summary

Fixes #646.

This PR does two related things for itemized-deduction calibration targets:

  • aligns IRS SOI itemized-deduction targets to the itemizing-filer population
  • aligns JCT national deduction/QBI targets to their correct economic meaning as tax expenditures

Changes:

  • add tax_unit_itemizes == 1 to the SOI strata for salt, real_estate_taxes, and medical_expense_deduction
  • load national JCT deduction and QBI rows as reform-based tax expenditure targets instead of baseline filer/itemizer targets
  • carry reform_id through target_overview, target querying, matrix building, and staging validation
  • compare reform targets to repeal-based income_tax deltas rather than raw deduction totals
  • deactivate stale baseline rows from older ETL runs and ignore inactive rows in matrix target selection
  • add regression coverage for reform_id passthrough and reform target querying/naming

Why

The original SOI fix was directionally correct, but the JCT side still treated tax expenditures as deduction aggregates. JCT publishes revenue-loss estimates, not raw deduction totals, so calibration needs to target repeal-based income_tax deltas for those rows.

Validation

  • python3 -m py_compile on all edited files
  • SQLite smoke test of the updated target_overview and best-period target query logic

Environment limitation

I could not run the repo's normal pytest path in this workspace because the available environments are broken on torch and missing sqlmodel in the fallback Python.

baogorek and others added 2 commits March 26, 2026 11:06
SOI targets for SALT, real estate taxes, and medical expense deduction
are reported only for the ~10% of filers who itemize, but the existing
`variable > 0` constraint captures everyone with economic exposure
(~80-90% of filers). This mismatch causes massive count and dollar
overestimates. Adding `tax_unit_itemizes == 1` fixes the population
alignment.

Changes:
- etl_irs_soi.py: For salt, real_estate_taxes, and
  medical_expense_deduction, append a `tax_unit_itemizes == 1`
  constraint to child strata in the generic target loop.
- etl_national_targets.py: Split JCT itemized deduction targets
  (salt_deduction, medical_expense_deduction, charitable_deduction,
  interest_deduction) into a separate itemizer_targets list loaded
  into a new "United States - Itemizing Tax Filers" stratum with
  both filer and itemizer constraints. QBI deduction remains in the
  plain filer stratum (above-the-line).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MaxGhenis MaxGhenis changed the title Add tax_unit_itemizes constraint for itemized deduction targets Align itemized deduction targets with SOI itemizers and JCT tax expenditures Mar 27, 2026
@MaxGhenis
Copy link
Copy Markdown
Contributor

Reviewed the economics and the calibration-path changes in detail.

The SOI-side itemizer constraint fix is correct, and the follow-up JCT changes now fix the real concept mismatch: these national JCT rows are tax expenditures, not baseline deduction aggregates. With the reform_id plumbing in place, the PR is now targeting repeal-based income_tax deltas rather than raw deduction sums, which is the right object.

I don't see a remaining blocking correctness bug. The follow-up items I still want are non-blocking cleanup work:

  • simplify the validate_staging.py reform cache/helper split so it can't be misused later
  • add tests for stale-row deactivation in etl_national_targets.py
  • add a test around reform-delta computation / sign convention
  • optionally add explicit source comments for the hardcoded JCT table values

I'm planning to merge this and open a focused cleanup PR for those items.

@MaxGhenis MaxGhenis merged commit 94abbf9 into main Mar 27, 2026
5 of 6 checks passed
@MaxGhenis MaxGhenis deleted the fix/itemizer-stratum-constraint branch March 27, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Treat JCT deduction targets as tax expenditures, not deduction aggregates

2 participants