The Codex failure-audit of the Claude 5 models' 322 wrong cases (PR #90) flags 16 cases where the judge suspects the PolicyEngine reference value rather than the models — clustering in CHIP/Medicaid income-threshold handling, three state income tax cells, two SNAP cells, and Head Start eligibility. Each needs human triage against policyengine-us; the June audit's equivalents produced real upstream fixes.
The verdicts are conservative by prompt design (default is models-wrong), so these are worth taking seriously. Reproduction: every case is a frozen-manifest cell — scenario inputs in paper/snapshot/20260501/us_scenarios.csv, reference values in us_reference_outputs.csv (policyengine.py 4.16.1 / policyengine-us 1.723.0).
scenario_032 · child1_chip_eligible
PolicyEngine's CHIP reference appears to use an implausibly low Minnesota child CHIP income threshold, treating 275% FPL for a family of three as about $38,000 when it should be far higher, making the stated $42,664 income likely below rather than above the threshold.
scenario_026 · child1_medicaid_eligible
The recorded reference value appears inverted or mismatched: it says child1_medicaid_eligible = Yes while the supplied PolicyEngine derivation explicitly says PolicyEngine determined Child 1 is not Medicaid eligible (False).
scenario_064 · child1_medicaid_eligible
The recorded reference value is No, but the provided PolicyEngine derivation explicitly says PolicyEngine determined child1_medicaid_eligible is True, suggesting a reference extraction or label mismatch for this variable.
scenario_038 · child2_chip_eligible
PolicyEngine appears to apply an implausibly low Louisiana CHIP income threshold, treating 200% FPL for a four-person household as about $21,600 when 200% FPL should be far higher than this household's roughly $22,992 income.
scenario_100 · child2_head_start_eligible
PolicyEngine appears to be denying Head Start eligibility despite a 5-year-old child in a three-person household with $5,915 annual wages, which is plainly below the federal poverty guideline threshold described in the reference rationale.
scenario_008 · child3_chip_eligible
The derivation appears to apply the CHIP income threshold backwards or uses an incorrect New Jersey 2026 threshold, since an 8-person household with about $30,915 income is far below any plausible child CHIP/NJ FamilyCare income limit expressed as a multiple of FPL.
scenario_109 · child3_chip_eligible
PolicyEngine appears to be using an implausibly low 200% FPL threshold for a five-person household in 2026, causing Florida CHIP eligibility to be denied despite $30,000 being below the likely applicable income limit.
scenario_026 · child3_medicaid_eligible
PolicyEngine appears to be returning Medicaid eligibility for a 9-year-old NC child despite household income exceeding the stated child Medicaid/CHIP income threshold, possibly conflating CHIP eligibility with Medicaid eligibility or applying the wrong income limit.
scenario_008 · child5_head_start_eligible
PolicyEngine appears to compare the household's roughly $30,915 income against an incorrect Head Start poverty threshold for an 8-person household, since that income is well below the federal poverty guideline rather than above it.
scenario_015 · head_medicaid_eligible
PolicyEngine appears to have evaluated the head under Indiana's non-disabled adult Medicaid pathway despite the prompt setting is disabled=true, potentially missing a disability or working-disabled Medicaid category.
scenario_031 · head_medicaid_eligible
PolicyEngine appears to have applied California's MAGI Medicaid expansion logic for non-elderly adults despite the head being age 67, where aged Medicaid rules should govern.
scenario_038 · snap
The stated PolicyEngine derivation says the monthly SNAP allotment is $994 minus a $392.70 expected contribution, or $601.30, but $601.30 times 12 is $7,215.60 rather than the reported $7,286.94.
scenario_121 · snap
PolicyEngine appears to be applying the SNAP gross income limit to a household with a disabled member, even though elderly/disabled SNAP households are generally subject to the net income test rather than the gross income test, and the stated medical expenses could reduce net income enough to produce a positive benefit.
scenario_046 · state_income_tax_before_refundable_credits
PolicyEngine's derivation says Oklahoma itemized deductions of $22,429.78 exceeded the $12,700 standard deduction, but the taxable income of $75,303 appears to subtract the standard deduction rather than the larger itemized deduction.
scenario_056 · state_income_tax_before_refundable_credits
PolicyEngine appears to apply New Jersey's rate schedule to taxable income after the $1,000 exemption while omitting New Jersey's low-income filing/taxability threshold that eliminates liability for a single filer with gross income under $10,000.
scenario_085 · state_income_tax_before_refundable_credits
PolicyEngine appears to have incorrectly zeroed Pennsylvania income tax before refundable credits, likely by applying an erroneous income threshold/tax-forgiveness rule or excluding taxable interest income that Pennsylvania taxes at its flat rate.
🤖 Generated with Claude Code
The Codex failure-audit of the Claude 5 models' 322 wrong cases (PR #90) flags 16 cases where the judge suspects the PolicyEngine reference value rather than the models — clustering in CHIP/Medicaid income-threshold handling, three state income tax cells, two SNAP cells, and Head Start eligibility. Each needs human triage against policyengine-us; the June audit's equivalents produced real upstream fixes.
The verdicts are conservative by prompt design (default is models-wrong), so these are worth taking seriously. Reproduction: every case is a frozen-manifest cell — scenario inputs in
paper/snapshot/20260501/us_scenarios.csv, reference values inus_reference_outputs.csv(policyengine.py 4.16.1 / policyengine-us 1.723.0).scenario_032 ·
child1_chip_eligiblePolicyEngine's CHIP reference appears to use an implausibly low Minnesota child CHIP income threshold, treating 275% FPL for a family of three as about $38,000 when it should be far higher, making the stated $42,664 income likely below rather than above the threshold.
scenario_026 ·
child1_medicaid_eligibleThe recorded reference value appears inverted or mismatched: it says child1_medicaid_eligible = Yes while the supplied PolicyEngine derivation explicitly says PolicyEngine determined Child 1 is not Medicaid eligible (False).
scenario_064 ·
child1_medicaid_eligibleThe recorded reference value is No, but the provided PolicyEngine derivation explicitly says PolicyEngine determined child1_medicaid_eligible is True, suggesting a reference extraction or label mismatch for this variable.
scenario_038 ·
child2_chip_eligiblePolicyEngine appears to apply an implausibly low Louisiana CHIP income threshold, treating 200% FPL for a four-person household as about $21,600 when 200% FPL should be far higher than this household's roughly $22,992 income.
scenario_100 ·
child2_head_start_eligiblePolicyEngine appears to be denying Head Start eligibility despite a 5-year-old child in a three-person household with $5,915 annual wages, which is plainly below the federal poverty guideline threshold described in the reference rationale.
scenario_008 ·
child3_chip_eligibleThe derivation appears to apply the CHIP income threshold backwards or uses an incorrect New Jersey 2026 threshold, since an 8-person household with about $30,915 income is far below any plausible child CHIP/NJ FamilyCare income limit expressed as a multiple of FPL.
scenario_109 ·
child3_chip_eligiblePolicyEngine appears to be using an implausibly low 200% FPL threshold for a five-person household in 2026, causing Florida CHIP eligibility to be denied despite $30,000 being below the likely applicable income limit.
scenario_026 ·
child3_medicaid_eligiblePolicyEngine appears to be returning Medicaid eligibility for a 9-year-old NC child despite household income exceeding the stated child Medicaid/CHIP income threshold, possibly conflating CHIP eligibility with Medicaid eligibility or applying the wrong income limit.
scenario_008 ·
child5_head_start_eligiblePolicyEngine appears to compare the household's roughly $30,915 income against an incorrect Head Start poverty threshold for an 8-person household, since that income is well below the federal poverty guideline rather than above it.
scenario_015 ·
head_medicaid_eligiblePolicyEngine appears to have evaluated the head under Indiana's non-disabled adult Medicaid pathway despite the prompt setting
is disabled=true, potentially missing a disability or working-disabled Medicaid category.scenario_031 ·
head_medicaid_eligiblePolicyEngine appears to have applied California's MAGI Medicaid expansion logic for non-elderly adults despite the head being age 67, where aged Medicaid rules should govern.
scenario_038 ·
snapThe stated PolicyEngine derivation says the monthly SNAP allotment is $994 minus a $392.70 expected contribution, or $601.30, but $601.30 times 12 is $7,215.60 rather than the reported $7,286.94.
scenario_121 ·
snapPolicyEngine appears to be applying the SNAP gross income limit to a household with a disabled member, even though elderly/disabled SNAP households are generally subject to the net income test rather than the gross income test, and the stated medical expenses could reduce net income enough to produce a positive benefit.
scenario_046 ·
state_income_tax_before_refundable_creditsPolicyEngine's derivation says Oklahoma itemized deductions of $22,429.78 exceeded the $12,700 standard deduction, but the taxable income of $75,303 appears to subtract the standard deduction rather than the larger itemized deduction.
scenario_056 ·
state_income_tax_before_refundable_creditsPolicyEngine appears to apply New Jersey's rate schedule to taxable income after the $1,000 exemption while omitting New Jersey's low-income filing/taxability threshold that eliminates liability for a single filer with gross income under $10,000.
scenario_085 ·
state_income_tax_before_refundable_creditsPolicyEngine appears to have incorrectly zeroed Pennsylvania income tax before refundable credits, likely by applying an erroneous income threshold/tax-forgiveness rule or excluding taxable interest income that Pennsylvania taxes at its flat rate.
🤖 Generated with Claude Code