Problem
scenario_121 (SC, single 53-year-old) intends a disabled household: the prompt narrates disability, and the inputs set is_disabled: true plus disability_benefits: 13044. But neither input is USDA-qualifying disability under 7 CFR 271.2 (which is receipt-based: SSI, SSDI, VA disability, etc.):
disability_benefits is PolicyEngine's employer/private disability income variable — not a qualifying federal programis_disabled is a bare boolean that PE reads for the SNAP work-registration exemption but not for is_usda_disabled
So the reference pipeline computes a non-disabled SNAP unit: the household fails the 130% gross-income test (ratio 2.55) and gets $0, and the $30k of medical expenses never unlocks the excess-medical deduction. Force is_usda_disabled=true and the same household is eligible with $286/mo ($3,457/yr): elderly/disabled units are exempt from the gross test, and net income collapses to ~$41 after the medical deduction.
The scenario is internally ill-posed rather than PE being wrong — the ground-truth household doesn't represent what the prompt describes. Full trace in #91 (row 13).
Options for the next reference refresh
- Preferred: re-encode the disability as a receipt-based input PE consumes for
is_usda_disabled — e.g. route the disability income to social_security_disability (SSDI receipt, SGA-independent). Note the SSI route does not work here: $25,665 wages = $2,139/mo > the 2026 SGA limit.
- Alternatively, drop the SNAP output for this scenario as a scope gap.
Either changes a frozen scenario/reference, so it belongs in the next snapshot/reference-refresh cycle, not a live patch. Models that treated the household as disabled-for-SNAP and answered a positive amount are currently scored wrong on this cell.
Also worth considering
A generator-side lint: any scenario whose prompt narrates disability should set at least one receipt-based disabled_programs input (is_ssi_disabled chain, social_security_disability, veteran flags), since is_disabled alone propagates inconsistently in policyengine-us (work-registration yes, gross-test exemption no).
Problem
scenario_121 (SC, single 53-year-old) intends a disabled household: the prompt narrates disability, and the inputs set
is_disabled: trueplusdisability_benefits: 13044. But neither input is USDA-qualifying disability under 7 CFR 271.2 (which is receipt-based: SSI, SSDI, VA disability, etc.):disability_benefitsis PolicyEngine's employer/private disability income variable — not a qualifying federal programis_disabledis a bare boolean that PE reads for the SNAP work-registration exemption but not foris_usda_disabledSo the reference pipeline computes a non-disabled SNAP unit: the household fails the 130% gross-income test (ratio 2.55) and gets $0, and the $30k of medical expenses never unlocks the excess-medical deduction. Force
is_usda_disabled=trueand the same household is eligible with$286/mo ($3,457/yr): elderly/disabled units are exempt from the gross test, and net income collapses to ~$41 after the medical deduction.The scenario is internally ill-posed rather than PE being wrong — the ground-truth household doesn't represent what the prompt describes. Full trace in #91 (row 13).
Options for the next reference refresh
is_usda_disabled— e.g. route the disability income tosocial_security_disability(SSDI receipt, SGA-independent). Note the SSI route does not work here: $25,665 wages = $2,139/mo > the 2026 SGA limit.Either changes a frozen scenario/reference, so it belongs in the next snapshot/reference-refresh cycle, not a live patch. Models that treated the household as disabled-for-SNAP and answered a positive amount are currently scored wrong on this cell.
Also worth considering
A generator-side lint: any scenario whose prompt narrates disability should set at least one receipt-based
disabled_programsinput (is_ssi_disabledchain,social_security_disability, veteran flags), sinceis_disabledalone propagates inconsistently in policyengine-us (work-registration yes, gross-test exemption no).