|
| 1 | +We're excited to announce the full launch of the Enhanced Current Population Survey (Enhanced CPS). This comprehensive dataset powers PolicyEngine's microsimulation modeling with unprecedented accuracy, enabling precise analysis of tax and benefit reforms across the United States. |
| 2 | + |
| 3 | +Building on extensive development and testing since our [beta launch](/us/research/enhanced-cps-beta), the Enhanced CPS now includes sophisticated imputation models for income sources and household characteristics that were previously unavailable or underreported in survey data. These improvements provide a more complete picture of American households' economic circumstances. Learn more about our [methodology](https://policyengine.github.io/policyengine-us-data/methodology) and [data sources](https://policyengine.github.io/policyengine-us-data/data). |
| 4 | + |
| 5 | +## New features |
| 6 | + |
| 7 | +### Income source imputations |
| 8 | + |
| 9 | +The Enhanced CPS now includes machine learning-based imputations for income sources that are frequently underreported in surveys: |
| 10 | + |
| 11 | +**Tip income**: Using employer-reported data from the Survey of Income and Program Participation (SIPP), we impute tip income based on employment income, age, and household composition. This enhancement enables accurate analysis of proposals to exempt tips from taxation. See our [imputation methodology documentation](https://policyengine.github.io/policyengine-us-data/methodology#imputation) for technical details. |
| 12 | + |
| 13 | +**Overtime premiums**: We calculate overtime income using hours worked, occupation codes, and Fair Labor Standards Act exemption status, allowing accurate modeling of overtime exemption proposals. |
| 14 | + |
| 15 | +**Auto loan interest**: Imputed from the Survey of Consumer Finances (SCF), this addition enables analysis of proposals to make auto loan interest deductible. |
| 16 | + |
| 17 | +### Immigration status imputation |
| 18 | + |
| 19 | +We've implemented the ASEC Undocumented Algorithm to impute Social Security Number card types, enabling more accurate modeling of policies with citizenship or work authorization requirements. This process-of-elimination approach examines 14 conditions to identify likely undocumented individuals, calibrated to match external population estimates. Details are available in our [demographic imputation section](https://policyengine.github.io/policyengine-us-data/methodology#demographic-imputation). |
| 20 | + |
| 21 | +### Technical infrastructure improvements |
| 22 | + |
| 23 | +**Two-stage methodology**: Our approach combines sophisticated imputation with advanced reweighting techniques. First, we use Quantile Regression Forests (QRF) to impute missing variables from multiple data sources, preserving realistic variation and capturing conditional distribution tails. Second, we apply gradient-based optimization with PyTorch to reweight households, matching administrative targets while maintaining the survey's statistical properties. |
| 24 | + |
| 25 | + |
| 26 | + |
| 27 | +The process flow integrates five source datasets (CPS ASEC, IRS PUF, SIPP, SCF, ACS) that are aged to the target year. Through QRF imputation, we create two enhanced CPS variants: one with missing PUF variables filled in, and another with existing variables replaced by PUF values. These datasets then undergo reweighting optimization to produce the final Enhanced CPS dataset. [View our detailed methodology documentation](https://policyengine.github.io/policyengine-us-data/methodology). |
| 28 | + |
| 29 | +**Microimpute package**: We've developed and adopted [`microimpute`](https://github.com/PolicyEngine/microimpute), a new open-source Python package that automates our QRF-based imputation methods. This package makes our imputation methodology more transparent and reusable. |
| 30 | + |
| 31 | +**Advanced reweighting with L0 regularization**: Our reweighting process uses log-transformed weights with dropout regularization and incorporates an L0 penalty for sparsity. This ensures positive weights while preventing overfitting and maintaining interpretability. The optimization minimizes mean squared relative error using the Adam optimizer. |
| 32 | + |
| 33 | +**Enhanced validation**: Our [calibration process](https://policyengine.github.io/policyengine-us-data/methodology#calibration) targets 9,168 administrative totals from sources including IRS SOI, Census, CBO/Treasury, and JCT data, ensuring the Enhanced CPS accurately represents: |
| 34 | + |
| 35 | +- Income components by source |
| 36 | +- Benefit program enrollment |
| 37 | +- Demographic distributions |
| 38 | +- Geographic population counts |
| 39 | + |
| 40 | +## Upcoming developments |
| 41 | + |
| 42 | +### State and local calibration |
| 43 | + |
| 44 | +With support from [Arnold Ventures](https://www.arnoldventures.org/), we're extending the Enhanced CPS to provide accurate estimates for every state and congressional district. This follows our successful implementation of local-area microsimulation in the UK, funded by the [Nuffield Foundation](https://www.nuffieldfoundation.org/). |
| 45 | + |
| 46 | +Once complete, the Enhanced CPS will become the default for state-level analysis as well, and PolicyEngine users will be able to analyze the impacts of federal and state policy reforms on: |
| 47 | + |
| 48 | +- Poverty rates by state and congressional district |
| 49 | +- Income inequality measures for local areas |
| 50 | +- Winners and losers from reforms in specific districts |
| 51 | +- Distributional impacts by income decile for each state |
| 52 | + |
| 53 | +### Microcalibrate package |
| 54 | + |
| 55 | +We're developing [`microcalibrate`](https://github.com/PolicyEngine/microcalibrate), a next-generation reweighting package that enhances our current gradient descent approach. This package will offer: |
| 56 | + |
| 57 | +- Faster convergence to calibration targets |
| 58 | +- Better preservation of the original survey's covariance structure |
| 59 | +- More flexible loss functions for different use cases |
| 60 | +- Easier extension to multi-area calibration |
| 61 | + |
| 62 | +### Additional data enhancements |
| 63 | + |
| 64 | +We currently integrate data from the Survey of Consumer Finances (for auto loan interest) and American Community Survey (for housing costs). Future enhancements will expand these integrations: |
| 65 | + |
| 66 | +- **Wealth modeling from SCF**: Comprehensive asset and debt data for modeling asset limits in SNAP, SSI, and other means-tested programs, similar to our [wealth modeling in the UK](https://policyengine.org/uk/research/uk-the-new-policyengine) |
| 67 | +- **Consumer Expenditure Survey**: Consumption patterns for modeling sales taxes, carbon pricing, and other consumption-based policies |
| 68 | +- **Expanded ACS integration**: Additional geographic and demographic detail for state and local policy analysis |
| 69 | + |
| 70 | +## Using the Enhanced CPS |
| 71 | + |
| 72 | +The Enhanced CPS is now the exclusive dataset for nationwide PolicyEngine US analyses. We've removed the dataset selector to streamline the user experience—the Enhanced CPS automatically powers all federal policy simulations, while state-specific analyses continue to use the standard CPS until our local calibration is complete. |
| 73 | + |
| 74 | +You can access the Enhanced CPS through: |
| 75 | + |
| 76 | +**Web interface**: The Enhanced CPS powers all nationwide calculations at [policyengine.org/us](https://policyengine.org/us) |
| 77 | + |
| 78 | +**Python package**: Works by default for our `Microsimulation` calls. |
| 79 | + |
| 80 | +**Direct download**: For Python users, the data automatically downloads from our Hugging Face repository when you instantiate a simulation. The files are stored at [`hf://policyengine/policyengine-us-data`](https://huggingface.co/policyengine/policyengine-us-data) |
| 81 | + |
| 82 | +## Technical details |
| 83 | + |
| 84 | +For researchers interested in our methodology: |
| 85 | + |
| 86 | +- **Full technical documentation**: [PolicyEngine US Data documentation](https://policyengine.github.io/policyengine-us-data) |
| 87 | +- **Data integration methodology**: [Imputation and fusion techniques](https://policyengine.github.io/policyengine-us-data/methodology#data-fusion) |
| 88 | +- **Calibration approach**: [Reweighting methodology](https://policyengine.github.io/policyengine-us-data/methodology#reweighting) |
| 89 | +- **Validation results**: [Comparison with administrative data](https://policyengine.github.io/policyengine-us-data/discussion) |
| 90 | +- **Implementation code**: [Microimpute package](https://github.com/PolicyEngine/microimpute) |
| 91 | +- **Source code**: [Enhanced CPS on GitHub](https://github.com/PolicyEngine/policyengine-us-data/tree/main/policyengine_us_data/datasets/cps) |
| 92 | + |
| 93 | +## Conclusion |
| 94 | + |
| 95 | +The Enhanced CPS represents a major advancement in open-source microsimulation data. By [combining the demographic richness of the Current Population Survey with tax detail from IRS records](https://policyengine.github.io/policyengine-us-data/background) and sophisticated imputation techniques, we've created a dataset that supports comprehensive analysis of both tax and benefit policies. |
| 96 | + |
| 97 | +As we expand to state and local calibration, the Enhanced CPS will enable unprecedented granularity in policy analysis—empowering lawmakers, researchers, and citizens to understand how proposed reforms would affect their communities. |
| 98 | + |
| 99 | +We welcome feedback and collaboration as we continue improving this foundational infrastructure for evidence-based policymaking. For questions or to contribute to development, please visit our [GitHub repositories](https://github.com/PolicyEngine) or [contact us](mailto:hello@policyengine.org). |
0 commit comments