address merge conflict

Author: chusloj

Makefile

@@ -121,16 +121,26 @@ puf_stage3/puf_ratios.csv: puf_stage3/stage3.py \
 	cd puf_stage3 ; python stage3.py
 
 .PHONY=cps-files
-cps-files: cps_data/cps.csv.gz \
+cps-files: cps_data/pycps/cps_raw.csv.gz \
            cps_stage1/stage_2_targets.csv \
            cps_stage2/cps_weights.csv.gz
 
-cps_data/cps.csv.gz: cps_data/finalprep.py \
-                     cps_data/cps_raw.csv.gz \
-                     cps_data/adjustment_targets.csv \
-                     cps_data/benefitprograms.csv
-	cd cps_data ; python finalprep.py && \
-        gunzip cps.csv.gz && gzip -n cps.csv
+cps_data/pycps/cps_raw.csv.gz: cps_data/pycps/create.py \
+                               cps_data/pycps/benefits.py \
+                               cps_data/pycps/filing_rules.json \
+                               cps_data/pycps/finalprep.py \
+                               cps_data/pycps/helpers.py \
+                               cps_data/pycps/impute.py \
+                               cps_data/pycps/pycps.py \
+                               cps_data/pycps/splitincome.py \
+                               cps_data/pycps/targeting.py \
+                               cps_data/pycps/taxunit.py \
+                               cps_data/pycps/template.txt \
+                               cps_data/pycps/transform_sas.py \
+                               cps_data/pycps/adjustment_targets.csv \
+                               cps_data/benefitprograms.csv
+	cd cps_data/pycps ; python create.py && \
+	gunzip cps.csv.gz && gzip -n cps.csv
 
 cps_stage1/stage_2_targets.csv: cps_stage1/stage1.py \
                                 cps_stage1/SOI_estimates.csv \
@@ -140,11 +150,11 @@ cps_stage1/stage_2_targets.csv: cps_stage1/stage1.py \
 
 cps_stage2/cps_weights.csv.gz: cps_stage2/stage2.py \
                                cps_stage2/solve_lp_for_year.py \
-                               cps_data/cps_raw.csv.gz \
+                               cps_data/pycps/cps_raw.csv.gz \
                                puf_stage1/Stage_I_factors.csv \
                                cps_stage1/stage_2_targets.csv
 	cd cps_stage2 ; python stage2.py && \
-        gunzip cps_weights.csv.gz && gzip -n cps_weights.csv
+	gunzip cps_weights.csv.gz && gzip -n cps_weights.csv
 
 .PHONY=all
 all: puf-files cps-files

README.md

@@ -35,6 +35,21 @@ sample data being used.  But the weights, ratios, and benefits files
 do depend on the data file, so they are different in the two sets of
 data files.
 
+Installation
+-----------
+
+Currently, the only way to install `taxdata` currently is to clone the git
+repo locally.
+```
+git clone https://github.com/PSLmodels/taxdata.git
+```
+Next navigate to the directory and install the `taxdata-dev` conda environment
+```
+cd taxdata
+conda env create -f environment.yml
+```
+To run the scripts that produce `puf.csv` and `cps.csv.gz`, activate the
+`taxdata-dev` conda environment and follow the workflow laid out below.
 
 Data-Preparation Documentation and Workflow
 -------------------------------------------
@@ -51,10 +66,8 @@ running the `make help` command.  If you want more background on the
 make utility and makefiles, search for Internet links with the
 keywords `makefile` and `automate`.
 
-Please see the [Makefile Documentation](MAKEFILE_doc.md) for more information
-
-Note that the stage2 linear program that generates the weights file is
-very long-running, taking five or more hours depending on your
+Note that the stage2 linear program that generates the weights file for the PUF
+is very long-running, taking five or more hours depending on your
 computer's CPU speed.  We are considering options for speeding up this
 stage2 work, but for the time being you can execute `make puf-files`
 and `make cps-files` in separate terminal windows to have the two
@@ -73,6 +86,14 @@ the corresponding files in the Tax-Calculator directory tree) without
 actually doing the file copies.  At the terminal command-prompt in the
 top-level taxdata directory, execute `./csvcopy.sh` to get help.
 
+### Example
+
+To create `cps.csv.gz`, run
+```
+conda activate taxdata-dev
+make cps-files
+```
+
 
 Contributing to taxdata Repository
 ----------------------------------
@@ -83,9 +104,17 @@ to make sure your proposed code is consistent with the repository's
 coding style and then run `make pytest` to ensure that all the tests
 pass.
 
+Disclaimer
+----------
+
+`taxdata` is under continuous development. As such, results will change as the
+underlying data and logic improves.
+
 
 Contributors
 ------------
-- Anderson Frailey
-- John O'Hare
-- Amy Xu
+
+A full list of contributors on GitHub can be found 
+here](https://github.com/PSLmodels/taxdata/graphs/contributors). John O'Hare
+of Quantria Strategies has also made significant contributions to the
+development of `taxdata`.

cps_data/README.md

@@ -1,17 +1,92 @@
 About cps_data
 ==============
 
-This directory contains the following script:
+This directory contains the python scripts used to create `cps.csv.gz`. You
+can run all of the scripts with the command `python create.py`. By default,
+you will get a CPS file composed of the 2013, 2014, and 2015 March CPS Supplemental
+files. If you would like to use another combination of the 2013, 2014, 2015,
+2016, 2017, and 2018 files, there are two ways to do so.
 
-* Python script **finalprep.py**, which reads/writes:
+1. You can modify `create.py` by adding the `cps_files` argument to the `create()`
+function call at the bottom of the file to specify which files you would like to
+use. For example, to use the 2016, 2017, and 2018 files, the function call would
+now be
+```python
+if __name__ == "__main__":
+    create(
+        exportcsv=False, exportpkl=True, exportraw=False, validate=False,
+        benefits=True, verbose=True, cps_files=[2016, 2017, 2018]
+    )
+```
 
-  Input files:
-    - cps_raw.csv.gz
-    - adjustment_targets.csv
-    - benefitprograms.csv
+2. You could write a separate python file that imports the `create()` function
+and calls it in the same way as above.
 
-  Output files:
-    - cps.csv
+## Input files:
+With the exception of the CPS March Supplements, all input files can be found
+in the `pycps/data` directory.
+
+### CPS March Supplements
+* asec2013_pubuse.dat
+* asec2014_pubuse_tax_fix_5x8_2017.dat
+* asec2015_pubuse.dat
+* asec2016_pubuse.dat
+* asec2017_pubuse.dat
+* asec2018_pubuse.dat
+
+### C-TAM Benefit Imputations
+
+Note that we only have C-TAM imputations for the 2013, 2014, and 2015 files.
+For other years, we just use the benefit program information in the CPS
+* Housing_Imputation_logreg_2013.csv
+* Housing_Imputation_logreg_2014.csv
+* Housing_Imputation_logreg_2015.csv
+* medicaid2013.csv
+* medicaid2014.csv
+* medicaid2015.csv
+* medicare2013.csv
+* medicare2014.csv
+* medicare2015.csv
+* otherbenefitprograms.csv
+* SNAP_Imputation_2013.csv
+* SNAP_Imputation_2014.csv
+* SNAP_Imputation_2015.csv
+* SS_augmentation_2013.csv
+* SS_augmentation_2014.csv
+* SS_augmentation_2015.csv
+* SSI_Imputation2013.csv
+* SSI_Imputation2014.csv
+* SSI_Imputation2015.csv
+* TANF_Imputation_2013.csv
+* TANF_Imputation_2014.csv
+* TANF_Imputation_2015.csv
+* UI_imputation_logreg_2013.csv
+* UI_imputation_logreg_2014.csv
+* UI_imputation_logreg_2015.csv
+* VB_Imputation2013.csv
+* VB_Imputation2014.csv
+* VB_Imputation2015.csv
+* WIC_imputation_children_logreg_2013.csv
+* WIC_imputation_children_logreg_2014.csv
+* WIC_imputation_children_logreg_2015.csv
+* WIC_imputation_infants_logreg_2013.csv
+* WIC_imputation_infants_logreg_2014.csv
+* WIC_imputation_infants_logreg_2015.csv
+* WIC_imputation_women_logreg_2013.csv
+* WIC_imputation_women_logreg_2014.csv
+* WIC_imputation_women_logreg_2015.csv
+
+### Imputation Parameters
+
+These parameters are used in the imputations found in `pycps/impute.py`
+* logit_beta.csv
+* ols_betas.csv
+
+## Output Files
+
+Only `cps.csv.gz` is included in the repository due to the size of `cps_raw.csv.gz`.
+* cps.csv.gz
+* cps_raw.csv.gz
 
 
 Documentation