You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Switch from per capita employment income to average earnings forecast (#1205)
* Switch from per capita employment income to average earnings forecast
* Add changelog entry
* Add reform tests
* Add updateable reform impact tests
* Add printout in tests
* Remove duplication of microsim runs
* Run these first so they always run if tests fails
* Update verbosity
* Remove backup file, also add printout of changes to growfactors
* Format
* Don't pass tests by definition!
* Update tests
* Try locking dependencies
* Remove system arg
* Use python 3.11
* Use different toml syntax
* Remove ubuntu test
* Add UV sync to main action
---------
Co-authored-by: Nikhil Woodruff <nikhil.woodruff@outlook.com>
This directory contains automated tests to monitor when model changes affect the fiscal impacts of policy reforms. It helps ensure that updates to the PolicyEngine UK model don't unexpectedly change reform outcomes.
4
+
5
+
## Overview
6
+
7
+
The testing framework consists of three main components:
8
+
9
+
1.**reforms_config.yaml** - Configuration file containing reform definitions and their expected impacts
10
+
2.**test_reform_impacts.py** - Pytest test suite that verifies reforms produce expected impacts
11
+
3.**update_reform_impacts.py** - Script to update expected impacts when model changes are intentional
12
+
13
+
## Quick start
14
+
15
+
### Running tests
16
+
17
+
To check if current model produces expected reform impacts:
18
+
19
+
```bash
20
+
pytest test_reform_impacts.py
21
+
```
22
+
23
+
To run with more detail:
24
+
25
+
```bash
26
+
pytest test_reform_impacts.py -v
27
+
```
28
+
29
+
### Updating expected impacts
30
+
31
+
When model changes are intentional and you need to update the baseline expectations:
32
+
33
+
```bash
34
+
# Preview changes without updating
35
+
python update_reform_impacts.py --dry-run
36
+
37
+
# Update the configuration file
38
+
python update_reform_impacts.py
39
+
40
+
# Update with detailed output
41
+
python update_reform_impacts.py --verbose
42
+
```
43
+
44
+
## Configuration file structure
45
+
46
+
The `reforms_config.yaml` file defines each reform with:
47
+
48
+
```yaml
49
+
reforms:
50
+
- name: "Description of the reform"
51
+
expected_impact: 10.5# Fiscal impact in billions
52
+
parameters:
53
+
"gov.some.parameter": 0.25
54
+
"gov.another.parameter": 1000
55
+
```
56
+
57
+
## Workflow
58
+
59
+
### Regular testing
60
+
61
+
1. Run `pytest policyengine_uk/tests/microsimulation/test_reform_impacts.py` as part of your CI/CD pipeline
62
+
2. Tests will fail if any reform impact differs by more than £0.1 billion from expected
63
+
3. This catches unintended consequences of model changes
64
+
65
+
### After intentional model changes
66
+
67
+
1. Run `python policyengine_uk/tests/microsimulation/update_reform_impacts.py --dry-run` to preview impact changes
68
+
2. Review the changes to ensure they're expected
69
+
3. Run `python policyengine_uk/tests/microsimulation/update_reform_impacts.py` to update the configuration
70
+
4. Commit both the model changes and updated `reforms_config.yaml`
71
+
72
+
### Adding new reforms
73
+
74
+
1. Add a new entry to `reforms_config.yaml`:
75
+
```yaml
76
+
- name: "Your new reform description"
77
+
expected_impact: 0.0 # Initial placeholder
78
+
parameters:
79
+
"gov.parameter.to.change": new_value
80
+
```
81
+
82
+
2. Run `python policyengine_uk/tests/microsimulation/update_reform_impacts.py` to calculate the actual impact
83
+
3. Run `pytest policyengine_uk/tests/microsimulation/test_reform_impacts.py` to verify the test passes
84
+
85
+
## Test tolerance
86
+
87
+
Tests allow for a tolerance of £0.1 billion in fiscal impacts. This accounts for:
88
+
- Minor numerical differences in calculations
89
+
- Small variations from microsimulation sampling
90
+
91
+
If you need to adjust this tolerance, modify the assertion in `test_reform_impacts.py`.
92
+
93
+
## Troubleshooting
94
+
95
+
### Tests failing after model update
96
+
97
+
1. Check if the changes are expected by running the update script with `--dry-run`
98
+
2. If changes are expected, update the configuration file
99
+
3. If changes are unexpected, investigate the model modifications
100
+
101
+
### Import errors
102
+
103
+
Ensure you have the required dependencies:
104
+
```bash
105
+
pip install policyengine-uk pytest pyyaml
106
+
```
107
+
108
+
### Configuration file not found
109
+
110
+
The tests expect `reforms_config.yaml` to be in the same directory as the test file. You can specify a different path:
0 commit comments