Improve uprating growthfactors (#1218)

* Remove duplicate docs deployment

* Just run mac

* Uprating improvements

* Add catch

* Enable entity table downloads

* Re-add VAT to gov tax?

Author: nikhilwoodruff

changelog_entry.yaml

@@ -0,0 +1,7 @@
+- bump: minor
+  changes:
+    added:
+    - Documentation on growth factors.
+    - Cleaned up non-standard uprating factors for wealth variables.
+    - Added triple lock uprating detail and reform switches.
+    - Added ability to download entity datasets from HuggingFace.

docs/book/assumptions/growthfactors.ipynb

@@ -0,0 +1,96 @@
+# Economic assumptions
+
+We project economic variables using year-over-year growth rates stored in `parameters/gov/economic_assumptions/yoy_growth.yaml`. We generate index values from these rates to update household variables. We source all values from the OBR's Economic and Fiscal Outlook (March 2025) unless we specify otherwise.
+
+## Consumer price index
+
+We use the OBR's CPI projections to drive most benefit and consumption variables.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 10.0% | 5.7% | 2.3% | 3.2% | 1.9% | 2.0% | 2.0% | 2.0% |
+
+We apply CPI to these variables:
+- All reported benefits– `attendance_allowance_reported`, `carers_allowance_reported`, `child_benefit_reported`, `child_tax_credit_reported`, `dla_m_reported`, `dla_sc_reported`, `esa_contrib_reported`, `esa_income_reported`, `housing_benefit_reported`, `iidb_reported`, `incapacity_benefit_reported`, `income_support_reported`, `jsa_contrib_reported`, `jsa_income_reported`, `maternity_allowance_reported`, `pension_credit_reported`, `pip_dl_reported`, `pip_m_reported`, `sda_reported`, `state_pension_reported`, `universal_credit_reported`, `winter_fuel_allowance_reported`, `working_tax_credit_reported`
+- All consumption categories– `alcohol_and_tobacco_consumption`, `clothing_and_footwear_consumption`, `communication_consumption`, `domestic_energy_consumption`, `education_consumption`, `food_and_non_alcoholic_beverages_consumption`, `health_consumption`, `household_furnishings_consumption`, `housing_water_and_electricity_consumption`, `miscellaneous_consumption`, `recreation_consumption`, `restaurants_and_hotels_consumption`, `transport_consumption`
+- Other variables– `afcs_reported`, `bsp_reported`, `childcare_expenses`, `diesel_spending`, `free_school_fruit_veg`, `free_school_meals`, `free_school_milk`, `maintenance_expenses`, `petrol_spending`, `statutory_maternity_pay`, `statutory_paternity_pay`, `statutory_sick_pay`, `state_pension`
+
+## Average earnings
+
+We apply the OBR's wage growth forecasts to employment-related variables.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 6.4% | 6.9% | 4.7% | 3.7% | 2.2% | 2.1% | 2.3% | 2.5% |
+
+We use average earnings for these variables: `employee_pension_contributions`, `employer_pension_contributions`, `employment_income`, `employment_income_before_lsr`, `personal_pension_contributions`, `student_loan_repayments`
+
+## Per capita GDP
+
+We derive these rates from OBR GDP growth and ONS population projections.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 9.2% | 5.0% | 3.8% | 2.8% | 2.8% | 3.1% | 3.3% | 3.3% |
+
+We use per capita GDP for these variables: `capital_gains`, `capital_gains_before_response`, `corporate_wealth`, `dividend_income`, `gross_financial_wealth`, `lump_sum_income`, `main_residence_value`, `maintenance_income`, `miscellaneous_income`, `mortgage_capital_repayment`, `net_financial_wealth`, `non_residential_property_value`, `other_investment_income`, `other_residential_property_value`, `owned_land`, `pension_income`, `private_transfer_income`, `property_income`, `savings`, `savings_interest_income`, `sublet_income`
+
+## Council tax
+
+We use the OBR's Council Tax receipts and projections.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 5.3% | 5.6% | 6.4% | 4.6% | 4.5% | 4.6% | 4.5% | 4.5% |
+
+We apply this to: `council_tax`
+
+## Mortgage interest
+
+We use the OBR's mortgage interest rate index growth rates.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 26.2% | 48.5% | 22.1% | 13.6% | 12.6% | 8.2% | 4.2% | 4.7% |
+
+We apply this to: `mortgage_interest_repayment`
+
+## Private pension index
+
+We use RPI year-on-year change from the previous year, capped at 5%.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 5.0% | 5.0% | 5.0% | 4.7% | 3.7% | 2.2% | 2.1% | 2.3% |
+
+We apply this to: `private_pension_income`
+
+## Rent
+
+We use the OBR's rental growth projections.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 4.0% | 6.3% | 7.4% | 5.7% | 3.6% | 2.7% | 2.3% | 2.4% |
+
+We apply this to: `rent`
+
+## Per capita mixed income
+
+We derive these rates from OBR mixed income growth and ONS population projections.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 6.3% | 2.4% | 4.8% | 4.7% | 3.1% | 3.1% | 3.6% | 3.8% |
+
+We apply this to: `self_employment_income`
+
+## Population
+
+We use ONS population projections.
+
+| Fiscal year | 2022 | 2023 | 2024 | 2025 | 2026 | 2027 | 2028 | 2029 |
+|-------------|------|------|------|------|------|------|------|------|
+| Growth rate | 0.3% | 1.4% | 1.0% | 1.1% | 0.7% | 0.8% | 0.4% | 0.5% |
+
+We apply this to: `household_weight`

docs/book/assumptions/growthfactors.md

@@ -13,5 +13,9 @@
 from pathlib import Path
 import os
 from policyengine_core.taxbenefitsystems import TaxBenefitSystem
+from policyengine_uk.data.economic_assumptions import (
+    BASELINE_GROWFACTORS,
+    apply_growth_factors,
+)
 
 REPO = Path(__file__).parent

policyengine_uk/__init__.py

@@ -1,5 +1 @@
 from policyengine_uk.data.dataset_schema import UKDataset
-from policyengine_uk.data.economic_assumptions import (
-    BASELINE_GROWFACTORS,
-    apply_growth_factors,
-)

policyengine_uk/data/__init__.py

@@ -1,12 +1,39 @@
 import pandas as pd
-from policyengine_uk import Microsimulation
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from policyengine_uk import Microsimulation
+
+from pathlib import Path
+import h5py
 
 
 class UKDataset:
     person: pd.DataFrame
     benunit: pd.DataFrame
     household: pd.DataFrame
 
+    @staticmethod
+    def validate_file_path(file_path: str):
+        if not file_path.endswith(".h5"):
+            raise ValueError("File path must end with '.h5' for UKDataset.")
+        if not Path(file_path).exists():
+            raise FileNotFoundError(f"File not found: {file_path}")
+
+        # Check if the file contains time_period, person, benunit, and household datasets
+        with h5py.File(file_path, "r") as f:
+            required_datasets = [
+                "time_period",
+                "person",
+                "benunit",
+                "household",
+            ]
+            for dataset in required_datasets:
+                if dataset not in f:
+                    raise ValueError(
+                        f"Dataset '{dataset}' not found in the file: {file_path}"
+                    )
+
     def __init__(
         self,
         file_path: str = None,
@@ -16,6 +43,7 @@ def __init__(
         fiscal_year: int = 2025,
     ):
         if file_path is not None:
+            self.validate_file_path(file_path)
             with pd.HDFStore(file_path) as f:
                 self.person = f["person"]
                 self.benunit = f["benunit"]
@@ -59,7 +87,9 @@ def copy(self):
         )
 
     @staticmethod
-    def from_simulation(simulation: Microsimulation, fiscal_year: int = 2025):
+    def from_simulation(
+        simulation: "Microsimulation", fiscal_year: int = 2025
+    ):
         entity_dfs = {}
 
         for entity in ["person", "benunit", "household"]:

policyengine_uk/data/dataset_schema.py

@@ -2,7 +2,7 @@
 from pathlib import Path
 from policyengine_uk.data.dataset_schema import UKDataset
 
-START_YEAR = 2020
+START_YEAR = 2022
 END_YEAR = 2029
 
 
@@ -13,40 +13,29 @@ def create_policyengine_uprating_factors_table(print_diff=True):
 
     variable_names = []
     years = []
-    index_values = []
+    yoy_values = []
 
     parameter_by_variable = {}
 
     for variable in system.variables.values():
         if variable.uprating is not None:
-            parameter = system.parameters.get_child(variable.uprating)
+            parameter = system.parameters.get_child(
+                variable.uprating.replace("indices", "yoy_growth")
+            )
             parameter_by_variable[variable.name] = parameter.name
-            start_value = parameter(START_YEAR)
             for year in range(START_YEAR, END_YEAR + 1):
                 variable_names.append(variable.name)
-                years.append(str(year))  # Convert to string here
-                growth = parameter(year) / start_value
-                index_values.append(round(growth, 3))
+                years.append(str(year))
+                yoy_values.append(round(parameter(year), 3))
 
     df["Variable"] = variable_names
     df["Year"] = years
-    df["Value"] = index_values
+    df["Value"] = yoy_values
 
     # Convert to there is a column for each year
     df = df.pivot(index="Variable", columns="Year", values="Value")
     df = df.sort_values("Variable")
 
-    # Create a table with growth factors by year
-
-    df_growth = df.copy()
-    for year in range(END_YEAR, START_YEAR, -1):
-        year_str = str(year)
-        prev_year_str = str(year - 1)
-        df_growth[year_str] = round(
-            df_growth[year_str] / df_growth[prev_year_str] - 1, 3
-        )
-    df_growth[str(START_YEAR)] = 0
-
     file_path = Path(__file__).parent / "uprating_growth_factors.csv"
 
     # Read old CSV if it exists
@@ -57,16 +46,16 @@ def create_policyengine_uprating_factors_table(print_diff=True):
         old_df.columns = old_df.columns.astype(str)
 
     # Prepare new dataframe
-    df_growth["Parameter"] = df.index.map(parameter_by_variable)
-    df_growth = df_growth[
+    df["Parameter"] = df.index.map(parameter_by_variable)
+    df = df[
         ["Parameter"] + [str(year) for year in range(START_YEAR, END_YEAR + 1)]
     ]
 
     # Print diff if old CSV existed and print_diff is True
     if old_df is not None and print_diff:
-        print_csv_diff(old_df, df_growth)
+        print_csv_diff(old_df, df)
         # Save new CSV
-        df_growth.to_csv(file_path)
+        df.to_csv(file_path)
 
     return pd.read_csv(file_path)