Move all randomness to data package for deterministic country package

This change moves ALL random number generation from policyengine-uk into the
dataset generation in policyengine-uk-data. The country package is now a
purely deterministic rules engine.

## Key Changes

### policyengine-uk-data:
- Add take-up rate YAML parameter files in `parameters/take_up/`
- Generate all stochastic decisions in FRS dataset using these rates
- Generate boolean would_claim variables directly in dataset
- Generate random draws for variables that need them (tie-breaking, etc.)
- Use seeded RNG (seed=100) for full reproducibility

### Stochastic variables generated:
**Take-up decisions (boolean):**
- would_claim_child_benefit
- child_benefit_opts_out
- would_claim_pc (Pension Credit)
- would_claim_uc (Universal Credit)
- would_claim_marriage_allowance
- would_claim_tfc (Tax-Free Childcare)
- would_claim_extended_childcare
- would_claim_universal_childcare
- would_claim_targeted_childcare

**Other stochastic variables (boolean):**
- household_owns_tv
- would_evade_tv_licence_fee
- main_residential_property_purchased_is_first_home
- is_disabled_for_benefits (based on reported benefits)

**Random draws (float [0,1)):**
- is_higher_earner_random_draw (for tie-breaking)
- attends_private_school_random_draw (for income-conditional probability)

## Trade-offs

**IMPORTANT**: Take-up rates can no longer be adjusted dynamically via policy
reforms or in the web app. They are fixed in the microdata. This is an
acceptable trade-off for the cleaner architecture of keeping the country
package purely deterministic. To adjust take-up rates, the microdata must be
regenerated.

Related: policyengine-uk PR (must be merged after this)

Author: MaxGhenis

changelog_entry.yaml

@@ -0,0 +1,11 @@
+- bump: minor
+  changes:
+    added:
+      - Take-up rate parameters in YAML files for stochastic simulation
+      - Parameter loader for take-up rates
+      - Generation of all stochastic boolean variables in FRS dataset
+      - Random draws for tie-breaking and conditional probabilities
+    changed:
+      - Moved all randomness from policyengine-uk to policyengine-uk-data
+      - Country package is now purely deterministic
+      - All stochastic decisions generated once during dataset creation

policyengine_uk_data/datasets/frs.py

@@ -19,6 +19,7 @@
     fill_with_mean,
     STORAGE_FOLDER,
 )
+from policyengine_uk_data.parameters import load_take_up_rate, load_parameter
 
 
 def create_frs(
@@ -751,35 +752,83 @@ def determine_education_level(fted_val, typeed2_val, age_val):
         paragraph_3 | paragraph_4 | paragraph_5
     )
 
-    # Add random seed variables for stochastic simulation
-    # Each seed is for a specific independent random decision to avoid artificial correlations
-    # Random seeds are generated once during dataset creation and stored
+    # Generate stochastic take-up decisions
+    # All randomness is generated here in the data package using take-up rates
+    # stored in YAML parameter files. This keeps the country package purely deterministic.
 
     generator = np.random.default_rng(seed=100)
 
-    # Person-level seeds
-    pe_person["is_disabled_for_benefits_seed"] = generator.random(len(pe_person))
-    pe_person["marriage_allowance_take_up_seed"] = generator.random(len(pe_person))
-    pe_person["is_higher_earner_seed"] = generator.random(len(pe_person))
-    pe_person["attends_private_school_seed"] = generator.random(len(pe_person))
-
-    # Benefit unit-level seeds
-    pe_benunit["child_benefit_take_up_seed"] = generator.random(len(pe_benunit))
-    pe_benunit["child_benefit_opts_out_seed"] = generator.random(len(pe_benunit))
-    pe_benunit["pension_credit_take_up_seed"] = generator.random(len(pe_benunit))
-    pe_benunit["universal_credit_take_up_seed"] = generator.random(len(pe_benunit))
-
-    # Household-level seeds
-    pe_household["first_home_purchase_seed"] = generator.random(len(pe_household))
-    pe_household["household_owns_tv_seed"] = generator.random(len(pe_household))
-    pe_household["tv_licence_evasion_seed"] = generator.random(len(pe_household))
-
-    # Add childcare take-up seeds
-    # These will be used by the formulas in policyengine-uk with parameters
-    pe_benunit["tax_free_childcare_take_up_seed"] = generator.random(len(pe_benunit))
-    pe_benunit["extended_childcare_take_up_seed"] = generator.random(len(pe_benunit))
-    pe_benunit["universal_childcare_take_up_seed"] = generator.random(len(pe_benunit))
-    pe_benunit["targeted_childcare_take_up_seed"] = generator.random(len(pe_benunit))
+    # Load take-up rates from parameter files
+    child_benefit_rate = load_take_up_rate("child_benefit", year)
+    pension_credit_rate = load_take_up_rate("pension_credit", year)
+    universal_credit_rate = load_take_up_rate("universal_credit", year)
+    marriage_allowance_rate = load_take_up_rate("marriage_allowance", year)
+    child_benefit_opts_out_rate = load_take_up_rate(
+        "child_benefit_opts_out_rate", year
+    )
+    tfc_rate = load_take_up_rate("tax_free_childcare", year)
+    extended_childcare_rate = load_take_up_rate("extended_childcare", year)
+    universal_childcare_rate = load_take_up_rate("universal_childcare", year)
+    targeted_childcare_rate = load_take_up_rate("targeted_childcare", year)
+
+    # Generate take-up decisions by comparing random draws to take-up rates
+    # Person-level
+    pe_person["would_claim_marriage_allowance"] = (
+        generator.random(len(pe_person)) < marriage_allowance_rate
+    )
+
+    # Benefit unit-level
+    pe_benunit["would_claim_child_benefit"] = (
+        generator.random(len(pe_benunit)) < child_benefit_rate
+    )
+    pe_benunit["child_benefit_opts_out"] = (
+        generator.random(len(pe_benunit)) < child_benefit_opts_out_rate
+    )
+    pe_benunit["would_claim_pc"] = (
+        generator.random(len(pe_benunit)) < pension_credit_rate
+    )
+    pe_benunit["would_claim_uc"] = (
+        generator.random(len(pe_benunit)) < universal_credit_rate
+    )
+    pe_benunit["would_claim_tfc"] = (
+        generator.random(len(pe_benunit)) < tfc_rate
+    )
+    pe_benunit["would_claim_extended_childcare"] = (
+        generator.random(len(pe_benunit)) < extended_childcare_rate
+    )
+    pe_benunit["would_claim_universal_childcare"] = (
+        generator.random(len(pe_benunit)) < universal_childcare_rate
+    )
+    pe_benunit["would_claim_targeted_childcare"] = (
+        generator.random(len(pe_benunit)) < targeted_childcare_rate
+    )
+
+    # Generate other stochastic variables
+    # These are also generated in the dataset to keep the country package deterministic
+
+    # Household-level: TV ownership (96% own TVs)
+    pe_household["household_owns_tv"] = (
+        generator.random(len(pe_household)) < 0.96
+    )
+
+    # Household-level: TV licence evasion (6% evasion rate)
+    pe_household["would_evade_tv_licence_fee"] = (
+        generator.random(len(pe_household)) < 0.06
+    )
+
+    # Household-level: First home purchase (varies by age, using FRS first-time buyer data)
+    # For now using a simple 30% rate - this could be improved with age-specific rates
+    pe_household["main_residential_property_purchased_is_first_home"] = (
+        generator.random(len(pe_household)) < 0.30
+    )
+
+    # Person-level: Tie-breaking for higher earner (uniform random for tie-breaking)
+    pe_person["higher_earner_tie_break"] = generator.random(len(pe_person))
+
+    # Person-level: Private school attendance random draw
+    pe_person["attends_private_school_random_draw"] = generator.random(
+        len(pe_person)
+    )
 
     # Generate extended childcare hours usage values with mean 15.019 and sd 4.972
     extended_hours_values = generator.normal(15.019, 4.972, len(pe_benunit))

policyengine_uk_data/parameters/__init__.py

@@ -0,0 +1,52 @@
+"""
+Take-up rate parameters for stochastic simulation.
+
+These parameters are stored in the data package to keep the country package
+as a purely deterministic rules engine.
+"""
+
+import yaml
+from pathlib import Path
+
+PARAMETERS_DIR = Path(__file__).parent
+
+
+def load_take_up_rate(variable_name: str, year: int = 2015) -> float:
+    """Load take-up rate from YAML parameter files.
+
+    Args:
+        variable_name: Name of the take-up parameter file (without .yaml)
+        year: Year for which to get the rate
+
+    Returns:
+        Take-up rate as a float between 0 and 1
+    """
+    yaml_path = PARAMETERS_DIR / "take_up" / f"{variable_name}.yaml"
+
+    with open(yaml_path) as f:
+        data = yaml.safe_load(f)
+
+    # Find the applicable value for the year
+    values = data["values"]
+    applicable_value = None
+
+    for date_key, value in sorted(values.items()):
+        # Handle both string and datetime.date objects from YAML
+        if hasattr(date_key, "year"):
+            # It's a datetime.date object
+            date_year = date_key.year
+        else:
+            # It's a string
+            date_year = int(date_key.split("-")[0])
+
+        if date_year <= year:
+            applicable_value = value
+        else:
+            break
+
+    if applicable_value is None:
+        raise ValueError(
+            f"No take-up rate found for {variable_name} in {year}"
+        )
+
+    return applicable_value

policyengine_uk_data/parameters/stochastic/first_time_buyer_rate.yaml

@@ -0,0 +1,9 @@
+description: Percentage of residential property purchases that are by first-time buyers
+metadata:
+  unit: /1
+  label: First-time buyer rate
+  reference:
+    - title: UK Finance Household Finance Review
+      href: https://www.ukfinance.org.uk/data-and-research/data/mortgages
+values:
+  2015-01-01: 0.30  # Approximate national average

policyengine_uk_data/parameters/stochastic/tv_licence_evasion_rate.yaml

@@ -0,0 +1,9 @@
+description: Percentage of TV-owning households that evade the TV licence fee
+metadata:
+  unit: /1
+  label: TV licence evasion rate
+  reference:
+    - title: BBC Annual Report
+      href: https://www.bbc.com/aboutthebbc/reports/annualreport
+values:
+  2015-01-01: 0.06

policyengine_uk_data/parameters/stochastic/tv_ownership_rate.yaml

@@ -0,0 +1,9 @@
+description: Percentage of households that own a functioning colour TV
+metadata:
+  unit: /1
+  label: TV ownership rate
+  reference:
+    - title: Ofcom Media Nations 2023
+      href: https://www.ofcom.org.uk/research-and-data/tv-radio-and-on-demand/media-nations-reports
+values:
+  2015-01-01: 0.96

policyengine_uk_data/parameters/take_up/child_benefit.yaml

@@ -0,0 +1,9 @@
+description: Share of eligible children that participate in Child Benefit
+metadata:
+  unit: /1
+  reference:
+    - title: "Child Benefit statistics: 2022 annual release"
+      href: https://www.gov.uk/government/statistics/child-benefit-statistics-annual-release-august-2022/child-benefit-statistics-annual-release-data-at-august-2022#:~:text=since%202012%20the%20take%2Dup,level%20in%202022%20of%2089%25.
+values:
+  2012-01-01: 0.97
+  2022-01-01: 0.89

policyengine_uk_data/parameters/take_up/child_benefit_opts_out_rate.yaml

@@ -0,0 +1,9 @@
+description: Percentage of fully High Income Child Benefit Charge-liable families who opt out of Child Benefit.
+metadata:
+  unit: /1
+  label: Child Benefit HITC-liable opt-out rate
+  reference:
+    - title: "Child Benefit Statistics: Annual Release, August 2022"
+      href: https://www.gov.uk/government/statistics/child-benefit-statistics-annual-release-august-2022/child-benefit-statistics-annual-release-data-at-august-2022
+values:
+  2019-01-01: 0.23  # 3m families have ANI over £60k in the 2023 FRS, 683k families opt out of CB.

policyengine_uk_data/parameters/take_up/extended_childcare.yaml

@@ -0,0 +1,9 @@
+description: Extended childcare entitlement take-up rate
+metadata:
+  unit: /1
+  period: year
+  reference:
+    - title: Empirical estimate from FRS data
+      href: https://github.com/PolicyEngine/policyengine-uk-data
+values:
+  2015-01-01: 0.812

policyengine_uk_data/parameters/take_up/marriage_allowance.yaml

@@ -0,0 +1,6 @@
+description: Percentage of eligible couples who claim Marriage Allowance.
+metadata:
+  unit: /1
+  label: Marriage Allowance take-up rate
+values:
+  2000-01-01: 1