Thread time_period through local-area loss matrices (#345 step 4)

Fixes two concrete bugs that would have prevented calibrating the same
base dataset at a year other than its stored `time_period`:

- `local_authorities/loss.py` read household weights at a hard-coded 2025
  when computing the national-total fallbacks used for LAs missing ONS
  data. Now uses the explicit `time_period` argument.
- `constituencies/loss.py` passed `dataset.time_period` to
  `get_national_income_projections` and `sim.default_calculation_period`
  even when the caller supplied a different `time_period`. Same fix.

Also extracts the year-resolution logic from `build_loss_matrix._resolve_value`
into a documented public function `resolve_target_value`, names the
three-year tolerance as a constant, and adds 12 unit tests covering the
fallback policy (exact match, nearest past year, tolerance limit, no
backwards extrapolation, VOA population scaling).

Ships `docs/targets_coverage.md` documenting year coverage across every
target category and where the real gaps are (DWP 2026+, local-area CSV
refreshes). No new data sourced in this PR — sourcing is deferred.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: vahid-ahmadi

changelog.d/345.md

@@ -1 +1 @@
-Add panel ID contract, `create_yearly_snapshots` helper and `age_dataset` demographic ageing module as the first three steps towards per-year snapshots (#345).
+Add panel ID contract, `create_yearly_snapshots` helper, `age_dataset` demographic ageing module and year-aware loss matrices with a documented `resolve_target_value` fallback policy as the first four steps towards per-year snapshots (#345).

docs/targets_coverage.md

@@ -0,0 +1,57 @@
+# Calibration target coverage by year
+
+Per-year calibration (step 4 of [#345](https://github.com/PolicyEngine/policyengine-uk-data/issues/345)) depends on target values being available for every year we want to calibrate against. This document is the snapshot of where coverage is today so it is easy to see where future data-sourcing work is needed.
+
+The pipeline resolves a requested year against each target's available years using `resolve_target_value` in `policyengine_uk_data/targets/build_loss_matrix.py`. The policy is: exact match → nearest past year within three years → `None`. There is no backwards extrapolation and no forwards extrapolation beyond three years — if the tolerance is exceeded the target is silently skipped.
+
+## National and country-level targets
+
+| Category | Source | Year range in repo | Year-keyed? | Notes |
+| --- | --- | --- | --- | --- |
+| Population by sex × age band | ONS mid-year population estimates | 2022-2029 | Yes (`ons_demographics.py`) | Downloaded as multi-year at registry build time |
+| Regional population by age | ONS subnational estimates | 2018-2029 | Yes (`demographics.csv`) | Multi-year CSV keyed by `year` column |
+| UK total population | ONS | 2018-2029 | Yes (`demographics.csv`) | Used by `resolve_target_value` for VOA scaling |
+| Scotland demographics (children, babies, 3+ child households) | ONS / Scottish Government | 2025 + some 2029 | Partial | Missing 2026-2028 |
+| Income by band (SPI) — national | HMRC SPI | 2022-2029 | Yes (`incomes_projection.csv`) | Projected from 2021 SPI via microsimulation |
+| Income tax, NICs, VAT, CGT, SDLT, fuel duty totals | OBR Economic and Fiscal Outlook | 2024-2030 | Yes (`obr.py`) | Live download, multi-year |
+| Council tax totals | OBR | 2024-2030 | Yes | OBR line items |
+| Council tax band counts | VOA | 2024 | No | Population-scaled by `resolve_target_value` for adjacent years |
+| Housing totals (mortgage, private rent, social rent) | ONS / EHS | 2025 | No | Single-year only — needs 2026+ refresh |
+| Tenure totals | ONS / EHS | 2025 | No | Single-year only |
+| Savings interest | ONS | 2025 | No | Single-year only |
+| Land values (household, corporate, total) | ONS National Balance Sheet | 2025 | No | Single-year only |
+| Regional household land values | MHCLG | 2025 | No | Single-year only |
+| DWP benefit caseloads (UC, ESA, PIP, JSA, benefit cap, UC by children / family type) | DWP Stat-Xplore / benefit statistics | 2025 (a few 2026) | Mostly no | **Primary gap**: 2026+ needs DWP forecasts or policy extrapolation |
+| Salary sacrifice (IT relief, contributions, NI relief) | HMRC / OBR | 2025 | No | OBR has 2024-2030 on some items |
+| Salary sacrifice headcount | OBR | 2024-2030 | Yes | Multi-year |
+| UC jobseeker splits, UC outside cap | OBR | 2024-2030 | Yes | Multi-year |
+| Two-child limit | DWP | 2025 | No | Single-year only |
+| Student loan plan borrower counts | SLC | 2025 | No | Single-year only |
+| Student loan repayment | SLC | 2025 | No | Single-year only |
+| NTS vehicle ownership | DfT National Travel Survey | 2024 | No | Single-year only |
+| TV licence | OBR | 2024 + 3% pa extrapolation | Implicit | Hard-coded extrapolation in `obr.py` |
+
+## Constituency- and LA-level targets
+
+| Category | Source | Year | Notes |
+| --- | --- | --- | --- |
+| Age bands per constituency / LA | ONS subnational population estimates | Snapshot (no year column) | `age.csv` files under `datasets/local_areas/*/targets/`; need annual refresh |
+| Income by area (employment, self-employment; count + amount) | HMRC SPI table 3.15 | Snapshot | `spi_by_constituency.csv`, `spi_by_la.csv`; HMRC publishes annually |
+| UC household counts by area | DWP Stat-Xplore | November 2023 | Scaled to 2025 national totals via `_scaled_uc_children_by_country` |
+| UC households by number of children (area level) | DWP Stat-Xplore | November 2023 base + 2025 scaling | In `local_uc.py` |
+| ONS small-area income estimates (LA only) | ONS | FYE 2020 + uprating | Uprated per-year via `get_ons_income_uprating_factors(year)` |
+| Tenure by LA | English Housing Survey 2023 | Snapshot | `la_tenure.xlsx` |
+| Private rent median by LA | VOA / ONS | Snapshot | `la_private_rents_median.xlsx` |
+
+## Known gaps — what blocks full per-year calibration
+
+1. **DWP benefit caseloads for 2026+**. The DWP statistical releases publish mostly current-year snapshots; forecasts are internal. Getting these requires coordination with the policy team or an agreed extrapolation policy from 2025 onwards.
+2. **Local-area CSVs (age, SPI, UC)**. Single-year snapshots stored as CSVs without a `year` column. For panel calibration these need an annual refresh process and a filename convention that includes the source year (e.g. `spi_by_constituency_2024.csv`).
+3. **Small-scale single-year sources**. NTS vehicles (2024), housing totals, SLC student loans, land values — each individually small but collectively relevant.
+
+## Related code
+
+- `policyengine_uk_data/targets/build_loss_matrix.py` — `resolve_target_value` and the national loss matrix builder.
+- `policyengine_uk_data/datasets/local_areas/constituencies/loss.py` — constituency loss matrix; now honours `time_period`.
+- `policyengine_uk_data/datasets/local_areas/local_authorities/loss.py` — LA loss matrix; now honours `time_period` (previously read weights at hard-coded 2025).
+- `policyengine_uk_data/targets/sources/` — individual target modules. The multi-year ones (`obr.py`, `ons_demographics.py`) are the template for converting the rest.

policyengine_uk_data/datasets/local_areas/constituencies/loss.py

@@ -46,14 +46,18 @@ def create_constituency_target_matrix(
         time_period = dataset.time_period
 
     sim = Microsimulation(dataset=dataset, reform=reform)
-    sim.default_calculation_period = dataset.time_period
+    # Honour the explicit ``time_period`` argument so that calibrating the
+    # same base dataset to a different year (needed for panel output in
+    # #345) reads variables at the requested year rather than the dataset's
+    # stored year.
+    sim.default_calculation_period = time_period
 
     matrix = pd.DataFrame()
     y = pd.DataFrame()
 
     # ── Income targets ─────────────────────────────────────────────
     incomes = get_constituency_income_targets()
-    national_incomes = get_national_income_projections(int(dataset.time_period))
+    national_incomes = get_national_income_projections(int(time_period))
 
     for income_variable in INCOME_VARIABLES:
         income_values = sim.calculate(income_variable).values

policyengine_uk_data/datasets/local_areas/local_authorities/loss.py

@@ -51,15 +51,20 @@ def create_local_authority_target_matrix(
     la_codes = pd.read_csv(STORAGE_FOLDER / "local_authorities_2021.csv")
 
     sim = Microsimulation(dataset=dataset, reform=reform)
-    original_weights = sim.calculate("household_weight", 2025).values
+    # Read the uncalibrated weights at the requested calibration year rather
+    # than a hard-coded 2025. The downstream national totals on lines
+    # ~160-170 are used as fall-back per-LA targets when ONS data is
+    # missing, and they must be expressed at the same year as the targets
+    # we are calibrating against.
+    original_weights = sim.calculate("household_weight", int(time_period)).values
     sim.default_calculation_period = time_period
 
     matrix = pd.DataFrame()
     y = pd.DataFrame()
 
     # ── Income targets ─────────────────────────────────────────────
     incomes = get_la_income_targets()
-    national_incomes = get_national_income_projections(int(dataset.time_period))
+    national_incomes = get_national_income_projections(int(time_period))
 
     for income_variable in INCOME_VARIABLES:
         income_values = sim.calculate(income_variable).values

policyengine_uk_data/targets/build_loss_matrix.py

@@ -122,24 +122,59 @@ def create_target_matrix(
     return df, pd.Series(target_values, index=target_names)
 
 
-def _resolve_value(target: Target, year: int) -> float | None:
-    """Get the target value for a year, falling back to nearest year.
+YEAR_FALLBACK_TOLERANCE = 3
+"""Maximum allowed gap between the requested year and the nearest available
+target year before ``resolve_target_value`` gives up and returns ``None``.
+
+Three years matches the typical publication lag on HMRC/DWP/ONS series and
+the length of the OBR's rolling forecast window. It is a deliberately
+generous ceiling — the goal is to keep the calibration functional when a
+dataset runs slightly ahead of the latest published target, not to make up
+numbers several years out."""
+
+
+def resolve_target_value(
+    target: Target,
+    year: int,
+    *,
+    tolerance: int = YEAR_FALLBACK_TOLERANCE,
+) -> float | None:
+    """Return the calibration value for ``target`` at ``year``.
+
+    Policy, in order:
+
+    1. **Exact year available** → return it.
+    2. **Requested year is in the past relative to all available years**
+       (e.g. ask 2022, data starts 2024) → return ``None``. We do not
+       extrapolate backwards, because doing so would quietly misreport
+       historical reality.
+    3. **Nearest past year is within ``tolerance`` years** → use that value.
+       For VOA-sourced targets only, the value is scaled by the UK
+       population ratio between the two years so that count-type targets
+       track demographic growth.
+    4. **Nearest past year is further than ``tolerance``** → return
+       ``None`` rather than make up a number.
 
-    VOA council tax targets are population-uprated when extrapolating
-    from their base year (2024).
+    Args:
+        target: a ``Target`` whose ``values`` map maps year → scalar.
+        year: the calendar year being requested.
+        tolerance: maximum number of years the fallback may look back.
+            Defaults to ``YEAR_FALLBACK_TOLERANCE``.
+
+    Returns:
+        The target value, or ``None`` if no usable year is available.
     """
     if year in target.values:
         return target.values[year]
     available = sorted(target.values.keys())
     if not available:
         return None
     closest = min(available, key=lambda y: abs(y - year))
-    if abs(closest - year) > 3:
+    if abs(closest - year) > tolerance:
         return None
     if closest > year:
         return None
     base_value = target.values[closest]
-    # VOA council tax counts scale with population
     if target.source == "voa" and year != closest:
         from policyengine_uk_data.targets.sources.local_age import (
             get_uk_total_population,
@@ -152,6 +187,11 @@ def _resolve_value(target: Target, year: int) -> float | None:
     return base_value
 
 
+# Kept as a private alias so existing call sites (this module only) do not
+# need to be rewritten. New code should prefer ``resolve_target_value``.
+_resolve_value = resolve_target_value
+
+
 class _SimContext:
     """Holds the simulation and lazily computed intermediate arrays."""
 

policyengine_uk_data/tests/test_resolve_target_value.py

@@ -0,0 +1,130 @@
+"""Tests for the year-resolution policy in build_loss_matrix (#345, step 4)."""
+
+import pytest
+
+from policyengine_uk_data.targets.build_loss_matrix import (
+    YEAR_FALLBACK_TOLERANCE,
+    resolve_target_value,
+)
+from policyengine_uk_data.targets.schema import (
+    GeographicLevel,
+    Target,
+    Unit,
+)
+
+
+def _target(values: dict[int, float], *, source: str = "ons") -> Target:
+    """Minimal Target instance for year-resolution tests."""
+    return Target(
+        name=f"{source}/test",
+        variable="test_variable",
+        source=source,
+        unit=Unit.COUNT,
+        geographic_level=GeographicLevel.NATIONAL,
+        values=values,
+    )
+
+
+def test_exact_year_match_returns_that_value():
+    t = _target({2024: 10.0, 2025: 20.0, 2026: 30.0})
+    assert resolve_target_value(t, 2025) == 20.0
+
+
+def test_exact_match_preferred_over_fallback():
+    t = _target({2024: 10.0, 2025: 20.0})
+    # Even though 2024 is "nearest" to 2024, the exact match for 2025
+    # must win outright.
+    assert resolve_target_value(t, 2024) == 10.0
+
+
+def test_falls_back_to_nearest_past_year_within_tolerance():
+    t = _target({2023: 7.0})
+    # 2024 and 2025 fall back to 2023 within tolerance.
+    assert resolve_target_value(t, 2024) == 7.0
+    assert resolve_target_value(t, 2025) == 7.0
+    assert resolve_target_value(t, 2026) == 7.0
+
+
+def test_returns_none_when_only_future_years_available():
+    """Extrapolating backwards would misreport historical reality."""
+    t = _target({2025: 50.0, 2026: 60.0})
+    assert resolve_target_value(t, 2023) is None
+    assert resolve_target_value(t, 2024) is None
+
+
+def test_returns_none_when_fallback_exceeds_tolerance():
+    t = _target({2020: 5.0})
+    # 2024 is four years away — beyond the three-year default.
+    assert resolve_target_value(t, 2024) is None
+
+
+def test_custom_tolerance_is_honoured():
+    t = _target({2020: 5.0})
+    # Explicitly widen the tolerance.
+    assert resolve_target_value(t, 2024, tolerance=4) == 5.0
+    # Or tighten it.
+    assert resolve_target_value(t, 2022, tolerance=1) is None
+
+
+def test_empty_values_returns_none():
+    t = _target({})
+    assert resolve_target_value(t, 2025) is None
+
+
+def test_default_tolerance_is_three_years():
+    """Lock the public tolerance constant so a change is deliberate."""
+    assert YEAR_FALLBACK_TOLERANCE == 3
+
+
+def test_non_voa_target_does_not_get_population_scaled():
+    """Only VOA council-tax counts should track population growth."""
+    t = _target({2024: 100.0}, source="dwp")
+    # 2025 is within tolerance but DWP data must not be rescaled.
+    assert resolve_target_value(t, 2025) == 100.0
+
+
+def test_voa_target_scales_with_population_when_extrapolating(monkeypatch):
+    """VOA counts must move roughly in line with population."""
+    t = _target({2024: 100.0}, source="voa")
+
+    fake_pop = {2024: 67_000_000.0, 2025: 68_000_000.0}
+
+    def fake_total_population(year):
+        return fake_pop[year]
+
+    monkeypatch.setattr(
+        "policyengine_uk_data.targets.sources.local_age.get_uk_total_population",
+        fake_total_population,
+    )
+
+    resolved = resolve_target_value(t, 2025)
+    expected = 100.0 * 68_000_000.0 / 67_000_000.0
+    assert resolved == pytest.approx(expected)
+
+
+def test_voa_target_returns_base_when_year_matches_exactly(monkeypatch):
+    """Population scaling only kicks in when we actually extrapolate."""
+    t = _target({2025: 123.0}, source="voa")
+
+    # If the scaler is called, blow up — it must not be touched here.
+    def explode(year):
+        raise AssertionError("Population scaler called on exact match")
+
+    monkeypatch.setattr(
+        "policyengine_uk_data.targets.sources.local_age.get_uk_total_population",
+        explode,
+    )
+
+    assert resolve_target_value(t, 2025) == 123.0
+
+
+def test_voa_guards_against_zero_population_base(monkeypatch):
+    """If the population lookup returns zero, fall back to the raw value."""
+    t = _target({2024: 100.0}, source="voa")
+
+    monkeypatch.setattr(
+        "policyengine_uk_data.targets.sources.local_age.get_uk_total_population",
+        lambda year: 0.0,
+    )
+    # Division by zero must be avoided; raw value returned as-is.
+    assert resolve_target_value(t, 2025) == 100.0