Add defensive load_weights helper to normalise calibration weight shape#351
Merged
Conversation
Contributor
Author
|
Self-review: APPROVE. See |
vahid-ahmadi
added a commit
that referenced
this pull request
Apr 21, 2026
Conflict was in `policyengine_uk_data/utils/calibrate.py`: main added `load_weights` (#351 defensive h5-weight loader) at the same file position where this branch added `compute_log_weight_smoothness_penalty` (#346 step 5). Both functions are independent and both stay. All 150 tests in the panel-pipeline suite + the calibrate smoothness tests pass after the merge, including `load_weights` consumers pulled in from main (test_calibrate_save, test_la_land_value_targets).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The bug hunt (finding U4) flagged that the L2 calibrator in
calibrate_local_areaswrites 2D(n_areas, n_records)weights under a dataset key, while the L0-regularised variant (in a separate PR branch) writes a flat(n_records,)array under the same key. Downstream consumers cannot tell the shapes apart unless they explicitly check axes;.sum(axis=0)on a 1D array is a no-op, which silently produces wrong answers.This PR adds a shape-normalising loader in the stable module so any future consumer — including whichever calibrator ships first — can load weights safely:
load_weights(weight_file, dataset_key, n_areas=None, n_records=None)returns a 2D(n_areas, n_records)array. A 1D input is reshaped to(1, n_records)so matrix ops behave identically across back-ends.n_areasandn_recordskwargs raise a clearValueErroron shape mismatch instead of letting the error surface as garbage weighted totals downstream.KeyErrorwith the list of available keys. 3D+ inputs raiseValueErrorexplicitly.No call site is changed here — this is a detection and normalisation tool that consumers can adopt as they are touched (e.g.
matrix_builder, validation notebooks). The L0 calibrator PR should route through this helper before merging to main so both shapes stay interchangeable.Test plan
uv run pytest policyengine_uk_data/tests/test_load_weights.py -qpasses (5 cases: 2D pass-through, 1D promotion, explicit shape validation, missing key, higher-dim rejection).Finding U4 from the bug hunt.