|
| 1 | +# Time-varying λ models (breakpoint hazard) |
| 2 | + |
| 3 | +Status: **experimental / parked.** The constant-λ breakpoint model is implemented, |
| 4 | +tested, and was run nationwide once (2026-06-08). The nationwide fit did not |
| 5 | +support a meaningful breakpoint, so the planned random-effects extension was |
| 6 | +**not** built. This note records why we built it, how it works, how to run it, |
| 7 | +what we found, and why we stopped — so the work is reproducible and the dead end |
| 8 | +is not re-explored from scratch. |
| 9 | + |
| 10 | +The breakpoint code is fully **isolated and additive**: the production |
| 11 | +`constant`, `random_by_type`, and `random_effects` models are unchanged. See also |
| 12 | +[turnover-model-methodology.md §8](turnover-model-methodology.md) for the |
| 13 | +statistical derivation. |
| 14 | + |
| 15 | +## 1. Why we developed it |
| 16 | + |
| 17 | +The turnover model assumes a **constant** per-year hazard λ over each |
| 18 | +observation interval, so the cumulative hazard that drives the change |
| 19 | +probability (`P(change) = 1 − exp(−rate)`) is just `λ·Δt`. We wanted to test an |
| 20 | +"infant-mortality" hypothesis: that a POI tag churns at one rate while the tag |
| 21 | +value is young and a different rate once it has aged past some breakpoint `t_B` |
| 22 | +(prior centred at ~1 year). The breakpoint is the simplest time-varying λ with a |
| 23 | +**closed-form integral**, which is the design rule for any time-varying form here |
| 24 | +(no numerical quadrature in the NUTS inner loop). |
| 25 | + |
| 26 | +## 2. Key idea — integrated hazard + tag-age clock |
| 27 | + |
| 28 | +The whole likelihood consumes λ **only** through the integral over each interval, |
| 29 | +`H(t1, t2) = ∫ λ(a) da`. Constant λ is the `λ·Δt` special case; the breakpoint |
| 30 | +swaps in a different closed form. The time axis is the tag's **age since its |
| 31 | +current value was established** (`last_tag_timestamp`), so per-row hazards |
| 32 | +telescope to each POI's cumulative hazard. |
| 33 | + |
| 34 | +Two-rate breakpoint integral (`λ_1` before age `t_B`, `λ_2` after): |
| 35 | + |
| 36 | +``` |
| 37 | +crossing = clip(t_B, age_start, age_end) |
| 38 | +H(t1, t2) = λ_1·(crossing − age_start) + λ_2·(age_end − crossing) |
| 39 | +``` |
| 40 | + |
| 41 | +This one branchless expression covers all three cases (interval fully before / |
| 42 | +after / straddling `t_B`) and reduces to `λ·Δt` when `λ_1 == λ_2`. |
| 43 | + |
| 44 | +**Rating consequence.** Snapshot rating anchors at `last_edited`, but a tag-age |
| 45 | +hazard must be integrated at the POI's *true* age over the `[last_edited, now]` |
| 46 | +window: |
| 47 | + |
| 48 | +``` |
| 49 | +Λ(a) = λ_1·min(a, t_B) + λ_2·max(0, a − t_B) |
| 50 | +P(turnover) = 1 − exp( −( Λ(a_now) − Λ(a_last_edit) ) ) |
| 51 | +``` |
| 52 | + |
| 53 | +which needs a per-POI `tag_established` (joined from history); POIs absent from |
| 54 | +history fall back to `a_last_edit = 0`. |
| 55 | + |
| 56 | +## 3. Key files and functions |
| 57 | + |
| 58 | +| File | What | |
| 59 | +|---|---| |
| 60 | +| [src/openpois/models/setup.py](src/openpois/models/setup.py) | `prepare_data_for_model` emits `age_start` / `age_end` (tag-age bounds; `age_end − age_start == tag_years`, `age_start == 0` on first intervals). Additive — constant/RE models ignore them. | |
| 61 | +| [src/openpois/models/osm_models.py](src/openpois/models/osm_models.py) | `_breakpoint_integrated_hazard(lam1, lam2, t_b, age_start, age_end)` (the clamp formula); `ConstantBreakpointModel` (subclass of `ConstantModel`, reuses the ZIE δ machinery); registry key `constant_breakpoint`; `DEFAULT_T_BREAKPOINT_PRIOR = (0.0, 1.0)`. | |
| 62 | +| [scripts/models/osm_turnover.py](scripts/models/osm_turnover.py) | `--model-type constant_breakpoint`; wires `osm_turnover_model.t_breakpoint_prior` into model metadata. | |
| 63 | +| [config.yaml](config.yaml) | `osm_turnover_model.t_breakpoint_prior: [0.0, 1.0]` — (loc, scale) on log `t_B`; loc 0 → median `t_B` = 1 yr. | |
| 64 | +| [scripts/osm_data/add_turnover_columns.py](scripts/osm_data/add_turnover_columns.py) | Final-stage augmentation: reuses an existing `osm_observations.parquet` (no history re-download), adds `age_start`/`age_end`, and emits `osm_current_tag.parquet` — one row per element with `tag_established` (latest version's `last_tag_timestamp`) + `last_seen`. | |
| 65 | +| [scripts/osm_snapshot/apply_model_breakpoint.py](scripts/osm_snapshot/apply_model_breakpoint.py) | Rates the live snapshot via the closed-form `Λ`. `load_breakpoint_draws`, `cumulative_hazard`, `turnover_stats`. Output columns: `p_turnover_*` = **P(change)**, `conf_*` = 1 − P (P(no change)), `tag_age_years`, `matched_history`. | |
| 66 | +| [tests/test_osm_models.py](tests/test_osm_models.py) | `test_breakpoint_integrated_hazard`, `test_constant_breakpoint_recovery`, `test_constant_breakpoint_reduces_to_constant`, `test_predictions_schema_constant_breakpoint`, `test_constant_breakpoint_requires_age_columns`, `test_prepare_data_emits_age_columns`. | |
| 67 | + |
| 68 | +**Parameters:** `log_lambda_1`, `log_lambda_2` (each N(0, 3) on the log scale, |
| 69 | +same prior as the constant model's λ), `log_t_breakpoint` (N(0, 1) → log-normal |
| 70 | +`t_B`, median 1 yr), `logit_delta` (unchanged ZIE δ). Derived draws expose |
| 71 | +`lambda_1`, `lambda_2`, `t_breakpoint`, `delta`. |
| 72 | + |
| 73 | +## 4. How it's run (the nationwide test, end to end) |
| 74 | + |
| 75 | +No new history download — reuse the prepared `20260521` history through the final |
| 76 | +formatting stage: |
| 77 | + |
| 78 | +```bash |
| 79 | +# 1. Augment observations + emit tag_established lookup → osm_data/20260608 |
| 80 | +python scripts/osm_data/add_turnover_columns.py \ |
| 81 | + --source-version 20260521 --target-version 20260608 |
| 82 | + |
| 83 | +# 2. Fit the constant breakpoint model (dense; ~2 h over ~9.9M rows on CPU) |
| 84 | +python -u scripts/models/osm_turnover.py \ |
| 85 | + --model-type constant_breakpoint \ |
| 86 | + --observations ~/data/openpois/osm_data/20260608/osm_observations.parquet \ |
| 87 | + --model-version 2026-06-08-breakpoint-test |
| 88 | + |
| 89 | +# 3. Rate the live snapshot with the closed-form Λ |
| 90 | +python -u scripts/osm_snapshot/apply_model_breakpoint.py \ |
| 91 | + --model-version 2026-06-08-breakpoint-test \ |
| 92 | + --current-tag-version 20260608 |
| 93 | +``` |
| 94 | + |
| 95 | +Note: the cell **sufficient-statistics** fast path cannot be used — a sampled |
| 96 | +`t_B` enters the per-row integral through `clip`, so the likelihood is **dense** |
| 97 | +per-row. That is why the national constant fit took ~2 h, and why a |
| 98 | +random-effects + OOS version would cost many hours. |
| 99 | + |
| 100 | +## 5. What we found (nationwide, 2026-06-08) |
| 101 | + |
| 102 | +Input: `osm_data/20260608` (9,963,378 observation rows). Fit: 4 chains × |
| 103 | +(500 warmup + 500 draws), ~2 h 11 m wall. |
| 104 | + |
| 105 | +**The fit did not converge — and that is the finding.** |
| 106 | + |
| 107 | +- `max R-hat = 202.5`, `min ESS = 2` (0 divergences, accept 0.846, mean_steps 18.2). |
| 108 | +- `λ_2` is stable across all chains: **≈ 0.054 /yr**. |
| 109 | +- `λ_1` and `t_B` are **confounded** — per-chain `t_B` ∈ {0.0063, 0.015, 0.044, |
| 110 | + 0.044} yr (≈ **2–16 days**), with `λ_1` swinging 1.7 → 19.4 to compensate. |
| 111 | + When `t_B → 0`, only the *product* `λ_1·t_B` (the brief early-time mass, which |
| 112 | + also overlaps δ) is identified — not `λ_1` and `t_B` separately. |
| 113 | + |
| 114 | +So the national data wants a very short high-churn window (days) right after a |
| 115 | +name is set, then settles to `λ_2`. There is **no meaningful breakpoint at a |
| 116 | +≥6-month timescale**. |
| 117 | + |
| 118 | +Apply step (sanity, despite the non-converged posterior): all **8,799,633** POIs |
| 119 | +rated, no NaNs, `p_turnover` (= P(change)) mean 0.23 / median 0.20, range |
| 120 | +0.003–0.68. The tag-age mechanism works as designed — an old tag edited recently |
| 121 | +(e.g. age 10.6 yr, edited 1.2 yr ago) correctly sits in the low-`λ_2` regime |
| 122 | +(p ≈ 0.06). History coverage on the live snapshot was **21.3%** matched to a real |
| 123 | +`tag_established`; the other 78.7% used the last-edit fallback. |
| 124 | + |
| 125 | +## 6. Why we are not developing it further (for now) |
| 126 | + |
| 127 | +1. **The breakpoint collapses into δ.** The two-regime structure the data |
| 128 | + supports lives at a sub-month scale — i.e. "brief churn right after a name is |
| 129 | + set," which the **zero-inflation δ** term (instant-change mass at t = 0, |
| 130 | + methodology §1.7) already models. The breakpoint is largely redundant at the |
| 131 | + granularity the data supports. |
| 132 | +2. **Non-identifiability.** R-hat 202 / ESS 2 is the symptom: `λ_1` and `t_B` |
| 133 | + trade off as `t_B → 0`. The pooled parameter summary is not trustworthy. |
| 134 | +3. **A user-defined gate closed.** We agreed to extend to random effects **only |
| 135 | + if** the national constant fit showed (a) `λ_1` vs `λ_2` differing by > 10% |
| 136 | + **and** (b) `t_B ≥ 6 months`. Condition (a) passed (~99% apart), but |
| 137 | + (b) failed decisively (`t_B ≈ 0.027 yr ≈ 10 days`). Per the gate, we stopped. |
| 138 | +4. **Cost.** Without the suff-stats fast path the model is dense; a |
| 139 | + random-effects breakpoint with per-amenity `t_B` plus full OOS would run many |
| 140 | + hours per pass. |
| 141 | + |
| 142 | +### The random-effects breakpoint extension was scoped but NOT built |
| 143 | + |
| 144 | +Design (for reference if revisited): a new `breakpoint_models.py` with a |
| 145 | +`RandomEffectsBreakpointModel` adapted from `RandomEffectsModel` — the entire |
| 146 | +log-λ predictor duplicated into independent `λ_1` / `λ_2` regimes (global |
| 147 | +intercept + amenity/MSA/interaction random effects + urbanicity fixed effects), |
| 148 | +a global `t_B` with an optional amenity-level random intercept, δ kept from the |
| 149 | +best spec (Full + δ(amenity+MSA), `2026-06-06-oos-full-dmsa`), dense likelihood, |
| 150 | +fit in-sample and OOS with the standard per-fold/aggregate/subgroup metrics. |
| 151 | +None of this was implemented. |
| 152 | + |
| 153 | +### If revisited, options in rough order of promise |
| 154 | + |
| 155 | +- Drop δ and let `λ_1` absorb the early churn (does a breakpoint *replace* δ?). |
| 156 | +- Tighten/constrain the `t_B` prior, or add an ordering / identifiability |
| 157 | + constraint so `λ_1`/`t_B` separate. |
| 158 | +- Accept that a ≥6-month breakpoint is not supported nationally and keep the |
| 159 | + constant / random-effects λ + δ model (the production path). |
| 160 | + |
| 161 | +## 7. Reproducibility artifacts |
| 162 | + |
| 163 | +On disk (uncommitted; production versions untouched): |
| 164 | +- `~/data/openpois/osm_data/20260608/` — augmented `osm_observations.parquet` |
| 165 | + (+ `age_start`/`age_end`) and `osm_current_tag.parquet` (3,878,564 elements). |
| 166 | +- `~/data/openpois/osm_turnover_model/2026-06-08-breakpoint-test/` — fit |
| 167 | + (`fitted_params.csv`, `param_draws.parquet`, `diagnostics.csv`) + |
| 168 | + `osm_snapshot_turnover.parquet` (the rated snapshot). |
| 169 | +- Logs in `~/data/openpois/logs/` (`augment_20260608`, `fit_breakpoint_20260608`, |
| 170 | + `apply_breakpoint_20260608`). |
| 171 | + |
| 172 | +Production pins in [config.yaml](config.yaml) (`osm_data: 20260521`, |
| 173 | +`model_output: 20260422_by_shared_label`) and their outputs were never modified. |
0 commit comments