Skip to content

Commit d2ced8b

Browse files
authored
Fix resilience score intervals for active formula (koala73#3972)
1 parent 418bf73 commit d2ced8b

20 files changed

Lines changed: 772 additions & 197 deletions

api/health.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,7 @@ const STANDALONE_KEYS = {
188188
portwatchChokepointsRef: 'portwatch:chokepoints:ref:v1',
189189
chokepointFlows: 'energy:chokepoint-flows:v1',
190190
emberElectricity: 'energy:ember:v1:_all',
191-
resilienceIntervals: 'resilience:intervals:v2:US',
191+
resilienceIntervals: 'resilience:intervals:v3:US',
192192
sprPolicies: 'energy:spr-policies:v1',
193193
pipelinesGas: 'energy:pipelines:gas:v1',
194194
pipelinesOil: 'energy:pipelines:oil:v1',

docs/internal/country-resilience-upgrade-plan.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -237,7 +237,7 @@ intervals, change attribution, external expert review).
237237
- **Cache keys** (all in `_shared.ts`): `resilience:score:v7:<cc>` (6h),
238238
`resilience:ranking:v8` (6h, written only when all countries scored),
239239
`resilience:history:v4:<cc>` (daily sorted set, 30-day retention),
240-
`resilience:intervals:v2:<cc>` (95% CI from backtest).
240+
`resilience:intervals:v3:<cc>` (formula-tagged score sensitivity band).
241241
- **Warmup:** handler-owned, up to 200 missing countries per ranking request
242242
via `warmMissingResilienceScores()` in `get-resilience-ranking.ts`.
243243
- **Static seeding**, 11 slots in `scripts/seed-resilience-static.mjs` (WGI,
@@ -459,7 +459,7 @@ interval computation anywhere near the read path"*.)
459459
- Combine bootstrap and MC samples into a joint distribution; store
460460
per-pillar p05/p50/p95 and a joint overall p05/p50/p95.
461461
- Re-rank each sample and store per-country p05/p95 **rank band**.
462-
- Persist under `resilience:intervals:v2:<cc>` as
462+
- Persist under `resilience:intervals:v3:<cc>` as
463463
`{ computedAt, schemaVersion, pillarIntervals, overallInterval, rankBand }`.
464464
- **Read path** (`buildResilienceScore`): always reads the latest interval
465465
payload. If `computedAt` is older than 48 hours the response sets
@@ -740,7 +740,7 @@ tools.
740740
- **T3.1** Bootstrap + MC intervals implemented **fully offline** per
741741
*Technical Approach → Decomposed uncertainty*. Extend
742742
`scripts/seed-resilience-intervals.mjs` to write pillar-level intervals,
743-
joint overall intervals, and rank bands to `resilience:intervals:v2:<cc>`
743+
joint overall intervals, and rank bands to `resilience:intervals:v3:<cc>`
744744
every 6 hours. **Zero lazy computation on the read path**, missing key
745745
means intervals are omitted from the response; the read path never blocks
746746
on bootstrap or MC work. Add `staleIntervals: true` flag when payload is
@@ -825,7 +825,7 @@ tools.
825825
keys (UCDP, displacement, sanctions, WGI, RSF, etc.) → aggregates into
826826
domains → aggregates into pillars (new) → overall. History sorted set
827827
updated if current day not yet written. **Interval cache read only**: if
828-
`resilience:intervals:v2:<cc>` is present and fresh, it is attached; if
828+
`resilience:intervals:v3:<cc>` is present and fresh, it is attached; if
829829
the payload is >48h old the response sets `staleIntervals: true`; if the
830830
key is missing (warmup path), intervals are **omitted** from the response.
831831
The read path never computes intervals inline. All bootstrap and Monte

docs/methodology/country-resilience-index.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -603,7 +603,7 @@ The CRI is designed to be auditable end-to-end: given the Redis snapshot at any
603603
| `resilience:score:v18:{countryCode}` | JSON | 6 hours | `buildResilienceScore` in `server/worldmonitor/resilience/v1/_shared.ts` | `getResilienceScore` handler |
604604
| `resilience:ranking:v18` | JSON | 12 hours | `getResilienceRanking` warm path, only when at least 75% of countries are scored (`RANKING_CACHE_MIN_COVERAGE = 0.75`) | `getResilienceRanking` handler |
605605
| `resilience:history:v13:{countryCode}` | sorted set | indefinite, trimmed to 30 days | `appendHistory` during scoring | trend and `change30d` computation |
606-
| `resilience:intervals:v2:{countryCode}` | JSON | 6 hours | `scripts/seed-resilience-intervals.mjs` | `getResilienceScore` (optional `scoreInterval` field) |
606+
| `resilience:intervals:v3:{countryCode}` | JSON | 7 days; freshness monitored via `seed-meta`/health cadence | `scripts/seed-resilience-scores.mjs` and `scripts/seed-resilience-intervals.mjs` | `getResilienceScore` (optional `scoreInterval` field) and `getResilienceRanking` (`rankStable`) |
607607
| `seed-meta:resilience:static` | JSON | 2 hours | `scripts/seed-resilience-static.mjs` at the end of each successful seed run | scorer for `dataVersion` population, health checks |
608608
| `resilience:static:{countryCode}` | JSON | 400 days | `scripts/seed-resilience-static.mjs` | scorer for all baseline signals (WGI, WHO, FAO, GPI, RSF, and so on) |
609609
| `resilience:static:index:v1` | JSON | 400 days | `scripts/seed-resilience-static.mjs` | warmup path to enumerate countries |
@@ -703,7 +703,7 @@ Self-assessed against the standard composite-indicator review axes on a 0-10 sca
703703
| **Reproducibility** | 8.0 | Every dimension's sourceKey, cadence, and goalpost lives in `_indicator-registry.ts` and is linted against this doc. Cache keys are versioned (`resilience:score:v7`, `ranking:v8`, `history:v4`). `dataVersion` is written by the seed and plumbed to the widget footer. Gap: the benchmark and backtest scripts do not yet run on a CI cron; those land in Phase 2 T2.7. |
704704
| **Source quality** | 7.0 | World Bank, IMF, WHO, IEA, UNHCR, UCDP, IPC, BIS, FAO, RSF, GPI: all authoritative. Gap: curated-list sources (BIS ~40 economies, WTO) do not cover the full WorldMonitor country set, which is why the `unmonitored` imputation class exists. Phase 2 T2.9 adds language-normalized information signal to reduce English-press bias. |
705705
| **Timeliness** | 6.5 | Structural sources are annual (WGI, GPI, RSF, WHO, IMF macro) and dominate the total weight of the index. BIS EER is monthly. The Freshness classifier (T1.5) surfaces this at the dimension level so users can see which parts of a country score are 12 months old. Thirteen stress-side indicators already run at realtime or daily cadence via the cross-source stack (`ucdpConflict`, `internetOutages`, `infraOutages`, `unrestEvents`, `socialVelocity` at realtime; `sanctionCount`, `cyberThreats`, `gpsJamming`, `shippingStress`, `transitDisruption`, `gasStorageStress`, `energyPriceStress`, `newsThreatScore` at daily). Gap: the live-shock pillar relies on those signals but the structural pillar is still capped by annual sources; Phase 2 T2.2 adds FX volatility at daily cadence to narrow the cadence gap on the currency-external dimension and the Phase 3 reference-edition split will formalize annual vs rolling cadences per pillar. |
706-
| **Sensitivity** | 7.0 | Weight-perturbation Monte Carlo sensitivity (#2823) exists in the backtesting layer. Phase 1 did not add new sensitivity work. Overall p5/p95 score intervals are computed and exposed (#2877, #2885), and the widget renders the overall `[p05–p95]` range next to the score. Gap: no waterfall chart of individual signal contributions yet; that lands in Phase 3 T3.3. |
706+
| **Sensitivity** | 7.0 | Weight-perturbation Monte Carlo sensitivity (#2823) exists in the backtesting layer. Phase 1 did not add new sensitivity work. Overall p5/p95 score sensitivity bands are computed under the active score formula and exposed (#2877, #2885, #3967), and the widget renders the overall `[p05–p95]` range next to the score. The band is a formula-aware weight-perturbation sensitivity range, not an input-data uncertainty interval. Gap: no waterfall chart of individual signal contributions yet; that lands in Phase 3 T3.3. |
707707

708708
**Phase 1 acceptance gate status: met.** Both required thresholds (Methodology ≥7.5, Explainability ≥7.5) are satisfied with honest rationales. The two gaps flagged in each axis are tracked against Phase 2 and Phase 3 tasks in the upgrade plan.
709709

scripts/_resilience-intervals.mjs

Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
export const RESILIENCE_INTERVAL_KEY_PREFIX = 'resilience:intervals:v3:';
2+
export const RESILIENCE_INTERVAL_METHODOLOGY = 'weight-perturbation-sensitivity-v1';
3+
export const DRAWS = 100;
4+
5+
export const DOMAIN_WEIGHTS = {
6+
economic: 0.17,
7+
infrastructure: 0.15,
8+
energy: 0.11,
9+
'social-governance': 0.19,
10+
'health-food': 0.13,
11+
recovery: 0.25,
12+
};
13+
14+
export const DOMAIN_ORDER = [
15+
'economic',
16+
'infrastructure',
17+
'energy',
18+
'social-governance',
19+
'health-food',
20+
'recovery',
21+
];
22+
23+
export const PILLAR_WEIGHTS = {
24+
'structural-readiness': 0.40,
25+
'live-shock-exposure': 0.35,
26+
'recovery-capacity': 0.25,
27+
};
28+
29+
export const PILLAR_ORDER = [
30+
'structural-readiness',
31+
'live-shock-exposure',
32+
'recovery-capacity',
33+
];
34+
35+
export const PENALTY_ALPHA = 0.50;
36+
37+
function round(value, places = 2) {
38+
const factor = 10 ** places;
39+
return Math.round(value * factor) / factor;
40+
}
41+
42+
function roundInterval(value) {
43+
return Math.round(value * 10) / 10;
44+
}
45+
46+
function floorInterval(value) {
47+
return Math.floor(value * 10) / 10;
48+
}
49+
50+
function ceilInterval(value) {
51+
return Math.ceil(value * 10) / 10;
52+
}
53+
54+
export function createIntervalDiagnostics() {
55+
return {
56+
activeScoreClampCount: 0,
57+
activeScoreClampMaxDelta: 0,
58+
activeScoreClampSamples: [],
59+
};
60+
}
61+
62+
function normalizeFormula(value) {
63+
return value === 'pc' || value === 'd6' ? value : null;
64+
}
65+
66+
export function currentEnvFormula() {
67+
const pillarCombine = (process.env.RESILIENCE_PILLAR_COMBINE_ENABLED ?? 'false').toLowerCase() === 'true';
68+
const schemaV2 = (process.env.RESILIENCE_SCHEMA_V2_ENABLED ?? 'true').toLowerCase() === 'true';
69+
return pillarCombine && schemaV2 ? 'pc' : 'd6';
70+
}
71+
72+
function recordActiveScoreClamp(options, before, after, activeScore) {
73+
if (before.p05 === after.p05 && before.p95 === after.p95) return;
74+
const diagnostics = options?.diagnostics;
75+
if (!diagnostics || typeof diagnostics !== 'object') return;
76+
77+
const delta = activeScore < before.p05
78+
? before.p05 - activeScore
79+
: activeScore > before.p95
80+
? activeScore - before.p95
81+
: 0;
82+
diagnostics.activeScoreClampCount = (Number(diagnostics.activeScoreClampCount) || 0) + 1;
83+
diagnostics.activeScoreClampMaxDelta = Math.max(Number(diagnostics.activeScoreClampMaxDelta) || 0, round(delta, 4));
84+
if (Array.isArray(diagnostics.activeScoreClampSamples) && diagnostics.activeScoreClampSamples.length < 5) {
85+
diagnostics.activeScoreClampSamples.push({
86+
countryCode: typeof options.countryCode === 'string' ? options.countryCode : undefined,
87+
formula: normalizeFormula(options.formula) ?? undefined,
88+
activeScore: round(activeScore, 4),
89+
before,
90+
after,
91+
delta: round(delta, 4),
92+
});
93+
}
94+
}
95+
96+
function clampToActiveScore(interval, activeScore, options = {}) {
97+
if (!Number.isFinite(activeScore)) return interval;
98+
const before = { ...interval };
99+
let { p05, p95 } = before;
100+
if (activeScore < p05) p05 = floorInterval(activeScore);
101+
if (activeScore > p95) p95 = ceilInterval(activeScore);
102+
const after = { p05, p95 };
103+
recordActiveScoreClamp(options, before, after, activeScore);
104+
return after;
105+
}
106+
107+
function percentile(samples, quantile) {
108+
if (samples.length === 0) return 0;
109+
const index = Math.min(samples.length - 1, Math.max(0, Math.ceil(samples.length * quantile) - 1));
110+
return samples[index];
111+
}
112+
113+
function jitterWeights(weights, rng) {
114+
const jittered = weights.map((w) => w * (0.9 + rng() * 0.2));
115+
const sum = jittered.reduce((total, value) => total + value, 0);
116+
if (!Number.isFinite(sum) || sum <= 0) return weights;
117+
return jittered.map((w) => w / sum);
118+
}
119+
120+
export function computeIntervals(domainScores, domainWeights, draws = DRAWS, options = {}) {
121+
const rng = options.rng ?? Math.random;
122+
const activeScore = Number(options.activeScore);
123+
const samples = [];
124+
const count = Math.max(1, Math.floor(Number(draws) || DRAWS));
125+
for (let i = 0; i < count; i++) {
126+
const normalized = jitterWeights(domainWeights, rng);
127+
const score = domainScores.reduce((sum, value, index) => sum + value * normalized[index], 0);
128+
samples.push(score);
129+
}
130+
samples.sort((a, b) => a - b);
131+
return clampToActiveScore({
132+
p05: roundInterval(percentile(samples, 0.05)),
133+
p95: roundInterval(percentile(samples, 0.95)),
134+
}, activeScore, options);
135+
}
136+
137+
export function penalizedPillarScore(pillars) {
138+
if (!pillars.length) return 0;
139+
const weighted = pillars.reduce((sum, p) => sum + p.score * p.weight, 0);
140+
const minScore = Math.min(...pillars.map((p) => p.score));
141+
const penalty = 1 - PENALTY_ALPHA * (1 - minScore / 100);
142+
return round(weighted * penalty);
143+
}
144+
145+
export function computePillarIntervals(pillars, draws = DRAWS, options = {}) {
146+
const rng = options.rng ?? Math.random;
147+
const activeScore = Number(options.activeScore);
148+
const scores = pillars.map((pillar) => Number(pillar.score));
149+
const weights = pillars.map((pillar) => Number(pillar.weight));
150+
const samples = [];
151+
const count = Math.max(1, Math.floor(Number(draws) || DRAWS));
152+
for (let i = 0; i < count; i++) {
153+
const normalized = jitterWeights(weights, rng);
154+
samples.push(penalizedPillarScore(scores.map((score, index) => ({ score, weight: normalized[index] }))));
155+
}
156+
samples.sort((a, b) => a - b);
157+
return clampToActiveScore({
158+
p05: roundInterval(percentile(samples, 0.05)),
159+
p95: roundInterval(percentile(samples, 0.95)),
160+
}, activeScore, options);
161+
}
162+
163+
function extractDomains(scoreData) {
164+
const domains = Array.isArray(scoreData?.domains) ? scoreData.domains : [];
165+
return DOMAIN_ORDER.map((id) => {
166+
const domain = domains.find((entry) => entry?.id === id);
167+
const score = Number(domain?.score);
168+
const weight = Number(domain?.weight ?? DOMAIN_WEIGHTS[id]);
169+
if (!Number.isFinite(score) || !Number.isFinite(weight)) return null;
170+
return { id, score, weight };
171+
}).filter(Boolean);
172+
}
173+
174+
function extractPillars(scoreData) {
175+
const pillars = Array.isArray(scoreData?.pillars) ? scoreData.pillars : [];
176+
return PILLAR_ORDER.map((id) => {
177+
const pillar = pillars.find((entry) => entry?.id === id);
178+
const score = Number(pillar?.score);
179+
const weight = Number(pillar?.weight ?? PILLAR_WEIGHTS[id]);
180+
if (!Number.isFinite(score) || !Number.isFinite(weight)) return null;
181+
return { id, score, weight };
182+
}).filter(Boolean);
183+
}
184+
185+
export function domainAggregate(domains) {
186+
if (!domains.length) return null;
187+
return round(domains.reduce((sum, domain) => sum + domain.score * domain.weight, 0));
188+
}
189+
190+
export function inferScoreFormula(scoreData, options = {}) {
191+
const cached = normalizeFormula(scoreData?._formula);
192+
if (cached) return cached;
193+
194+
const fallback = normalizeFormula(options.fallbackFormula) ?? currentEnvFormula();
195+
const overallScore = Number(scoreData?.overallScore);
196+
if (!Number.isFinite(overallScore)) return fallback;
197+
198+
const domains = options.domains ?? extractDomains(scoreData);
199+
const pillars = options.pillars ?? extractPillars(scoreData);
200+
const d6Score = domainAggregate(domains);
201+
const pcScore = pillars.length > 0 ? penalizedPillarScore(pillars) : null;
202+
const d6Diff = d6Score == null ? Number.POSITIVE_INFINITY : Math.abs(overallScore - d6Score);
203+
const pcDiff = pcScore == null ? Number.POSITIVE_INFINITY : Math.abs(overallScore - pcScore);
204+
const tolerance = Number(options.tolerance ?? 0.2);
205+
206+
if (pcDiff <= tolerance && d6Diff > tolerance) return 'pc';
207+
if (d6Diff <= tolerance && pcDiff > tolerance) return 'd6';
208+
if (Number.isFinite(pcDiff) || Number.isFinite(d6Diff)) {
209+
if (pcDiff + 0.05 < d6Diff) return 'pc';
210+
if (d6Diff + 0.05 < pcDiff) return 'd6';
211+
}
212+
// Ambiguous d6 ~= pc ties intentionally fall back to the active seed env.
213+
// Railway seeders must keep RESILIENCE_PILLAR_COMBINE_ENABLED and
214+
// RESILIENCE_SCHEMA_V2_ENABLED aligned with the server rollout flags;
215+
// otherwise an untagged score payload that fits both formulas closely
216+
// can only be resolved by those env flags.
217+
return fallback;
218+
}
219+
220+
export function buildScoreIntervalPayload(scoreData, options = {}) {
221+
const draws = Math.max(1, Math.floor(Number(options.draws) || DRAWS));
222+
const overallScore = Number(scoreData?.overallScore);
223+
if (!Number.isFinite(overallScore)) return null;
224+
225+
const domains = extractDomains(scoreData);
226+
const pillars = extractPillars(scoreData);
227+
const formula = inferScoreFormula(scoreData, { ...options, domains, pillars });
228+
229+
const interval = formula === 'pc'
230+
? (pillars.length > 0
231+
? computePillarIntervals(pillars, draws, {
232+
rng: options.rng,
233+
activeScore: overallScore,
234+
diagnostics: options.diagnostics,
235+
countryCode: scoreData?.countryCode,
236+
formula,
237+
})
238+
: null)
239+
: (domains.length > 0
240+
? computeIntervals(
241+
domains.map((domain) => domain.score),
242+
domains.map((domain) => domain.weight),
243+
draws,
244+
{
245+
rng: options.rng,
246+
activeScore: overallScore,
247+
diagnostics: options.diagnostics,
248+
countryCode: scoreData?.countryCode,
249+
formula,
250+
},
251+
)
252+
: null);
253+
if (!interval) return null;
254+
255+
return {
256+
p05: interval.p05,
257+
p95: interval.p95,
258+
_formula: formula,
259+
draws,
260+
computedAt: options.computedAt ?? new Date().toISOString(),
261+
methodology: RESILIENCE_INTERVAL_METHODOLOGY,
262+
};
263+
}

0 commit comments

Comments
 (0)