Adopt IMF data alongside World Bank (Economic Data Contract v2.0)#1872
Adopt IMF data alongside World Bank (Economic Data Contract v2.0)#1872
Conversation
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/42cd2b09-4743-4205-8419-1779c10843b2 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/42cd2b09-4743-4205-8419-1779c10843b2 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
…omment Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/42cd2b09-4743-4205-8419-1779c10843b2 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
🏷️ Automatic Labeling SummaryThis PR has been automatically labeled based on the files changed and PR metadata. Applied Labels: documentation,security,workflow,riksdag-data,schema,isms,ci-cd,testing,refactor,size-xl,news,agentic-workflow Label Categories
For more information, see |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
… SBOM, agents Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/67025cd7-cc47-45cc-84a8-593a64cfac43 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/9a9789ea-fd17-4cff-b0e5-bbe93c327192 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
…ale schema description Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/48ad87bd-eaf9-41e8-907f-ecfc8895487a Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
There was a problem hiding this comment.
Pull request overview
This PR migrates IMF data ingestion from an external Python MCP (uvx/stdio) to an in-repo TypeScript client + thin CLI, updating the Economic Data Contract to v2.0 so workflows, validators, persistence, and documentation treat IMF as a first-class provider alongside World Bank and SCB.
Changes:
- Add a pure-TypeScript IMF client + CLI (
scripts/imf-client.ts,scripts/imf-fetch.ts) and persist IMF responses underanalysis/data/imf/…with sidecar provenance. - Extend the economic-context loader/validator and statistical-claims mapping to support IMF provider/projection metadata and attribution.
- Remove IMF MCP/PyPI runtime expectations from workflows/setup and update docs, schema, ADR, and threat models accordingly.
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/validate-economic-context.test.ts | Updates validator expectations for IMF-inclusive attribution messaging. |
| tests/statistical-claims-detector.test.ts | Adds coverage ensuring IMF indicator codes propagate into detected claims. |
| tests/imf-context.test.ts | New tests for IMF policy/committee indicator mapping utilities. |
| tests/imf-codes.test.ts | New tests for ISO3↔IMF AREA mappings and related helpers. |
| tests/imf-client.test.ts | New tests for IMF client parsing, retry/backoff behavior, projections, and SDMX passthrough. |
| tests/economic-context-multi-provider.test.ts | New tests validating v1/v2 economic-data loader behavior with mixed providers. |
| tests/data-persistence.test.ts | Adds tests for persistIMFData() persistence layout + metadata sidecar. |
| scripts/validate-economic-context.ts | Extends attribution + validation messaging to include IMF and structured attribution logic. |
| scripts/statistical-claims-detector.ts | Adds optional imfIndicator field to claim patterns and outputs. |
| scripts/parliamentary-data/data-persistence.ts | Adds persistIMFData() and registers imf persistence type. |
| scripts/imf-fetch.ts | New tsx CLI to fetch/persist IMF WEO/compare/SDMX data for workflows. |
| scripts/imf-context.ts | New IMF indicator catalogue + lookup helpers for domains/committees/citations. |
| scripts/imf-codes.ts | New canonical code mappings for IMF datamapper vs SDMX area codes. |
| scripts/imf-client.ts | New REST client for IMF Datamapper JSON + SDMX 3.0 with retries/timeouts. |
| scripts/data-transformers/load-economic-context.ts | Extends loader to accept v2 provider/projection metadata and source.imf[] (with v1 back-compat). |
| package.json | Empties/notes x-external-mcp to reflect TS-only IMF integration under npm/SBOM governance. |
| docs/adr/0001-adopt-imf-data-alongside-world-bank.md | Adds ADR documenting the TS-only IMF approach and its consequences. |
| analysis/schemas/economic-data.schema.json | Updates schema descriptions and adds optional provider/projection metadata + source.imf. |
| analysis/imf/use-cases.md | New IMF integration use-cases documenting CLI usage and example artefacts. |
| analysis/imf/indicator-policy-mapping.md | New mapping document aligning committees/policy areas with IMF indicators. |
| analysis/imf/README.md | New IMF integration overview (why/strategy/surface/rate-limit discipline). |
| analysis/economic-indicators-inventory.json | New multi-provider inventory v4.0 including IMF entries and provider-selection rules. |
| THREAT_MODEL.md | Adds TB-6a threat model entry for direct IMF HTTP access via TS client (no MCP). |
| SECURITY_ARCHITECTURE.md | Notes x-external-mcp tracking and that IMF is now in npm/SBOM scope. |
| README.md | Updates data-source list to include IMF and clarifies narrowed WB role. |
| FUTURE_THREAT_MODEL.md | Adds future-threat row for IMF integration and reduces risk rating. |
| .github/workflows/news-weekly-review.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-week-ahead.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-translate.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-realtime-monitor.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-propositions.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-motions.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-monthly-review.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-month-ahead.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-interpellations.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-evening-analysis.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-committee-reports.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/news-article-generator.md | Adds IMF origins to network/safe-outputs allowlists after MCP removal. |
| .github/workflows/economic-context-audit.yml | Documents the v1→v2 grace window for attribution/schema evolution. |
| .github/workflows/copilot-setup-steps.yml | Removes Python runtime setup and documents IMF as TS client invoked via tsx. |
| .github/aw/SHARED_PROMPT_PATTERNS.md | Updates economic indicator discovery/fetch guidance to use TS CLI for IMF. |
| .github/aw/ECONOMIC_DATA_CONTRACT.md | Updates contract to v2.0: IMF-first macro/fiscal, provider/projection rules, grace window. |
| .github/agents/news-journalist.md | Updates agent guidance to include IMF projections where relevant. |
| .github/agents/intelligence-operative.md | Updates tooling matrix to describe IMF access via bash/tsx instead of MCP. |
| .github/agents/data-pipeline-specialist.md | Updates pipeline guidance to include IMF TS client/CLI, caching, and rate-limit rules. |
| export function hasAttribution(html: string): boolean { | ||
| return /World Bank|världsbanken|verdensbank|weltbank|banco mundial|banque mondiale|wereldbank|البنك الدولي|הבנק העולמי|世界銀行|세계은행|世界银行|SCB|Statistics Sweden|Statistiska centralbyrån/i.test(html); | ||
| return /IMF|International Monetary Fund|Internationella valutafonden|Internationaler Währungsfonds|Fondo Monetario Internacional|Fonds monétaire international|Internationaal Monetair Fonds|صندوق النقد الدولي|קרן המטבע הבינלאומית|国際通貨基金|국제통화기금|国际货币基金组织|World Bank|världsbanken|verdensbank|weltbank|banco mundial|banque mondiale|wereldbank|البنك الدولي|הבנק העולמי|世界銀行|세계은행|世界银行|SCB|Statistics Sweden|Statistiska centralbyrån/i.test(html); |
| if (response.status === 429 && attempt < this.maxRetries) { | ||
| // Respect IMF advertised rate limit (~10 req / 5 s). | ||
| const delay = 1_000 * (attempt + 1) + 1_000; | ||
| clearTimeout(timeoutId); | ||
| await new Promise((resolve) => setTimeout(resolve, delay)); | ||
| return this.fetchWithRetry(url, attempt + 1, extraHeaders); | ||
| } |
| if (booleans.has('persist')) { | ||
| // Derive a reasonable cache key from --indicator / --country when given; | ||
| // otherwise fall back to hashing the SDMX path so caches collide cleanly | ||
| // on re-runs with identical queries. | ||
| const indicator = flags.get('indicator') ?? pathWithQuery.split('/').slice(-2)[0] ?? 'sdmx'; | ||
| const country = flags.get('country') ?? 'all'; |
| import { ImfClient, IMF_WEO_INDICATORS, IMF_FM_INDICATORS } from './imf-client.ts'; | ||
| import { persistIMFData } from './parliamentary-data/data-persistence.ts'; |
| * The loader bridges the data produced by workflow agents (which call | ||
| * World Bank / SCB MCP tools) and the HTML renderer in | ||
| * World Bank / IMF / SCB MCP tools) and the HTML renderer in | ||
| * `content-generators/economic-dashboard-section.ts`. When the JSON file |
| export interface EconomicContextSource { | ||
| /** World Bank indicator IDs actually queried (e.g. `NY.GDP.MKTP.KD.ZG`). */ | ||
| worldBank: string[]; | ||
| /** SCB table IDs actually queried (e.g. `TAB1291`). */ | ||
| scb: string[]; | ||
| /** | ||
| * IMF citation strings actually queried (e.g. `WEO:NGDP_RPCH`, | ||
| * `FM:GGXWDG_NGDP`). Present when schema version ≥ 2.0. Always | ||
| * populated as an array by the loader — may be empty on v1 files. | ||
| */ | ||
| imf: string[]; | ||
| } |
|
@copilot apply changes based on the comments in this thread and fix TypeScript & JavaScript Testing / Unit Tests (Vitest) (pull_request) |
Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/6c23f335-6d79-49c8-a414-6180b4941768 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Addressed all six review comments and the failing
Targeted tests (imf-client, data-persistence, imf-context, imf-codes, validate-economic-context, economic-context-multi-provider): 133 / 133 pass. Both |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
|
@copilot TypeScript & JavaScript Testing / Unit Tests (Vitest) (pull_request) |
…ons manifest Agent-Logs-Url: https://github.com/Hack23/riksdagsmonitor/sessions/cd31fd04-e81b-4ac4-bc87-1b538d7b9743 Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>
Fixed in ba56841. The failing test was |
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR migrates IMF data access from a third-party Python MCP/uvx runtime to an in-repo, pure-TypeScript client + thin CLI, expanding the economic-data contract to v2.0 (IMF + provider/projection metadata) and updating validators, persistence, workflows, and documentation accordingly.
Changes:
- Add TypeScript-based IMF client (
scripts/imf-client.ts), supporting Datamapper (WEO) plus SDMX 3.0 passthrough, and expose it viascripts/imf-fetch.ts. - Extend economic context loading/validation/persistence to support
source.imf[]and per-pointprovider/projection/projectionVintage. - Remove Python/
uvxsetup and MCP references; update workflows allowlists, docs, ADR, schema, threat model, and tests.
Reviewed changes
Copilot reviewed 46 out of 46 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/validate-economic-context.test.ts | Updates expectations for IMF-inclusive attribution messaging. |
| tests/statistical-claims-detector.test.ts | Adds assertions for IMF indicator propagation in detected claims. |
| tests/imf-context.test.ts | New tests for IMF policy/committee mapping helpers. |
| tests/imf-codes.test.ts | New tests for IMF ISO3↔AREA mappings and helpers. |
| tests/imf-client.test.ts | New tests for IMF client parsing, retry/backoff, and SDMX passthrough. |
| tests/economic-context-multi-provider.test.ts | New tests validating schema v1/v2 back-compat and mixed providers. |
| tests/data-persistence.test.ts | Adds coverage for persistIMFData pathing and sidecar metadata. |
| scripts/validate-economic-context.ts | Accepts IMF attribution and IMF sources in economic-data validation. |
| scripts/statistical-claims-detector.ts | Adds optional imfIndicator metadata to detected claims. |
| scripts/parliamentary-data/data-persistence.ts | Adds persistIMFData and updates persistence document types. |
| scripts/imf-fetch.ts | New CLI wrapper for agentic workflows (weo/compare/sdmx/list-indicators). |
| scripts/imf-context.ts | New IMF indicator catalogue + lookups for domains/committees/peers. |
| scripts/imf-codes.ts | New canonical IMF code mapping utilities for datasets/country codes. |
| scripts/imf-client.ts | New TypeScript REST client for IMF Datamapper + SDMX with retries. |
| scripts/data-transformers/load-economic-context.ts | Loads schema v2 fields and normalizes v1 artefacts to v2 defaults. |
| package.json | Records empty external MCP manifest (x-external-mcp) post-migration. |
| docs/adr/0001-adopt-imf-data-alongside-world-bank.md | ADR documenting TS-only IMF approach and consequences. |
| analysis/schemas/economic-data.schema.json | Schema v2: adds source.imf[] and per-point provider/projection fields. |
| analysis/imf/use-cases.md | Documents IMF CLI usage patterns and example artefacts. |
| analysis/imf/indicator-policy-mapping.md | Defines committee→IMF indicator mapping reference. |
| analysis/imf/README.md | Explains IMF integration rationale, code surface, and rate limits. |
| analysis/economic-indicators-inventory.json | New multi-provider inventory (IMF inline + WB reference + SCB). |
| analysis/daily/2026-04-20/propositions/data-download-manifest.md | Rewords placeholder self-audit line to satisfy placeholder detector. |
| THREAT_MODEL.md | Adds TB-6a threat boundary for IMF TS client (no MCP). |
| SECURITY_ARCHITECTURE.md | Notes external MCP manifest is empty; IMF is npm/SBOM covered. |
| README.md | Adds IMF as an integrated data source and clarifies WB’s narrowed role. |
| FUTURE_THREAT_MODEL.md | Lowers IMF integration risk profile (TypeScript client, no new runtime). |
| .github/workflows/news-weekly-review.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-week-ahead.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-translate.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-realtime-monitor.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-propositions.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-motions.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-monthly-review.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-month-ahead.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-interpellations.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-evening-analysis.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-committee-reports.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/news-article-generator.md | Adds IMF origins to workflow allowlists. |
| .github/workflows/economic-context-audit.yml | Documents v2 cutover grace window and mixed v1/v2 acceptance. |
| .github/workflows/copilot-setup-steps.yml | Removes Python setup; documents IMF as TS client invoked via tsx. |
| .github/aw/SHARED_PROMPT_PATTERNS.md | Updates discovery protocol and IMF fetching idioms for agents. |
| .github/aw/ECONOMIC_DATA_CONTRACT.md | Contract v2.0: IMF-first macro/fiscal + projection metadata rules. |
| .github/agents/news-journalist.md | Updates agent guidance to incorporate IMF projections where relevant. |
| .github/agents/intelligence-operative.md | Documents IMF usage via bash+tsx CLI (no MCP). |
| .github/agents/data-pipeline-specialist.md | Documents IMF client/CLI usage, caching, and rate-limit discipline. |
Comments suppressed due to low confidence (5)
scripts/imf-client.ts:1
- The
catchpath retries for all thrown errors, including non-retriable HTTP errors (e.g. 400/401/403/404) because!response.okthrows and is then retried here. This can cause unnecessary repeated requests and slower failures. Consider only retrying in thecatchfor network/abort errors, and for HTTP responses only retrying on 5xx (and 429, which you already handle explicitly), while letting other 4xx fail immediately.
scripts/imf-client.ts:1 - In the 429 path you
clearTimeout(timeoutId)before waiting, but in the genericcatchretry path the timeout isn’t cleared until the recursive retry returns (because recursion happens insidecatch, delayingfinally). This can keep timers alive longer than necessary and adds avoidable wakeups. Align the generic retry path with the 429 path by clearing the timeout before sleeping/recursing (and/or restructuring to ensurefinallyruns before scheduling retries).
scripts/imf-fetch.ts:1 - The header documents exit code
2for bad CLI arguments, but an unknown command currently falls intodefaultand exits successfully (code 0) after printing help. Treat unknown commands as invalid args (print help + error to stderr andprocess.exit(2)) to avoid workflows silently succeeding on typos.
scripts/imf-fetch.ts:1 comparecurrently triggers one request per country (viacompareCountriesWeo→getLatestWeoIndicator→getWeoIndicator). Several docs/comments in this PR describecompareas “one batched Datamapper call across several countries”; that’s not what the implementation does, which matters for IMF rate limits and workflow latency. Either (mandatory) update the documentation/comments to reflect the N-request behavior, or (optional, if the API supports it) implement a true batched fetch; additionally consider inserting a small per-country delay in the loop to better respect the published rate limit whencountriesis large.
scripts/data-transformers/load-economic-context.ts:1projectionVintageis added whenever present, even ifprojectionis false. This contradicts the local type doc (“Present only when projection=true”) and the contract language, and can confuse downstream consumers. Consider only copyingprojectionVintagewhenenriched.projection === true(and optionally dropping/ignoring it otherwise).
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: James Pether Sörling <pethers@users.noreply.github.com>
🔍 Lighthouse Performance Audit
📥 Download full Lighthouse report Budget Compliance: Performance budgets enforced via |
Swap the third-party
c-cf/imf-data-mcpPython MCP (stdio,uvx) for the repo's pure-TypeScriptscripts/imf-client.ts, exposed to agentic workflows through a new thin CLI (scripts/imf-fetch.ts) invoked via thebashtool. Eliminates Python /uvxfrom the agentic runtime, dropspypi.org+files.pythonhosted.orgfrom every workflow firewall allowlist, and brings IMF under the same npm / SBOM governance as World Bank and SCB.Code
scripts/imf-fetch.ts— thin CLI overImfClient+persistIMFData. Commands:weo,compare,sdmx,list-indicators,help.scripts/parliamentary-data/data-persistence.ts—persistIMFData()provenance tag flippedimf-data-mcp→imf-ts-client.scripts/imf-client.ts+scripts/imf-context.ts— JSDoc pruned of references to the retired MCP.Configs / workflows (pure removal)
.github/copilot-mcp.json—imfMCP server entry removed..github/workflows/copilot-setup-steps.yml—python3/pipx/uvinstall +uvxpre-warm + verification block dropped..github/workflows/news-*.mdfrontmatters —imf:MCP block removed;pypi.org+files.pythonhosted.orgremoved from bothnetwork:andsafe-outputs:allowlists.Docs / governance
ECONOMIC_DATA_CONTRACT.mdv2.0 +SHARED_PROMPT_PATTERNS.md— MCP 4-call discovery flow replaced bytsx scripts/imf-fetch.ts …idiom.0001— Decision + Consequences rewritten around the TS-only stack;c-cf/imf-data-mcpmoved to "Alternatives rejected".THREAT_MODEL.mdTB-6a — PyPI / upstream-repo /uvxvectors retired.FUTURE_THREAT_MODEL.mdIMF row MEDIUM → LOW.SECURITY_ARCHITECTURE.md+package.json—x-external-mcpmanifest emptied.analysis/schemas/economic-data.schema.json—source.imfdescription updated.analysis/imf/{README,use-cases}.md,analysis/economic-indicators-inventory.json, agents — all narrate IMF as a TS client + CLI, not an MCP.PR review feedback addressed
validate-economic-context.ts) — added\bIMF\b/\bSCB\bword boundaries tohasAttribution().imf-client.ts) — 429 back-off rewritten to documented exponential 1 s → 2 s → 4 s, honoursRetry-After(capped at 30 s).imf-fetch.ts) — SDMX persist-key comment rewritten to match the path-segment heuristic.imf-fetch.ts) — local module imports switched to.jsspecifiers. Fixed theTS5097CI error on theTypeScript type-check (scripts)step.load-economic-context.ts) — module header no longer claims IMF is an MCP tool.load-economic-context.ts) — file vs loaded types split (EconomicContextSourceFilewith optionalimf;EconomicContextSourcewith requiredimf).CI failures fixed
TypeScript type-check (scripts)— fixed by comment Fix MCP server paths and add rule-based Copilot instructions #4 (.jsimport specifiers).Unit Tests (Vitest)—tests/analysis-quality-validation.test.tswas failing becauseanalysis/daily/2026-04-20/propositions/data-download-manifest.mdline 99 contained the literal string"[PLACEHOLDER]"in a self-audit bullet ("✅ 0[PLACEHOLDER]patterns"), which the test's\[PLACEHOLDER\]detector regex correctly flagged as an unfilled placeholder. Rephrased the bullet to✅ 0 bracketed "PLACEHOLDER" patterns.— preserves the audit claim without the literal token.tests/analysis-quality-validation.test.tsnow passes: 35 / 35.Validation
npx tsc --project tsconfig.scripts.json --noEmit— clean.npx tsc --project tsconfig.lib.json --noEmit— clean.Net result
IMF access is now exactly parallel to World Bank and SCB: a plain npm TypeScript module. No Python, no
uvx, no third-party MCP. Firewall egress shrinks to the three IMF origins (data.imf.org,api.imf.org,www.imf.org). Supply-chain surface drops accordingly — TB-6a is now LOW rather than MEDIUM.