diff --git a/.github/skills/data-science-for-intelligence/SKILL.md b/.github/skills/data-science-for-intelligence/SKILL.md index 12315c626..7dbeb1f13 100644 --- a/.github/skills/data-science-for-intelligence/SKILL.md +++ b/.github/skills/data-science-for-intelligence/SKILL.md @@ -25,11 +25,13 @@ Apply this skill when: - ✅ Constructing influence networks from voting alignment patterns - ✅ Forecasting trends in party support or politician effectiveness - ✅ Clustering politicians or parties by voting similarity +- ✅ Modelling Swedish economic transmission chains using IMF macro context plus SCB KPI/AKU/HEK/fuel-price series and Riksbank policy-rate/minutes signals Do NOT use for: - ❌ Simple aggregation queries (use SQL views instead) - ❌ Real-time operational dashboards (use materialized views) - ❌ Causal inference without proper experimental design or quasi-experimental methods +- ❌ Replacing IMF cross-country macro/fiscal indicators with SCB or Riksbank series; use SCB/Riksbank only as Swedish ground-truth complements ## Data Science Framework for CIA Platform diff --git a/.github/skills/riksdag-regering-mcp/SKILL.md b/.github/skills/riksdag-regering-mcp/SKILL.md index f9053e719..cdf6cfe3a 100644 --- a/.github/skills/riksdag-regering-mcp/SKILL.md +++ b/.github/skills/riksdag-regering-mcp/SKILL.md @@ -23,6 +23,7 @@ Provide comprehensive access to Swedish political data through the `riksdag-rege 4. **Real-Time Access**: Latest parliamentary and government activities 5. **GDPR Compliance**: Public interest basis (Article 6(1)(e)), no personal data beyond official capacity 6. **Multi-Source Integration**: Riksdag documents + Government publications +7. **Quantitative Context Handoff**: Pair parliamentary evidence with SCB/Riksbank helper outputs when domestic economic transmission matters; keep IMF as cross-country macro canon and SCB/Riksbank as Swedish ground truth. ## Available Tools (32 Total) @@ -112,6 +113,7 @@ Provide comprehensive access to Swedish political data through the `riksdag-rege - **Electoral Research**: Historical trends, party evolution - **Transparency Dashboards**: Real-time political metrics - **Risk Assessment**: Identify democratic accountability gaps +- **Economic Transmission Analysis**: Combine Riksdag measures with `scripts/scb-fetch.ts` KPI/AKU/HEK/fuel presets and `scripts/riksbank-fetch.ts` policy-rate/minutes context for Swedish-specific impact estimates. ## Calling MCP Tools in Agentic Workflows diff --git a/analysis/templates/comparative-international.md b/analysis/templates/comparative-international.md index a67c36a06..9f0addaae 100644 --- a/analysis/templates/comparative-international.md +++ b/analysis/templates/comparative-international.md @@ -159,6 +159,21 @@ graph TB --- +## 🇸🇪 Swedish Ground-Truth Transmission Layer (SCB + Riksbank) + +> Use this table whenever the Swedish measure depends on domestic price, labour-market, household-income or monetary-policy transmission. IMF remains primary for cross-country macro; SCB and Riksbank supply the Swedish-specific ground truth. Fetch with `tsx scripts/scb-fetch.ts preset --preset --persist` and `tsx scripts/riksbank-fetch.ts --persist`. + +| Transmission step | Swedish source | Required provenance | Current value | Timing / lag | Analytical use | +|-------------------|----------------|---------------------|:-------------:|--------------|----------------| +| Pump price / energy component → CPI | SCB `fuel-prices` (`PR0101A`) + SCB `cpi` (`0000003N`) | `economicProvenance.provider="scb"` | — | Monthly | Estimate direct CPI and disposable-income effect | +| Labour-market slack | SCB `aku` (`000003V8`) | `economicProvenance.provider="scb"` | — | Quarterly | Judge whether Swedish unemployment divergence is structural or cyclical | +| Household disposable-income exposure | SCB `household-economy` (`HE0110A`) | `economicProvenance.provider="scb"` | — | Annual / latest | Distributional incidence and voter-segment exposure | +| Monetary-policy pass-through | Riksbank `repo-rate-path` + `minutes` | `economicProvenance.provider="riksbank"` | — | Per monetary-policy decision | Check whether fiscal relief conflicts with the Riksbank inflation path | + +> **Interpretation:** State the Swedish mechanism explicitly: instrument → SCB/Riksbank metric → household / labour / sentiment channel → expected political salience. Never relabel SCB or Riksbank data as IMF; use IMF only for cross-country macro/fiscal comparators. + +--- + ## 📘 Policy-Transfer Assessment | Question | Answer | Evidence | @@ -216,4 +231,3 @@ graph TB - [ ] **Cross-references resolve** — every `[link](file.md)` in this artifact points to a file that exists in the run folder (`analysis/daily/$ARTICLE_DATE/$SUBFOLDER/`) or to a methodology / template under `analysis/`. - [ ] **Mermaid renders** — every fenced ` ```mermaid ` block parses (no missing class definitions, no orphan nodes, no >40-node graphs that overflow viewport on mobile). - [ ] **Line-floor check** — artifact length ≥ the per-artifact floor in [`reference-quality-thresholds.json`](../methodologies/reference-quality-thresholds.json); shorter artifacts trigger Pass-2 rewrite, never a `[truncated]` note. - diff --git a/analysis/templates/intelligence-assessment.md b/analysis/templates/intelligence-assessment.md index 29ac8c9cb..1f25a2671 100644 --- a/analysis/templates/intelligence-assessment.md +++ b/analysis/templates/intelligence-assessment.md @@ -156,6 +156,21 @@ flowchart TD --- +## 📊 Quantitative Transmission Check (SCB + Riksbank) + +> Complete this block for assessments where economic transmission affects intent, timing or voter impact. It is a Swedish ground-truth layer and complements — but does not replace — IMF macro/fiscal context. + +| Mechanism | Required source | economicProvenance provider | Latest signal | Intelligence implication | +|-----------|-----------------|-----------------------------|:-------------:|--------------------------| +| Price shock / relief channel | `tsx scripts/scb-fetch.ts preset --preset cpi` + `tsx scripts/scb-fetch.ts preset --preset fuel-prices` | `scb` | — | Does the measure materially affect household cost pressure? | +| Labour-market sensitivity | `tsx scripts/scb-fetch.ts preset --preset aku` | `scb` | — | Does unemployment amplify or mute the political signal? | +| Household exposure | `tsx scripts/scb-fetch.ts preset --preset household-economy` | `scb` | — | Which voter segments face the strongest disposable-income effect? | +| Monetary-policy reaction function | `tsx scripts/riksbank-fetch.ts repo-rate-path` + `tsx scripts/riksbank-fetch.ts minutes` | `riksbank` | — | Does the Riksbank path support or contradict the claimed transmission? | + +**Assessment note:** Cite concrete SCB/Riksbank values before making a forecast about disposable income, consumer sentiment or polling impact. If either source is unavailable, label the gap and fall back to cached `analysis/data/scb/` or `analysis/data/riksbank/` artifacts. + +--- + ## 🚩 Red Flags (elevate scrutiny) | Signal | Meaning | Recommended action | @@ -211,4 +226,3 @@ flowchart TD - [ ] **Cross-references resolve** — every `[link](file.md)` in this artifact points to a file that exists in the run folder (`analysis/daily/$ARTICLE_DATE/$SUBFOLDER/`) or to a methodology / template under `analysis/`. - [ ] **Mermaid renders** — every fenced ` ```mermaid ` block parses (no missing class definitions, no orphan nodes, no >40-node graphs that overflow viewport on mobile). - [ ] **Line-floor check** — artifact length ≥ the per-artifact floor in [`reference-quality-thresholds.json`](../methodologies/reference-quality-thresholds.json); shorter artifacts trigger Pass-2 rewrite, never a `[truncated]` note. - diff --git a/scripts/parliamentary-data/data-persistence.ts b/scripts/parliamentary-data/data-persistence.ts index ff39e126f..f8cf18a06 100644 --- a/scripts/parliamentary-data/data-persistence.ts +++ b/scripts/parliamentary-data/data-persistence.ts @@ -84,6 +84,7 @@ export type PersistenceDocumentType = | 'imf' | 'statskontoret' | 'scb' + | 'riksbank' | string; // extensible for generic MCP servers /** Sidecar metadata written alongside data files. */ @@ -617,6 +618,55 @@ export function persistSCBData( return path.join(dir, filename); } +/** + * Persist Riksbank public web/JSON artifacts. + * + * Stored under `analysis/data/riksbank/{kind}.json`. Riksbank data is public + * and unauthenticated; provenance sidecars record the source URL/kind and the + * TypeScript CLI used to retrieve it. + * + * @param kind - Logical artifact kind (e.g. 'repo-rate-path', 'minutes'). + * @param response - Raw or normalized Riksbank payload. + * @param dataRoot - Override for the data root directory (for testing). + * @returns Absolute path to the persisted data file. + */ +export function persistRiksbankData( + kind: string, + response: unknown, + dataRoot: string = DATA_ROOT, +): string { + const dir = path.join(dataRoot, 'riksbank'); + ensureDir(dir); + + const sanitized = sanitizeDokId(kind); + const filename = `${sanitized}.json`; + fs.writeFileSync( + path.join(dir, filename), + JSON.stringify(response, null, 2), + 'utf8', + ); + + const candidateUrl = typeof response === 'object' && response !== null && 'url' in response + ? (response as { url?: unknown }).url + : undefined; + const sourceUrl = typeof candidateUrl === 'string' && candidateUrl.length > 0 + ? candidateUrl + : undefined; + const metaFilename = `${sanitized}.meta.json`; + fs.writeFileSync( + path.join(dir, metaFilename), + JSON.stringify({ + fetchedAt: new Date().toISOString(), + mcpTool: 'riksbank-ts-client', + kind, + ...(sourceUrl ? { url: sourceUrl } : {}), + }, null, 2), + 'utf8', + ); + + return path.join(dir, filename); +} + /** * Return the absolute path to the data repository root. * Useful for callers that need to reference persisted files. diff --git a/scripts/riksbank-fetch.ts b/scripts/riksbank-fetch.ts new file mode 100644 index 000000000..3f24f4472 --- /dev/null +++ b/scripts/riksbank-fetch.ts @@ -0,0 +1,438 @@ +#!/usr/bin/env tsx +/** + * @module scripts/riksbank-fetch + * @description Fetches public Riksbank web/JSON artifacts for Swedish + * monetary-policy and fuel-price transmission context. + */ + +import path from 'node:path'; +import { pathToFileURL } from 'node:url'; + +import { persistRiksbankData } from './parliamentary-data/data-persistence.js'; + +export type RiksbankArtifactKind = 'repo-rate-path' | 'minutes' | 'fuel-price-context'; + +export interface RiksbankEconomicProvenance { + readonly provider: 'riksbank'; + readonly dataflow: 'riksbank-web'; + readonly indicator: RiksbankArtifactKind; + readonly url: string; + readonly retrieved_at: string; +} + +export interface RiksbankFetchPayload { + readonly provider: 'riksbank'; + readonly kind: RiksbankArtifactKind; + readonly url: string; + readonly contentType: string; + readonly retrievedAt: string; + readonly status: 'ok' | 'no-data'; + readonly warning?: string; + readonly title?: string; + readonly json?: unknown; + readonly text?: string; + /** Base64-encoded payload for binary responses (e.g. PDF). Capped at PDF_MAX_BYTES. */ + readonly pdfBase64?: string; + /** Byte length of the binary payload before base64 encoding. */ + readonly pdfBytes?: number; + readonly economicProvenance: RiksbankEconomicProvenance; +} + +interface ParsedArgs { + readonly command: 'repo-rate-path' | 'minutes' | 'fuel-price-context' | 'fetch' | 'help'; + readonly flags: ReadonlyMap; + readonly booleans: ReadonlySet; +} + +const DEFAULT_URLS: Readonly> = Object.freeze({ + 'repo-rate-path': 'https://www.riksbank.se/en-gb/monetary-policy/the-policy-rate/', + minutes: 'https://www.riksbank.se/en-gb/monetary-policy/monetary-policy-minutes/', + 'fuel-price-context': 'https://www.riksbank.se/en-gb/monetary-policy/monetary-policy-reports/', +}); + +const HELP = `tsx scripts/riksbank-fetch.ts [flags] + +Commands: + repo-rate-path Fetch the policy-rate path source page unless --url overrides it + minutes Fetch monetary-policy minutes source page/PDF/JSON + fuel-price-context Fetch monetary-policy report context for fuel/energy assumptions + fetch Fetch a custom --kind and --url pair from www.riksbank.se + help Show this message + +Flags: + --url Riksbank HTTPS URL (host must be www.riksbank.se or riksbank.se) + --kind repo-rate-path | minutes | fuel-price-context (fetch command) + --persist Write output under analysis/data/riksbank/ +`; + +export function parseRiksbankArgs(argv: readonly string[]): ParsedArgs { + const command = (argv[0] ?? 'help') as ParsedArgs['command']; + const validCommands: readonly ParsedArgs['command'][] = [ + 'repo-rate-path', 'minutes', 'fuel-price-context', 'fetch', 'help', + ]; + if (!validCommands.includes(command)) throw new Error(`unknown command ${command}`); + const flags = new Map(); + const booleans = new Set(); + for (let i = 1; i < argv.length; i++) { + const token = argv[i]; + if (!token.startsWith('--')) throw new Error(`unexpected positional argument ${token}`); + const key = token.slice(2); + const next = argv[i + 1]; + if (next !== undefined && !next.startsWith('--')) { + flags.set(key, next); + i++; + } else { + booleans.add(key); + } + } + return { command, flags, booleans }; +} + +export function parseRiksbankKind(value: string): RiksbankArtifactKind { + if (value === 'repo-rate-path' || value === 'minutes' || value === 'fuel-price-context') return value; + throw new Error(`unknown Riksbank artifact kind ${value}`); +} + +export function assertRiksbankFetchTarget(url: string): URL { + let parsed: URL; + try { + parsed = new URL(url); + } catch { + throw new Error(`invalid Riksbank URL: ${url}`); + } + if (parsed.protocol !== 'https:') throw new Error('Riksbank fetch URL must use HTTPS'); + if (parsed.hostname !== 'www.riksbank.se' && parsed.hostname !== 'riksbank.se') { + throw new Error(`Riksbank host ${parsed.hostname} is not in allowlist`); + } + return parsed; +} + +function extractTitle(text: string): string | undefined { + const title = /]*>(.*?)<\/title>/is.exec(text)?.[1] + ?.replace(/\s+/g, ' ') + .trim(); + return title && title.length > 0 ? title : undefined; +} + +const DEFAULT_RIKSBANK_TIMEOUT_MS = 15_000; +const TEXT_MAX_BYTES = 20_000; +/** Hard cap on HTML/text body size before truncation. Prevents pathological + * HTML pages from exhausting memory while still allowing a generous slice. */ +const TEXT_RESPONSE_MAX_BYTES = 2 * 1024 * 1024; +/** Hard cap on PDF size accepted from Riksbank. Matches a generous monetary-policy + * report size (~5 MB) without allowing pathological responses to exhaust memory. */ +const PDF_MAX_BYTES = 5 * 1024 * 1024; +/** Maximum number of redirects to follow manually while re-validating each host. */ +const MAX_REDIRECTS = 3; + +async function safeCancel(target: ReadableStreamDefaultReader | ReadableStream | null | undefined): Promise { + if (!target) return; + try { await target.cancel(); } catch { /* ignore cancel errors */ } +} + +async function safeReleaseLock(reader: ReadableStreamDefaultReader): Promise { + try { reader.releaseLock(); } catch { /* ignore release errors */ } +} + +function buildProvenance(kind: RiksbankArtifactKind, url: string, retrievedAt: string): RiksbankEconomicProvenance { + return { + provider: 'riksbank', + dataflow: 'riksbank-web', + indicator: kind, + url, + retrieved_at: retrievedAt, + }; +} + +function buildOutagePayload( + kind: RiksbankArtifactKind, + url: string, + contentType: string, + warning: string, +): RiksbankFetchPayload { + const retrievedAt = new Date().toISOString(); + return { + provider: 'riksbank', + kind, + url, + contentType, + retrievedAt, + status: 'no-data', + warning, + economicProvenance: buildProvenance(kind, url, retrievedAt), + }; +} + +function parseContentLength(header: string | null): number | undefined { + if (!header) return undefined; + const value = Number.parseInt(header, 10); + return Number.isFinite(value) && value >= 0 ? value : undefined; +} + +/** Read a `ReadableStream` body with a hard byte cap. The returned + * `bytes` holds the data read so far; `exceeded === true` means the cap was + * hit and `bytes` should not be consumed. */ +async function readBodyWithCap( + body: ReadableStream | null, + maxBytes: number, +): Promise<{ exceeded: boolean; bytes: Uint8Array; bytesRead: number }> { + if (!body) return { exceeded: false, bytes: new Uint8Array(0), bytesRead: 0 }; + const reader = body.getReader(); + const chunks: Uint8Array[] = []; + let total = 0; + let exceeded = false; + try { + for (;;) { + const { value, done } = await reader.read(); + if (done) break; + if (!value) continue; + total += value.byteLength; + if (total > maxBytes) { + exceeded = true; + await safeCancel(reader); + break; + } + chunks.push(value); + } + } finally { + await safeReleaseLock(reader); + } + if (exceeded) { + return { exceeded: true, bytes: new Uint8Array(0), bytesRead: total }; + } + const out = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { + out.set(chunk, offset); + offset += chunk.byteLength; + } + return { exceeded: false, bytes: out, bytesRead: total }; +} + +/** Perform a single fetch with manual-redirect handling against the Riksbank + * host allowlist. Returns the final 2xx Response or throws. */ +async function fetchWithManualRedirects( + target: URL, + signal: AbortSignal, +): Promise<{ response: Response; finalUrl: URL }> { + let current = target; + for (let hop = 0; hop <= MAX_REDIRECTS; hop++) { + const response = await fetch(current, { + headers: { Accept: 'application/json, text/html, application/pdf;q=0.8, text/plain;q=0.7' }, + signal, + redirect: 'manual', + }); + if (response.status >= 300 && response.status < 400) { + const location = response.headers.get('location'); + if (!location) { + throw new Error(`Riksbank redirect ${response.status} without Location header`); + } + // Drain redirect body to free the connection. + await safeCancel(response.body); + const next = new URL(location, current); + assertRiksbankFetchTarget(next.toString()); + current = next; + continue; + } + return { response, finalUrl: current }; + } + throw new Error(`Riksbank fetch exceeded ${MAX_REDIRECTS} redirects`); +} + +export async function fetchRiksbankPayload( + kind: RiksbankArtifactKind, + url: string, + options: { timeoutMs?: number } = {}, +): Promise { + const target = assertRiksbankFetchTarget(url); + const timeoutMs = options.timeoutMs ?? DEFAULT_RIKSBANK_TIMEOUT_MS; + const controller = new AbortController(); + const timeoutId = setTimeout(() => controller.abort(), timeoutMs); + + try { + let response: Response; + let finalUrl: URL; + try { + ({ response, finalUrl } = await fetchWithManualRedirects(target, controller.signal)); + } catch (error) { + const detail = error instanceof Error ? error.message : String(error); + return buildOutagePayload( + kind, + target.toString(), + 'application/octet-stream', + `Riksbank fetch failed (${detail}); callers should fall back to cached analysis/data/riksbank/ artifacts.`, + ); + } + + const finalUrlStr = finalUrl.toString(); + + if (!response.ok) { + await safeCancel(response.body); + return buildOutagePayload( + kind, + finalUrlStr, + response.headers.get('content-type') ?? 'application/octet-stream', + `Riksbank fetch returned HTTP ${response.status} ${response.statusText}; callers should fall back to cached analysis/data/riksbank/ artifacts.`, + ); + } + + const contentType = response.headers.get('content-type') ?? 'application/octet-stream'; + const contentLength = parseContentLength(response.headers.get('content-length')); + const retrievedAt = new Date().toISOString(); + + if (contentType.includes('json')) { + // JSON responses are typically small; still cap to TEXT_RESPONSE_MAX_BYTES. + if (contentLength !== undefined && contentLength > TEXT_RESPONSE_MAX_BYTES) { + await safeCancel(response.body); + return buildOutagePayload( + kind, + finalUrlStr, + contentType, + `Riksbank JSON Content-Length ${contentLength} exceeds cap ${TEXT_RESPONSE_MAX_BYTES}; persisted as no-data.`, + ); + } + const capped = await readBodyWithCap(response.body, TEXT_RESPONSE_MAX_BYTES); + if (capped.exceeded) { + return buildOutagePayload( + kind, + finalUrlStr, + contentType, + `Riksbank JSON body exceeded ${TEXT_RESPONSE_MAX_BYTES} bytes (read ${capped.bytesRead}); persisted as no-data.`, + ); + } + const jsonBytes = capped.bytes; + let json: unknown; + try { + json = JSON.parse(new TextDecoder('utf-8').decode(jsonBytes)); + } catch (error) { + const detail = error instanceof Error ? error.message : String(error); + return buildOutagePayload(kind, finalUrlStr, contentType, `Riksbank JSON parse failed (${detail}).`); + } + return { + provider: 'riksbank', + kind, + url: finalUrlStr, + contentType, + retrievedAt, + status: 'ok', + json, + economicProvenance: buildProvenance(kind, finalUrlStr, retrievedAt), + }; + } + + if (contentType.includes('pdf')) { + if (contentLength !== undefined && contentLength > PDF_MAX_BYTES) { + await safeCancel(response.body); + return buildOutagePayload( + kind, + finalUrlStr, + contentType, + `Riksbank PDF Content-Length ${contentLength} exceeds cap ${PDF_MAX_BYTES}; persisted as no-data.`, + ); + } + const capped = await readBodyWithCap(response.body, PDF_MAX_BYTES); + if (capped.exceeded) { + return buildOutagePayload( + kind, + finalUrlStr, + contentType, + `Riksbank PDF body exceeded ${PDF_MAX_BYTES} bytes (read ${capped.bytesRead}); persisted as no-data.`, + ); + } + const pdfBytesRaw = capped.bytes; + const pdfBase64 = Buffer.from(pdfBytesRaw).toString('base64'); + return { + provider: 'riksbank', + kind, + url: finalUrlStr, + contentType, + retrievedAt, + status: 'ok', + pdfBase64, + pdfBytes: pdfBytesRaw.byteLength, + economicProvenance: buildProvenance(kind, finalUrlStr, retrievedAt), + }; + } + + if (contentLength !== undefined && contentLength > TEXT_RESPONSE_MAX_BYTES) { + await safeCancel(response.body); + return buildOutagePayload( + kind, + finalUrlStr, + contentType, + `Riksbank text Content-Length ${contentLength} exceeds cap ${TEXT_RESPONSE_MAX_BYTES}; persisted as no-data.`, + ); + } + const capped = await readBodyWithCap(response.body, TEXT_RESPONSE_MAX_BYTES); + if (capped.exceeded) { + return buildOutagePayload( + kind, + finalUrlStr, + contentType, + `Riksbank text body exceeded ${TEXT_RESPONSE_MAX_BYTES} bytes (read ${capped.bytesRead}); persisted as no-data.`, + ); + } + const textBytes = capped.bytes; + const text = new TextDecoder('utf-8').decode(textBytes); + const title = extractTitle(text); + return { + provider: 'riksbank', + kind, + url: finalUrlStr, + contentType, + retrievedAt, + status: 'ok', + ...(title ? { title } : {}), + text: text.slice(0, TEXT_MAX_BYTES), + economicProvenance: buildProvenance(kind, finalUrlStr, retrievedAt), + }; + } finally { + clearTimeout(timeoutId); + } +} + +async function runKind( + kind: RiksbankArtifactKind, + url: string, + booleans: ReadonlySet, +): Promise { + const payload = await fetchRiksbankPayload(kind, url); + process.stdout.write(`${JSON.stringify(payload, null, 2)}\n`); + if (booleans.has('persist')) persistRiksbankData(kind, payload); +} + +function requireRiksbankFlag( + flags: ReadonlyMap, + name: string, +): string { + const value = flags.get(name)?.trim(); + if (!value) throw new Error(`missing required flag --${name}`); + return value; +} + +async function main(): Promise { + const { command, flags, booleans } = parseRiksbankArgs(process.argv.slice(2)); + if (command === 'help') { + process.stdout.write(HELP); + return; + } + const kind = + command === 'fetch' ? parseRiksbankKind(requireRiksbankFlag(flags, 'kind')) : command; + const urlFlag = flags.get('url')?.trim(); + const url = urlFlag && urlFlag.length > 0 ? urlFlag : DEFAULT_URLS[kind]; + await runKind(kind, url, booleans); +} + +function isDirectExecution(): boolean { + const entry = process.argv[1]; + if (!entry) return false; + return import.meta.url === pathToFileURL(path.resolve(entry)).href; +} + +if (isDirectExecution()) { + main().catch((error: unknown) => { + const message = error instanceof Error ? error.message : String(error); + process.stderr.write(`riksbank-fetch: ${message}\n`); + process.exit(/^(missing|unknown|unexpected|invalid|Riksbank fetch URL)/i.test(message) ? 2 : 1); + }); +} diff --git a/scripts/scb-fetch.ts b/scripts/scb-fetch.ts new file mode 100644 index 000000000..de6251321 --- /dev/null +++ b/scripts/scb-fetch.ts @@ -0,0 +1,239 @@ +#!/usr/bin/env tsx +/** + * @module scripts/scb-fetch + * @description CLI wrapper around the SCB MCP client for Swedish ground-truth + * quantitative layers used alongside the IMF economic canon. + */ + +import path from 'node:path'; +import { pathToFileURL } from 'node:url'; + +import { SCBClient } from './scb-client.js'; +import { persistSCBData } from './parliamentary-data/data-persistence.js'; + +export type SCBPresetKey = 'cpi' | 'aku' | 'household-economy' | 'fuel-prices'; + +export interface SCBPreset { + readonly key: SCBPresetKey; + readonly tableId: string; + readonly label: string; + readonly domain: string; + readonly defaultValueCodes: Readonly>; + readonly notes: string; +} + +export interface SCBEconomicProvenance { + readonly provider: 'scb'; + readonly dataflow: 'SCB PxWeb'; + readonly indicator: string; + readonly tableId: string; + readonly retrieved_at: string; + readonly mcpTool: 'query_table'; +} + +export interface SCBFetchPayload { + readonly provider: 'scb'; + readonly preset?: SCBPresetKey; + readonly tableId: string; + readonly label?: string; + readonly valueCodes: Readonly>; + readonly data: readonly unknown[]; + readonly status: 'ok' | 'no-data'; + readonly warning?: string; + readonly economicProvenance: SCBEconomicProvenance; +} + +interface ParsedArgs { + readonly command: 'list-presets' | 'preset' | 'table' | 'help'; + readonly flags: ReadonlyMap; + readonly booleans: ReadonlySet; +} + +export const SCB_PRESETS: readonly SCBPreset[] = Object.freeze([ + { + key: 'cpi', + tableId: '0000003N', + label: 'Consumer Price Index (KPI)', + domain: 'inflation', + defaultValueCodes: Object.freeze({ Tid: 'top(12)' }), + notes: 'Monthly CPI / KPI layer for disposable-income and cost-of-living transmission analysis.', + }, + { + key: 'aku', + tableId: '000003V8', + label: 'Labour Force Survey (AKU)', + domain: 'labour', + defaultValueCodes: Object.freeze({ Tid: 'top(8)' }), + notes: 'Quarterly labour-market layer for AKU unemployment and employment comparisons.', + }, + { + key: 'household-economy', + tableId: 'HE0110A', + label: 'Household economy (HEK / income distribution)', + domain: 'household economy', + defaultValueCodes: Object.freeze({ Tid: 'top(5)' }), + notes: 'Household income and distribution layer for disposable-income impact estimates.', + }, + { + key: 'fuel-prices', + tableId: 'PR0101A', + label: 'Fuel and energy consumer-price components', + domain: 'prices', + defaultValueCodes: Object.freeze({ Tid: 'top(12)' }), + notes: 'Fuel-price component layer for pump-price to CPI transmission analysis.', + }, +] as const); + +const HELP = `tsx scripts/scb-fetch.ts [flags] + +Commands: + list-presets Print curated KPI / AKU / HEK / fuel-price presets + preset Fetch one curated preset by --preset + table Fetch one SCB table by --table-id + help Show this message + +Flags: + --preset cpi | aku | household-economy | fuel-prices + --table-id SCB table ID for table command + --value-codes PxWeb value_codes JSON, e.g. '{"Tid":"top(10)"}' + --periods Convenience fallback for Tid=top(N) + --persist Write output under analysis/data/scb/ +`; + +export function parseSCBArgs(argv: readonly string[]): ParsedArgs { + const command = (argv[0] ?? 'help') as ParsedArgs['command']; + const validCommands: readonly ParsedArgs['command'][] = ['list-presets', 'preset', 'table', 'help']; + if (!validCommands.includes(command)) throw new Error(`unknown command ${command}`); + + const flags = new Map(); + const booleans = new Set(); + for (let i = 1; i < argv.length; i++) { + const token = argv[i]; + if (!token.startsWith('--')) throw new Error(`unexpected positional argument ${token}`); + const key = token.slice(2); + const next = argv[i + 1]; + if (next !== undefined && !next.startsWith('--')) { + flags.set(key, next); + i++; + } else { + booleans.add(key); + } + } + return { command, flags, booleans }; +} + +export function requireSCBFlag(flags: ReadonlyMap, key: string): string { + const value = flags.get(key); + if (!value) throw new Error(`missing required flag --${key}`); + return value; +} + +export function parseSCBPreset(value: string): SCBPreset { + const preset = SCB_PRESETS.find((item) => item.key === value); + if (!preset) throw new Error(`unknown SCB preset ${value}`); + return preset; +} + +export function parseSCBValueCodes(raw: string | undefined, periods: string | undefined): Record { + if (raw) { + const parsed = JSON.parse(raw) as unknown; + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + throw new Error('--value-codes must be a JSON object'); + } + return Object.fromEntries( + Object.entries(parsed as Record).map(([key, value]) => [key, String(value)]), + ); + } + if (periods) { + const count = Number.parseInt(periods, 10); + if (!Number.isInteger(count) || count < 1) throw new Error('--periods must be a positive integer'); + return { Tid: `top(${count})` }; + } + return {}; +} + +export async function fetchSCBTablePayload( + tableId: string, + valueCodes: Readonly>, + options: { preset?: SCBPreset; client?: SCBClient } = {}, +): Promise { + const client = options.client ?? new SCBClient(); + const data = await client.getTableData(tableId, { ...valueCodes }); + const retrievedAt = new Date().toISOString(); + const status = data.length > 0 ? 'ok' : 'no-data'; + return { + provider: 'scb', + ...(options.preset ? { preset: options.preset.key, label: options.preset.label } : {}), + tableId, + valueCodes, + data, + status, + ...(status === 'no-data' ? { warning: 'SCB returned no rows; callers should fall back to cached data if available.' } : {}), + economicProvenance: { + provider: 'scb', + dataflow: 'SCB PxWeb', + indicator: tableId, + tableId, + retrieved_at: retrievedAt, + mcpTool: 'query_table', + }, + }; +} + +async function runTable( + tableId: string, + valueCodes: Readonly>, + booleans: ReadonlySet, + preset?: SCBPreset, +): Promise { + const payload = await fetchSCBTablePayload(tableId, valueCodes, { ...(preset ? { preset } : {}) }); + process.stdout.write(`${JSON.stringify(payload, null, 2)}\n`); + if (booleans.has('persist')) { + persistSCBData(tableId, payload, { + provider: 'scb', + ...(preset ? { preset: preset.key } : {}), + valueCodes, + }); + } +} + +async function main(): Promise { + const { command, flags, booleans } = parseSCBArgs(process.argv.slice(2)); + switch (command) { + case 'list-presets': + process.stdout.write(`${JSON.stringify({ presets: SCB_PRESETS }, null, 2)}\n`); + return; + case 'preset': { + const preset = parseSCBPreset(requireSCBFlag(flags, 'preset')); + const valueCodes = { + ...preset.defaultValueCodes, + ...parseSCBValueCodes(flags.get('value-codes'), flags.get('periods')), + }; + await runTable(preset.tableId, valueCodes, booleans, preset); + return; + } + case 'table': { + const tableId = requireSCBFlag(flags, 'table-id'); + const valueCodes = parseSCBValueCodes(flags.get('value-codes'), flags.get('periods')); + await runTable(tableId, valueCodes, booleans); + return; + } + case 'help': + default: + process.stdout.write(HELP); + } +} + +function isDirectExecution(): boolean { + const entry = process.argv[1]; + if (!entry) return false; + return import.meta.url === pathToFileURL(path.resolve(entry)).href; +} + +if (isDirectExecution()) { + main().catch((error: unknown) => { + const message = error instanceof Error ? error.message : String(error); + process.stderr.write(`scb-fetch: ${message}\n`); + process.exit(/^(missing|unknown|unexpected|--)/i.test(message) ? 2 : 1); + }); +} diff --git a/tests/data-persistence.test.ts b/tests/data-persistence.test.ts index cca3e18cc..2627e5ae7 100644 --- a/tests/data-persistence.test.ts +++ b/tests/data-persistence.test.ts @@ -28,6 +28,7 @@ import { persistWorldBankData, persistIMFData, persistSCBData, + persistRiksbankData, getDataRoot, } from '../scripts/parliamentary-data/data-persistence.js'; @@ -496,6 +497,24 @@ describe('data-persistence', () => { }); }); + describe('persistRiksbankData', () => { + it('should store Riksbank artifacts with sidecar', () => { + const resultPath = persistRiksbankData( + 'repo-rate-path', + { provider: 'riksbank', url: 'https://www.riksbank.se/en-gb/monetary-policy/' }, + tmpDir, + ); + expect(fs.existsSync(resultPath)).toBe(true); + expect(resultPath).toContain(path.join('riksbank', 'repo-rate-path.json')); + + const metaPath = resultPath.replace('.json', '.meta.json'); + const meta = JSON.parse(fs.readFileSync(metaPath, 'utf8')); + expect(meta.kind).toBe('repo-rate-path'); + expect(meta.url).toBe('https://www.riksbank.se/en-gb/monetary-policy/'); + expect(meta.mcpTool).toBe('riksbank-ts-client'); + }); + }); + // ── Collision avoidance ─────────────────────────────────────────────────── describe('collision avoidance', () => { diff --git a/tests/riksbank-fetch.test.ts b/tests/riksbank-fetch.test.ts new file mode 100644 index 000000000..2298bc8ff --- /dev/null +++ b/tests/riksbank-fetch.test.ts @@ -0,0 +1,134 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; + +import { + assertRiksbankFetchTarget, + fetchRiksbankPayload, + parseRiksbankArgs, + parseRiksbankKind, +} from '../scripts/riksbank-fetch.js'; + +describe('Riksbank fetch CLI helpers', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('parses commands and artifact kinds', () => { + const parsed = parseRiksbankArgs(['fetch', '--kind', 'minutes', '--url', 'https://www.riksbank.se/en-gb/x']); + expect(parsed.command).toBe('fetch'); + expect(parsed.flags.get('kind')).toBe('minutes'); + expect(parseRiksbankKind('repo-rate-path')).toBe('repo-rate-path'); + }); + + it('rejects unsafe or non-Riksbank URLs', () => { + expect(() => assertRiksbankFetchTarget('https://www.riksbank.se/en-gb/')).not.toThrow(); + expect(() => assertRiksbankFetchTarget('http://www.riksbank.se/en-gb/')).toThrow(/HTTPS/); + expect(() => assertRiksbankFetchTarget('https://example.com/')).toThrow(/allowlist/); + expect(() => assertRiksbankFetchTarget('not a url')).toThrow(/invalid/); + }); + + it('builds provenance for HTML responses', async () => { + vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response('Minutes - Riksbankcontent', { + status: 200, + headers: { 'content-type': 'text/html; charset=utf-8' }, + }), + ); + const payload = await fetchRiksbankPayload('minutes', 'https://www.riksbank.se/en-gb/minutes/'); + expect(payload.provider).toBe('riksbank'); + expect(payload.status).toBe('ok'); + expect(payload.title).toBe('Minutes - Riksbank'); + expect(payload.economicProvenance.provider).toBe('riksbank'); + expect(payload.economicProvenance.indicator).toBe('minutes'); + }); + + it('builds provenance for JSON responses', async () => { + vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response(JSON.stringify({ repoRate: 2.25 }), { + status: 200, + headers: { 'content-type': 'application/json' }, + }), + ); + const payload = await fetchRiksbankPayload('repo-rate-path', 'https://www.riksbank.se/en-gb/rate.json'); + expect(payload.status).toBe('ok'); + expect(payload.json).toEqual({ repoRate: 2.25 }); + expect(payload.text).toBeUndefined(); + }); + + it('encodes PDF responses as base64 with a byte-length cap', async () => { + const pdfBytes = new Uint8Array([0x25, 0x50, 0x44, 0x46, 0x2d, 0x31, 0x2e, 0x34]); // "%PDF-1.4" + vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response(pdfBytes, { + status: 200, + headers: { 'content-type': 'application/pdf' }, + }), + ); + const payload = await fetchRiksbankPayload('minutes', 'https://www.riksbank.se/en-gb/min.pdf'); + expect(payload.status).toBe('ok'); + expect(payload.contentType).toContain('pdf'); + expect(payload.pdfBytes).toBe(pdfBytes.byteLength); + expect(payload.pdfBase64).toBe(Buffer.from(pdfBytes).toString('base64')); + expect(payload.text).toBeUndefined(); + }); + + it('fail-softs to a no-data payload on network outage', async () => { + vi.spyOn(globalThis, 'fetch').mockRejectedValue(new Error('Riksbank unreachable')); + const payload = await fetchRiksbankPayload('repo-rate-path', 'https://www.riksbank.se/en-gb/rate/'); + expect(payload.status).toBe('no-data'); + expect(payload.warning).toMatch(/cached analysis\/data\/riksbank/i); + expect(payload.economicProvenance.provider).toBe('riksbank'); + }); + + it('fail-softs to a no-data payload on non-2xx HTTP status', async () => { + vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response('Server Error', { status: 503, statusText: 'Service Unavailable' }), + ); + const payload = await fetchRiksbankPayload('minutes', 'https://www.riksbank.se/en-gb/m/'); + expect(payload.status).toBe('no-data'); + expect(payload.warning).toMatch(/HTTP 503/); + }); + + it('fail-softs when Content-Length exceeds the PDF cap before downloading', async () => { + const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response(null, { + status: 200, + headers: { 'content-type': 'application/pdf', 'content-length': String(10 * 1024 * 1024) }, + }), + ); + const payload = await fetchRiksbankPayload('minutes', 'https://www.riksbank.se/en-gb/m.pdf'); + expect(payload.status).toBe('no-data'); + expect(payload.warning).toMatch(/Content-Length/); + expect(fetchSpy).toHaveBeenCalledTimes(1); + }); + + it('follows redirects manually and validates each Location host', async () => { + const fetchSpy = vi.spyOn(globalThis, 'fetch') + .mockResolvedValueOnce( + new Response(null, { + status: 302, + headers: { location: 'https://www.riksbank.se/en-gb/final/' }, + }), + ) + .mockResolvedValueOnce( + new Response('Final', { + status: 200, + headers: { 'content-type': 'text/html; charset=utf-8' }, + }), + ); + const payload = await fetchRiksbankPayload('minutes', 'https://www.riksbank.se/en-gb/start/'); + expect(payload.status).toBe('ok'); + expect(payload.url).toBe('https://www.riksbank.se/en-gb/final/'); + expect(fetchSpy).toHaveBeenCalledTimes(2); + }); + + it('rejects redirects to off-allowlist hosts', async () => { + vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response(null, { + status: 302, + headers: { location: 'https://evil.example.com/leak' }, + }), + ); + const payload = await fetchRiksbankPayload('minutes', 'https://www.riksbank.se/en-gb/m/'); + expect(payload.status).toBe('no-data'); + expect(payload.warning).toMatch(/allowlist|fetch failed/i); + }); +}); diff --git a/tests/scb-fetch.test.ts b/tests/scb-fetch.test.ts new file mode 100644 index 000000000..adb6821c9 --- /dev/null +++ b/tests/scb-fetch.test.ts @@ -0,0 +1,61 @@ +import { describe, it, expect, vi, afterEach } from 'vitest'; + +import { + SCB_PRESETS, + fetchSCBTablePayload, + parseSCBArgs, + parseSCBPreset, + parseSCBValueCodes, + requireSCBFlag, +} from '../scripts/scb-fetch.js'; +import { SCBClient } from '../scripts/scb-client.js'; + +describe('SCB fetch CLI helpers', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('exposes KPI, AKU, household-economy and fuel-price presets', () => { + expect(SCB_PRESETS.map((preset) => preset.key)).toEqual([ + 'cpi', + 'aku', + 'household-economy', + 'fuel-prices', + ]); + expect(parseSCBPreset('cpi').tableId).toBe('0000003N'); + expect(parseSCBPreset('aku').tableId).toBe('000003V8'); + }); + + it('parses command flags and JSON value codes', () => { + const parsed = parseSCBArgs(['table', '--table-id', 'TAB5765', '--value-codes', '{"Tid":"top(4)"}', '--persist']); + expect(parsed.command).toBe('table'); + expect(requireSCBFlag(parsed.flags, 'table-id')).toBe('TAB5765'); + expect(parsed.booleans.has('persist')).toBe(true); + expect(parseSCBValueCodes(parsed.flags.get('value-codes'), undefined)).toEqual({ Tid: 'top(4)' }); + }); + + it('builds a periods fallback value code', () => { + expect(parseSCBValueCodes(undefined, '6')).toEqual({ Tid: 'top(6)' }); + }); + + it('throws for invalid CLI input', () => { + expect(() => parseSCBArgs(['bad-command'])).toThrow(/unknown command/); + expect(() => requireSCBFlag(new Map(), 'table-id')).toThrow(/missing required flag/); + expect(() => parseSCBPreset('bad')).toThrow(/unknown SCB preset/); + expect(() => parseSCBValueCodes('[]', undefined)).toThrow(/JSON object/); + }); + + it('emits SCB provenance and fail-soft no-data payload on outage', async () => { + vi.spyOn(globalThis, 'fetch').mockRejectedValue(new Error('SCB API down')); + const payload = await fetchSCBTablePayload( + '0000003N', + { Tid: 'top(2)' }, + { client: new SCBClient({ maxRetries: 0 }) }, + ); + expect(payload.status).toBe('no-data'); + expect(payload.data).toEqual([]); + expect(payload.warning).toMatch(/cached data/i); + expect(payload.economicProvenance.provider).toBe('scb'); + expect(payload.economicProvenance.indicator).toBe('0000003N'); + }); +});