diff --git a/analysis/methodologies/ai-driven-analysis-guide.md b/analysis/methodologies/ai-driven-analysis-guide.md index 9aec4eeea..7da05a828 100644 --- a/analysis/methodologies/ai-driven-analysis-guide.md +++ b/analysis/methodologies/ai-driven-analysis-guide.md @@ -93,6 +93,28 @@ npx tsx scripts/download-parliamentary-data.ts \ **Write `data-download-manifest.md`** using the [manifest template](../templates/data-download-manifest.md). It records what arrived, from which MCP tools, with what data-depth distribution (FULL-TEXT / SUMMARY / METADATA-ONLY). +After `download-parliamentary-data.ts` completes for `committeeReports`, also run the voting-records script to capture party-level vote counts and defector detection for each betänkande: + +```bash +npx tsx scripts/fetch-voting-records.ts \ + --date ${ARTICLE_DATE} \ + --doc-type committeeReports \ + --persist +``` + +This writes `data/voteringar/${ARTICLE_DATE}/{bet}.json` and injects voting-record summaries into `analysis/daily/${ARTICLE_DATE}/committeeReports/voting-records/`. Each record carries an explicit `status` field. `fetchVotingForBet` emits one of three statuses: `"fetched"` (full table available), `"not_found"` (MCP returned successfully with zero rows — e.g. referral, procedural decision, or committee item without a chamber vote), or `"error"` (transient MCP/network failure with `errorMessage`). Editorial tooling that *knows* a vote is upcoming may also persist `"vote_pending"` annotations manually. The script emits a matching injection template for every non-`fetched` status (``, ``, ``), so the coalition-mathematics section can paste the template verbatim and rerun the script to upgrade `error` / `not_found` to `fetched` once data is available. + +To fetch the parliamentary forward calendar for week-ahead or month-ahead forecasting, run: + +```bash +npx tsx scripts/fetch-calendar.ts \ + --from ${ARTICLE_DATE} \ + --tom ${TOM_DATE} \ + --persist +``` + +This writes `analysis/data/calendar/${ARTICLE_DATE}_${TOM_DATE}.json` using the MCP `get_calendar_events` primary path with automatic fallback to HTML parsing of riksdagen.se/sv/kalendarium/. + If the date yields 0 documents, apply the **Empty-Day Protocol** (§ Empty-Day Handling below) — never publish a "0 documents" file. --- @@ -168,7 +190,7 @@ Every run produces **all five Family C files and all seven Family D files**. The | ⭐ `methodology-reflection.md` | **VITAL run-audit gate.** Evidence sufficiency, confidence distribution, source diversity, party-neutrality arithmetic, **ICD 203 compliance audit**, three concrete methodology improvements for the next cycle. Skipping it breaks the self-correction loop. | [`methodology-reflection.md`](../templates/methodology-reflection.md) | Key Assumptions Check, Quality of Information Check | | `election-2026-analysis.md` | Seat-projection deltas + coalition viability for every run through 2026-09; after the election it converts to a permanent "post-2026 government-formation context" file | [`election-2026-analysis.md`](../templates/election-2026-analysis.md) | Morphological | | `voter-segmentation.md` | Demographic / regional / ideological segment impact; when the day's docs are procedural, documents baseline segment positions | [`voter-segmentation.md`](../templates/voter-segmentation.md) | Outside-In Thinking | -| `coalition-mathematics.md` | Current seat map + pivotal votes + Sainte-Laguë scenarios; stable structure regardless of daily contentiousness | [`coalition-mathematics.md`](../templates/coalition-mathematics.md) | Morphological | +| `coalition-mathematics.md` | Current seat map + pivotal votes + Sainte-Laguë scenarios; stable structure regardless of daily contentiousness. **MUST** include a voting-record table sourced from `fetch-voting-records` output (`data/voteringar/{date}/{bet}.json`) for every betänkande cited, or one of the explicit annotations: `` when `status: "not_found"` (MCP returned successfully with zero data — referral or procedural vote), `` when `status: "error"` (transient MCP/network failure; rerun once the service is back), or — set manually by editorial tooling that knows a vote is upcoming — ``. | [`coalition-mathematics.md`](../templates/coalition-mathematics.md) | Morphological | | `historical-parallels.md` | Named precedent(s) ≤ 40 years with similarity score; when no obvious parallel exists, documents the "no-precedent" finding with reasoning | [`historical-parallels.md`](../templates/historical-parallels.md) | Outside-In Thinking | | `media-framing-analysis.md` | How each party, press quadrant, and platform frames the day; runs every cycle to build the longitudinal frame record | [`media-framing-analysis.md`](../templates/media-framing-analysis.md) | Outside-In Thinking | | `implementation-feasibility.md` | Delivery-risk view (budget / IT / regulatory / workforce); when no new bill lands, audits the backlog of in-flight commitments | [`implementation-feasibility.md`](../templates/implementation-feasibility.md) | Premortem Analysis | @@ -281,6 +303,12 @@ graph TB |----------|:--------:|:--------:|:--------:|:--------:|:--------:| | Morning per-type (propositions, motions, betänkanden, interpellationer, frågor) | ✅ All 9 | ✅ Both | ✅ All 5 | ✅ All 7 | ✅ Every doc | | Midday week-ahead / month-ahead forecasts | ✅ All 9 | ✅ Both | ✅ All 5 | ✅ All 7 | ✅ Every forecast item | + +> 📅 **Week-ahead / month-ahead calendar enrichment**: For midday forecasting runs, run `fetch-calendar.ts` **before** analysis to pre-populate forward events: +> ```bash +> npx tsx scripts/fetch-calendar.ts --from ${ARTICLE_DATE} --tom ${TOM_DATE} --persist +> ``` +> The resulting `analysis/data/calendar/${ARTICLE_DATE}_${TOM_DATE}.json` feeds `forward-indicators.md` (horizon items) and `coalition-mathematics.md` (scheduled votes). Use the `source` field to cite whether events came from MCP (`"mcp"`) or the web fallback (`"web_fallback"`), and apply the appropriate Admiralty reliability code. | Evening analysis | ✅ All 9 | ✅ Both | ✅ All 5 | ✅ All 7 | ✅ Every doc | | Realtime monitor | ✅ All 9 | ✅ Both | ✅ All 5 | ✅ All 7 | ✅ Every doc | | Weekly review | ✅ All 9 | ✅ Both | ✅ All 5 | ✅ All 7 | Top 20 | diff --git a/scripts/fetch-calendar.ts b/scripts/fetch-calendar.ts new file mode 100644 index 000000000..bf21f49df --- /dev/null +++ b/scripts/fetch-calendar.ts @@ -0,0 +1,359 @@ +#!/usr/bin/env tsx +/** + * @module scripts/fetch-calendar + * @description Fetch riksdag calendar events using a primary (MCP) → + * fallback (web fetch + HTML parsing) chain. + * + * Usage: + * npx tsx scripts/fetch-calendar.ts --from 2026-04-27 --tom 2026-05-27 [--org UTSK] [--akt bet] [--persist] + * + * Output: + * stdout — always written (JSON) + * analysis/data/calendar/{from}_{tom}.json — written only when --persist is set + * + * Exit codes: + * 0 — success + * 1 — runtime / network error + * 2 — bad CLI arguments + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { MCPClient } from './mcp-client.js'; + +// --------------------------------------------------------------------------- +// Paths +// --------------------------------------------------------------------------- + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const REPO_ROOT = path.resolve(__dirname, '..'); + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +export interface ParsedCalendarArgs { + readonly from: string; + readonly tom: string; + readonly org: string | null; + readonly akt: string | null; + readonly persist: boolean; +} + +export interface CalendarEvent { + datum: string; + tid: string; + org: string; + titel: string; + typ: string; +} + +export interface CalendarOutput { + from: string; + tom: string; + fetchedAt: string; + source: 'mcp' | 'web_fallback'; + events: CalendarEvent[]; +} + +// --------------------------------------------------------------------------- +// CLI argument parsing +// --------------------------------------------------------------------------- + +const DATE_RE = /^\d{4}-\d{2}-\d{2}$/; + +export interface CalendarParseArgsResult { + readonly args: ParsedCalendarArgs; + readonly error: string | null; +} + +export function parseArgs(argv: readonly string[]): CalendarParseArgsResult { + const flags = new Map(); + const booleans = new Set(); + + for (let i = 0; i < argv.length; i++) { + const token = argv[i]; + if (!token.startsWith('--')) continue; + const key = token.slice(2); + const next = argv[i + 1]; + if (next !== undefined && !next.startsWith('--')) { + flags.set(key, next); + i++; + } else { + booleans.add(key); + } + } + + const fromVal = flags.get('from'); + const tomVal = flags.get('tom'); + + if (!fromVal) { + return { + args: { from: '', tom: '', org: null, akt: null, persist: false }, + error: 'missing required flag --from', + }; + } + if (!tomVal) { + return { + args: { from: fromVal, tom: '', org: null, akt: null, persist: false }, + error: 'missing required flag --tom', + }; + } + + if (!DATE_RE.test(fromVal)) { + return { + args: { from: '', tom: '', org: null, akt: null, persist: false }, + error: `--from must be YYYY-MM-DD, got: ${fromVal}`, + }; + } + if (!DATE_RE.test(tomVal)) { + return { + args: { from: fromVal, tom: '', org: null, akt: null, persist: false }, + error: `--tom must be YYYY-MM-DD, got: ${tomVal}`, + }; + } + + return { + args: { + from: fromVal, + tom: tomVal, + org: flags.get('org') ?? null, + akt: flags.get('akt') ?? null, + persist: booleans.has('persist'), + }, + error: null, + }; +} + +// --------------------------------------------------------------------------- +// MCP primary path +// --------------------------------------------------------------------------- + +async function fetchViaMcp(client: MCPClient, args: ParsedCalendarArgs): Promise { + const raw = await client.fetchCalendarEvents(args.from, args.tom, args.org, args.akt); + return raw.map((item) => { + const r = item as Record; + return { + datum: String(r['datum'] ?? r['date'] ?? r['dtstart'] ?? ''), + tid: String(r['tid'] ?? r['time'] ?? r['starttid'] ?? ''), + org: String(r['org'] ?? r['organ'] ?? r['organisation'] ?? ''), + titel: String(r['titel'] ?? r['summary'] ?? r['title'] ?? r['rubrik'] ?? ''), + typ: String(r['typ'] ?? r['type'] ?? r['akt'] ?? r['aktivitet'] ?? ''), + }; + }); +} + +// --------------------------------------------------------------------------- +// Web fallback — parse riksdagen.se/sv/kalendarium/ HTML +// --------------------------------------------------------------------------- + +const RIKSDAGEN_CALENDAR_URL = 'https://www.riksdagen.se/sv/kalendarium/'; + +/** + * Parse calendar events from riksdagen.se HTML using regex patterns. + * Since cheerio may not be available, we use Node's built-in fetch + * and regex-based HTML extraction. + */ +export function parseCalendarHtml(html: string): CalendarEvent[] { + const events: CalendarEvent[] = []; + + // Pattern: extract event blocks. The page wraps events in article/li + // elements with class like "event-item", "event", "calendar-item". + // We extract: date, time, organ, title, type using several heuristics. + + // Strategy 1: JSON-LD structured data (most reliable) + const jsonLdRe = /]*type="application\/ld\+json"[^>]*>([\s\S]*?)<\/script>/gi; + for (const m of html.matchAll(jsonLdRe)) { + try { + const raw = m[1]; + if (!raw) continue; + const obj = JSON.parse(raw) as Record; + const items = Array.isArray(obj) ? obj : [obj]; + for (const item of items) { + if (typeof item !== 'object' || item === null) continue; + const ev = item as Record; + if (ev['@type'] === 'Event' || ev['@type'] === 'SocialEvent') { + events.push({ + datum: String(ev['startDate'] ?? ev['startdate'] ?? '').slice(0, 10), + tid: String(ev['startDate'] ?? '').slice(11, 16), + org: String( + (ev['organizer'] as Record)?.['name'] ?? '', + ), + titel: String(ev['name'] ?? ev['headline'] ?? ''), + typ: String(ev['eventType'] ?? ev['category'] ?? ''), + }); + } + } + } catch { + // JSON parse failed — skip this block + } + } + + if (events.length > 0) return events; + + // Strategy 2: Scan for common HTML patterns in riksdagen.se + // Event title typically in or

+ const titleRe = /<(?:a|h[23])[^>]*class="[^"]*(?:event-title|calendar-title|event-name)[^"]*"[^>]*>([\s\S]*?)<\/(?:a|h[23])>/gi; + const dateRe = /(?:data-date|datetime)="(\d{4}-\d{2}-\d{2})"/gi; + const timeRe = /(\d{2}:\d{2})/g; + + const dates = [...html.matchAll(dateRe)].map((m) => m[1] ?? ''); + const titles = [...html.matchAll(titleRe)].map((m) => + // Use [\s\S]*? to match newlines inside tags (prevents incomplete sanitization) + (m[1] ?? '').replace(/<[\s\S]*?>/g, '').trim(), + ); + // Pre-compute time matches once outside the loop (O(N) instead of + // O(titles × times)). Keep an index pointer (`timeCursor`) similar to + // `usedDates`'s sequential walk so each match is consumed at most once. + const allTimes = [...html.matchAll(timeRe)].map((m) => m[1] ?? ''); + + const usedDates = new Set(); + let timeCursor = 0; + + for (let i = 0; i < titles.length; i++) { + const title = titles[i]; + if (!title) continue; + + // Find nearest unused date + let datum = ''; + for (let d = i; d < dates.length; d++) { + if (!usedDates.has(d) && dates[d]) { + datum = dates[d]!; + usedDates.add(d); + break; + } + } + + // Consume the next available time match (linear pointer scan) + let tid = ''; + while (timeCursor < allTimes.length) { + const candidate = allTimes[timeCursor++]; + if (candidate) { + tid = candidate; + break; + } + } + + events.push({ datum, tid, org: '', titel: title, typ: '' }); + } + + return events; +} + +async function fetchViaWeb(args: ParsedCalendarArgs): Promise { + const url = new URL(RIKSDAGEN_CALENDAR_URL); + if (args.from) url.searchParams.set('from', args.from); + if (args.tom) url.searchParams.set('tom', args.tom); + if (args.org) url.searchParams.set('org', args.org); + if (args.akt) url.searchParams.set('akt', args.akt); + + const response = await fetch(url.toString(), { + headers: { 'User-Agent': 'riksdagsmonitor/1.0 (+https://hack23.com)' }, + signal: AbortSignal.timeout(15_000), + }); + + if (!response.ok) { + throw new Error(`web_fallback: HTTP ${response.status} from ${url.toString()}`); + } + + const html = await response.text(); + return parseCalendarHtml(html); +} + +// --------------------------------------------------------------------------- +// Orchestrator — MCP primary → web fallback → graceful empty +// --------------------------------------------------------------------------- + +/** + * Dependencies for the calendar fetch orchestrator. Allows tests to inject + * deterministic stubs in place of the real MCP and web paths. + */ +export interface FetchCalendarDeps { + readonly fetchViaMcp: (args: ParsedCalendarArgs) => Promise; + readonly fetchViaWeb: (args: ParsedCalendarArgs) => Promise; + readonly now?: () => Date; + readonly logger?: (msg: string) => void; +} + +/** + * Run the MCP-primary → web-fallback chain and return a `CalendarOutput`. + * Source is `'mcp'` if the MCP path returned ≥1 event, otherwise + * `'web_fallback'` (regardless of whether the web fallback itself returned + * events or had to degrade gracefully to an empty array). + */ +export async function fetchCalendarEvents( + args: ParsedCalendarArgs, + deps: FetchCalendarDeps, +): Promise { + const log = deps.logger ?? (() => {}); + const fetchedAt = (deps.now?.() ?? new Date()).toISOString(); + + let events: CalendarEvent[] = []; + let source: 'mcp' | 'web_fallback' = 'mcp'; + + try { + events = await deps.fetchViaMcp(args); + log(`fetch-calendar: MCP returned ${events.length} event(s)`); + } catch (mcpErr) { + log(`fetch-calendar: MCP failed (${String(mcpErr)}), trying web fallback`); + } + + if (events.length === 0) { + source = 'web_fallback'; + try { + events = await deps.fetchViaWeb(args); + log(`fetch-calendar: web_fallback returned ${events.length} event(s)`); + } catch (webErr) { + log(`fetch-calendar: web_fallback also failed (${String(webErr)}), returning empty`); + events = []; + } + } + + return { from: args.from, tom: args.tom, fetchedAt, source, events }; +} + +// --------------------------------------------------------------------------- +// Main entry point +// --------------------------------------------------------------------------- + +async function main(): Promise { + const { args, error } = parseArgs(process.argv.slice(2)); + if (error) { + process.stderr.write(`fetch-calendar: ${error}\n`); + process.exit(2); + } + + const { from, tom, persist } = args; + + const client = new MCPClient(); + const output = await fetchCalendarEvents(args, { + fetchViaMcp: (a) => fetchViaMcp(client, a), + fetchViaWeb, + logger: (msg) => process.stderr.write(`${msg}\n`), + }); + + process.stdout.write(JSON.stringify(output, null, 2) + '\n'); + + if (persist) { + const calendarDir = path.join(REPO_ROOT, 'analysis', 'data', 'calendar'); + fs.mkdirSync(calendarDir, { recursive: true }); + const outFile = path.join(calendarDir, `${from}_${tom}.json`); + fs.writeFileSync(outFile, JSON.stringify(output, null, 2) + '\n', 'utf8'); + process.stderr.write(`fetch-calendar: persisted → ${path.relative(REPO_ROOT, outFile)}\n`); + } +} + +// Run if this is the entry point +const isMain = + process.argv[1] !== undefined && + (process.argv[1].endsWith('fetch-calendar.ts') || + process.argv[1].endsWith('fetch-calendar.js')); + +if (isMain) { + main().catch((err: unknown) => { + process.stderr.write(`fetch-calendar: fatal error: ${String(err)}\n`); + process.exit(1); + }); +} diff --git a/scripts/fetch-voting-records.ts b/scripts/fetch-voting-records.ts new file mode 100644 index 000000000..9b44467fb --- /dev/null +++ b/scripts/fetch-voting-records.ts @@ -0,0 +1,536 @@ +#!/usr/bin/env tsx +/** + * @module scripts/fetch-voting-records + * @description Fetch party-level and individual voting records from the + * riksdag-regering MCP for betänkanden listed in an analysis manifest. + * + * Usage: + * npx tsx scripts/fetch-voting-records.ts --date 2026-04-27 --doc-type committeeReports [--persist] + * npx tsx scripts/fetch-voting-records.ts --date 2026-04-27 [--persist] + * + * Output: + * data/voteringar/{date}/{sanitized_bet}.json — one file per betänkande + * analysis/daily/{date}/{docType}/voting-records/ — injection templates (--persist) + * + * Exit codes: + * 0 — success + * 1 — runtime / network error + * 2 — bad CLI arguments + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { MCPClient } from './mcp-client.js'; + +// --------------------------------------------------------------------------- +// Paths +// --------------------------------------------------------------------------- + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const REPO_ROOT = path.resolve(__dirname, '..'); + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +export interface ParsedVotingArgs { + readonly date: string; + readonly docType: string | null; + readonly persist: boolean; +} + +export interface PartyVoteRow { + parti: string; + ja: number; + nej: number; + avstar: number; + franvarande: number; +} + +export interface DefectorRecord { + iid: string; + intressentNamn: string; + parti: string; + rost: string; + partyMajority: string; +} + +export interface VotingRecordOutput { + bet: string; + rm: string | null; + fetchedAt: string; + status: 'fetched' | 'vote_pending' | 'not_found' | 'error'; + partyVotes: PartyVoteRow[]; + defectors: DefectorRecord[]; + mermaidDiagram: string; + errorMessage?: string; + injectionMarkdown?: string; +} + +// --------------------------------------------------------------------------- +// CLI argument parsing +// --------------------------------------------------------------------------- + +const DATE_RE = /^\d{4}-\d{2}-\d{2}$/; + +export interface ParseArgsResult { + readonly args: ParsedVotingArgs; + readonly error: string | null; +} + +export function parseArgs(argv: readonly string[]): ParseArgsResult { + const flags = new Map(); + const booleans = new Set(); + + for (let i = 0; i < argv.length; i++) { + const token = argv[i]; + if (!token.startsWith('--')) continue; + const key = token.slice(2); + const next = argv[i + 1]; + if (next !== undefined && !next.startsWith('--')) { + flags.set(key, next); + i++; + } else { + booleans.add(key); + } + } + + const dateRaw = flags.get('date') ?? new Date().toISOString().slice(0, 10); + if (!DATE_RE.test(dateRaw)) { + return { + args: { date: '', docType: null, persist: false }, + error: `--date must be YYYY-MM-DD, got: ${dateRaw}`, + }; + } + + return { + args: { + date: dateRaw, + docType: flags.get('doc-type') ?? null, + persist: booleans.has('persist'), + }, + error: null, + }; +} + +// --------------------------------------------------------------------------- +// Manifest parsing — extract `bet` values from data-download-manifest.md +// --------------------------------------------------------------------------- + +export function extractBetValues(manifestText: string): string[] { + const bets = new Set(); + + // Pattern: beteckning like "FiU48", "AU10", "KU20", "MJU15", "SoU12" etc. + // Swedish committee abbreviations can be mixed-case (e.g. Fi=Finansutskottet, + // So=Socialutskottet). Pattern: starts with uppercase, up to 4 more + // letters (upper or lower), then one or more digits. + const BET_RE = /\b([A-ZÅÄÖ][a-zåäöA-ZÅÄÖ]{0,4}\d+)\b/g; + for (const match of manifestText.matchAll(BET_RE)) { + const candidate = match[1]; + if (!candidate) continue; + // Require at least one letter before the digits (already guaranteed by regex) + // Filter year/version-like tokens that are not committee designations, + // for example a single-letter prefix followed by a 4-digit year: "A2026". + if (/^[A-ZÅÄÖ]\d{4}$/.test(candidate)) continue; + // Skip if the digit-free prefix alone is a known false-positive acronym + const letterPart = candidate.replace(/\d+$/, ''); + if (/^(Se|En|Sv|Da|No|Fi|De|Fr|Es|Nl|Ar|He|Ja|Ko|Zh|Id|Ok|In|As|At|By|Be|Do|Go|Is|It|If|Of|On|Or|To|Up|Us|We)$/i.test(letterPart)) continue; + bets.add(candidate); + } + + return [...bets]; +} + +// --------------------------------------------------------------------------- +// Defector detection +// --------------------------------------------------------------------------- + +/** + * Given individual voting records for a betänkande, compute defectors. + * An MP is a defector if they voted differently from their party's majority. + */ +export function detectDefectors(votes: unknown[]): { + defectors: DefectorRecord[]; + status: 'fetched' | 'vote_pending'; +} { + if (!votes.length) { + return { defectors: [], status: 'vote_pending' }; + } + + // Tally party majority votes + const partyTally = new Map>(); + for (const rawVote of votes) { + const vote = rawVote as Record; + const parti = String(vote['parti'] ?? vote['party'] ?? ''); + const rost = String(vote['rost'] ?? vote['vote'] ?? ''); + if (!parti || !rost) continue; + + if (!partyTally.has(parti)) partyTally.set(parti, new Map()); + const tally = partyTally.get(parti)!; + tally.set(rost, (tally.get(rost) ?? 0) + 1); + } + + // Determine majority vote per party. Ties for the maximum count are + // ambiguous (e.g. equal Ja/Nej splits), so we leave the majority + // *undefined* for those parties rather than picking an arbitrary winner + // based on iteration order — that previously produced false defectors. + const partyMajority = new Map(); + for (const [parti, tally] of partyTally) { + let maxCount = 0; + let majorityVote = ''; + let isTied = false; + for (const [rost, count] of tally) { + if (count > maxCount) { + maxCount = count; + majorityVote = rost; + isTied = false; + } else if (count === maxCount && count > 0 && rost !== majorityVote) { + isTied = true; + } + } + if (!isTied && majorityVote) { + partyMajority.set(parti, majorityVote); + } + } + + // Find defectors + const defectors: DefectorRecord[] = []; + for (const rawVote of votes) { + const vote = rawVote as Record; + const parti = String(vote['parti'] ?? vote['party'] ?? ''); + const rost = String(vote['rost'] ?? vote['vote'] ?? ''); + const iid = String(vote['iid'] ?? vote['intressent_id'] ?? ''); + const namn = String( + vote['intressentNamn'] ?? + vote['namn'] ?? + vote['name'] ?? + vote['tilltalsnamn'] ?? + '', + ); + + if (!parti || !rost) continue; + const majority = partyMajority.get(parti) ?? ''; + if (majority && rost !== majority && rost !== 'Frånvarande') { + defectors.push({ + iid, + intressentNamn: namn, + parti, + rost, + partyMajority: majority, + }); + } + } + + return { defectors, status: 'fetched' }; +} + +// --------------------------------------------------------------------------- +// Mermaid diagram generation +// --------------------------------------------------------------------------- + +/** + * Generate a Mermaid xychart-beta (bar chart) showing party vote distribution. + * Falls back to a markdown table comment if there are no rows. + */ +export function generateMermaidVoteChart(partyVotes: PartyVoteRow[], bet: string): string { + if (!partyVotes.length) { + return `%%{init: {"theme": "base"}}%%\nflowchart LR\n NA["No voting data for ${bet}"]`; + } + + // Sort by total votes descending + const sorted = [...partyVotes].sort( + (a, b) => (b.ja + b.nej + b.avstar) - (a.ja + a.nej + a.avstar), + ); + + const labels = sorted.map((r) => `"${r.parti}"`).join(', '); + const jaValues = sorted.map((r) => r.ja).join(', '); + const nejValues = sorted.map((r) => r.nej).join(', '); + const avstarValues = sorted.map((r) => r.avstar).join(', '); + + return [ + `%%{init: {"theme": "base"}}%%`, + `xychart-beta`, + ` title "Omröstning: ${bet}"`, + ` x-axis [${labels}]`, + ` y-axis "Röster" 0 --> ${Math.max(...sorted.map((r) => r.ja + r.nej + r.avstar + r.franvarande), 1)}`, + ` bar [${jaValues}]`, + ` bar [${nejValues}]`, + ` bar [${avstarValues}]`, + ].join('\n'); +} + +// --------------------------------------------------------------------------- +// Markdown injection template +// --------------------------------------------------------------------------- + +function buildInjectionMarkdown(record: VotingRecordOutput): string { + const { bet, status, partyVotes, defectors, mermaidDiagram, errorMessage } = record; + + if (status === 'vote_pending') { + return [ + ``, + `> **Omröstning ej genomförd** — betänkande \`${bet}\` har ännu inte röstats igenom.`, + `> Uppdatera med \`fetch-voting-records.ts --date {date}\` när omröstningen är avslutad.`, + ].join('\n'); + } + + if (status === 'not_found') { + return [ + ``, + `> **Ingen omröstning registrerad** — betänkande \`${bet}\` saknar röstresultat i Riksdagens öppna data`, + `> (kan vara remiss, procedurellt beslut eller utskottsärende utan kammaravgörande).`, + ].join('\n'); + } + + if (status === 'error') { + const detail = errorMessage ? `: ${errorMessage}` : ''; + return [ + ``, + `> **Hämtning av omröstning misslyckades** — \`${bet}\`${detail}.`, + `> Återhämta med \`fetch-voting-records.ts --date {date}\` när MCP/nätverket är tillgängligt.`, + ].join('\n'); + } + + const tableRows = partyVotes + .map((r) => `| ${r.parti} | ${r.ja} | ${r.nej} | ${r.avstar} | ${r.franvarande} |`) + .join('\n'); + + const defectorSection = + defectors.length > 0 + ? [ + '', + '#### Partiavvikare', + '', + '| Ledamot | Parti | Röstade | Partiets majoritet |', + '|---------|-------|---------|-------------------|', + ...defectors.map( + (d) => `| ${d.intressentNamn || d.iid} | ${d.parti} | ${d.rost} | ${d.partyMajority} |`, + ), + ].join('\n') + : '\n_Inga partiavvikare registrerade._'; + + return [ + `### Omröstning: ${bet}`, + '', + '| Parti | Ja | Nej | Avstår | Frånv. |', + '|-------|:--:|:---:|:------:|:------:|', + tableRows, + '', + '```mermaid', + mermaidDiagram, + '```', + defectorSection, + ].join('\n'); +} + +// --------------------------------------------------------------------------- +// Sanitize bet for filesystem use +// --------------------------------------------------------------------------- + +export function sanitizeBet(bet: string): string { + return bet.toLowerCase().replace(/[^a-z0-9åäö]/g, '-').replace(/-+/g, '-').replace(/^-|-$/g, ''); +} + +// --------------------------------------------------------------------------- +// Fetch a single bet's voting data +// --------------------------------------------------------------------------- + +async function fetchVotingForBet(client: MCPClient, bet: string): Promise { + const fetchedAt = new Date().toISOString(); + + let partyVotes: PartyVoteRow[] = []; + let rawIndividualVotes: unknown[]; + let rm: string | null = null; + + try { + const groupData = await client.fetchVotingGroup({ bet, groupBy: 'parti', limit: 200 }); + + if (groupData.length > 0) { + partyVotes = groupData.map((row) => { + const r = row as Record; + const parseNum = (v: unknown): number => Number.parseInt(String(v ?? '0'), 10) || 0; + const parti = String(r['parti'] ?? r['party'] ?? r['Parti'] ?? ''); + rm = rm ?? String(r['rm'] ?? r['riksmote'] ?? ''); + return { + parti, + ja: parseNum(r['ja'] ?? r['Ja'] ?? r['yes']), + nej: parseNum(r['nej'] ?? r['Nej'] ?? r['no']), + avstar: parseNum(r['avstar'] ?? r['Avstår'] ?? r['abstain'] ?? r['avstar_antal']), + franvarande: parseNum(r['franvarande'] ?? r['Frånvarande'] ?? r['absent']), + }; + }).filter((r) => r.parti !== ''); + } + + rawIndividualVotes = await client.fetchVotingRecords({ bet, limit: 200 }); + } catch (err) { + // Distinguish fetch failures (transient/network/MCP errors) from a + // confirmed empty voting record. `'not_found'` is reserved for the + // case where the MCP returned successfully but had zero data, while + // `'error'` carries the error message for downstream annotation. + const message = err instanceof Error ? err.message : String(err); + process.stderr.write(`⚠️ fetch-voting-records: MCP error for bet=${bet}: ${message}\n`); + const errorRecord: VotingRecordOutput = { + bet, + rm: null, + fetchedAt, + status: 'error', + partyVotes: [], + defectors: [], + mermaidDiagram: generateMermaidVoteChart([], bet), + errorMessage: message, + }; + errorRecord.injectionMarkdown = buildInjectionMarkdown(errorRecord); + return errorRecord; + } + + if (partyVotes.length === 0 && rawIndividualVotes.length === 0) { + // MCP returned successfully but with zero data — treat as a confirmed + // empty result (`not_found`) so downstream can distinguish from + // `vote_pending` (used by editorial tooling that *knows* a vote is + // upcoming) and `error` (transient MCP/network failure). + const notFoundRecord: VotingRecordOutput = { + bet, + rm: rm || null, + fetchedAt, + status: 'not_found', + partyVotes: [], + defectors: [], + mermaidDiagram: generateMermaidVoteChart([], bet), + }; + notFoundRecord.injectionMarkdown = buildInjectionMarkdown(notFoundRecord); + return notFoundRecord; + } + + const { defectors } = detectDefectors(rawIndividualVotes); + const mermaidDiagram = generateMermaidVoteChart(partyVotes, bet); + + const record: VotingRecordOutput = { + bet, + rm: rm || null, + fetchedAt, + status: 'fetched', + partyVotes, + defectors, + mermaidDiagram, + }; + + record.injectionMarkdown = buildInjectionMarkdown(record); + return record; +} + +// --------------------------------------------------------------------------- +// Main entry point +// --------------------------------------------------------------------------- + +async function main(): Promise { + const { args, error } = parseArgs(process.argv.slice(2)); + if (error) { + process.stderr.write(`fetch-voting-records: ${error}\n`); + process.exit(2); + } + + const { date, docType, persist } = args; + + // Locate manifests + const dailyRoot = path.join(REPO_ROOT, 'analysis', 'daily', date); + const manifestPaths: string[] = []; + + if (docType) { + const p = path.join(dailyRoot, docType, 'data-download-manifest.md'); + if (fs.existsSync(p)) manifestPaths.push(p); + } else { + // Scan all subdirectories + if (fs.existsSync(dailyRoot)) { + for (const entry of fs.readdirSync(dailyRoot, { withFileTypes: true })) { + if (entry.isDirectory()) { + const p = path.join(dailyRoot, entry.name, 'data-download-manifest.md'); + if (fs.existsSync(p)) manifestPaths.push(p); + } + } + } + } + + // Also check root manifest + const rootManifest = path.join(dailyRoot, 'data-download-manifest.md'); + if (fs.existsSync(rootManifest)) manifestPaths.push(rootManifest); + + if (manifestPaths.length === 0) { + process.stderr.write( + `fetch-voting-records: no data-download-manifest.md found under analysis/daily/${date}/\n`, + ); + process.exit(1); + } + + // Collect all unique bet values + const allBets = new Set(); + for (const manifestPath of manifestPaths) { + const text = fs.readFileSync(manifestPath, 'utf8'); + for (const bet of extractBetValues(text)) { + allBets.add(bet); + } + } + + if (allBets.size === 0) { + process.stderr.write(`fetch-voting-records: no beteckning values found in manifests\n`); + process.stdout.write(JSON.stringify({ date, bets: [], results: [] }, null, 2) + '\n'); + process.exit(0); + } + + process.stderr.write( + `fetch-voting-records: fetching ${allBets.size} bet(s): ${[...allBets].join(', ')}\n`, + ); + + const client = new MCPClient(); + const results: VotingRecordOutput[] = []; + const outDir = path.join(REPO_ROOT, 'data', 'voteringar', date); + fs.mkdirSync(outDir, { recursive: true }); + + for (const bet of allBets) { + const record = await fetchVotingForBet(client, bet); + results.push(record); + + // Write per-bet output + const outFile = path.join(outDir, `${sanitizeBet(bet)}.json`); + fs.writeFileSync(outFile, JSON.stringify(record, null, 2) + '\n', 'utf8'); + process.stderr.write(` ✓ ${bet} → ${path.relative(REPO_ROOT, outFile)} [${record.status}]\n`); + + // Write injection templates if --persist + if (persist && record.injectionMarkdown) { + // Determine which docType directory to write into + const docTypeDirs = docType + ? [path.join(dailyRoot, docType)] + : manifestPaths + .map((p) => path.dirname(p)) + .filter((d) => !d.endsWith(date)); // skip root + + for (const dtDir of docTypeDirs) { + const injectionDir = path.join(dtDir, 'voting-records'); + fs.mkdirSync(injectionDir, { recursive: true }); + const injectionFile = path.join(injectionDir, `${sanitizeBet(bet)}.md`); + fs.writeFileSync(injectionFile, record.injectionMarkdown + '\n', 'utf8'); + process.stderr.write( + ` 📄 injection → ${path.relative(REPO_ROOT, injectionFile)}\n`, + ); + } + } + } + + process.stdout.write( + JSON.stringify({ date, bets: [...allBets], results }, null, 2) + '\n', + ); +} + +// Run if this is the entry point +const isMain = + process.argv[1] !== undefined && + (process.argv[1].endsWith('fetch-voting-records.ts') || + process.argv[1].endsWith('fetch-voting-records.js')); + +if (isMain) { + main().catch((err: unknown) => { + process.stderr.write(`fetch-voting-records: fatal error: ${String(err)}\n`); + process.exit(1); + }); +} diff --git a/tests/fetch-calendar.test.ts b/tests/fetch-calendar.test.ts new file mode 100644 index 000000000..bc7f00ac9 --- /dev/null +++ b/tests/fetch-calendar.test.ts @@ -0,0 +1,387 @@ +/** + * @file tests/fetch-calendar.test.ts + * @description Vitest unit tests for fetch-calendar.ts + */ + +import { describe, it, expect, vi } from 'vitest'; + +import { + parseArgs, + parseCalendarHtml, + fetchCalendarEvents, + type CalendarEvent, + type CalendarOutput, + type FetchCalendarDeps, + type ParsedCalendarArgs, +} from '../scripts/fetch-calendar.js'; + +// --------------------------------------------------------------------------- +// parseArgs tests +// --------------------------------------------------------------------------- + +describe('parseArgs — fetch-calendar', () => { + it('parses --from and --tom happy path', () => { + const { args, error } = parseArgs(['--from', '2026-04-27', '--tom', '2026-05-27']); + expect(error).toBeNull(); + expect(args.from).toBe('2026-04-27'); + expect(args.tom).toBe('2026-05-27'); + expect(args.org).toBeNull(); + expect(args.akt).toBeNull(); + expect(args.persist).toBe(false); + }); + + it('parses optional --org, --akt, --persist flags', () => { + const { args, error } = parseArgs([ + '--from', '2026-04-27', + '--tom', '2026-05-27', + '--org', 'UTSK', + '--akt', 'bet', + '--persist', + ]); + expect(error).toBeNull(); + expect(args.org).toBe('UTSK'); + expect(args.akt).toBe('bet'); + expect(args.persist).toBe(true); + }); + + it('returns error when --from is missing', () => { + const { error } = parseArgs(['--tom', '2026-05-27']); + expect(error).not.toBeNull(); + expect(error).toMatch(/--from/); + }); + + it('returns error when --tom is missing', () => { + const { error } = parseArgs(['--from', '2026-04-27']); + expect(error).not.toBeNull(); + expect(error).toMatch(/--tom/); + }); + + it('returns error for invalid --from date format', () => { + const { error } = parseArgs(['--from', '04/27/2026', '--tom', '2026-05-27']); + expect(error).not.toBeNull(); + expect(error).toMatch(/YYYY-MM-DD/); + }); + + it('returns error for invalid --tom date format', () => { + const { error } = parseArgs(['--from', '2026-04-27', '--tom', 'next-month']); + expect(error).not.toBeNull(); + expect(error).toMatch(/YYYY-MM-DD/); + }); + + it('persist defaults to false when flag is absent', () => { + const { args } = parseArgs(['--from', '2026-04-27', '--tom', '2026-05-27']); + expect(args.persist).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// parseCalendarHtml tests +// --------------------------------------------------------------------------- + +describe('parseCalendarHtml', () => { + it('returns empty array for blank HTML', () => { + expect(parseCalendarHtml('')).toHaveLength(0); + expect(parseCalendarHtml('')).toHaveLength(0); + }); + + it('extracts events from JSON-LD structured data', () => { + const html = ` + + + + + +`; + + const events = parseCalendarHtml(html); + expect(events.length).toBeGreaterThanOrEqual(1); + const ev = events[0]!; + expect(ev.titel).toBe('Finansutskottets öppna utfrågning'); + expect(ev.datum).toBe('2026-05-05'); + expect(ev.tid).toBe('10:00'); + expect(ev.org).toBe('Finansutskottet'); + expect(ev.typ).toBe('Utfrågning'); + }); + + it('extracts events from multiple JSON-LD Event objects', () => { + const html = ` +`; + + const events = parseCalendarHtml(html); + expect(events.length).toBeGreaterThanOrEqual(2); + const titles = events.map((e) => e.titel); + expect(titles).toContain('Event A'); + expect(titles).toContain('Event B'); + }); + + it('falls back gracefully when JSON-LD parse fails', () => { + // Malformed JSON-LD should not throw + const html = ``; + expect(() => parseCalendarHtml(html)).not.toThrow(); + }); + + it('extracts events using HTML title patterns when no JSON-LD present', () => { + const html = ` + +
+ Debatt om budgeten + +
+`; + + // Should not throw; events may or may not be found depending on HTML pattern + expect(() => parseCalendarHtml(html)).not.toThrow(); + const events = parseCalendarHtml(html); + expect(Array.isArray(events)).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// fetchCalendarEvents — orchestrator (real logic exercised via injected deps) +// --------------------------------------------------------------------------- + +const baseArgs: ParsedCalendarArgs = { + from: '2026-04-27', + tom: '2026-05-27', + org: null, + akt: null, + persist: false, +}; + +function makeDeps(overrides: Partial): FetchCalendarDeps { + return { + fetchViaMcp: overrides.fetchViaMcp ?? (async () => []), + fetchViaWeb: overrides.fetchViaWeb ?? (async () => []), + now: overrides.now ?? (() => new Date('2026-04-27T00:00:00.000Z')), + logger: overrides.logger, + }; +} + +describe('fetchCalendarEvents — MCP primary path', () => { + it('source is "mcp" when MCP returns events and web is never called', async () => { + const mcpEvents: CalendarEvent[] = [ + { datum: '2026-05-05', tid: '10:00', org: 'FiU', titel: 'Utfrågning', typ: 'Öppet' }, + ]; + const webSpy = vi.fn(async () => [] as CalendarEvent[]); + + const out = await fetchCalendarEvents( + baseArgs, + makeDeps({ fetchViaMcp: async () => mcpEvents, fetchViaWeb: webSpy }), + ); + + expect(out.source).toBe('mcp'); + expect(out.events).toEqual(mcpEvents); + expect(webSpy).not.toHaveBeenCalled(); + expect(out.fetchedAt).toMatch(/^\d{4}-\d{2}-\d{2}T/); + }); + + it('falls back to web when MCP throws', async () => { + const webEvents: CalendarEvent[] = [ + { datum: '2026-05-06', tid: '14:00', org: 'KU', titel: 'Möte', typ: '' }, + ]; + + const out = await fetchCalendarEvents( + baseArgs, + makeDeps({ + fetchViaMcp: async () => { + throw new Error('mcp down'); + }, + fetchViaWeb: async () => webEvents, + }), + ); + + expect(out.source).toBe('web_fallback'); + expect(out.events).toEqual(webEvents); + }); +}); + +describe('fetchCalendarEvents — web fallback path', () => { + it('source is "web_fallback" when MCP returns empty', async () => { + const webEvents: CalendarEvent[] = [ + { datum: '2026-05-07', tid: '09:00', org: 'AU', titel: 'Debatt', typ: '' }, + ]; + + const out = await fetchCalendarEvents( + baseArgs, + makeDeps({ + fetchViaMcp: async () => [], + fetchViaWeb: async () => webEvents, + }), + ); + + expect(out.source).toBe('web_fallback'); + expect(out.events).toEqual(webEvents); + }); + + it('gracefully degrades to empty events array when web fetch fails', async () => { + const out = await fetchCalendarEvents( + baseArgs, + makeDeps({ + fetchViaMcp: async () => [], + fetchViaWeb: async () => { + throw new Error('web down'); + }, + }), + ); + + expect(out.source).toBe('web_fallback'); + expect(out.events).toEqual([]); + expect(out.fetchedAt).toMatch(/^\d{4}-\d{2}-\d{2}T/); + }); + + it('preserves args.from / args.tom on the output', async () => { + const out = await fetchCalendarEvents( + { ...baseArgs, from: '2026-06-01', tom: '2026-06-30' }, + makeDeps({}), + ); + expect(out.from).toBe('2026-06-01'); + expect(out.tom).toBe('2026-06-30'); + }); +}); + +// Backwards-compatibility shape assertions (object literal, no logic) +describe('CalendarOutput object shape', () => { + it('accepts a fully-populated CalendarOutput literal', () => { + const out: CalendarOutput = { + from: '2026-04-27', + tom: '2026-05-27', + fetchedAt: new Date().toISOString(), + source: 'mcp', + events: [], + }; + expect(out.source).toBe('mcp'); + }); +}); + +// --------------------------------------------------------------------------- +// HTML error response fixture +// --------------------------------------------------------------------------- + +describe('HTML error response handling', () => { + it('parseCalendarHtml handles 404-style HTML body gracefully', () => { + const notFoundHtml = ` + + +404 – Sidan hittades inte + +

Sidan hittades inte

+

Den begärda sidan kunde inte hittas.

+ +`; + + // Should not throw; should return empty or near-empty events + expect(() => parseCalendarHtml(notFoundHtml)).not.toThrow(); + const events = parseCalendarHtml(notFoundHtml); + expect(Array.isArray(events)).toBe(true); + }); + + it('parseCalendarHtml handles server-error HTML gracefully', () => { + const errorHtml = ` + + +500 Internal Server Error + +

Internal Server Error

+ +`; + + expect(() => parseCalendarHtml(errorHtml)).not.toThrow(); + const events = parseCalendarHtml(errorHtml); + expect(Array.isArray(events)).toBe(true); + }); + + it('parseCalendarHtml handles empty string without crashing', () => { + expect(() => parseCalendarHtml('')).not.toThrow(); + expect(parseCalendarHtml('')).toEqual([]); + }); +}); + +// --------------------------------------------------------------------------- +// Output structure validation +// --------------------------------------------------------------------------- + +describe('CalendarOutput structure', () => { + it('output has all required fields', () => { + const output: CalendarOutput = { + from: '2026-04-27', + tom: '2026-05-27', + fetchedAt: '2026-04-27T10:00:00.000Z', + source: 'mcp', + events: [ + { + datum: '2026-05-05', + tid: '10:00', + org: 'FiU', + titel: 'Utfrågning om statsbudgeten', + typ: 'Öppet', + }, + ], + }; + + expect(output).toHaveProperty('from'); + expect(output).toHaveProperty('tom'); + expect(output).toHaveProperty('fetchedAt'); + expect(output).toHaveProperty('source'); + expect(output).toHaveProperty('events'); + expect(Array.isArray(output.events)).toBe(true); + }); + + it('event has all required fields', () => { + const event = { + datum: '2026-05-05', + tid: '10:00', + org: 'FiU', + titel: 'Utfrågning', + typ: 'Öppet', + }; + + expect(event).toHaveProperty('datum'); + expect(event).toHaveProperty('tid'); + expect(event).toHaveProperty('org'); + expect(event).toHaveProperty('titel'); + expect(event).toHaveProperty('typ'); + }); + + it('source must be "mcp" or "web_fallback"', () => { + const validSources: string[] = ['mcp', 'web_fallback']; + const output: CalendarOutput = { + from: '2026-04-27', + tom: '2026-05-27', + fetchedAt: new Date().toISOString(), + source: 'mcp', + events: [], + }; + + expect(validSources).toContain(output.source); + }); + + it('fetchedAt is a valid ISO timestamp', () => { + const output: CalendarOutput = { + from: '2026-04-27', + tom: '2026-05-27', + fetchedAt: new Date().toISOString(), + source: 'web_fallback', + events: [], + }; + + expect(output.fetchedAt).toMatch(/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}/); + // `new Date(string)` never throws (it returns Invalid Date), so use + // Date.parse to actually validate the timestamp. + expect(Number.isNaN(Date.parse(output.fetchedAt))).toBe(false); + }); +}); diff --git a/tests/fetch-voting-records.test.ts b/tests/fetch-voting-records.test.ts new file mode 100644 index 000000000..c2945eeaf --- /dev/null +++ b/tests/fetch-voting-records.test.ts @@ -0,0 +1,377 @@ +/** + * @file tests/fetch-voting-records.test.ts + * @description Vitest unit tests for fetch-voting-records.ts + */ + +import { describe, it, expect } from 'vitest'; +import fs from 'node:fs'; +import path from 'node:path'; + +import { + parseArgs, + detectDefectors, + generateMermaidVoteChart, + extractBetValues, + sanitizeBet, + type PartyVoteRow, +} from '../scripts/fetch-voting-records.js'; + +// --------------------------------------------------------------------------- +// parseArgs tests +// --------------------------------------------------------------------------- + +describe('parseArgs — fetch-voting-records', () => { + it('parses --date and --doc-type and --persist happy path', () => { + const { args, error } = parseArgs(['--date', '2026-04-27', '--doc-type', 'committeeReports', '--persist']); + expect(error).toBeNull(); + expect(args.date).toBe('2026-04-27'); + expect(args.docType).toBe('committeeReports'); + expect(args.persist).toBe(true); + }); + + it('defaults date to today when not provided', () => { + const { args, error } = parseArgs([]); + expect(error).toBeNull(); + // Today is always YYYY-MM-DD format + expect(args.date).toMatch(/^\d{4}-\d{2}-\d{2}$/); + expect(args.docType).toBeNull(); + expect(args.persist).toBe(false); + }); + + it('returns error for invalid date format', () => { + const { error } = parseArgs(['--date', '27-04-2026']); + expect(error).not.toBeNull(); + expect(error).toMatch(/YYYY-MM-DD/); + }); + + it('returns error for non-date string', () => { + const { error } = parseArgs(['--date', 'yesterday']); + expect(error).not.toBeNull(); + expect(error).toMatch(/YYYY-MM-DD/); + }); + + it('parses --date without --doc-type', () => { + const { args, error } = parseArgs(['--date', '2026-01-15']); + expect(error).toBeNull(); + expect(args.date).toBe('2026-01-15'); + expect(args.docType).toBeNull(); + }); + + it('persist defaults to false when flag is absent', () => { + const { args, error } = parseArgs(['--date', '2026-04-27']); + expect(error).toBeNull(); + expect(args.persist).toBe(false); + }); +}); + +// --------------------------------------------------------------------------- +// detectDefectors tests +// --------------------------------------------------------------------------- + +describe('detectDefectors', () => { + it('returns vote_pending status when input is empty', () => { + const { defectors, status } = detectDefectors([]); + expect(status).toBe('vote_pending'); + expect(defectors).toHaveLength(0); + }); + + it('detects an MP voting Nej when party majority is Ja', () => { + const votes = [ + { parti: 'S', rost: 'Ja', iid: '001', intressentNamn: 'Anna Svensson' }, + { parti: 'S', rost: 'Ja', iid: '002', intressentNamn: 'Björn Karlsson' }, + { parti: 'S', rost: 'Ja', iid: '003', intressentNamn: 'Carla Lindqvist' }, + { parti: 'S', rost: 'Nej', iid: '004', intressentNamn: 'David Eriksson' }, + { parti: 'M', rost: 'Nej', iid: '005', intressentNamn: 'Eva Johansson' }, + { parti: 'M', rost: 'Nej', iid: '006', intressentNamn: 'Fredrik Nilsson' }, + ]; + + const { defectors, status } = detectDefectors(votes); + expect(status).toBe('fetched'); + expect(defectors).toHaveLength(1); + expect(defectors[0]).toMatchObject({ + iid: '004', + intressentNamn: 'David Eriksson', + parti: 'S', + rost: 'Nej', + partyMajority: 'Ja', + }); + }); + + it('detects no defectors when all MPs vote with their party', () => { + const votes = [ + { parti: 'S', rost: 'Ja', iid: '001', intressentNamn: 'A' }, + { parti: 'S', rost: 'Ja', iid: '002', intressentNamn: 'B' }, + { parti: 'M', rost: 'Nej', iid: '003', intressentNamn: 'C' }, + { parti: 'M', rost: 'Nej', iid: '004', intressentNamn: 'D' }, + ]; + + const { defectors, status } = detectDefectors(votes); + expect(status).toBe('fetched'); + expect(defectors).toHaveLength(0); + }); + + it('excludes Frånvarande from defector list', () => { + const votes = [ + { parti: 'SD', rost: 'Ja', iid: '001', intressentNamn: 'A' }, + { parti: 'SD', rost: 'Ja', iid: '002', intressentNamn: 'B' }, + { parti: 'SD', rost: 'Frånvarande', iid: '003', intressentNamn: 'C' }, + ]; + + const { defectors } = detectDefectors(votes); + expect(defectors).toHaveLength(0); + }); + + it('handles multiple defectors across multiple parties', () => { + const votes = [ + { parti: 'C', rost: 'Ja', iid: '1', intressentNamn: 'X' }, + { parti: 'C', rost: 'Ja', iid: '2', intressentNamn: 'Y' }, + { parti: 'C', rost: 'Nej', iid: '3', intressentNamn: 'Z' }, // defector + { parti: 'L', rost: 'Avstår', iid: '4', intressentNamn: 'W' }, + { parti: 'L', rost: 'Ja', iid: '5', intressentNamn: 'V' }, // defector + { parti: 'L', rost: 'Ja', iid: '6', intressentNamn: 'U' }, // defector + ]; + + const { defectors } = detectDefectors(votes); + // Z defects from C (majority Ja), W defects from L (majority Ja) + const defectorIds = defectors.map((d) => d.iid).sort(); + expect(defectorIds).toContain('3'); // Z (C) voted Nej, majority Ja + expect(defectorIds).toContain('4'); // W (L) voted Avstår, majority Ja + }); + + it('handles votes with alternative field names', () => { + const votes = [ + { party: 'V', vote: 'Ja', iid: 'v1', namn: 'Person A' }, + { party: 'V', vote: 'Ja', iid: 'v2', namn: 'Person B' }, + { party: 'V', vote: 'Nej', iid: 'v3', namn: 'Person C' }, + ]; + + const { defectors, status } = detectDefectors(votes); + expect(status).toBe('fetched'); + expect(defectors).toHaveLength(1); + expect(defectors[0]?.rost).toBe('Nej'); + }); +}); + +// --------------------------------------------------------------------------- +// generateMermaidVoteChart tests +// --------------------------------------------------------------------------- + +describe('generateMermaidVoteChart', () => { + it('produces a valid mermaid string for party votes', () => { + const partyVotes: PartyVoteRow[] = [ + { parti: 'S', ja: 100, nej: 0, avstar: 0, franvarande: 7 }, + { parti: 'M', ja: 0, nej: 68, avstar: 0, franvarande: 3 }, + { parti: 'SD', ja: 0, nej: 54, avstar: 0, franvarande: 2 }, + ]; + + const diagram = generateMermaidVoteChart(partyVotes, 'FiU48'); + expect(diagram).toContain('xychart-beta'); + expect(diagram).toContain('FiU48'); + expect(diagram).toContain('"S"'); + expect(diagram).toContain('"M"'); + expect(diagram).toContain('"SD"'); + expect(diagram).toContain('100'); + }); + + it('returns a flowchart-style diagram when there are no party votes', () => { + const diagram = generateMermaidVoteChart([], 'AU10'); + expect(diagram).toContain('AU10'); + // Should be a valid mermaid snippet (not throw) + expect(typeof diagram).toBe('string'); + expect(diagram.length).toBeGreaterThan(0); + }); + + it('includes x-axis and y-axis labels', () => { + const partyVotes: PartyVoteRow[] = [ + { parti: 'KD', ja: 16, nej: 0, avstar: 2, franvarande: 1 }, + ]; + const diagram = generateMermaidVoteChart(partyVotes, 'SoU12'); + expect(diagram).toContain('x-axis'); + expect(diagram).toContain('y-axis'); + }); +}); + +// --------------------------------------------------------------------------- +// extractBetValues tests +// --------------------------------------------------------------------------- + +describe('extractBetValues', () => { + it('extracts beteckning values from manifest text', () => { + const manifest = ` +# Data Download Manifest + +Focus betänkanden: FiU48, AU10, KU20 + +| dok_id | Title | +|--------|-------| +| FiU48 | Budget | +| AU10 | Work | +`; + const bets = extractBetValues(manifest); + expect(bets).toContain('FiU48'); + expect(bets).toContain('AU10'); + expect(bets).toContain('KU20'); + }); + + it('filters out common false-positive acronyms', () => { + const manifest = 'See ISO standards and HTTP protocol for CSV format. Also JSON and MCP.'; + const bets = extractBetValues(manifest); + expect(bets).not.toContain('ISO'); + expect(bets).not.toContain('HTTP'); + expect(bets).not.toContain('CSV'); + expect(bets).not.toContain('JSON'); + expect(bets).not.toContain('MCP'); + }); + + it('returns empty array for manifest with no beteckning values', () => { + const manifest = '# Daily Report\n\nNo committee reports today.'; + const bets = extractBetValues(manifest); + expect(bets).toHaveLength(0); + }); +}); + +// --------------------------------------------------------------------------- +// sanitizeBet tests +// --------------------------------------------------------------------------- + +describe('sanitizeBet', () => { + it('lowercases and preserves alphanumeric content', () => { + expect(sanitizeBet('FiU48')).toBe('fiu48'); + }); + + it('replaces special characters with hyphens', () => { + expect(sanitizeBet('AU 10')).toBe('au-10'); + }); + + it('collapses multiple hyphens', () => { + expect(sanitizeBet('KU--20')).toBe('ku-20'); + }); +}); + +// --------------------------------------------------------------------------- +// Contract test — any betänkande cited in intelligence-assessment.md +// must have a voting-record with a status field +// --------------------------------------------------------------------------- + +describe('contract: voting records have status field', () => { + const dailyRoot = path.resolve('analysis', 'daily'); + const voterRoot = path.resolve('data', 'voteringar'); + + it('every voting record JSON in data/voteringar/ has a status field', () => { + if (!fs.existsSync(voterRoot)) { + // No voting records yet — pass vacuously + return; + } + + const dateEntries = fs + .readdirSync(voterRoot, { withFileTypes: true }) + .filter((e) => e.isDirectory()); + + for (const dateEntry of dateEntries) { + const dateDir = path.join(voterRoot, dateEntry.name); + const jsonFiles = fs + .readdirSync(dateDir) + .filter((f) => f.endsWith('.json')); + + for (const jsonFile of jsonFiles) { + const filePath = path.join(dateDir, jsonFile); + let parsed: unknown; + try { + parsed = JSON.parse(fs.readFileSync(filePath, 'utf8')); + } catch (error: unknown) { + // Contract test: malformed voting-record JSON is a hard failure, + // not something to silently skip. + const message = error instanceof Error ? error.message : String(error); + throw new Error( + `Malformed voting record JSON in ${filePath}: ${message}`, + { cause: error }, + ); + } + + // `typeof null === 'object'` and arrays are also objects, so + // `toBeTypeOf('object')` would silently accept either and then + // fail with a confusing TypeError on the next line. Require a + // non-null, non-array plain object up front. + expect( + parsed !== null && typeof parsed === 'object' && !Array.isArray(parsed), + `${jsonFile} should be a non-null plain object, got: ${JSON.stringify(parsed)}`, + ).toBe(true); + + const record = parsed as Record; + expect( + record['status'], + `${jsonFile} must have a status field`, + ).toBeDefined(); + + expect( + ['fetched', 'vote_pending', 'not_found', 'error'], + `${jsonFile} status must be a valid value`, + ).toContain(record['status']); + } + } + }); + + it('cited betänkanden have either a voting record or an explicit pending/not-found/error annotation', () => { + if (!fs.existsSync(dailyRoot)) return; + + const dateDirs = fs + .readdirSync(dailyRoot, { withFileTypes: true }) + .filter((e) => e.isDirectory()); + + for (const dateDir of dateDirs) { + const date = dateDir.name; + const baseDir = path.join(dailyRoot, date); + const voterDir = path.join(voterRoot, date); + + // Only enforce the annotation contract for dates where + // fetch-voting-records has actually run (i.e. a per-date voter + // directory exists). This avoids retroactive failures on historical + // assessments authored before this script was introduced. + if (!fs.existsSync(voterDir)) continue; + + // Find all intelligence-assessment.md files + const assessmentFiles: string[] = []; + for (const entry of fs.readdirSync(baseDir, { withFileTypes: true })) { + if (entry.isDirectory()) { + const asmPath = path.join(baseDir, entry.name, 'intelligence-assessment.md'); + if (fs.existsSync(asmPath)) assessmentFiles.push(asmPath); + } + } + const rootAsm = path.join(baseDir, 'intelligence-assessment.md'); + if (fs.existsSync(rootAsm)) assessmentFiles.push(rootAsm); + + for (const asmFile of assessmentFiles) { + const content = fs.readFileSync(asmFile, 'utf8'); + const bets = extractBetValues(content); + + for (const bet of bets) { + const votingFile = path.join(voterDir, `${sanitizeBet(bet)}.json`); + + if (fs.existsSync(votingFile)) { + const record = JSON.parse(fs.readFileSync(votingFile, 'utf8')) as Record; + expect( + record['status'], + `Voting record for ${bet} (${date}) must have a status field`, + ).toBeDefined(); + } else { + // No voting file → assessment must carry an explicit annotation + // so readers know whether the vote is pending, missing from the + // API, or failed to fetch. + const votePendingAnnotation = ``; + const voteNotFoundAnnotation = ``; + const voteFetchErrorAnnotation = ``; + + expect( + content.includes(votePendingAnnotation) || + content.includes(voteNotFoundAnnotation) || + content.includes(voteFetchErrorAnnotation), + `Missing voting record for ${bet} (${date}) must be annotated with ` + + `${votePendingAnnotation}, ${voteNotFoundAnnotation} or ` + + `${voteFetchErrorAnnotation} in ${asmFile}`, + ).toBe(true); + } + } + } + } + }); +}); diff --git a/vitest.config.js b/vitest.config.js index f897b81b0..637715dc1 100644 --- a/vitest.config.js +++ b/vitest.config.js @@ -121,6 +121,8 @@ export default defineConfig({ 'scripts/validate-methodology-reflection.ts', 'scripts/catalog-downloaded-data.ts', 'scripts/download-parliamentary-data.ts', + 'scripts/fetch-calendar.ts', + 'scripts/fetch-voting-records.ts', 'scripts/imf-fetch.ts', 'scripts/statskontoret-fetch.ts', 'scripts/mcp-query-cli.ts',