diff --git a/devtools/compare-rendering/.gitignore b/devtools/compare-rendering/.gitignore new file mode 100644 index 0000000000..ceddaa37f1 --- /dev/null +++ b/devtools/compare-rendering/.gitignore @@ -0,0 +1 @@ +.cache/ diff --git a/devtools/compare-rendering/README.md b/devtools/compare-rendering/README.md new file mode 100644 index 0000000000..e3e35f1d63 --- /dev/null +++ b/devtools/compare-rendering/README.md @@ -0,0 +1,121 @@ +# compare-rendering + +Diffs Word and SuperDoc rendering of the same `.docx` at the *resolved schema* level — text, page assignment, and (in later milestones) font/indent/color/numbering. Emits typed `Finding[]` so an agent can route fixes to specific SuperDoc modules. + +This is a dev tool, not a pass/fail test. It surfaces concrete divergences so you don't have to compare screenshots by eye. + +## Scope (M1) + +- **Supported:** paragraph-only documents (text-heavy memos, letters, policies). +- **Short-circuited with a reason:** docs containing tables, inline/floating shapes, or tracked changes. The report emits an `unsupported` finding and skips the diff — honest boundary rather than a misleading "everything looks fine." +- **Categories emitted in M1:** `text`, `pagination`, `structure`, `unsupported`. Style/indent/color/numbering come in M2 once the SuperDoc-side normalizer pulls resolved values out of `measures[]` and `runs[]`. + +## Quick start + +```bash +export WORD_API_URL="https://word-mcp.superdoc.workers.dev" +export WORD_API_TOKEN="" + +pnpm compare-rendering -- \ + --input evals/fixtures/docs/memorandum.docx \ + --format md +``` + +Run directly without the wrapper: + +```bash +bun devtools/compare-rendering/src/cli.ts --input --format md +``` + +Example output (truncated): + +```markdown +# compare-rendering: memorandum.docx + +- Word pages: 3, SuperDoc pages: 3 +- Word paragraphs: 94, SuperDoc paragraphs: 94 + +## Findings (2) + +### pagination (2) +- **[visible]** Paragraph #39 landed on page 1 in SuperDoc but page 2 in Word (empty line) + - spec: ECMA-376 §17.3.1.16 (keepNext/keepLines/pageBreakBefore) + - code: `layout-engine/layout-engine/src/pagination` +- **[visible]** Paragraph #80 landed on page 2 in SuperDoc but page 3 in Word (" - Any press releases…") + - spec: ECMA-376 §17.3.1.16 (keepNext/keepLines/pageBreakBefore) + - code: `layout-engine/layout-engine/src/pagination` +``` + +## How it works + +``` +docx + ├── word adapter (POST /v1/executions to word-api) ─► word.json (cached) + └── superdoc adapter (spawn pnpm layout:export-one) ─► sd.layout.json + │ + normalize both sides + │ + NormalizedParagraph[] × 2 + │ + differ + taxonomy + │ + Finding[] report +``` + +- Word extraction is **cached** by `sha256(docx) + sha256(extract-layout.ps1)`. Editing SuperDoc code and re-running the tool only re-runs the SuperDoc side — no re-hit to the VM (~25s saved per iteration). Editing the PowerShell script busts the cache automatically. +- Bypass the cache for a single run with `--no-cache`. + +## Env + +| Variable | Purpose | +|------------------|------------------------------------------------------| +| `WORD_API_URL` | Base URL of the word-api worker | +| `WORD_API_TOKEN` | Bearer token | + +## Exit codes + +- `0` — ran successfully; findings are at most `visible`/`cosmetic` (or no findings at all) +- `1` — tool error (network, missing input, bad args) +- `2` — ran successfully but emitted at least one `blocking` finding + +Makes it CI-usable later without rework. + +## Non-goals + +- Pixel diffing (see `tests/visual/`). +- Tables, images, shapes, track changes, headers/footers, comments, TOC — deferred past M5. +- Auto-fix generation. +- Publishing as a package. + +## Milestones + +- **M1** ✅ — CLI on paragraph-only docs. 4 categories (`text`, `pagination`, `structure`, `unsupported`). Word-extraction cache. +- **M2** ✅ — Baseline + delta reporting (`--input-dir`, `--save-baseline`, `--baseline`). Findings get a stable `fingerprint` (`category:paragraphOrdinal`). Delta mode emits only `resolved` / `new` / `unchanged` vs baseline; exits `2` on any new finding. This is what makes the tool **agent-usable** — signal is "my change fixed N, broke M" instead of absolute findings. +- **M3** — LLM screenshot judge for docs where schema diff is silent or near-silent. Catches rendering divergences that don't surface in layout data at all (e.g. `w:val="wave"` border styles rendered as plain lines, font substitution, painter-level overflow). +- **M4** — Populate `NormalizedParagraph.resolved` on SuperDoc side. Taxonomy extends to `style`, `indent`, `font`, `color`, `alignment`, `spacing`, `numbering`. Safe to add now that M2 absorbs the "new field adds findings everywhere" noise. +- **M5** — Table support. Non-trivial; needs parallel table walks on both sides. + +## Insights from M1 corpus batch (75 docs, April 2026) + +- **Pagination findings compound.** Many "N pagination findings" collapse to one underlying bug expressed N times. `memorandum.docx` (3 findings) and `sd-1741-paragraph-between-borders` (36 findings) share the same root cause — SuperDoc fits slightly more content per page than Word; drift accumulates across pages. One fix likely eliminates most findings at once. +- **Schema diff has real false negatives.** `sd-1741` reports 0 text/style findings, but visually SuperDoc renders every border-between style (`wave`, `doubleWave`, `dashDotStroked`, `triple`, …) as a plain line while Word renders each correctly. Schema-level comparison will never catch this class without the M3 screenshot judge. +- **~27 % of the corpus is in M1 scope.** 13 / 75 docs are short-circuited for tables/shapes/comments/revisions; the rest yield meaningful findings. Real-world DOCX coverage unlocks at M5 (tables). + +## Corpus sweep + baselines + +Pass `--input-dir` to run a whole directory of docs. Combine with `--save-baseline` to snapshot the current findings, and `--baseline` to diff a later run against that snapshot. + +```bash +# Snapshot current state as the main-branch baseline (once, on main). +pnpm compare-rendering -- \ + --input-dir test-corpus/rendering \ + --save-baseline test-corpus/.baseline.json + +# On a feature branch: what did my change actually affect? +pnpm compare-rendering -- \ + --input-dir test-corpus/rendering \ + --baseline test-corpus/.baseline.json \ + --format md +``` + +Delta output names the docs with `resolved` (baseline had it, current doesn't → you fixed it) and `new` (current has it, baseline didn't → you introduced or didn't fix it). `unchanged` is counted but not listed. Exit `2` when any new finding shows up — CI-friendly gate. diff --git a/devtools/compare-rendering/package.json b/devtools/compare-rendering/package.json new file mode 100644 index 0000000000..efcbbdcb52 --- /dev/null +++ b/devtools/compare-rendering/package.json @@ -0,0 +1,10 @@ +{ + "private": true, + "type": "module", + "name": "compare-rendering", + "scripts": { + "start": "bun src/cli.ts", + "typecheck": "tsc --noEmit", + "test": "vitest run" + } +} diff --git a/devtools/compare-rendering/src/baseline.ts b/devtools/compare-rendering/src/baseline.ts new file mode 100644 index 0000000000..763244b9f7 --- /dev/null +++ b/devtools/compare-rendering/src/baseline.ts @@ -0,0 +1,85 @@ +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import { dirname } from 'node:path'; +import type { Baseline, CompareReport, DeltaReport, Finding } from './types.ts'; + +const CURRENT_SCHEMA_VERSION = 1 as const; + +export async function readBaseline(path: string): Promise { + const raw = JSON.parse(await readFile(path, 'utf8')); + if (raw?.schemaVersion !== CURRENT_SCHEMA_VERSION) { + throw new Error(`baseline ${path}: unsupported schemaVersion ${raw?.schemaVersion}`); + } + return raw as Baseline; +} + +export async function writeBaseline(path: string, reports: CompareReport[]): Promise { + const baseline: Baseline = { + schemaVersion: CURRENT_SCHEMA_VERSION, + capturedAt: new Date().toISOString(), + docs: {}, + }; + for (const r of reports) { + const key = baselineKey(r.docxPath); + baseline.docs[key] = { docxSha: r.docxSha, findings: r.findings }; + } + await mkdir(dirname(path), { recursive: true }); + await writeFile(path, `${JSON.stringify(baseline, null, 2)}\n`, 'utf8'); +} + +/** + * Diff a fresh set of reports against a baseline. Findings are keyed by + * `fingerprint` within each doc — same fingerprint in both → unchanged; + * only in baseline → resolved; only in current → new. + * + * Docs present in current but not in baseline contribute all their findings + * as new (the doc itself is new to the corpus). Docs present in baseline + * but not in current are ignored — they're a batch-scope issue, not a + * regression in behavior. + */ +export function diffAgainstBaseline(reports: CompareReport[], baseline: Baseline): DeltaReport { + const docs: DeltaReport['docs'] = []; + let totalResolved = 0; + let totalNew = 0; + let totalUnchanged = 0; + + for (const r of reports) { + const key = baselineKey(r.docxPath); + const baselineDoc = baseline.docs[key]; + const baselineByFp = new Map(); + if (baselineDoc) for (const f of baselineDoc.findings) baselineByFp.set(f.fingerprint, f); + + const currentByFp = new Map(); + for (const f of r.findings) currentByFp.set(f.fingerprint, f); + + const resolved: Finding[] = []; + const fresh: Finding[] = []; + let unchanged = 0; + + for (const [fp, f] of baselineByFp) { + if (!currentByFp.has(fp)) resolved.push(f); + } + for (const [fp, f] of currentByFp) { + if (baselineByFp.has(fp)) unchanged += 1; + else fresh.push(f); + } + + if (resolved.length || fresh.length || unchanged) { + docs.push({ file: key, resolved, new: fresh, unchangedCount: unchanged }); + } + totalResolved += resolved.length; + totalNew += fresh.length; + totalUnchanged += unchanged; + } + + return { + baselineCapturedAt: baseline.capturedAt, + totals: { resolved: totalResolved, new: totalNew, unchanged: totalUnchanged }, + docs, + }; +} + +/** Normalize a docx path to a stable baseline key (basename). */ +function baselineKey(docxPath: string): string { + const i = Math.max(docxPath.lastIndexOf('/'), docxPath.lastIndexOf('\\')); + return i === -1 ? docxPath : docxPath.slice(i + 1); +} diff --git a/devtools/compare-rendering/src/cache.ts b/devtools/compare-rendering/src/cache.ts new file mode 100644 index 0000000000..284bd94134 --- /dev/null +++ b/devtools/compare-rendering/src/cache.ts @@ -0,0 +1,36 @@ +import { createHash } from 'node:crypto'; +import { mkdir, readFile, stat, writeFile } from 'node:fs/promises'; +import { fileURLToPath } from 'node:url'; +import { dirname, join } from 'node:path'; + +const CACHE_DIR = fileURLToPath(new URL('../.cache/word', import.meta.url)); + +export function sha256(bytes: Uint8Array | string): string { + const h = createHash('sha256'); + h.update(bytes); + return h.digest('hex'); +} + +export async function hashFile(path: string): Promise { + return sha256(await readFile(path)); +} + +function cachePath(sha: string, keySuffix: string): string { + return join(CACHE_DIR, `${sha}-${keySuffix}.json`); +} + +export async function readCache(sha: string, keySuffix: string): Promise { + const p = cachePath(sha, keySuffix); + try { + await stat(p); + } catch { + return null; + } + return JSON.parse(await readFile(p, 'utf8')) as T; +} + +export async function writeCache(sha: string, keySuffix: string, value: T): Promise { + const p = cachePath(sha, keySuffix); + await mkdir(dirname(p), { recursive: true }); + await writeFile(p, JSON.stringify(value), 'utf8'); +} diff --git a/devtools/compare-rendering/src/cli.ts b/devtools/compare-rendering/src/cli.ts new file mode 100644 index 0000000000..b8fc01df30 --- /dev/null +++ b/devtools/compare-rendering/src/cli.ts @@ -0,0 +1,239 @@ +#!/usr/bin/env bun +import { parseArgs as nodeParseArgs } from 'node:util'; +import { readdir, writeFile } from 'node:fs/promises'; +import { resolve, join } from 'node:path'; +import { extractWord } from './word.ts'; +import { extractSuperDoc } from './superdoc.ts'; +import { normalizeSuperDoc, normalizeWord } from './normalize.ts'; +import { diffParagraphs } from './differ.ts'; +import { formatDeltaJson, formatDeltaMarkdown, formatJson, formatMarkdown } from './format.ts'; +import { diffAgainstBaseline, readBaseline, writeBaseline } from './baseline.ts'; +import type { CompareReport, DeltaReport, Finding } from './types.ts'; + +type Args = { + input?: string; + inputDir?: string; + output?: string; + format: 'json' | 'md'; + pipeline: 'presentation' | 'headless'; + cache: boolean; + baseline?: string; + saveBaseline?: string; +}; + +const USAGE = `compare-rendering — diff Word vs SuperDoc rendering (paragraph-only scope) + +Usage: + pnpm compare-rendering -- --input [options] + pnpm compare-rendering -- --input-dir [options] + +Options: + --input Path to a .docx file (single-doc mode). + --input-dir Directory of .docx files (corpus mode). + --output Write report to file (default: stdout). + --format json|md Output format (default: json). + --pipeline presentation|headless SuperDoc layout pipeline (default: presentation). + --no-cache Bypass the Word extraction cache. + --baseline Compare current run against a baseline; emit delta. + --save-baseline Run and write findings as a baseline snapshot. + -h, --help Show this help. + +Env: + WORD_API_URL Base URL of the word-api worker. + WORD_API_TOKEN Bearer token for the worker. + +Exit codes: + 0 — ran; no blocking findings, no regressions vs baseline. + 1 — tool error (network, missing file, bad args). + 2 — ran; emitted blocking finding, or new findings vs baseline.`; + +function parseArgs(argv: string[]): Args { + const { values } = nodeParseArgs({ + args: argv, + options: { + input: { type: 'string' }, + 'input-dir': { type: 'string' }, + output: { type: 'string' }, + format: { type: 'string', default: 'json' }, + pipeline: { type: 'string', default: 'presentation' }, + 'no-cache': { type: 'boolean', default: false }, + baseline: { type: 'string' }, + 'save-baseline': { type: 'string' }, + help: { type: 'boolean', short: 'h', default: false }, + }, + strict: true, + allowPositionals: false, + }); + + if (values.help) { + console.log(USAGE); + process.exit(0); + } + + if (!values.input && !values['input-dir']) { + throw new Error('one of --input or --input-dir is required'); + } + if (values.input && values['input-dir']) { + throw new Error('--input and --input-dir are mutually exclusive'); + } + if (values.format !== 'json' && values.format !== 'md') { + throw new Error(`--format must be json or md, got "${values.format}"`); + } + if (values.pipeline !== 'presentation' && values.pipeline !== 'headless') { + throw new Error(`--pipeline must be presentation or headless, got "${values.pipeline}"`); + } + if (values.baseline && values['save-baseline']) { + throw new Error('--baseline and --save-baseline are mutually exclusive'); + } + + return { + input: values.input, + inputDir: values['input-dir'], + output: values.output, + format: values.format, + pipeline: values.pipeline, + cache: !values['no-cache'], + baseline: values.baseline, + saveBaseline: values['save-baseline'], + }; +} + +function hasBlocking(findings: Finding[]): boolean { + return findings.some((f) => f.severity === 'blocking'); +} + +const log = (msg: string) => console.error(`[compare-rendering] ${msg}`); + +async function compareOne(docxPath: string, args: Args): Promise { + const absPath = resolve(docxPath); + const wordStart = Date.now(); + const { extraction: wordExtraction, sha, cached } = await extractWord(absPath, { cache: args.cache }); + log(`word: ${cached ? 'cached' : 'fresh'} extraction in ${Date.now() - wordStart}ms (sha=${sha.slice(0, 12)})`); + + if (!wordExtraction.supported) { + return { + docxPath: absPath, + docxSha: sha, + wordSupported: false, + unsupportedReason: wordExtraction.unsupportedReason, + counts: { + wordParagraphs: 0, + superdocParagraphs: 0, + wordPages: wordExtraction.pageCount, + superdocPages: 0, + }, + findings: [ + { + fingerprint: 'unsupported:0', + category: 'unsupported', + severity: 'cosmetic', + paragraphOrdinal: 0, + word: wordExtraction.unsupportedReason, + superdoc: null, + message: `Document skipped: ${wordExtraction.unsupportedReason ?? 'unsupported'}`, + }, + ], + }; + } + + const sdStart = Date.now(); + const sdExtraction = await extractSuperDoc(absPath, { pipeline: args.pipeline }); + log(`superdoc: extracted in ${Date.now() - sdStart}ms`); + + const wordParas = normalizeWord(wordExtraction); + const sdParas = normalizeSuperDoc(sdExtraction); + const findings = diffParagraphs(wordParas, sdParas); + + return { + docxPath: absPath, + docxSha: sha, + wordSupported: true, + counts: { + wordParagraphs: wordParas.length, + superdocParagraphs: sdParas.length, + wordPages: wordExtraction.pageCount, + superdocPages: sdExtraction.pageCount, + }, + findings, + }; +} + +async function listDocxFiles(dir: string): Promise { + const entries = await readdir(dir); + return entries + .filter((f) => f.toLowerCase().endsWith('.docx')) + .sort() + .map((f) => join(dir, f)); +} + +async function main(): Promise { + const args = parseArgs(process.argv.slice(2)); + + const files = args.input ? [resolve(args.input)] : await listDocxFiles(resolve(args.inputDir!)); + if (files.length === 0) { + throw new Error(`no .docx files found in ${args.inputDir}`); + } + + log(`running ${files.length} doc(s)`); + const reports: CompareReport[] = []; + for (let i = 0; i < files.length; i += 1) { + const f = files[i]!; + log(`[${i + 1}/${files.length}] ${f.split('/').pop()}`); + reports.push(await compareOne(f, args)); + } + + if (args.saveBaseline) { + await writeBaseline(resolve(args.saveBaseline), reports); + log(`wrote baseline to ${resolve(args.saveBaseline)}`); + return; + } + + if (args.baseline) { + const baseline = await readBaseline(resolve(args.baseline)); + const delta = diffAgainstBaseline(reports, baseline); + await emitDelta(delta, args); + log(`resolved=${delta.totals.resolved} new=${delta.totals.new} unchanged=${delta.totals.unchanged}`); + if (delta.totals.new > 0) process.exitCode = 2; + return; + } + + // Default: emit one report (single-doc) or a JSON array (corpus). + if (reports.length === 1) { + await emit(reports[0]!, args); + if (hasBlocking(reports[0]!.findings)) process.exitCode = 2; + } else { + const out = `${JSON.stringify(reports, null, 2)}\n`; + if (args.output) { + await writeFile(resolve(args.output), out, 'utf8'); + log(`wrote ${resolve(args.output)}`); + } else { + process.stdout.write(out); + } + if (reports.some((r) => hasBlocking(r.findings))) process.exitCode = 2; + } +} + +async function emit(report: CompareReport, args: Args): Promise { + const out = args.format === 'md' ? formatMarkdown(report) : formatJson(report); + if (args.output) { + await writeFile(resolve(args.output), out, 'utf8'); + log(`wrote ${resolve(args.output)}`); + } else { + process.stdout.write(out); + } +} + +async function emitDelta(delta: DeltaReport, args: Args): Promise { + const out = args.format === 'md' ? formatDeltaMarkdown(delta) : formatDeltaJson(delta); + if (args.output) { + await writeFile(resolve(args.output), out, 'utf8'); + log(`wrote ${resolve(args.output)}`); + } else { + process.stdout.write(out); + } +} + +main().catch((e) => { + console.error(`[compare-rendering] error: ${(e as Error).message}`); + process.exit(1); +}); diff --git a/devtools/compare-rendering/src/differ.test.ts b/devtools/compare-rendering/src/differ.test.ts new file mode 100644 index 0000000000..5bfc9a5e1c --- /dev/null +++ b/devtools/compare-rendering/src/differ.test.ts @@ -0,0 +1,208 @@ +import { describe, expect, it } from 'vitest'; +import { diffParagraphs, fingerprintOf } from './differ.ts'; +import { oleToHex, pxToPt, wordAlignment, wordTri } from './normalize.ts'; +import { codeAreaFor, specRefFor } from './taxonomy.ts'; +import { diffAgainstBaseline } from './baseline.ts'; +import type { Baseline, CompareReport, NormalizedParagraph } from './types.ts'; + +const para = (overrides: Partial = {}): NormalizedParagraph => ({ + ordinal: 1, + text: 'hello', + style: 'Normal', + page: 1, + y: 72, + ...overrides, +}); + +describe('oleToHex', () => { + it('converts 0x00BBGGRR to #RRGGBB', () => { + expect(oleToHex(0x000000ff)).toBe('#FF0000'); // red + expect(oleToHex(0x0000ff00)).toBe('#00FF00'); // green + expect(oleToHex(0x00ff0000)).toBe('#0000FF'); // blue + expect(oleToHex(0)).toBe('#000000'); + }); + + it("handles Word's negative wdColorAutomatic sentinel", () => { + // -16777216 === 0xFF000000 in 32-bit; low 24 bits are 0 → maps to #000000 today. + // Flagged as M2 work: distinguish auto-color from explicit black. + expect(oleToHex(-16777216)).toBe('#000000'); + }); +}); + +describe('wordTri', () => { + it("maps 9999999 (wdUndefined) to 'mixed'", () => { + expect(wordTri(9999999)).toBe('mixed'); + }); + it('coerces other numbers to boolean', () => { + expect(wordTri(0)).toBe(false); + expect(wordTri(-1)).toBe(true); + expect(wordTri(1)).toBe(true); + }); +}); + +describe('wordAlignment', () => { + it("maps 0/1/2/3 and falls back to 'unknown'", () => { + expect(wordAlignment(0)).toBe('left'); + expect(wordAlignment(1)).toBe('center'); + expect(wordAlignment(2)).toBe('right'); + expect(wordAlignment(3)).toBe('justify'); + expect(wordAlignment(42)).toBe('unknown'); + }); +}); + +describe('pxToPt', () => { + it('converts 96px to 72pt', () => { + expect(pxToPt(96)).toBe(72); + expect(pxToPt(0)).toBe(0); + }); +}); + +describe('diffParagraphs', () => { + it('emits no findings when both sides agree', () => { + const p = [para({ ordinal: 1, text: 'a' }), para({ ordinal: 2, text: 'b', y: 100 })]; + expect(diffParagraphs(p, p)).toEqual([]); + }); + + it('ignores whitespace-only text differences', () => { + const w = [para({ text: ' hello world ' })]; + const s = [para({ text: 'hello world' })]; + expect(diffParagraphs(w, s)).toEqual([]); + }); + + it("emits one blocking 'structure' finding on paragraph-count mismatch and suppresses per-paragraph findings", () => { + const w = [para({ text: 'a' }), para({ text: 'b', page: 1 })]; + const s = [para({ text: 'different' })]; + const findings = diffParagraphs(w, s); + expect(findings).toHaveLength(1); + expect(findings[0]!.category).toBe('structure'); + expect(findings[0]!.severity).toBe('blocking'); + expect(findings[0]!.codeAreaHint).toBeDefined(); + }); + + it("emits a blocking 'text' finding when aligned paragraphs differ in text", () => { + const w = [para({ text: 'word side' })]; + const s = [para({ text: 'superdoc side' })]; + const findings = diffParagraphs(w, s); + expect(findings).toHaveLength(1); + expect(findings[0]!.category).toBe('text'); + expect(findings[0]!.severity).toBe('blocking'); + expect(findings[0]!.specRef).toMatch(/ECMA-376/); + }); + + it("emits a visible 'pagination' finding with rounded y and page info when pages differ", () => { + const w = [para({ text: 'same', page: 2, y: 100.456 })]; + const s = [para({ text: 'same', page: 1, y: 700.123 })]; + const findings = diffParagraphs(w, s); + expect(findings).toHaveLength(1); + const f = findings[0]!; + expect(f.category).toBe('pagination'); + expect(f.severity).toBe('visible'); + expect(f.word).toEqual({ page: 2, y: 100.5 }); + expect(f.superdoc).toEqual({ page: 1, y: 700.1 }); + expect(f.message).toContain('"same"'); + }); + + it("uses '(empty line)' message when the Word-side text is empty", () => { + const w = [para({ text: '', page: 2 })]; + const s = [para({ text: '', page: 1 })]; + const findings = diffParagraphs(w, s); + expect(findings[0]!.message).toContain('(empty line)'); + }); +}); + +describe('taxonomy', () => { + it('returns non-undefined hints for M1 categories', () => { + expect(codeAreaFor('pagination')).toBeDefined(); + expect(codeAreaFor('text')).toBeDefined(); + expect(codeAreaFor('structure')).toBeDefined(); + expect(specRefFor('text')).toMatch(/ECMA-376/); + expect(specRefFor('pagination')).toMatch(/ECMA-376/); + }); +}); + +describe('fingerprintOf', () => { + it('is stable and collision-free per (category, ordinal)', () => { + expect(fingerprintOf('pagination', 39)).toBe('pagination:39'); + expect(fingerprintOf('text', 39)).not.toBe(fingerprintOf('pagination', 39)); + expect(fingerprintOf('pagination', 39)).toBe(fingerprintOf('pagination', 39)); + }); + + it('is set on every finding emitted by diffParagraphs', () => { + const w = [para({ text: 'a', page: 1 }), para({ ordinal: 2, text: 'b', page: 2 })]; + const s = [para({ text: 'a', page: 1 }), para({ ordinal: 2, text: 'DIFFERENT', page: 1 })]; + for (const f of diffParagraphs(w, s)) { + expect(f.fingerprint).toBeTruthy(); + expect(f.fingerprint).toBe(`${f.category}:${f.paragraphOrdinal}`); + } + }); +}); + +describe('diffAgainstBaseline', () => { + const makeReport = ( + file: string, + findings: NormalizedParagraph[] extends unknown ? number[] : never = [], + ): CompareReport => { + // findings argument not used below; we construct findings explicitly per test + void findings; + return { + docxPath: `/abs/path/${file}`, + docxSha: 'sha', + wordSupported: true, + counts: { wordParagraphs: 0, superdocParagraphs: 0, wordPages: 1, superdocPages: 1 }, + findings: [], + }; + }; + + const mkFinding = (cat: 'pagination' | 'text', ordinal: number) => ({ + fingerprint: fingerprintOf(cat, ordinal), + category: cat, + severity: 'visible' as const, + paragraphOrdinal: ordinal, + word: null, + superdoc: null, + message: `${cat} at #${ordinal}`, + }); + + it('classifies findings as resolved / new / unchanged', () => { + const baseline: Baseline = { + schemaVersion: 1, + capturedAt: '2026-01-01T00:00:00Z', + docs: { + 'memo.docx': { + docxSha: 'sha', + findings: [mkFinding('pagination', 39), mkFinding('pagination', 80)], + }, + }, + }; + const report = makeReport('memo.docx'); + report.findings = [mkFinding('pagination', 39), mkFinding('text', 42)]; + const delta = diffAgainstBaseline([report], baseline); + + expect(delta.totals).toEqual({ resolved: 1, new: 1, unchanged: 1 }); + expect(delta.docs).toHaveLength(1); + expect(delta.docs[0]!.resolved.map((f) => f.fingerprint)).toEqual(['pagination:80']); + expect(delta.docs[0]!.new.map((f) => f.fingerprint)).toEqual(['text:42']); + expect(delta.docs[0]!.unchangedCount).toBe(1); + }); + + it('treats docs not in baseline as all-new', () => { + const baseline: Baseline = { schemaVersion: 1, capturedAt: 'x', docs: {} }; + const report = makeReport('new-doc.docx'); + report.findings = [mkFinding('pagination', 5)]; + const delta = diffAgainstBaseline([report], baseline); + expect(delta.totals.new).toBe(1); + expect(delta.totals.resolved).toBe(0); + }); + + it('emits empty delta when nothing changed', () => { + const baseline: Baseline = { + schemaVersion: 1, + capturedAt: 'x', + docs: { 'x.docx': { docxSha: 's', findings: [mkFinding('pagination', 1)] } }, + }; + const report = makeReport('x.docx'); + report.findings = [mkFinding('pagination', 1)]; + const delta = diffAgainstBaseline([report], baseline); + expect(delta.totals).toEqual({ resolved: 0, new: 0, unchanged: 1 }); + }); +}); diff --git a/devtools/compare-rendering/src/differ.ts b/devtools/compare-rendering/src/differ.ts new file mode 100644 index 0000000000..6df6e322cf --- /dev/null +++ b/devtools/compare-rendering/src/differ.ts @@ -0,0 +1,87 @@ +import type { Finding, FindingCategory, NormalizedParagraph } from './types.ts'; +import { codeAreaFor, specRefFor } from './taxonomy.ts'; + +/** + * A finding's stable key across runs. Same paragraph + same category = + * same fingerprint, even if the value (e.g. y-offset) drifts. This is + * what baseline diffing keys on. + */ +export function fingerprintOf(category: FindingCategory, paragraphOrdinal: number): string { + return `${category}:${paragraphOrdinal}`; +} + +export function diffParagraphs(word: NormalizedParagraph[], superdoc: NormalizedParagraph[]): Finding[] { + const findings: Finding[] = []; + + if (word.length !== superdoc.length) { + findings.push({ + fingerprint: fingerprintOf('structure', 0), + category: 'structure', + severity: 'blocking', + paragraphOrdinal: 0, + word: word.length, + superdoc: superdoc.length, + message: + `Paragraph count differs: Word=${word.length} SuperDoc=${superdoc.length}. ` + + `Per-paragraph findings suppressed because ordinal alignment cannot be trusted.`, + codeAreaHint: codeAreaFor('structure'), + }); + return findings; + } + + const n = word.length; + for (let i = 0; i < n; i += 1) { + const w = word[i]!; + const s = superdoc[i]!; + + if (!textsMatch(w.text, s.text)) { + findings.push({ + fingerprint: fingerprintOf('text', w.ordinal), + category: 'text', + severity: 'blocking', + paragraphOrdinal: w.ordinal, + word: truncate(w.text), + superdoc: truncate(s.text), + message: `Paragraph #${w.ordinal} text differs.`, + specRef: specRefFor('text'), + codeAreaHint: codeAreaFor('text'), + }); + } + + if (w.page !== s.page) { + findings.push({ + fingerprint: fingerprintOf('pagination', w.ordinal), + category: 'pagination', + severity: 'visible', + paragraphOrdinal: w.ordinal, + word: { page: w.page, y: round(w.y) }, + superdoc: { page: s.page, y: round(s.y) }, + message: + `Paragraph #${w.ordinal} landed on page ${s.page} in SuperDoc but page ${w.page} in Word` + + (w.text ? ` ("${truncate(w.text, 40)}")` : ' (empty line)'), + specRef: specRefFor('pagination'), + codeAreaHint: codeAreaFor('pagination'), + }); + } + } + + return findings; +} + +function textsMatch(a: string, b: string): boolean { + return normalizeText(a) === normalizeText(b); +} + +function normalizeText(s: string): string { + return s.replace(/\s+/g, ' ').trim(); +} + +function truncate(s: string, n = 80): string { + if (!s) return ''; + return s.length > n ? `${s.slice(0, n)}…` : s; +} + +function round(v: number, digits = 1): number { + const m = 10 ** digits; + return Math.round(v * m) / m; +} diff --git a/devtools/compare-rendering/src/extract-layout.ps1 b/devtools/compare-rendering/src/extract-layout.ps1 new file mode 100644 index 0000000000..7de238ef47 --- /dev/null +++ b/devtools/compare-rendering/src/extract-layout.ps1 @@ -0,0 +1,103 @@ +# extract-layout.ps1 — SuperDoc compare-rendering +# +# Reads $b64 (docx as base64) from the calling scope, decodes to a temp file, +# opens it in Word, and emits a JSON snapshot of resolved paragraph-level state +# between JSON_BEGIN / JSON_END markers. Tables/shapes/revisions short-circuit. +# +# Cache invalidation is automatic: word.ts hashes this file's bytes into the +# cache key, so any edit here busts the cache on the next run. + +$ErrorActionPreference = 'Stop' +$word = $null +$doc = $null +$inputPath = "C:\word-mcp\compare-input-$([guid]::NewGuid().ToString('N')).docx" + +try { + [IO.File]::WriteAllBytes($inputPath, [Convert]::FromBase64String($b64)) + + $word = New-Object -ComObject Word.Application + $word.Visible = $false + $word.DisplayAlerts = 0 + + $doc = $word.Documents.Open($inputPath) + [void]$doc.Fields.Update() + try { $word.ActiveWindow.View.RevisionsView = 0 } catch {} + + # Short-circuit unsupported features + $unsupported = $null + if ($doc.Tables.Count -gt 0) { $unsupported = "contains tables ($($doc.Tables.Count))" } + elseif ($doc.InlineShapes.Count -gt 0) { $unsupported = "contains inline shapes ($($doc.InlineShapes.Count))" } + elseif ($doc.Shapes.Count -gt 0) { $unsupported = "contains floating shapes ($($doc.Shapes.Count))" } + elseif ($doc.Revisions.Count -gt 0) { $unsupported = "contains tracked changes ($($doc.Revisions.Count))" } + elseif ($doc.Comments.Count -gt 0) { $unsupported = "contains comments ($($doc.Comments.Count))" } + + if ($unsupported) { + $result = [ordered]@{ + supported = $false + unsupportedReason = $unsupported + pageCount = [int]$doc.ComputeStatistics(2) + paragraphs = @() + } + Write-Output "JSON_BEGIN" + Write-Output ($result | ConvertTo-Json -Depth 25 -Compress) + Write-Output "JSON_END" + Write-Output "SUCCESS" + return + } + + $paragraphs = New-Object System.Collections.ArrayList + $pIdx = 0 + foreach ($p in $doc.Paragraphs) { + $pIdx++ + if ($p.Range.Information(12)) { continue } # wdWithInTable (defensive; already short-circuited) + + $r = $p.Range + $font = $r.Font + $fmt = $p.Format + $lf = $r.ListFormat + + $txt = $r.Text + if ($txt) { $txt = $txt.TrimEnd([char]13, [char]7, [char]11) } + + $rgb = -16777216 + try { $rgb = [int]$font.TextColor.RGB } catch {} + + [void]$paragraphs.Add([ordered]@{ + idx = $pIdx + text = $txt + style = $p.Style.NameLocal + fontName = $font.Name + fontSize = [double]$font.Size + bold = $font.Bold + italic = $font.Italic + colorRgb = $rgb + alignment = $fmt.Alignment + leftIndent = [double]$fmt.LeftIndent + firstLineIndent = [double]$fmt.FirstLineIndent + listString = $lf.ListString + listLevel = $lf.ListLevelNumber + page = [int]$r.Information(1) + y = [double]$r.Information(6) + }) + } + + $result = [ordered]@{ + supported = $true + pageCount = [int]$doc.ComputeStatistics(2) + paragraphs = $paragraphs + } + + Write-Output "JSON_BEGIN" + Write-Output ($result | ConvertTo-Json -Depth 25 -Compress) + Write-Output "JSON_END" + Write-Output "SUCCESS" +} +catch { + Write-Output ("ERROR: " + $_.Exception.Message) + Write-Output ("AT: " + $_.InvocationInfo.PositionMessage) +} +finally { + if ($doc) { try { [void]$doc.Close(0) } catch {} } + if ($word) { try { [void]$word.Quit(0) } catch {} } + Remove-Item $inputPath -Force -ErrorAction SilentlyContinue +} diff --git a/devtools/compare-rendering/src/format.ts b/devtools/compare-rendering/src/format.ts new file mode 100644 index 0000000000..ae537f16cb --- /dev/null +++ b/devtools/compare-rendering/src/format.ts @@ -0,0 +1,105 @@ +import type { CompareReport, DeltaReport } from './types.ts'; + +export function formatJson(report: CompareReport): string { + return `${JSON.stringify(report, null, 2)}\n`; +} + +export function formatMarkdown(report: CompareReport): string { + const lines: string[] = []; + lines.push(`# compare-rendering: ${basename(report.docxPath)}`); + lines.push(''); + lines.push(`- sha256: \`${report.docxSha}\``); + lines.push(`- Word pages: ${report.counts.wordPages}, SuperDoc pages: ${report.counts.superdocPages}`); + lines.push( + `- Word paragraphs: ${report.counts.wordParagraphs}, SuperDoc paragraphs: ${report.counts.superdocParagraphs}`, + ); + lines.push(''); + + if (!report.wordSupported) { + lines.push(`> **Skipped**: ${report.unsupportedReason ?? 'unsupported document'}`); + lines.push(''); + return lines.join('\n'); + } + + if (report.findings.length === 0) { + lines.push('No findings — Word and SuperDoc agree on paragraph text and page assignment.'); + lines.push(''); + return lines.join('\n'); + } + + const byCategory = groupBy(report.findings, (f) => f.category); + lines.push(`## Findings (${report.findings.length})`); + for (const [cat, findings] of byCategory) { + lines.push(''); + lines.push(`### ${cat} (${findings.length})`); + for (const f of findings) { + lines.push(`- **[${f.severity}]** ${f.message}`); + if (f.specRef) lines.push(` - spec: ${f.specRef}`); + if (f.codeAreaHint) lines.push(` - code: \`${f.codeAreaHint}\``); + } + } + lines.push(''); + return lines.join('\n'); +} + +export function formatDeltaJson(delta: DeltaReport): string { + return `${JSON.stringify(delta, null, 2)}\n`; +} + +export function formatDeltaMarkdown(delta: DeltaReport): string { + const lines: string[] = []; + const { resolved, new: fresh, unchanged } = delta.totals; + lines.push(`# compare-rendering: delta vs baseline`); + lines.push(''); + lines.push(`Baseline captured: ${delta.baselineCapturedAt}`); + lines.push(''); + lines.push(`**Resolved**: ${resolved} · **New**: ${fresh} · **Unchanged**: ${unchanged}`); + lines.push(''); + + const withResolved = delta.docs.filter((d) => d.resolved.length); + const withNew = delta.docs.filter((d) => d.new.length); + + if (withResolved.length) { + lines.push(`## Resolved (${resolved}) — your change fixed these`); + for (const d of withResolved) { + lines.push(`- ${d.file} (${d.resolved.length})`); + for (const f of d.resolved) lines.push(` - [${f.severity}] ${f.message}`); + } + lines.push(''); + } + + if (withNew.length) { + lines.push(`## New (${fresh}) — your change introduced these or didn't fix them`); + for (const d of withNew) { + lines.push(`- ${d.file} (${d.new.length})`); + for (const f of d.new) { + lines.push(` - [${f.severity}] ${f.message}`); + if (f.codeAreaHint) lines.push(` - code: \`${f.codeAreaHint}\``); + } + } + lines.push(''); + } + + if (!withResolved.length && !withNew.length) { + lines.push('No change vs baseline — nothing fixed, nothing broken.'); + lines.push(''); + } + + return lines.join('\n'); +} + +function basename(p: string): string { + const i = Math.max(p.lastIndexOf('/'), p.lastIndexOf('\\')); + return i === -1 ? p : p.slice(i + 1); +} + +function groupBy(xs: T[], key: (x: T) => K): Map { + const m = new Map(); + for (const x of xs) { + const k = key(x); + const bucket = m.get(k) ?? []; + bucket.push(x); + m.set(k, bucket); + } + return m; +} diff --git a/devtools/compare-rendering/src/normalize.ts b/devtools/compare-rendering/src/normalize.ts new file mode 100644 index 0000000000..59bf59422e --- /dev/null +++ b/devtools/compare-rendering/src/normalize.ts @@ -0,0 +1,88 @@ +import type { NormalizedParagraph, ResolvedStyle, TriState, WordExtraction, WordParagraph } from './types.ts'; +import type { SuperDocExtraction } from './superdoc.ts'; + +const WD_UNDEFINED = 9999999; + +export function normalizeWord(extraction: WordExtraction): NormalizedParagraph[] { + return extraction.paragraphs.map((p, i) => wordParagraphToNormalized(p, i + 1)); +} + +function wordParagraphToNormalized(p: WordParagraph, ordinal: number): NormalizedParagraph { + return { + ordinal, + text: p.text ?? '', + style: p.style, + resolved: { + fontName: p.fontName, + fontSize: p.fontSize, + bold: wordTri(p.bold), + italic: wordTri(p.italic), + color: oleToHex(p.colorRgb), + leftIndent: p.leftIndent, + firstLineIndent: p.firstLineIndent, + alignment: wordAlignment(p.alignment), + listString: p.listString ?? '', + listLevel: p.listLevel, + }, + page: p.page, + y: p.y, + }; +} + +function wordTri(v: number): TriState { + if (v === WD_UNDEFINED) return 'mixed'; + return Boolean(v); +} + +function wordAlignment(v: number): ResolvedStyle['alignment'] { + switch (v) { + case 0: + return 'left'; + case 1: + return 'center'; + case 2: + return 'right'; + case 3: + return 'justify'; + default: + return 'unknown'; + } +} + +function oleToHex(ole: number): string { + // Word's Font.TextColor.RGB returns 0x00BBGGRR. Convert to #RRGGBB. + const r = ole & 0xff; + const g = (ole >> 8) & 0xff; + const b = (ole >> 16) & 0xff; + return `#${[r, g, b] + .map((n) => n.toString(16).padStart(2, '0')) + .join('') + .toUpperCase()}`; +} + +export function normalizeSuperDoc(extraction: SuperDocExtraction): NormalizedParagraph[] { + const out: NormalizedParagraph[] = []; + let ordinal = 0; + for (const block of extraction.blocks) { + if (block.kind !== 'paragraph') continue; + ordinal += 1; + const bid = block.id ?? ''; + const text = (block.runs ?? []).map((r) => r.text ?? '').join(''); + const y_px = extraction.blockY[bid]; + const page = extraction.blockPage[bid] ?? 0; + out.push({ + ordinal, + text, + style: '', + page, + y: y_px !== undefined ? pxToPt(y_px) : 0, + }); + } + return out; +} + +export { pxToPt, oleToHex, wordTri, wordAlignment }; + +function pxToPt(px: number): number { + return (px * 72) / 96; +} diff --git a/devtools/compare-rendering/src/superdoc.ts b/devtools/compare-rendering/src/superdoc.ts new file mode 100644 index 0000000000..3371c9f452 --- /dev/null +++ b/devtools/compare-rendering/src/superdoc.ts @@ -0,0 +1,92 @@ +import { spawn } from 'node:child_process'; +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +type SuperDocSnapshot = { + layoutSnapshot: { + layout: { pages: SuperDocPage[] }; + blocks: SuperDocBlock[]; + }; +}; + +type SuperDocPage = { + number: number; + fragments: Array<{ blockId: string; y: number }>; +}; + +type SuperDocBlock = { + kind: 'paragraph' | 'sectionBreak' | string; + id?: string; + runs?: Array<{ text?: string }>; + attrs?: Record; +}; + +export type SuperDocExtraction = { + blocks: SuperDocBlock[]; + blockPage: Record; + blockY: Record; + pageCount: number; +}; + +const REPO_ROOT = fileURLToPath(new URL('../../../', import.meta.url)); + +export async function extractSuperDoc( + docxPath: string, + opts: { pipeline?: 'presentation' | 'headless' } = {}, +): Promise { + const pipeline = opts.pipeline ?? 'presentation'; + const tmp = await mkdtemp(join(tmpdir(), 'compare-sd-')); + const outputPath = join(tmp, 'snapshot.layout.json'); + + try { + await runLayoutExport(docxPath, outputPath, pipeline); + const raw = JSON.parse(await readFile(outputPath, 'utf8')) as SuperDocSnapshot; + return normalizeSuperDocSnapshot(raw); + } finally { + await rm(tmp, { recursive: true, force: true }).catch(() => {}); + } +} + +function runLayoutExport(input: string, output: string, pipeline: string): Promise { + return new Promise((resolve, reject) => { + const child = spawn( + 'pnpm', + ['layout:export-one', '--', '--input', input, '--output', output, '--pipeline', pipeline], + { cwd: REPO_ROOT, stdio: ['ignore', 'pipe', 'pipe'] }, + ); + + let stderr = ''; + child.stderr.on('data', (c) => { + stderr += String(c); + }); + child.on('error', reject); + child.on('close', (code) => { + if (code === 0) resolve(); + else reject(new Error(`layout:export-one exited ${code}:\n${stderr.trim().slice(-1000)}`)); + }); + }); +} + +function normalizeSuperDocSnapshot(raw: SuperDocSnapshot): SuperDocExtraction { + const blocks = raw.layoutSnapshot.blocks; + const blockPage: Record = {}; + const blockY: Record = {}; + + for (const pg of raw.layoutSnapshot.layout.pages) { + for (const frag of pg.fragments) { + if (!(frag.blockId in blockPage)) { + blockPage[frag.blockId] = pg.number; + blockY[frag.blockId] = frag.y; + } + } + } + + return { + blocks, + blockPage, + blockY, + pageCount: raw.layoutSnapshot.layout.pages.length, + }; +} diff --git a/devtools/compare-rendering/src/taxonomy.ts b/devtools/compare-rendering/src/taxonomy.ts new file mode 100644 index 0000000000..6054114bdb --- /dev/null +++ b/devtools/compare-rendering/src/taxonomy.ts @@ -0,0 +1,41 @@ +import type { FindingCategory } from './types.ts'; + +/** + * Seed taxonomy: maps a finding category to a SuperDoc code area the reader + * should inspect. The agent consuming these findings can use the hint to + * route investigation without re-deriving it from prose. + * + * Keep hints coarse on purpose — file trees move, packages do not. + */ +const CODE_AREAS: Partial> = { + text: 'super-editor/src/editors/v1/core/super-converter', + pagination: 'layout-engine/layout-engine/src/pagination', + structure: 'super-editor/src/editors/v1/core/super-converter', + style: 'layout-engine/style-engine', + indent: 'layout-engine/style-engine', + numbering: 'layout-engine/style-engine (numbering resolution)', + font: 'layout-engine/style-engine (font resolution)', + color: 'layout-engine/style-engine (color resolution)', + alignment: 'layout-engine/style-engine', + spacing: 'layout-engine/style-engine', +}; + +const SPEC_REFS: Partial> = { + text: 'ECMA-376 §17.3.1 (run content)', + pagination: 'ECMA-376 §17.3.1.16 (keepNext/keepLines/pageBreakBefore)', + style: 'ECMA-376 §17.7 (style definitions)', + indent: 'ECMA-376 §17.3.1.12 (w:ind)', + numbering: 'ECMA-376 §17.9 (numbering definitions)', + font: 'ECMA-376 §17.3.2 (run properties)', + color: 'ECMA-376 §17.3.2.6 (w:color)', + alignment: 'ECMA-376 §17.3.1.13 (w:jc)', + spacing: 'ECMA-376 §17.3.1.33 (w:spacing)', +}; + +export function codeAreaFor(category: FindingCategory): string | undefined { + return CODE_AREAS[category]; +} + +export function specRefFor(category: FindingCategory): string | undefined { + return SPEC_REFS[category]; +} diff --git a/devtools/compare-rendering/src/types.ts b/devtools/compare-rendering/src/types.ts new file mode 100644 index 0000000000..fd5c18806b --- /dev/null +++ b/devtools/compare-rendering/src/types.ts @@ -0,0 +1,116 @@ +export type TriState = boolean | 'mixed'; + +export type ResolvedStyle = { + fontName: string; + fontSize: number; + bold: TriState; + italic: TriState; + color: string; + leftIndent: number; + firstLineIndent: number; + alignment: 'left' | 'right' | 'center' | 'justify' | 'unknown'; + listString: string; + listLevel: number; +}; + +export type NormalizedParagraph = { + ordinal: number; + text: string; + style: string; + resolved?: ResolvedStyle; + page: number; + y: number; +}; + +export type Severity = 'blocking' | 'visible' | 'cosmetic'; + +export type FindingCategory = + | 'text' + | 'pagination' + | 'structure' + | 'style' + | 'indent' + | 'numbering' + | 'font' + | 'color' + | 'alignment' + | 'spacing' + | 'unsupported' + | 'unknown'; + +export type Finding = { + fingerprint: string; + category: FindingCategory; + severity: Severity; + paragraphOrdinal: number; + word: unknown; + superdoc: unknown; + message: string; + specRef?: string; + codeAreaHint?: string; +}; + +export type WordExtraction = { + supported: boolean; + unsupportedReason?: string; + pageCount: number; + paragraphs: WordParagraph[]; +}; + +export type WordParagraph = { + idx: number; + text: string; + style: string; + fontName: string; + fontSize: number; + bold: number; + italic: number; + colorRgb: number; + alignment: number; + leftIndent: number; + firstLineIndent: number; + listString: string; + listLevel: number; + page: number; + y: number; +}; + +export type CompareReport = { + docxPath: string; + docxSha: string; + wordSupported: boolean; + unsupportedReason?: string; + counts: { + wordParagraphs: number; + superdocParagraphs: number; + wordPages: number; + superdocPages: number; + }; + findings: Finding[]; +}; + +/** + * A frozen snapshot of findings for a whole corpus run. Written by + * `--save-baseline`, read by `--baseline` to compute deltas. + */ +export type Baseline = { + schemaVersion: 1; + capturedAt: string; + docs: Record; +}; + +/** + * Per-doc delta vs a baseline. Same fingerprint → unchanged; fingerprint + * only in baseline → resolved (your change fixed it); fingerprint only in + * current → new (your change introduced it or didn't fix it). + */ +export type DeltaReport = { + baselineCapturedAt: string; + totals: { resolved: number; new: number; unchanged: number }; + docs: Array<{ + file: string; + resolved: Finding[]; + new: Finding[]; + unchangedCount: number; + }>; +}; diff --git a/devtools/compare-rendering/src/word.ts b/devtools/compare-rendering/src/word.ts new file mode 100644 index 0000000000..6531c275c7 --- /dev/null +++ b/devtools/compare-rendering/src/word.ts @@ -0,0 +1,98 @@ +import { readFile } from 'node:fs/promises'; +import { fileURLToPath } from 'node:url'; +import type { WordExtraction } from './types.ts'; +import { hashFile, readCache, sha256, writeCache } from './cache.ts'; + +const SCRIPT_PATH = fileURLToPath(new URL('./extract-layout.ps1', import.meta.url)); + +type JobEnvelope = { + id: string; + status: 'queued' | 'running' | 'succeeded' | 'failed'; + result?: { output?: string } | null; + error?: { code: string; message: string } | null; +}; + +const POLL_INTERVAL_MS = 500; +const POLL_BUFFER_MS = 30_000; + +async function runPowerShell(script: string, timeoutSeconds: number): Promise { + const base = process.env.WORD_API_URL; + const token = process.env.WORD_API_TOKEN; + if (!base) throw new Error('WORD_API_URL not set'); + if (!token) throw new Error('WORD_API_TOKEN not set'); + + const root = base.replace(/\/$/, ''); + const authHeaders = { Authorization: `Bearer ${token}` } as const; + + const res = await fetch(`${root}/v1/executions`, { + method: 'POST', + headers: { ...authHeaders, 'Content-Type': 'application/json' }, + body: JSON.stringify({ script, timeout_seconds: timeoutSeconds }), + }); + + if (!res.ok) { + const body = await res.text().catch((e) => ``); + throw new Error(`word-api HTTP ${res.status}: ${body.slice(0, 5000)}`); + } + + let job = (await res.json()) as JobEnvelope; + const deadline = Date.now() + timeoutSeconds * 1000 + POLL_BUFFER_MS; + + while (job.status === 'queued' || job.status === 'running') { + if (Date.now() > deadline) { + throw new Error(`word-api job ${job.id} poll deadline exceeded (${job.status})`); + } + await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS)); + const pollRes = await fetch(`${root}/v1/jobs/${job.id}`, { headers: authHeaders }); + if (!pollRes.ok) { + const body = await pollRes.text().catch(() => ''); + throw new Error(`word-api poll HTTP ${pollRes.status}: ${body.slice(0, 500)}`); + } + job = (await pollRes.json()) as JobEnvelope; + } + + if (job.status !== 'succeeded') { + const code = job.error?.code ?? 'unknown'; + const message = job.error?.message ?? 'no error message'; + throw new Error(`word-api job ${job.id} ${job.status} (${code}): ${message}`); + } + return job.result?.output ?? ''; +} + +function parseExtractionOutput(output: string): WordExtraction { + const begin = output.indexOf('JSON_BEGIN'); + if (begin === -1) { + throw new Error(`extract-layout.ps1: missing JSON_BEGIN marker\n${output.slice(0, 800)}`); + } + const payloadStart = begin + 'JSON_BEGIN'.length; + const end = output.indexOf('JSON_END', payloadStart); + if (end === -1) { + throw new Error(`extract-layout.ps1: missing JSON_END marker\n${output.slice(0, 800)}`); + } + const json = output.slice(payloadStart, end).trim(); + return JSON.parse(json) as WordExtraction; +} + +export async function extractWord( + docxPath: string, + opts: { cache?: boolean } = {}, +): Promise<{ extraction: WordExtraction; sha: string; cached: boolean }> { + const [docxSha, psBody] = await Promise.all([hashFile(docxPath), readFile(SCRIPT_PATH, 'utf8')]); + const psSha = sha256(psBody).slice(0, 12); + const useCache = opts.cache !== false; + + if (useCache) { + const hit = await readCache(docxSha, psSha); + if (hit) return { extraction: hit, sha: docxSha, cached: true }; + } + + const docxBytes = await readFile(docxPath); + const b64 = docxBytes.toString('base64'); + const command = `$b64 = '${b64}'\n${psBody}`; + + const output = await runPowerShell(command, 600); + const extraction = parseExtractionOutput(output); + + if (useCache) await writeCache(docxSha, psSha, extraction); + return { extraction, sha: docxSha, cached: false }; +} diff --git a/devtools/compare-rendering/tsconfig.json b/devtools/compare-rendering/tsconfig.json new file mode 100644 index 0000000000..75ab79a444 --- /dev/null +++ b/devtools/compare-rendering/tsconfig.json @@ -0,0 +1,16 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "bundler", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "resolveJsonModule": true, + "allowImportingTsExtensions": true, + "noEmit": true, + "lib": ["ESNext"], + "types": ["node"] + }, + "include": ["src/**/*"] +} diff --git a/devtools/compare-rendering/vitest.config.ts b/devtools/compare-rendering/vitest.config.ts new file mode 100644 index 0000000000..bfaebe3ce6 --- /dev/null +++ b/devtools/compare-rendering/vitest.config.ts @@ -0,0 +1,8 @@ +import { defineConfig } from 'vitest/config'; + +export default defineConfig({ + test: { + include: ['src/**/*.test.ts'], + globals: true, + }, +}); diff --git a/package.json b/package.json index 499c44ab79..d1250e38c3 100644 --- a/package.json +++ b/package.json @@ -32,6 +32,7 @@ "dev:super-editor": "pnpm --prefix packages/super-editor run dev", "dev:collab": "pnpm --prefix packages/superdoc run dev:collab", "word-benchmark-sidecar": "pnpm --prefix packages/superdoc run word-benchmark-sidecar", + "compare-rendering": "bun devtools/compare-rendering/src/cli.ts", "dev:docs": "concurrently -k -n VITE,CDN,DOCS -c cyan,yellow,green \"pnpm --prefix packages/superdoc run dev\" \"pnpm --prefix packages/superdoc run watch:cdn\" \"pnpm --prefix apps/docs run dev\"", "build:superdoc": "pnpm --prefix packages/superdoc run build", "build:super-editor": "pnpm --prefix packages/super-editor run build",