feat(compare-rendering): add Word↔SuperDoc layout comparison

devtools/compare-rendering/.gitignore

@@ -0,0 +1 @@
+.cache/

devtools/compare-rendering/README.md

@@ -0,0 +1,111 @@
+# compare-rendering
+
+Diffs Word and SuperDoc rendering of the same `.docx` at the *resolved schema* level — text, page assignment, and (in later milestones) font/indent/color/numbering. Emits typed `Finding[]` so an agent can route fixes to specific SuperDoc modules.
+
+This is a dev tool, not a pass/fail test. It surfaces concrete divergences so you don't have to compare screenshots by eye.
+
+## Scope (M1)
+
+- **Supported:** paragraph-only documents (text-heavy memos, letters, policies).
+- **Short-circuited with a reason:** docs containing tables, inline/floating shapes, or tracked changes. The report emits an `unsupported` finding and skips the diff — honest boundary rather than a misleading "everything looks fine."
+- **Categories emitted in M1:** `text`, `pagination`, `structure`, `unsupported`. Style/indent/color/numbering come in M2 once the SuperDoc-side normalizer pulls resolved values out of `measures[]` and `runs[]`.
+
+## Quick start
+
+```bash
+export WORD_API_URL="https://word-mcp.superdoc.workers.dev"
+export WORD_API_TOKEN="<your-bearer-token>"
+
+pnpm compare-rendering -- \
+  --input evals/fixtures/docs/memorandum.docx \
+  --format md
+```
+
+Run directly without the wrapper:
+
+```bash
+bun devtools/compare-rendering/src/cli.ts --input <path> --format md
+```
+
+Example output (truncated):
+
+```markdown
+# compare-rendering: memorandum.docx
+
+- Word pages: 3, SuperDoc pages: 3
+- Word paragraphs: 94, SuperDoc paragraphs: 94
+
+## Findings (2)
+
+### pagination (2)
+- **[visible]** Paragraph #39 landed on page 1 in SuperDoc but page 2 in Word (empty line)
+  - spec: ECMA-376 §17.3.1.16 (keepNext/keepLines/pageBreakBefore)
+  - code: `layout-engine/layout-engine/src/pagination`
+- **[visible]** Paragraph #80 landed on page 2 in SuperDoc but page 3 in Word ("   - Any press releases…")
+  - spec: ECMA-376 §17.3.1.16 (keepNext/keepLines/pageBreakBefore)
+  - code: `layout-engine/layout-engine/src/pagination`
+```
+
+## How it works
+
+```
+docx
+ ├── word adapter (POST /v1/executions to word-api)         ─► word.json (cached)
+ └── superdoc adapter (spawn pnpm layout:export-one)        ─► sd.layout.json
+                                │
+                        normalize both sides
+                                │
+                     NormalizedParagraph[] × 2
+                                │
+                           differ + taxonomy
+                                │
+                           Finding[] report
+```
+
+- Word extraction is **cached** by `sha256(docx) + sha256(extract-layout.ps1)`. Editing SuperDoc code and re-running the tool only re-runs the SuperDoc side — no re-hit to the VM (~25s saved per iteration). Editing the PowerShell script busts the cache automatically.
+- Bypass the cache for a single run with `--no-cache`.
+
+## Env
+
+| Variable         | Purpose                                              |
+|------------------|------------------------------------------------------|
+| `WORD_API_URL`   | Base URL of the word-api worker                      |
+| `WORD_API_TOKEN` | Bearer token                                         |
+
+## Exit codes
+
+- `0` — ran successfully; findings are at most `visible`/`cosmetic` (or no findings at all)
+- `1` — tool error (network, missing input, bad args)
+- `2` — ran successfully but emitted at least one `blocking` finding
+
+Makes it CI-usable later without rework.
+
+## Non-goals
+
+- Pixel diffing (see `tests/visual/`).
+- Tables, images, shapes, track changes, headers/footers, comments, TOC — deferred past M5.
+- Auto-fix generation.
+- Publishing as a package.
+
+## Milestones (revised after M1 corpus-batch insights)
+
+- **M1** ✅ — CLI works end-to-end on paragraph-only docs. 4 categories (`text`, `pagination`, `structure`, `unsupported`). JSON + markdown output. Word-extraction cache. Ad-hoc `scripts/batch.ts` runner for whole-corpus sweeps.
+- **M2** — Baseline + delta reporting. Snapshot findings against a pinned SuperDoc sha, emit only `resolved` / `new` since baseline. This is what makes the tool **agent-usable**: signal becomes "my change fixed N, broke M" instead of "here are 367 absolute findings." Pin a `main`-branch baseline at `test-corpus/.baseline.json`.
+- **M3** — LLM screenshot judge for docs where schema diff is silent or near-silent. Catches rendering divergences that don't surface in layout data at all (e.g. `w:val="wave"` border styles rendered as plain lines, font substitution, painter-level overflow).
+- **M4** — Populate `NormalizedParagraph.resolved` on SuperDoc side. Taxonomy extends to `style`, `indent`, `font`, `color`, `alignment`, `spacing`, `numbering`. Safe to add once M2 absorbs the "new field adds findings everywhere" noise.
+- **M5** — Table support. Non-trivial; needs parallel table walks on both sides.
+
+## Insights from M1 corpus batch (75 docs, April 2026)
+
+- **Pagination findings compound.** Many "N pagination findings" collapse to one underlying bug expressed N times. `memorandum.docx` (3 findings) and `sd-1741-paragraph-between-borders` (36 findings) share the same root cause — SuperDoc fits slightly more content per page than Word; drift accumulates across pages. One fix likely eliminates most findings at once.
+- **Schema diff has real false negatives.** `sd-1741` reports 0 text/style findings, but visually SuperDoc renders every border-between style (`wave`, `doubleWave`, `dashDotStroked`, `triple`, …) as a plain line while Word renders each correctly. Schema-level comparison will never catch this class without the M3 screenshot judge.
+- **~27 % of the corpus is in M1 scope.** 13 / 75 docs are short-circuited for tables/shapes/comments/revisions; the rest yield meaningful findings. Real-world DOCX coverage unlocks at M5 (tables).
+
+## Corpus sweep
+
+Ad-hoc batch runner at `scripts/batch.ts` — iterates every `.docx` under a directory, writes per-doc JSON reports plus a `_summary.json`, and prints a one-line status per doc. Graduates to a proper `--input-dir` flag in M2 alongside baseline support.
+
+```bash
+WORD_API_URL=... WORD_API_TOKEN=... \
+  bun devtools/compare-rendering/scripts/batch.ts test-corpus/rendering
+```

devtools/compare-rendering/package.json

@@ -0,0 +1,10 @@
+{
+  "private": true,
+  "type": "module",
+  "name": "compare-rendering",
+  "scripts": {
+    "start": "bun src/cli.ts",
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run"
+  }
+}

devtools/compare-rendering/scripts/batch.ts

@@ -0,0 +1,137 @@
+#!/usr/bin/env bun
+// Ad-hoc batch runner — iterates every .docx under a directory, runs compare-rendering,
+// and prints a summary. Not part of M1; this is scaffolding for M3's --input-dir mode.
+
+import { readdir, readFile, writeFile, mkdir } from 'node:fs/promises';
+import { resolve, join, basename } from 'node:path';
+import { spawn } from 'node:child_process';
+
+const INPUT_DIR = resolve(process.argv[2] ?? 'test-corpus/rendering');
+const OUTPUT_DIR = resolve(process.argv[3] ?? 'test-corpus/.reports');
+const CLI = resolve(new URL('../src/cli.ts', import.meta.url).pathname);
+
+await mkdir(OUTPUT_DIR, { recursive: true });
+
+const files = (await readdir(INPUT_DIR)).filter((f) => f.endsWith('.docx')).sort();
+
+console.log(`[batch] ${files.length} docs under ${INPUT_DIR}\n`);
+
+type Summary = {
+  file: string;
+  status: 'match' | 'diffs' | 'skipped' | 'error';
+  supported: boolean;
+  unsupportedReason?: string;
+  findings: number;
+  byCategory: Record<string, number>;
+  durationMs: number;
+  note?: string;
+};
+
+const summaries: Summary[] = [];
+const start = Date.now();
+
+for (let i = 0; i < files.length; i += 1) {
+  const f = files[i]!;
+  const input = join(INPUT_DIR, f);
+  const output = join(OUTPUT_DIR, f.replace(/\.docx$/, '.json'));
+  const t0 = Date.now();
+
+  const status = await runOne(input, output);
+  const ms = Date.now() - t0;
+
+  try {
+    const raw = JSON.parse(await readFile(output, 'utf8'));
+    const byCategory: Record<string, number> = {};
+    for (const finding of raw.findings ?? []) {
+      byCategory[finding.category] = (byCategory[finding.category] ?? 0) + 1;
+    }
+    const supported = raw.wordSupported === true;
+    summaries.push({
+      file: f,
+      status: !supported ? 'skipped' : raw.findings.length === 0 ? 'match' : 'diffs',
+      supported,
+      unsupportedReason: raw.unsupportedReason,
+      findings: (raw.findings ?? []).length,
+      byCategory,
+      durationMs: ms,
+    });
+  } catch {
+    summaries.push({
+      file: f,
+      status: 'error',
+      supported: false,
+      findings: 0,
+      byCategory: {},
+      durationMs: ms,
+      note: `CLI exit ${status}`,
+    });
+  }
+
+  const last = summaries[summaries.length - 1]!;
+  const marker = last.status === 'match' ? '✓' : last.status === 'diffs' ? '⚠' : last.status === 'skipped' ? '–' : '✗';
+  const tail =
+    last.status === 'skipped'
+      ? `skipped: ${last.unsupportedReason}`
+      : last.status === 'diffs'
+        ? `${last.findings} finding(s) ${JSON.stringify(last.byCategory)}`
+        : last.status === 'match'
+          ? 'match'
+          : (last.note ?? 'error');
+  console.log(`[${i + 1}/${files.length}] ${marker} ${f} (${ms}ms) — ${tail}`);
+}
+
+const totalMs = Date.now() - start;
+
+console.log(`\n[batch] done in ${(totalMs / 1000).toFixed(1)}s`);
+
+const counts = { match: 0, diffs: 0, skipped: 0, error: 0 };
+for (const s of summaries) counts[s.status] += 1;
+console.log(`\nOverall:`);
+console.log(`  match:   ${counts.match}`);
+console.log(`  diffs:   ${counts.diffs}`);
+console.log(`  skipped: ${counts.skipped}`);
+console.log(`  error:   ${counts.error}`);
+
+const reasons: Record<string, number> = {};
+for (const s of summaries) {
+  if (s.status === 'skipped' && s.unsupportedReason) {
+    const key = s.unsupportedReason.replace(/\s*\(\d+\)$/, '');
+    reasons[key] = (reasons[key] ?? 0) + 1;
+  }
+}
+if (Object.keys(reasons).length) {
+  console.log(`\nSkip reasons:`);
+  for (const [k, v] of Object.entries(reasons).sort((a, b) => b[1] - a[1])) {
+    console.log(`  ${v.toString().padStart(3)} × ${k}`);
+  }
+}
+
+const categories: Record<string, number> = {};
+for (const s of summaries) {
+  for (const [cat, count] of Object.entries(s.byCategory)) {
+    categories[cat] = (categories[cat] ?? 0) + count;
+  }
+}
+if (Object.keys(categories).length) {
+  console.log(`\nFindings by category (across docs with diffs):`);
+  for (const [k, v] of Object.entries(categories).sort((a, b) => b[1] - a[1])) {
+    console.log(`  ${v.toString().padStart(3)} × ${k}`);
+  }
+}
+
+const summaryPath = join(OUTPUT_DIR, '_summary.json');
+await writeFile(summaryPath, JSON.stringify({ totalMs, counts, reasons, categories, summaries }, null, 2));
+console.log(`\n[batch] summary written to ${summaryPath}`);
+
+function runOne(input: string, output: string): Promise<number> {
+  return new Promise((resolve) => {
+    const child = spawn('bun', [CLI, '--input', input, '--output', output, '--format', 'json'], {
+      stdio: ['ignore', 'pipe', 'pipe'],
+      env: process.env,
+    });
+    // Drain stdio to avoid backpressure; we don't print the CLI's own logs.
+    child.stdout?.on('data', () => {});
+    child.stderr?.on('data', () => {});
+    child.on('close', (code) => resolve(code ?? 0));
+  });
+}

devtools/compare-rendering/src/cache.ts

@@ -0,0 +1,36 @@
+import { createHash } from 'node:crypto';
+import { mkdir, readFile, stat, writeFile } from 'node:fs/promises';
+import { fileURLToPath } from 'node:url';
+import { dirname, join } from 'node:path';
+
+const CACHE_DIR = fileURLToPath(new URL('../.cache/word', import.meta.url));
+
+export function sha256(bytes: Uint8Array | string): string {
+  const h = createHash('sha256');
+  h.update(bytes);
+  return h.digest('hex');
+}
+
+export async function hashFile(path: string): Promise<string> {
+  return sha256(await readFile(path));
+}
+
+function cachePath(sha: string, keySuffix: string): string {
+  return join(CACHE_DIR, `${sha}-${keySuffix}.json`);
+}
+
+export async function readCache<T>(sha: string, keySuffix: string): Promise<T | null> {
+  const p = cachePath(sha, keySuffix);
+  try {
+    await stat(p);
+  } catch {
+    return null;
+  }
+  return JSON.parse(await readFile(p, 'utf8')) as T;
+}
+
+export async function writeCache<T>(sha: string, keySuffix: string, value: T): Promise<void> {
+  const p = cachePath(sha, keySuffix);
+  await mkdir(dirname(p), { recursive: true });
+  await writeFile(p, JSON.stringify(value), 'utf8');
+}