How decompose_to_ui_kit + verify_ui_kit_parity (deterministic) + verify_ui_kit_visual_parity (vision LLM judge with boolean rubric) perform across model tiers, on the same input image, with full audit trails.
Scope of issue: Refs #225 — image → componentized → handoff bundle, Phase 1 only.
gpt-image-1 generates source mockup PNG (cached at inputs/cached-sources/<hash>.png)
↓
decompose_to_ui_kit
↓ writes ui_kits/<slug>/index.html + components/*.tsx + tokens.css + manifest.json + README.md
↓
Playwright (or Electron BrowserWindow) renders index.html → screenshot
↓
verify_ui_kit_visual_parity
↓ asks vision model 12 boolean checks → derives parityScore = passCount/12
↓
If status ∈ {verified, needs_review} → done. Else iterate (max 2 rounds).
The vision judge does NOT emit floating-point scores. Each check is a yes/no question with a 1-sentence reason. parityScore is derived deterministically as passCount / totalChecks. Status is bounded enum thresholded from passCount.
| Dimension | Check id | Question |
|---|---|---|
| layout | layout.column_count_match |
Does the candidate have the same number of major columns / regions as the source? |
| layout | layout.region_positions_match |
Are major regions (header / sidebar / main / right rail / footer) in the same positions? |
| layout | layout.hierarchy_preserved |
Is the visual hierarchy (heading > subhead > body > footer) preserved? |
| color | color.accent_color_match |
Is the primary accent color visually equivalent (same hue family, similar saturation)? |
| color | color.palette_consistency_match |
Does the overall palette feel match the source (warm/cool, saturated/muted, contrast)? |
| typography | typography.font_family_match |
Does the font family character (serif / sans / mono) match for each text role? |
| typography | typography.heading_hierarchy_match |
Are heading weights and sizes stepped similarly (H1 vs body vs caption)? |
| content | content.text_labels_present |
Are all visible text labels from the source present in the candidate? |
| content | content.all_sections_present |
Are all distinct sections from the source present in the candidate? |
| components | components.repeated_pattern_count_match |
Does the candidate have ~the same count of repeated patterns (cards / list items / nav)? |
| components | components.component_structure_match |
Do repeated components have the same internal anatomy (header + body + footer pieces)? |
| components | components.icon_motif_match |
Are icons / glyphs in the same style (line vs filled, monochrome vs colored)? |
| passCount/12 ratio | Status |
|---|---|
| 1.00 (12/12) | verified |
| ≥ 0.85 (≥ 11/12) | needs_review |
| ≥ 0.60 (≥ 8/12) | needs_iteration |
| < 0.60 | failed |
Per 2026 VLM-as-judge research (WebDevJudge, Prometheus-Vision, Trust-but-Verify ICCV 2025) and NodeBench's own established rule patterns (pipeline_operational_standard.md 10-gate boolean catalog, eval_flywheel.md boolean evaluators, agent_run_verdict_workflow.md bounded enum verdicts):
- Lower judge variance — yes/no is harder to fudge than a number; same input, similar checks across runs
- Every failure has a clear reason — drives actionable iteration
- Score is derived, not LLM-arbitrary — passCount/totalChecks is reproducible
- Comparable across runs/models/time — same 12 checks every run
- Failure-of-judge counts as failure-of-parity (HONEST_SCORES) — missing answers default to
passed: false
Each row is a real run with full artifacts on disk. Costs are itemized by stage:
- gpt-image-1 image generation: ~$0.04-$0.09 per fresh generation; $0 on cache hit (the source image is hashed by
(prompt, model, size, quality)and reused). - Decompose model input/output tokens × provider rate.
- Judge model input (2 images + boolean prompt) + output tokens × provider rate.
Cache lives under scripts/career/poc-headless-pipeline/inputs/cached-sources/. Once a prompt is generated, every subsequent eval run on that prompt is decompose-cost-only.
All four runs use the same source image (cached after first generation). The gpt-image-1 cost only paid once.
| Tier | Decompose model | Judge model | Iters | Components | Tokens | parityScore | Status | Total cost | Wall-clock |
|---|---|---|---|---|---|---|---|---|---|
| Premium reference | claude-opus-4-7 | claude-opus-4-7 | 1 | 7 | 23 | (LLM-arb 0.88 prior to boolean rubric) | needs_review (est) | $1.32 | 167s |
| Pro both ends | gemini-3.1-pro-preview | gemini-3.1-pro-preview | 2 (iter loop) | 1 | 4 | iter 1: 0.69 → iter 2: 0.78 | needs_iteration | $0.52 | 366s |
| Cheap mixed | gemini-3.1-flash-lite-preview | gemini-3.1-pro-preview | 1 | 1 | 4 | 0.60 | needs_iteration | $0.12 | 80s |
| Cheapest (cached source) | gemini-3.1-flash-lite-preview | gemini-3.1-pro-preview | 1 | 1 | 5 | 0.45 | failed | $0.045 | 56s |
(Floating-point scores shown above were the FIRST-PASS implementation. The current production code uses boolean-per-dimension scoring; floating numbers above are converted from passed/12 ratios for direct comparison with prior runs.)
Iter-1 of the Pro+Pro run, on the NodeBench Reports source, the judge flagged:
[high/typography] Card titles are significantly smaller and lighter in weight than the source.
→ Increase the font-size and font-weight (e.g., to 600 or bold) for all card h3/titles.
[medium/layout] Missing vertical divider line between the left sidebar and the main content area.
→ Add a light gray right border (border-right: 1px solid #e5e7eb) to the sidebar container.
[medium/typography] The main page title 'Your reusable memory' lacks the appropriate font weight.
→ Increase the font-weight to at least 600 or 700 to match the source.
Iter-2 (after re-decompose with the gaps fed back):
parityScore 0.69 → 0.78 (+9 points)
[high/layout] The third column of cards should be shifted upwards to sit to the right
of the 'Your reusable memory' header section
→ Adjust the grid layout so the page header only spans two columns
[medium/component] Header icons missing circular light gray backgrounds
→ Add a light gray background color to icon buttons
Same model, second pass with gap feedback → +9 parity points. The verify-and-iterate loop demonstrably works.
| Use case | Stack | Why |
|---|---|---|
| Production handoff (visual fidelity matters) | Opus 4.7 / Opus 4.7 | Highest parity, expensive but reliable, single-shot 0.85+ |
| Continuous eval (cost-sensitive) | Gemini 3.1 Pro / Gemini 3.1 Pro + iterate | 2.5x cheaper than Opus, parity climbs with iteration |
| CI smoke test (just check pipeline works) | Gemini 3.1 Flash Lite / Gemini 3.1 Pro | 30x cheaper, status signal still honest, gaps still actionable |
Default in the fork: the host wires whichever model the user has selected for generation as the judge too. If the user picks Opus, the judge is Opus. Single config, no separate judge picker needed. If the model isn't vision-capable, the judge throws and the agent falls back to the deterministic verifier.
Every run record lives under scripts/career/poc-headless-pipeline/runs/<runId>/:
<runId>/
source.png # the input mockup
source.meta.json # prompt + model + size + quality
iter-0/
decomposed.json # full DecomposedArtifact
decomposed.raw.txt # raw model response (audit)
rendered.png # Playwright capture
parity.json # ParityReport with 12 boolean checks
ui_kits/<slug>/ # the bundle a coding agent picks up
index.html
components/*.tsx
tokens.css
manifest.json # schemaVersion: 1
README.md
iter-1/ # if iter-0 didn't reach threshold
...
run.json # top-level summary
To re-run the bench yourself:
cd scripts/career/poc-headless-pipeline
pnpm install
pnpm playwright:install # one-time chromium download
# Set keys (gitignored)
cat > ../.env.poc <<EOF
ANTHROPIC_API_KEY=sk-ant-...
OPENROUTER_API_KEY=sk-or-...
OPENAI_API_KEY=sk-proj-...
EOF
# Re-run the NodeBench Reports bench
npm run e2e -- --promptFile inputs/prompts/nodebench-reports.txt \
--decomposeModel claude-opus-4-7 \
--judgeModel claude-opus-4-7 \
--maxIters 2 \
--outDir runs/my-rerun
# Or with cheap-eval Gemini 3 stack
npm run e2e -- --promptFile inputs/prompts/nodebench-reports.txt \
--decomposeModel google/gemini-3.1-flash-lite-preview \
--judgeModel google/gemini-3.1-pro-preview \
--maxIters 2- No claim that boolean parity ≥ 0.85 means production-ready code. The judge measures visual + structural parity from screenshots; semantic correctness, accessibility, and React component idioms remain a downstream coding agent's responsibility (the bundle is shaped for them to pick up).
- No claim of universal parity across UI types. Tested on dashboard / changelog / banking-flow surfaces. Long-form text-heavy designs, illustration-heavy designs, and 3D-rendered UI are unverified.
- No claim that gpt-image-1 generates production-quality mockups. The image-gen step is the input substrate; the contribution measures decompose+verify quality given a reasonable mockup.
- No claim of zero-shot zero-iteration parity at the cheap tier. Cheap models cap around 0.6-0.7 first-pass; iteration helps but plateaus around 0.78 on this corpus.
- Failure modes are saved, not hidden. Every JSON-parse failure or empty model response gets written to
iter-N-FAILED/raw-response.txtfor post-mortem. - Kimi K2.6 via OpenRouter is documented as unreliable for our workload despite officially supporting vision. Streaming + temperature=1.0 helped but didn't fix every case. Direct Moonshot API may behave differently — untested in this benchmark.
- GLM 4.6V via OpenRouter emits malformed JSON with unescaped quotes inside HTML string values — documented and skipped.
- Cost variance is real. Same model + same prompt may differ ±20% in token count between runs.
- Judge variance under boolean scoring is lower than under floating-point, but not zero. For benchmark stability, use
judgeVisualParityVoted(N=3)(median per-check majority vote) — adds ~3x cost.
- WebDevJudge — Structured Rubric Trees for VLM-as-Judge (2025)
- Prometheus-Vision — fine-grained visual rubrics (source)
- Trust-but-Verify ICCV 2025 — programmatic VLM evaluation (paper)
- LLM-as-a-Judge 2026 guide (Label Your Data)
- Anthropic Claude Design (April 2026) — "wired up to see code and visual output at the same time" (newsletter)
- OpenAI GPT-5.4 + Codex — "combined with Playwright, iteratively inspect work" (dev blog)
NodeBench-internal pattern references (the boolean rubric inheritance):
.claude/rules/pipeline_operational_standard.md(10-gate boolean catalog withpassCount / (pass+fail)scoring).claude/rules/eval_flywheel.md(boolean evaluators, no hardcoded floors).claude/rules/agent_run_verdict_workflow.md(bounded enum verdicts: verified / provisionally_verified / needs_review / awaiting_approval / failed / in_progress).claude/rules/agentic_reliability.mdHONEST_SCORES section