Skip to content

Commit f1d408e

Browse files
authored
test(producer): add variables-prod regression for the variables stack (PR 5/5) (#604)
## What End-to-end Docker regression test that exercises the **full variables chain** shipped in PRs #600-#603. Closes the loop between unit-tested seams and the actual rendered output. This is **PR 5 of 5**, the regression cap. Stacked on `feat/get-variables-skills` (PR #603). ## Why PRs 1-4 ship unit tests that cover the seams independently: - Engine: \`evaluateOnNewDocument\` injection (mocked Puppeteer) - Helper: \`getVariables()\` merge (jsdom) - CLI: \`parseVariablesArg\` validation (pure function) - Loader: \`__hfVariablesByComp\` population (jsdom) - Validator: \`validateVariables\` type-checking (pure) But none of those check that the chain actually works front-to-back inside the production Chrome+ffmpeg+harness combo. A type signature change on \`CaptureOptions.variables\`, a regression in \`evaluateOnNewDocument\` ordering, a bug in the runtime helper's attribute parsing — any of those could pass unit tests and silently render the wrong text. This regression catches it. ## How **Fixture** (\`packages/producer/tests/variables-prod/\`): - \`src/index.html\` — composition with three declared variables (\`title\`, \`subtitle\`, \`bgColor\`) read via \`window.__hyperframes.getVariables()\` and rendered as positioned text on a colored background. **No animation** — keeps the regression frame-stable so it isolates "did the variables flow through?" from motion concerns. - \`meta.json\` — tags \`[\"variables\", \"composition\"]\` (runs in the existing fast shard's tag filter, no workflow YAML changes needed). \`renderConfig.variables\` provides override values that the baseline reflects (\"Override Title\", \"Override subtitle\", \`#0a3d62\`). **If variables don't propagate**, the rendered frame shows declared defaults (\"Default Title\", black) — visibly different from the baseline, so PSNR fails on dozens of frames. - \`output/output.mp4\` — Docker-generated baseline per the project's CLAUDE.md golden-baseline rule. Host renders drift across Chrome/font versions and would fail PSNR even on green code. - \`src/silence.wav\` — copied from \`missing-host-comp-id\`'s silence track to satisfy the audio-correlation check. **Harness change** (\`packages/producer/src/regression-harness.ts\`): - \`TestMetadata.renderConfig\` gains an optional \`variables: Record<string, unknown>\` field, validated as a JSON object (not array, not null) in the \`meta.json\` validator. - The \`createRenderJob\` call site forwards \`renderConfig.variables\` to \`RenderConfig.variables\`, which the engine already consumes via \`evaluateOnNewDocument\` (PR #600). ## Test plan - [x] **\`docker:test variables-prod\` PASSED** — 100/100 visual checkpoints, audio correlation 1.000. - [x] **Defeated the bug it's meant to catch** — generated a baseline against an old image (without the harness change) and confirmed it shows defaults. After the harness change + image rebuild, the baseline correctly shows overrides. - [x] **CI auto-includes** — fast shard's \`--exclude-tags slow,render-compat,hdr\` lets \`[variables, composition]\` through. No \`.github/workflows/regression.yml\` edits needed. ## Backwards compatibility Additive. Existing fixtures don't set \`renderConfig.variables\` and behave identically. The harness validator only fires on the new field if it's present. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2 parents 62c2589 + c9d5fe6 commit f1d408e

6 files changed

Lines changed: 268 additions & 0 deletions

File tree

packages/producer/src/regression-harness.ts

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,14 @@ type TestMetadata = {
3636
workers?: number; // Optional: auto-calculates if omitted
3737
/** Force HDR in the harness; omitted/false preserves historical SDR-only test behavior. */
3838
hdr?: boolean;
39+
/**
40+
* Render-time variable overrides, equivalent to `hyperframes render
41+
* --variables '<json>'`. Injected as `window.__hfVariables` before any
42+
* page script runs so the runtime helper `getVariables()` returns the
43+
* merged result of declared defaults (`data-composition-variables`)
44+
* and these overrides. Omit when the test doesn't exercise variables.
45+
*/
46+
variables?: Record<string, unknown>;
3947
};
4048
};
4149

@@ -160,6 +168,12 @@ function validateMetadata(meta: unknown): TestMetadata {
160168
if (rc.hdr !== undefined && typeof rc.hdr !== "boolean") {
161169
throw new Error("meta.json: 'renderConfig.hdr' must be a boolean (or omit for false)");
162170
}
171+
if (
172+
rc.variables !== undefined &&
173+
(rc.variables === null || typeof rc.variables !== "object" || Array.isArray(rc.variables))
174+
) {
175+
throw new Error("meta.json: 'renderConfig.variables' must be a JSON object (or omitted)");
176+
}
163177

164178
return m as TestMetadata;
165179
}
@@ -601,6 +615,7 @@ async function runTestSuite(
601615
useGpu: false,
602616
debug: false,
603617
hdrMode: suite.meta.renderConfig.hdr ? "force-hdr" : "force-sdr",
618+
variables: suite.meta.renderConfig.variables,
604619
});
605620

606621
await executeRenderJob(job, tempSrcDir, renderedOutputPath);
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
"name": "Variables Prod",
3+
"description": "End-to-end regression for the variables system. Composition declares title/subtitle/bgColor in data-composition-variables and reads them via window.__hyperframes.getVariables(). renderConfig.variables provides overrides that flow CLI → producer → engine evaluateOnNewDocument → window.__hfVariables → helper merge → DOM text → rendered pixels. If any link in that chain breaks, the rendered text/background diverges from the baseline and PSNR fails.",
4+
"tags": ["variables", "composition"],
5+
"minPsnr": 30,
6+
"maxFrameFailures": 0,
7+
"minAudioCorrelation": 0.9,
8+
"maxAudioLagWindows": 120,
9+
"renderConfig": {
10+
"fps": 24,
11+
"variables": {
12+
"title": "Override Title",
13+
"subtitle": "Override subtitle",
14+
"bgColor": "#0a3d62"
15+
}
16+
}
17+
}

packages/producer/tests/variables-prod/output/compiled.html

Lines changed: 137 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:be9a5689e27272b191fa566843e45f88f728b36f677a3db49f89e9193627a667
3+
size 117111
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
<!doctype html>
2+
<html
3+
lang="en"
4+
data-composition-variables='[
5+
{"id":"title","type":"string","label":"Title","default":"Default Title"},
6+
{"id":"subtitle","type":"string","label":"Subtitle","default":"Default subtitle"},
7+
{"id":"bgColor","type":"color","label":"Background","default":"#0b0b0d"}
8+
]'
9+
>
10+
<head>
11+
<meta charset="UTF-8" />
12+
<meta name="viewport" content="width=1920, height=1080" />
13+
<script src="https://cdn.jsdelivr.net/npm/gsap@3.14.2/dist/gsap.min.js"></script>
14+
<style>
15+
* {
16+
margin: 0;
17+
padding: 0;
18+
box-sizing: border-box;
19+
}
20+
html,
21+
body {
22+
margin: 0;
23+
width: 1920px;
24+
height: 1080px;
25+
overflow: hidden;
26+
}
27+
body {
28+
background: #0b0b0d;
29+
font-family: -apple-system, "Helvetica Neue", Arial, sans-serif;
30+
}
31+
[data-composition-id="main"] {
32+
width: 100%;
33+
height: 100%;
34+
position: relative;
35+
}
36+
.title {
37+
position: absolute;
38+
top: 380px;
39+
left: 0;
40+
right: 0;
41+
font-size: 140px;
42+
font-weight: 700;
43+
color: #ffffff;
44+
text-align: center;
45+
line-height: 1;
46+
}
47+
.subtitle {
48+
position: absolute;
49+
top: 600px;
50+
left: 0;
51+
right: 0;
52+
font-size: 56px;
53+
color: #9aa0a6;
54+
text-align: center;
55+
line-height: 1;
56+
}
57+
</style>
58+
</head>
59+
<body>
60+
<div
61+
id="root"
62+
data-composition-id="main"
63+
data-start="0"
64+
data-duration="3"
65+
data-width="1920"
66+
data-height="1080"
67+
>
68+
<audio
69+
id="silence-track"
70+
src="silence.wav"
71+
data-start="0"
72+
data-duration="3"
73+
data-track-index="0"
74+
></audio>
75+
<h1 id="title-el" class="title clip" data-start="0" data-duration="3"></h1>
76+
<p id="subtitle-el" class="subtitle clip" data-start="0" data-duration="3"></p>
77+
</div>
78+
79+
<script>
80+
// Render-time values: declared defaults from data-composition-variables
81+
// merged with overrides from meta.json's renderConfig.variables (which
82+
// arrive via window.__hfVariables, injected by the engine).
83+
const vars = window.__hyperframes.getVariables();
84+
document.body.style.background = vars.bgColor;
85+
document.getElementById("title-el").textContent = vars.title;
86+
document.getElementById("subtitle-el").textContent = vars.subtitle;
87+
88+
window.__timelines = window.__timelines || {};
89+
const tl = gsap.timeline({ paused: true });
90+
// Static placement — no animation. Keeps the regression frame-stable
91+
// and isolates "did the variables flow through?" from motion concerns.
92+
tl.to({}, { duration: 3 });
93+
window.__timelines["main"] = tl;
94+
</script>
95+
</body>
96+
</html>
281 KB
Binary file not shown.

0 commit comments

Comments
 (0)