feat(producer): enable webm in distributed mode via concat-copy (#951)

* feat(producer): enable webm in distributed mode via concat-copy

PR 8.2 of the WebM distributed-rendering plan (v1.5 backlog #1; see
DISTRIBUTED-RENDERING-PLAN.md §7.2). Wires libvpx-vp9 webm through the
distributed pipeline now that PR 8.1 proved concat-copy works.

Architectural decision: Path A (concat-copy) — based on PR 8.1's smoke
test result (9/9 tests pass for both yuv420p and yuva420p VP9 streams).
The simpler architecture wins; no re-encode in assemble, no encode-
parallelism loss.

Changes:

- plan.ts:
  - DistributedRenderConfig.format and PlanResult.format now include
    "webm" — type-level acceptance matches the runtime gate.
  - rejectUnsupportedDistributedFormat() no longer trips on webm. HDR
    mp4 remains the only refused configuration.
  - resolveEncoderTriple() returns libvpx-vp9-software + yuva420p +
    preset="good" for format="webm". yuva420p preserves alpha — the
    format's main reason for existing for web delivery.
  - codec= remains rejected for non-mp4 formats (mov is always ProRes
    4444; webm is always libvpx-vp9). The error message lists all four
    distributed-supported formats.
  - FormatNotSupportedInDistributedError docstring updated to reflect
    the new reality (only HDR is unsupported).

- freezePlan.ts: LockedRenderConfig.encoder gains "libvpx-vp9-software".
  Mirrors libx265-software / prores-software / png-sequence in shape;
  the chunk worker reads this discriminant to decide encode args.

- renderChunk.ts: drops the now-incorrect cast that excluded webm from
  buildSyntheticRenderJob's format input; tightens the preset-format
  cast to include webm.

- assemble.ts: docstring + comment updates. The mp4/mov concat-copy
  path is format-agnostic — webm uses the exact same code (applyFaststart
  is a no-op for webm via the existing chunkEncoder.ts gate;
  muxVideoWithAudio already routes webm to libopus audio).

- planFormatBanlist.test.ts: webm-rejection tests removed; replaced with
  "accepts webm" tests + a HDR+webm combo test that verifies HDR is the
  trip regardless of format.

- plan.test.ts: new describe block pins the webm wiring contract:
  format="webm" produces an encoder=libvpx-vp9-software /
  pixelFormat=yuva420p planDir with closedGop=true and gopSize=chunkSize.

- webm-concat-copy.test.ts (smoke): extended with a yuva420p variant
  that proves the alpha pixel format the distributed pipeline actually
  emits also round-trips through concat-copy. 9/9 tests pass locally.

§8 format support matrix in DISTRIBUTED-RENDERING-PLAN.md is intentionally
left unchanged at this PR — it flips to ✓ in PR 8.4 once the end-to-end
fixture (PR 8.3) is green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(producer): include webm in plan-time needsAlpha + strengthen alpha smoke

PR review feedback from Miguel and Vai on #951 caught a real bug:
`plan.ts`'s `needsAlpha` disjunction excluded `"webm"`, so the plan
stage froze `forceScreenshot: false` into the `LockedRenderConfig`
even though distributed webm uses `yuva420p`. Every chunk worker
captured opaque RGB via BeginFrame (which doesn't preserve alpha on
Linux headless-shell), and libvpx-vp9 encoded uniformly-opaque alpha
that the encoder then dropped — producing un-keyable webm.

Two changes:

1. **plan.ts**: include `"webm"` in `needsAlpha`. Matches the
   in-process renderer's logic at `renderOrchestrator.ts:1469`
   (`const needsAlpha = isWebm || isMov || isPngSequence`); the two
   sites must stay in sync since the distributed pipeline's PSNR
   regression compares against the in-process baseline.

2. **Smoke test (yuva420p describe)**: source frames now use a real
   alpha gradient (`geq=a='X*255/W'` on top of `testsrc2`) instead of
   `testsrc2 + format=rgba` which was uniformly opaque. The decode-
   pix_fmt assertion is dropped (ffprobe reports `yuv420p` for
   VP9-with-alpha because the alpha lives in a Matroska
   `BlockAdditional` sidecar) and replaced with two stronger checks:
   - `TAG:ALPHA_MODE=1` is present on the stream — proves the
     encoder was actually configured for alpha
   - alpha plane variance after `-c:v libvpx-vp9 -i ... -pix_fmt rgba
     -vf extractplanes=a,signalstats` — proves the alpha sub-stream
     round-trips through concat-copy with spatially-varying content,
     not uniform/dropped alpha
   - decode-test gate is now exit-code-only (was `exitCode || stderr`
     which would flake on chatty ffmpeg `-v error` builds emitting
     non-fatal DTS/container notes)

These checks would have caught the `needsAlpha` bug before review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(aws-lambda): widen narrow format types to include webm

CI on PR #951 was failing at typecheck/build because the producer's
`DistributedRenderConfig.format` widened to include webm in this PR
but the aws-lambda package's narrow `"mp4" | "mov" | "png-sequence"`
type literals in `events.ts`, `handler.ts`, and `validateConfig.ts`
hadn't kept up. `renderToLambda.ts:87` passed `config.format` (now
including webm) into a parameter typed against the narrow union,
producing TS2345.

This widening originally landed in PR #952 (test fixture PR) but
needs to be atomic with the producer's widening here to keep each
PR independently typecheck-clean.

Also refactor `formatExtension` from a switch dispatch to a
`Record<DistributedFormat, string>` lookup. Adding the webm case
tipped the switch's CRAP to the 30.0 fallow threshold; the lookup
table drops cyclomatic from 5 to 1 with the same compile-time
exhaustiveness guarantee (TS errors on missing entries when
`DistributedFormat` adds a new format). The runtime
`_exhaustive: never` throw was only protecting against a string
slipping past TS; `validateConfig.ts`'s `ALLOWED_FORMATS` already
gates untrusted input at the SDK boundary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: jrusso1020

packages/aws-lambda/src/events.ts

@@ -65,7 +65,7 @@ export interface RenderChunkEvent {
   /** S3 URI prefix where the chunk output should be uploaded (`s3://bucket/{prefix}/`). */
   ChunkOutputS3Prefix: string;
   /** Output container format from the plan's encoder.json; drives file vs frame-dir handling. */
-  Format: "mp4" | "mov" | "png-sequence";
+  Format: "mp4" | "mov" | "png-sequence" | "webm";
 }
 
 /** Activity C: fetch planDir + all chunks + audio, assemble, upload final. */
@@ -80,7 +80,7 @@ export interface AssembleEvent {
   /** Final output S3 URI (`s3://bucket/key.mp4`). */
   OutputS3Uri: string;
   /** Output container format; drives file vs frame-dir handling. */
-  Format: "mp4" | "mov" | "png-sequence";
+  Format: "mp4" | "mov" | "png-sequence" | "webm";
 }
 
 /**
@@ -106,7 +106,7 @@ export interface PlanLambdaResult {
   Fps: 24 | 30 | 60;
   Width: number;
   Height: number;
-  Format: "mp4" | "mov" | "png-sequence";
+  Format: "mp4" | "mov" | "png-sequence" | "webm";
   HasAudio: boolean;
   AudioS3Uri: string | null;
   FfmpegVersion: string;

packages/aws-lambda/src/formatExtension.ts

@@ -6,19 +6,20 @@
  * looks like vs a png-sequence.
  */
 
-export type DistributedFormat = "mp4" | "mov" | "png-sequence";
+export type DistributedFormat = "mp4" | "mov" | "png-sequence" | "webm";
+
+// Closed-enum lookup table. TS enforces exhaustiveness via the
+// `Record<DistributedFormat, string>` annotation — adding a format to
+// `DistributedFormat` without adding the matching key here fails to
+// typecheck, which is the same exhaustiveness guarantee a switch +
+// `_exhaustive: never` arm provides but at lower complexity.
+const FORMAT_EXTENSIONS: Record<DistributedFormat, string> = {
+  mp4: ".mp4",
+  mov: ".mov",
+  webm: ".webm",
+  "png-sequence": "",
+};
 
 export function formatExtension(format: DistributedFormat): string {
-  switch (format) {
-    case "mp4":
-      return ".mp4";
-    case "mov":
-      return ".mov";
-    case "png-sequence":
-      return "";
-    default: {
-      const _exhaustive: never = format;
-      throw new Error(`[formatExtension] unsupported format: ${_exhaustive as string}`);
-    }
-  }
+  return FORMAT_EXTENSIONS[format];
 }

packages/aws-lambda/src/handler.ts

@@ -433,7 +433,7 @@ async function downloadChunkObjects(
   s3: S3Client,
   uris: string[],
   workDir: string,
-  format: "mp4" | "mov" | "png-sequence",
+  format: "mp4" | "mov" | "png-sequence" | "webm",
 ): Promise<string[]> {
   const chunksDir = join(workDir, "chunks");
   mkdirSync(chunksDir, { recursive: true });

packages/aws-lambda/src/sdk/validateConfig.test.ts

@@ -54,7 +54,7 @@ describe("validateDistributedRenderConfig", () => {
       "unsupported format",
       {
         ...VALID,
-        format: "webm",
+        format: "gif",
       } as unknown as SerializableDistributedRenderConfig,
       "config.format",
     ],

packages/aws-lambda/src/sdk/validateConfig.ts

@@ -31,7 +31,7 @@ export class InvalidConfigError extends Error {
 }
 
 const ALLOWED_FPS = [24, 30, 60] as const;
-const ALLOWED_FORMATS = ["mp4", "mov", "png-sequence"] as const;
+const ALLOWED_FORMATS = ["mp4", "mov", "png-sequence", "webm"] as const;
 const ALLOWED_CODECS = ["h264", "h265"] as const;
 const ALLOWED_QUALITIES = ["draft", "standard", "high"] as const;
 const ALLOWED_RUNTIME_CAPS = ["lambda", "temporal", "cloud-run-job", "k8s-job", "none"] as const;

packages/producer/src/services/distributed/assemble.ts

@@ -2,16 +2,20 @@
  * Activity C of the distributed render pipeline.
  *
  * `assemble(planDir, chunkPaths, audioPath, outputPath)` stitches per-chunk
- * outputs into the final deliverable. For mp4/mov this is `ffmpeg -f concat
- * -c copy` (free of re-encode loss because every chunk's first frame is an
- * IDR keyframe — the chunk encoder sets `lockGopForChunkConcat` to
- * enforce this). For png-sequence chunks (each chunk is a directory of
+ * outputs into the final deliverable. For mp4 / mov / webm this is
+ * `ffmpeg -f concat -c copy` (free of re-encode loss because every
+ * chunk's first frame is an IDR keyframe — the chunk encoder sets
+ * `lockGopForChunkConcat` to enforce this, which for libvpx-vp9 also
+ * disables alt-ref frames so concat seams remain independently
+ * decodable). For png-sequence chunks (each chunk is a directory of
  * frames) this is a straight directory merge with global re-numbering.
  *
- * Mux + faststart for mp4/mov go through the engine's `muxVideoWithAudio`
- * + `applyFaststart` helpers — same path the in-process renderer uses; we
- * just feed concat output rather than streaming-encoder output. Audio
- * length is pad-or-trimmed to `frameCount / fps` via
+ * Mux + faststart for mp4 / mov / webm go through the engine's
+ * `muxVideoWithAudio` + `applyFaststart` helpers — same path the
+ * in-process renderer uses; we just feed concat output rather than
+ * streaming-encoder output. (Faststart is a no-op for webm and mov —
+ * applyFaststart copies the input verbatim.) Audio length is
+ * pad-or-trimmed to `frameCount / fps` via
  * `padOrTrimAudioToVideoFrameCount` so the mux step doesn't introduce
  * sub-millisecond drift at the end of long renders.
  *
@@ -116,7 +120,7 @@ export async function assemble(
     return mergePngFrameDirs(chunkPaths, outputPath, plan.totalFrames, audioPath, start);
   }
 
-  // ── 2b. mp4 / mov: concat-copy then mux + faststart ────────────────────
+  // ── 2b. mp4 / mov / webm: concat-copy then mux + faststart ────────────
   if (!existsSync(dirname(outputPath))) {
     mkdirSync(dirname(outputPath), { recursive: true });
   }

packages/producer/src/services/distributed/plan.test.ts

@@ -460,3 +460,58 @@ describe("plan() — codec knob", () => {
     expect((caught as Error).message).toMatch(/codec must be "h264" or "h265"/);
   });
 });
+
+describe("plan() — webm format (distributed VP9)", () => {
+  const TIMEOUT_MS = 30_000;
+
+  it(
+    'maps `format: "webm"` to libvpx-vp9-software + yuva420p',
+    async () => {
+      // Webm is distributed-supported via closed-GOP concat-copy (PR 8.1
+      // proved the contract; this test pins the plan-time encoder choice).
+      // yuva420p preserves the format's reason for existing — alpha video
+      // for web playback over colored backgrounds.
+      const planDir = join(runRoot, "plan-webm-vp9");
+      mkdirSync(planDir, { recursive: true });
+      const result = await plan(
+        projectDir,
+        { fps: 30, width: 320, height: 240, format: "webm" },
+        planDir,
+      );
+      expect(result.format).toBe("webm");
+
+      const encoder = JSON.parse(
+        readFileSync(join(planDir, "meta", "encoder.json"), "utf-8"),
+      ) as Record<string, unknown>;
+      expect(encoder.encoder).toBe("libvpx-vp9-software");
+      expect(encoder.pixelFormat).toBe("yuva420p");
+      // Closed-GOP must be on so concat-copy at assemble time works.
+      // gopSize equals the chunkSize so every chunk's first frame is a
+      // keyframe with no alt-ref references reaching back across seams.
+      expect(encoder.closedGop).toBe(true);
+      expect(encoder.gopSize).toBe(encoder.chunkSize);
+    },
+    TIMEOUT_MS,
+  );
+
+  it("rejects `codec` with format=webm", async () => {
+    // webm is always libvpx-vp9 — same shape as mov (always ProRes 4444).
+    // A JS caller building config from JSON who passes `codec: "vp8"` or
+    // `codec: "av1"` must hit a typed error, not silently encode VP9.
+    const planDir = join(runRoot, "plan-webm-bad-codec");
+    mkdirSync(planDir, { recursive: true });
+    let caught: unknown;
+    try {
+      await plan(
+        projectDir,
+        // @ts-expect-error — runtime check is the test's purpose.
+        { fps: 30, width: 320, height: 240, format: "webm", codec: "vp8" },
+        planDir,
+      );
+    } catch (err) {
+      caught = err;
+    }
+    expect(caught).toBeInstanceOf(Error);
+    expect((caught as Error).message).toMatch(/codec.*only valid for format="mp4"/);
+  });
+});

packages/producer/src/services/distributed/plan.ts

@@ -74,12 +74,19 @@ export interface DistributedRenderConfig {
   width: number;
   height: number;
   /**
-   * Output container format. webm and HDR mp4 are not supported in
-   * distributed mode — `plan()` refuses them up front with a typed
+   * Output container format. HDR mp4 is not supported in distributed
+   * mode — `plan()` refuses it up front with a typed
    * `FormatNotSupportedInDistributedError`. The in-process renderer
-   * supports both.
+   * supports it.
+   *
+   * `"webm"` (VP9 + Opus) is distributed-supported via closed-GOP
+   * concat-copy: `lockGopForChunkConcat=true` forces a keyframe at every
+   * chunk boundary and disables libvpx-vp9's alt-ref frames so chunk
+   * files stitch losslessly. See `chunkEncoder.ts` for the VP9 args and
+   * `tests/distributed/_smoke/webm-concat-copy.test.ts` for the gating
+   * experiment that proved the contract.
    */
-  format: "mp4" | "mov" | "png-sequence";
+  format: "mp4" | "mov" | "png-sequence" | "webm";
   /**
    * Codec selection for `format: "mp4"`. `"h264"` (the default) → libx264 +
    * yuv420p; `"h265"` → libx265 + yuv420p with closed-GOP keyint params
@@ -169,7 +176,7 @@ export interface PlanResult {
   fps: 24 | 30 | 60;
   width: number;
   height: number;
-  format: "mp4" | "mov" | "png-sequence";
+  format: "mp4" | "mov" | "png-sequence" | "webm";
   ffmpegVersion: string;
   producerVersion: string;
 }
@@ -247,22 +254,25 @@ export class PlanTooLargeError extends Error {
 
 /**
  * Non-retryable error code raised when `plan()` is asked for an output
- * format that distributed mode doesn't support (webm, HDR mp4). The same
- * config would fail on every retry, so the failure must not auto-retry.
+ * format that distributed mode doesn't support (currently: HDR mp4). The
+ * same config would fail on every retry, so the failure must not
+ * auto-retry.
  */
 export const FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED = "FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED";
 
 /**
  * Typed error raised by `plan()` for outputs that distributed mode
  * refuses to ship.
  *
- *   - webm — VP9 + matroska concat-copy is fragile across libvpx-vp9
- *     builds, and the chunked pipeline can't guarantee bit-identical
- *     concat output across worker versions.
  *   - mp4 + HDR (PQ / HLG) — chunked HDR pre-extract + HDR signaling
  *     re-apply on the assembled file is not implemented yet.
  *
- * The in-process renderer (`executeRenderJob`) handles both natively.
+ * The in-process renderer (`executeRenderJob`) handles it natively.
+ *
+ * WebM was previously refused here; v0.7+ supports it via closed-GOP
+ * concat-copy. See {@link DistributedRenderConfig.format} for the
+ * supported set and {@link rejectUnsupportedDistributedFormat} for the
+ * gate.
  */
 export class FormatNotSupportedInDistributedError extends Error {
   readonly code: typeof FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED = FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED;
@@ -272,8 +282,8 @@ export class FormatNotSupportedInDistributedError extends Error {
     super(
       `[plan] format ${JSON.stringify(format)} is not supported in distributed mode: ${reason}. ` +
         `Render with the in-process renderer (\`executeRenderJob\`) — it has full format ` +
-        `support — or pick a distributed-supported format: mp4 SDR, mov ProRes 4444, or ` +
-        `png-sequence.`,
+        `support — or pick a distributed-supported format: mp4 SDR, mov ProRes 4444, ` +
+        `png-sequence, or webm VP9.`,
     );
     this.name = "FormatNotSupportedInDistributedError";
     this.format = format;
@@ -289,7 +299,9 @@ function formatBytes(bytes: number): string {
 }
 
 /**
- * Reject formats the distributed pipeline cannot ship (webm + HDR mp4).
+ * Reject formats the distributed pipeline cannot ship (HDR mp4 only —
+ * webm is supported as of v0.7 via closed-GOP concat-copy).
+ *
  * Throws {@link FormatNotSupportedInDistributedError} with a message
  * naming the rejected format. Runs at the very top of `plan()` so a
  * banned input never produces a partial planDir.
@@ -302,17 +314,6 @@ function formatBytes(bytes: number): string {
 export function rejectUnsupportedDistributedFormat(
   config: Pick<DistributedRenderConfig, "format" | "hdrMode">,
 ): void {
-  // The TypeScript type for `DistributedRenderConfig.format` already
-  // excludes webm, but a JS caller (or a caller that built the config
-  // dynamically from JSON) can still pass it. Belt-and-suspenders runtime
-  // check at the gate.
-  if ((config.format as string) === "webm") {
-    throw new FormatNotSupportedInDistributedError(
-      "webm",
-      "VP9 + matroska concat-copy is fragile across libvpx-vp9 builds, so chunked output " +
-        "can't be guaranteed byte-identical across workers",
-    );
-  }
   if ((config.hdrMode as string) === "force-hdr") {
     throw new FormatNotSupportedInDistributedError(
       "mp4-hdr",
@@ -513,7 +514,8 @@ function buildLockedRenderConfig(input: {
 /**
  * Resolve the encoder + pixel-format + preset triple for a distributed
  * render. Distributed mode is SDR-only: H.264 or H.265 8-bit for mp4,
- * ProRes 4444 for mov, raw RGBA for png-sequence.
+ * libvpx-vp9 + yuva420p (alpha) for webm, ProRes 4444 for mov, raw RGBA
+ * for png-sequence.
  *
  * `config.codec` is consulted only when `config.format === "mp4"`. Passing
  * `codec` with a non-mp4 format throws at plan time — surfaces the
@@ -547,12 +549,21 @@ function resolveEncoderTriple(config: DistributedRenderConfig): {
     throw new Error(
       `[plan] DistributedRenderConfig.codec is only valid for format="mp4"; received ` +
         `codec=${JSON.stringify(config.codec)} with format=${JSON.stringify(config.format)}. ` +
-        `Omit codec for non-mp4 formats — mov is always ProRes 4444 and png-sequence has no encoder.`,
+        `Omit codec for non-mp4 formats — mov is always ProRes 4444, webm is always ` +
+        `libvpx-vp9, and png-sequence has no encoder.`,
     );
   }
   if (config.format === "mov") {
     return { encoder: "prores-software", pixelFormat: "yuva444p10le", preset: "4444" };
   }
+  if (config.format === "webm") {
+    // webm distributes via closed-GOP libvpx-vp9 + concat-copy. yuva420p
+    // matches the in-process renderer's webm pixel format (alpha-capable
+    // — the format's main reason for existing). `getEncoderPreset` in
+    // the engine returns "good" for non-draft quality tiers; that becomes
+    // libvpx-vp9's `-deadline good` at encode time.
+    return { encoder: "libvpx-vp9-software", pixelFormat: "yuva420p", preset: "good" };
+  }
   return { encoder: "png-sequence", pixelFormat: "rgba", preset: "lossless" };
 }
 
@@ -642,10 +653,17 @@ export async function plan(
   // move the contents over once the staged work completes.
   const finalCompiledDir = join(planDir, "compiled");
 
-  // mov + png-sequence carry alpha — flip force-screenshot so compileStage
-  // takes the alpha-aware capture path (BeginFrame doesn't preserve alpha
-  // on Linux headless-shell).
-  const needsAlpha = config.format === "png-sequence" || config.format === "mov";
+  // webm + mov + png-sequence carry alpha — flip force-screenshot so
+  // compileStage takes the alpha-aware capture path (BeginFrame doesn't
+  // preserve alpha on Linux headless-shell). Must match the in-process
+  // renderer's needsAlpha logic in `renderOrchestrator.ts` so chunked
+  // webm output preserves the same alpha plane the in-process baseline
+  // does. Omitting webm here silently freezes `forceScreenshot: false`
+  // into the planDir and every chunk worker captures opaque RGB — the
+  // libvpx-vp9 alpha sub-stream then encodes either uniform alpha or
+  // gets downgraded by the encoder, producing un-keyable webm output.
+  const needsAlpha =
+    config.format === "png-sequence" || config.format === "mov" || config.format === "webm";
 
   // ── Compile ──
   const compileResult = await runCompileStage({