From e38c77f11ea7b09b8665803668ed8d6589997ea8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 3 Jun 2026 19:36:57 -0700 Subject: [PATCH 1/4] feat(watch): latency range with buffered playback Reframe playback latency as a range [latency-min, latency-max] in @moq/watch, unifying live (minimize) and buffered playback into one mechanism. The reference (wall-clock anchor) is re-anchored only when keeping it would push latency past the ceiling, so future-dated frames (e.g. TTS written faster than real-time) build up a buffer instead of dragging the playhead forward. Minimize latency is the degenerate case where the ceiling equals the floor, so there is no separate buffered code path. latency-max forms: - "real-time" (default) or <= floor: minimize, today's behavior. - a number (ms): buffer up to the cap, then skip ahead. - undefined / "none": uncapped, never skip. latency is sugar that collapses the floor and ceiling to one value; latency-min sets only the floor. reset() flushes the buffer and re-anchors at an utterance boundary. The decoded audio ring stays small (~1.5s). The lookahead lives upstream as encoded frames: the decoder awaits AudioBuffer.wait(timestamp) before decode(), which holds the frame as Opus until the playhead nears it, instead of buffering seconds of PCM. Backpressure works on both transports (Atomics poll on the SharedArrayBuffer path, worklet state messages on the postMessage fallback). A finite latency-max keeps sizing the ring to the cap and dropping via overflow. Follow-up: plumb latency-max into the wire SubscribeUpdate.maxLatency so the relay also reorders/drops groups past the cap (currently sends 0). Co-Authored-By: Claude Opus 4.8 (1M context) --- doc/lib/js/@moq/watch.md | 62 ++++++++ js/watch/src/audio/buffer.ts | 136 ++++++++++++++-- js/watch/src/audio/decoder.ts | 18 ++- js/watch/src/audio/render-worklet.ts | 3 + js/watch/src/audio/render.ts | 10 +- js/watch/src/audio/ring-buffer.test.ts | 60 ++++++++ js/watch/src/audio/ring-buffer.ts | 61 +++++++- js/watch/src/audio/shared-ring-buffer.test.ts | 66 ++++++++ js/watch/src/audio/shared-ring-buffer.ts | 41 ++++- js/watch/src/backend.ts | 51 +++++- js/watch/src/element.ts | 82 ++++++++-- js/watch/src/sync.test.ts | 62 ++++++++ js/watch/src/sync.ts | 145 ++++++++++++++---- js/watch/src/ui/components/buffer-control.ts | 4 +- 14 files changed, 726 insertions(+), 75 deletions(-) create mode 100644 js/watch/src/sync.test.ts diff --git a/doc/lib/js/@moq/watch.md b/doc/lib/js/@moq/watch.md index 89f0c57612..9d92f45e44 100644 --- a/doc/lib/js/@moq/watch.md +++ b/doc/lib/js/@moq/watch.md @@ -70,6 +70,9 @@ a real bundler (the examples below). - `paused`: Pause playback (boolean) - `muted`: Mute audio (boolean) - `volume`: Audio volume (0 to 1, default: 1) +- `latency`: Latency target. `"real-time"` (default) derives it from RTT, or a number sets a fixed jitter buffer in ms. Sugar for collapsing `latency-min` and `latency-max` to one value (minimize latency). +- `latency-min`: Latency floor (the jitter/startup buffer). Same units as `latency`; leaves the ceiling untouched. +- `latency-max`: Latency ceiling. `"real-time"` (default) minimizes latency, a number caps at that many ms, and `"none"` is uncapped. A ceiling above the floor enables [buffered playback](#buffered-playback): build up a buffer from future-dated frames instead of skipping ahead. - `catalog-format`: Catalog format. One of `"hang"`, `"msf"` (see [MSF](/concept/standard/msf)), or `"manual"` (supply the catalog yourself). When omitted, the format is auto-detected from the broadcast `name` extension (`.hang` or `.msf`), falling back to `"hang"`. ## Catalog Formats @@ -139,6 +142,65 @@ el.catalog = myCatalog; > `"msf"`) tears down the previous fetch loop, which clears `catalog`. Set the > catalog *after* switching to `"manual"`, not before. +## Buffered playback + +Latency is a **range**, `[latency-min, latency-max]`. By default the range is +collapsed (`latency` sets both to one value) and `@moq/watch` minimizes latency: +it anchors playback to the earliest frame seen relative to its timestamp and skips +ahead whenever the buffer grows past the target. That is right for live +conferencing, but wrong for content written *faster than real-time* with +timestamps in the future, such as a text-to-speech response streamed all at once. + +Open the range, by setting a `latency-max` above the floor, to anchor playback to +the first frame received and play through at the encoded pace. The buffer is +allowed to float anywhere between the floor and the ceiling without skipping: + +```html + + + +``` + +```typescript +const el = document.querySelector("moq-watch")!; +el.latencyMin = 100; // floor: start after 100ms buffered +el.latencyMax = 30_000; // ceiling: never skip until 30s buffered +``` + +`latency-min` is the jitter/startup buffer (it can also be `"real-time"` for an +adaptive floor). `latency-max` is the ceiling, and it has three forms: + +- a **number** (ms): buffer freely up to the cap, then skip ahead, so latency + stays at most that far behind the newest frame. +- **`"none"`** (`el.latencyMax = undefined` in JS): uncapped. Playback never skips + ahead. The decoded audio ring stays small (~1.5s); the lookahead is held upstream + as encoded frames, and the decoder applies backpressure (stops decoding ahead) + rather than dropping audio or buffering seconds of PCM. +- **`"real-time"`** (the default) or any value `<= latency-min`: collapsed, i.e. + today's minimize-latency behavior. + +The mechanism is the same in every case: the playhead is anchored on the first +frame and only re-anchored (skipped forward) when keeping it would push latency +past `latency-max`. Minimize is just the degenerate case where the ceiling equals +the floor. + +At each new utterance, call `reset()` to flush the audio buffer and re-anchor +playback to the next frame. The producer can interrupt by writing a new utterance +(optionally on a new track) and the viewer calls `reset()` to drop whatever was +still buffered: + +```typescript +el.reset(); +``` + +This removes the need to pace writes on the producer: emit the whole response as +fast as possible with correct (future) timestamps, and `reset()` on interruption. + +> Buffered playback uses the WebCodecs path (a `` child). It does not +> apply to the MSE `