docs: design for track archival + per-track moq-net cache

[kixelated]

Summary

Design-only (no implementation) for saving a MoQ track to durable storage and serving it back, plus the moq-net primitive it rides on. Two docs, refined through review:

• rs/moq-archive/DESIGN.md — the archive crate: record a track, serve old groups back through FETCH, tier RAM → disk → S3 via object_store, retention by media timestamp with wall-clock fallback. Includes the segment + manifest format, out-of-order group handling, usage mockups (standalone/VOD vs why the relay needs a different seam), and a prior-art survey (BookKeeper, Kafka KIP-405, Haystack/Bitcask, Parquet/Iceberg).
• rs/moq-net/CACHE.md — the per-track cache spike the archive builds on (see below).

Targets dev: it removes a public/wire field (TrackInfo.cache) and adds local API to TrackProducer.

The per-track cache (settled shape)

The relay/edge integration is not a Cache trait moq-net calls back into (rejected: no inversion of control). Instead:

• A concrete CacheConfig owned by TrackProducer only — local policy, never on the wire, never the original publisher's concern. TrackInfo.cache is removed.
• Per-track [min, max] bounds on size and duration. No shared/global LRU, so no shared lock; footprint is sum of per-track max.
• Watermark flush: groups accumulate to the high mark, then the max − min band flushes as one segment. This is the property an LRU lacks — it would emit one tiny object per group (fatal for audio). An interval backstop covers low-rate tracks.
• Tiers RAM → disk → remote via object_store, feature-gated so RAM-only stays dependency-free. On-tier bytes reuse the archive segment format, so a relay's spill is directly readable by an archive node.
• Served by ranged read from whichever tier holds the group; no fault-in.

Related design threads captured (not yet built)

• Splitting Origin into Announced (routing table, multi-connection) + registerable dynamic() handlers, so moq-relay/moq-cli/moq-edge compose without traits and the relay stays ~thin. The cache config threading onto auto-created tracks is part of this follow-up.
• Per-broadcast/track billing enablement (JWT archive claim + server-side policy + optional publisher signal).

Test plan

• N/A — documentation only. Review focuses on the cache shape (per-track bounds + watermark flush), removing TrackInfo.cache, the archive segment/manifest format, and the open questions before any code lands.

🤖 Generated with Claude Code

(Written by Claude)

[kixelated]

I think we just take a TrackDynamic.

[kixelated]

Note the caller will do this. For example, moq-relay might first try to use moq-archive to handle any dynamic requests.

I think moq-archive also needs it's own dynamic() to make that work? Because if something isn't on disk or on S3, we'll have to ask the original publisher for it.

That might be an argument for stuffing this logic directly into moq_net::TrackProducer and moq_net::TrackConsumer? It feels a bit gross but IDK the API is far more friendly. We would also definitely know when data is evicted from RAM cache too.

[kixelated]

Captured this as the lead open question and kept v1 wiring the chain in moq-archive (caller owns the TrackProducer, archive takes the TrackDynamic plus an upstream TrackConsumer to forward misses + record). Agreed the moq-net-native version is the friendlier API and the only one that reliably knows when a group is evicted from the RAM cache, which is the natural flush trigger.

The way I'd frame the decision: do we want moq_net::TrackProducer to accept a pluggable storage backend (a trait Cache it consults on a miss and notifies on eviction), so the archive becomes "attach storage to a track" rather than "wire a fallback chain"? That keeps storage code out of moq-net but gives moq-net the one hook (eviction) we can't get from the outside. If you're good with that shape I'll redesign the public API around it before any implementation; if you'd rather keep moq-net pure for now, the chain version stands. Which way do you want to go?

(Written by Claude)

---

Generated by Claude Code

[kixelated]

The broadcast and track names can have slashes. I guess we need to escape them.

[kixelated]

We might remove this from moq-lite-05. The current API doesn't even support it.

[kixelated]

I think we should merge multiple groups at each rollup step. So RAM can be fragmented, but each rollup becomes less fragmented.

ex. store for up to 30s in RAM, flush in 10s segments to disk. Store on disk for up to 5m, flush to S3 in 1m batches?

IDK if we want to get an LRU involved or we just eyeball RAM usage though. I feel like an LRU is correct in general for both RAM and disk though. We could use the used() and unused() state too I guess. No sense keeping unused stuff in RAM if we could combine it with the next flush to disk.

[kixelated]

We always keep the latest group in RAM.

[kixelated]

We should try to nail down the index file format for sure.

[kixelated]

Folded all the review comments into the latest commit:

• Take a TrackDynamic (not a TrackProducer): the archive is now a composable link in a cache chain (relay → archive → origin); caller owns the producer.
• Miss fallback / chaining: on a storage miss the archive forwards to an upstream TrackConsumer instead of returning NotFound; whether this should instead live inside moq_net core types is now the lead open question (replied on that thread for your call).
• Rollup: each tier concatenates multiple units from the tier above (RAM 30s → 10s disk segments → 1m S3 objects), so fragmentation drops downward. Resolves the old 1:1-vs-concatenate question toward concatenate-at-every-rollup.
• Eviction: LRU + byte budget per tier, not just age; always keep the latest group in RAM; use used()/unused() to flush unused groups early.
• Index format nailed down: per-segment postcard footer (group offset table) + per-track manifest, Parquet/Iceberg-shaped, replacing the JSONL sketch.
• Slashes: percent-encode broadcast/track names into single opaque path segments.
• Sub-group frame_start: dropped (likely removed from moq-lite-05, unsupported by the current API); the archive serves whole groups.

Also added a prior-art survey (BookKeeper ledgers, Kafka KIP-405 segments, Haystack/Bitcask, Parquet/Iceberg) and why no single embeddable crate covers batch + index + S3 tiering, in answer to the earlier "is there an existing library?" question.

(Written by Claude)

---

Generated by Claude Code

[kixelated]

Added a Usage mockups section with concrete call sites, and you're right that the relay doesn't fit the per-track API. Three scenarios:

A. Standalone / VOD node — works with serve(TrackDynamic) today, because the archive is the publisher: it publish_broadcasts, and for each requested_track it accepts the producer, takes producer.dynamic(), and serves from storage. No path collision since the live broadcast is gone.

B. Why moq-relay can't use it as-is — the relay only holds a single OriginProducer and forwards whole broadcasts; it never constructs a TrackProducer or calls .dynamic(). That happens inside moq-net's lite/ietf subscriber fan-out, so there's no point in relay code to hand the archive a TrackDynamic. Forcing it would mean republishing every broadcast through an archive-owned BroadcastProducer and re-accepting every track, i.e. reimplementing fan-out around the archive.

C. The moq-net seam (recommended) — a pluggable trait Cache that moq-net consults on a miss and notifies on eviction, attached at the origin where it already owns the per-track RAM cache. Then the whole relay integration is:

let origin = Origin::random().produce().with_cache(archive);

moq-net catches evictions (the natural flush trigger, which the chain version can't see) and serves misses before they cost an upstream FETCH. The public Archive becomes a Cache impl plus the standalone record/serve helpers.

This is the same fork as the open question I flagged, now made concrete: do you want me to redesign the API around a moq_net::Cache trait (relay-friendly, needs a small moq-net addition), or keep the per-track chain version (standalone/VOD only for now)? I'd recommend the Cache trait. Pushed in 93046420.

(Written by Claude)

---

Generated by Claude Code

[kixelated]

Extensive review (self-review with three finder passes + verification)

Ran a high-recall review over origin/dev...HEAD (correctness, removed-behavior, cross-file, reuse/simplification/efficiency, altitude, conventions). Fixes pushed in 71deaf25. Below: the public-API surface, what was fixed, and two open design calls for you.

Public API surface (all additive → not breaking; dev is correct)

New under moq_net::cache (pub mod cache):

• Config (#[non_exhaustive], Default), Bounds, Limit, Frame, Group, Batch
• Producer (intentionally not Clone), Consumer (Clone); Group::read / Group::produce bridges
• cache::segment: encode, rollup, group_from_blob, Segment, GroupEntry (#[non_exhaustive]), Error
• cache::store (behind cache-tiered): Store, Error
• TrackProducer::with_cache, TrackConsumer::with_cache

No existing signature changed. The one planned breaking change, removing the wire field TrackInfo.cache, is still pending and is dev-appropriate.

Correctness findings — fixed

1. Disk tier could wipe itself. index::promotion drained the entire disk tier when the low watermark was unset (floor of zero), so one over-max trip evicted even the newest flushed groups on the no-remote path. Now keeps the newest segment, mirroring the RAM tier's "keep latest" rule.
2. Crash-consistency. store::flush/compact mutated the index before the put/upload. A failed write stranded sequences pointing at a nonexistent object. Reordered to write-then-index (added Index::next_id() to reserve the key); a failure now leaves the index intact.
3. Overflow guard. store::get computed offset + length unchecked; a corrupt footer could overflow u64. Now checked_add, matching segment::blob.
4. fetch vs get inconsistency. fetch_group's cache path surfaced a produce() rebuild error (e.g. timescale mismatch on a not-yet-accepted wire consumer) as a hard fetch failure, while get_group treated it as a miss. Now both rebuild synchronously and miss-through on error.

Cleanup — applied

• store_of now expects a configured remote tier instead of silently reading disk for a remote location.
• index is internal orchestration (only the gated store uses it) → gated behind cache-tiered and dropped from the public surface; segment::GroupEntry got #[non_exhaustive].
• Varint coding correctly reuses VarInt::{encode,decode}_quic (not duplicated); both error enums use thiserror + #[from] + #[non_exhaustive]; no em dashes.

Two open decisions for you (not unilaterally changed)

A. TrackProducer::with_cache populate disables demand teardown. The populate path subscribes internally, so producer.unused() never resolves while a cache is attached, and a relay that drops idle tracks by demand (lite/ietf subscriber, session.rs) won't drop a cached one. That's intended for "keep recording when idle" (archive/DVR), but wrong for a transient relay cache. Options: keep as-is (documented now); populate on group eviction instead (no subscription, but needs a sync frame-snapshot primitive on groups); or populate from the relay's existing fan-out consumer. This is the same lifecycle question I flagged earlier — worth settling before wiring the relay.

B. Same populate loop head-of-line-blocks on each group's finish (Group::read resolves only when a group finishes), so a stalled group delays caching of later finished ones. Fixable with concurrent reads (FuturesUnordered) if we keep the subscribe approach — I held off pending the decision in A, since the populate design may change.

Both caveats are now documented on with_cache. Everything else is fixed; default suite 427 pass, cache-tiered 39 pass, clippy/rustdoc/fmt clean.

(Written by Claude)

---

Generated by Claude Code