Skip to content

feat(moq-video,libmoq): native H.264 decode (drop ffmpeg dependency)#1796

Merged
kixelated merged 8 commits into
devfrom
claude/moq-video-decode
Jun 20, 2026
Merged

feat(moq-video,libmoq): native H.264 decode (drop ffmpeg dependency)#1796
kixelated merged 8 commits into
devfrom
claude/moq-video-decode

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Summary

Adds a native H.264 decode path mirroring moq-video's existing native encode side, and exposes it through libmoq. This is the symmetric counterpart to how audio already works (moq-audio decodes Opus natively → moq_consume_audio_raw), and the prerequisite for removing ffmpeg from the in-tree OBS plugin (cpp/obs), which currently decodes subscribed video with libavcodec + libswscale.

This is Phase 1 (the Rust side) of that effort. The OBS plugin rewrite + CI/docs is a follow-up on top of #1787 (it can't compile against this API until it lands).

rs/moq-video — new decode module (mirror of encode)

  • decode::Consumer — the direct counterpart to moq_audio::AudioConsumer: subscribes to a moq-mux H.264 track and returns raw packed-I420 decode::Frames.
  • Pluggable backends behind the same trait/selector pattern as encode:
    • videotoolbox — macOS hardware (VTDecompressionSession, hand-written on the objc2-video-toolbox bindings).
    • openh264 — portable software fallback (the path that lets a Linux build ship without ffmpeg).
  • decode::Decoder does the access-unit prep: avc1 (length-prefixed + out-of-band avcC) → Annex-B with SPS/PPS injected ahead of keyframes (reusing moq_mux::codec::{annexb,h264} helpers); avc3 passes through. Keyframe-gated.
  • H.264 only (symmetric with what moq-video encodes); a non-H.264 rendition surfaces Error::UnsupportedCodec. Native decode of HEVC/VP9/AV1/VP8 (which ffmpeg did) is dropped.

rs/libmoq — new video-decode C API

  • moq_consume_video_raw (+ _close / _frame / _frame_free), the video twin of moq_consume_audio_raw. Decoding happens inside the staticlib, so callers get a native decoder with no ffmpeg. cbindgen auto-emits the new types/functions into moq.h.

Public API changes

  • rs/moq-video (additive): new pub mod decode with decode::{Consumer, Config, Kind, Frame}; new Error::{NoDecoder, UnsupportedCodec} variants (enum is #[non_exhaustive]). No existing signatures changed.
  • rs/libmoq (additive C ABI): moq_consume_video_raw{,_close,_frame,_frame_free}, moq_video_frame, moq_video_decoder_output; new Error::Video variant (code -36).

Both are additive, so non-breaking, but this targets dev per the branch-targeting rules (touches rs/moq-video / rs/libmoq public surface).

Test plan

  • decode round-trip unit tests (openh264 encode → decode) for both backends, incl. a macOS-gated VideoToolbox test.
  • End-to-end libmoq test (video_raw_decode): publishes real H.264 and decodes it through the C API, asserting a 320×240 packed-I420 frame.
  • cargo clippy clean, cargo fmt / taplo clean (via nix develop).
  • cargo check --workspace --all-targets green.
  • Runtime-verified on macOS (VideoToolbox + openh264). Windows/Linux-hardware decode backends are future follow-ups.

Follow-up (out of scope here)

  • OBS plugin (Phase 2, after feat(obs): vendor the OBS plugin in-tree under cpp/obs #1787): rewrite cpp/obs/src/moq-source.cpp onto moq_consume_video_raw (hand OBS VIDEO_FORMAT_I420 directly, drop avcodec/swscale), strip ffmpeg from cpp/obs/CMakeLists.txt, re-enable a portable Linux binary in the obs-build matrix, and update doc/bin/obs.md.
  • MediaFoundation / NVDEC / VAAPI decode backends (decode parity with encode).

🤖 Generated with Claude Code

(Written by Claude)

kixelated and others added 4 commits June 18, 2026 22:44
Add a native H.264 decode path mirroring the existing native encode side,
so consumers (the OBS plugin) no longer need ffmpeg to decode subscribed
video.

- moq-video: new `decode` module mirroring `encode`. `decode::Consumer`
  (the counterpart to moq-audio's AudioConsumer) subscribes to a moq-mux
  H.264 track and returns raw I420 frames. Backends are VideoToolbox
  (macOS) and openh264 (portable software fallback). avc1 payloads are
  converted to Annex-B with SPS/PPS injected ahead of keyframes; avc3
  passes through.
- libmoq: new `moq_consume_video_raw` (+ `_close`/`_frame`/`_frame_free`)
  C API, the video counterpart to `moq_consume_audio_raw`. Decoding now
  happens inside the staticlib, with no ffmpeg.

Only H.264 is supported (symmetric with what moq-video encodes); a
non-H.264 rendition surfaces UnsupportedCodec.

Verified on macOS: encode/decode round-trips for both backends, plus an
end-to-end libmoq test that publishes real H.264 and decodes it via the
C API.

The OBS plugin rewrite + CI/docs (removing ffmpeg from cpp/obs) is a
follow-up on top of #1787.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…module

`cargo doc -D warnings` rejects the intra-doc link (rustdoc::private-intra-doc-links);
drop it to plain text.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…coder

Replace the hand-rolled Annex-B start-code scanner (and its trailing-zero
heuristic) in the VideoToolbox backend with `moq_mux::codec::annexb::NalIterator`,
the same tested helper the rest of the codebase uses. The backend `decode` now
takes an owned `Bytes`, so NAL slices (incl. SPS/PPS) come out zero-copy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kixelated kixelated enabled auto-merge (squash) June 19, 2026 19:21
@kixelated kixelated merged commit 8eda0ea into dev Jun 20, 2026
1 check passed
@kixelated kixelated deleted the claude/moq-video-decode branch June 20, 2026 10:50
kixelated added a commit that referenced this pull request Jun 20, 2026
…-32ccb0

Brings the branch up to date with dev (HEVC #1802, native H.264 decode #1796,
async device capture #1807, srt/rtmp/rtc egress) and keeps this PR's deltas:
NVENC/VAAPI always-on for Linux, openh264 always-on (no `software` feature),
publish=false removed, the nvidia-video-codec-sdk fork repinned to the hardened
dynamic-loading commit (default-features off), and the NVIDIA-probe fallback.

Note: restores dev's H.265/HEVC codec-aware encode path (Codec enum, codec-aware
backend selection, hev1 producer) that the previous branch merge had dropped by
taking "ours" wholesale. Verified: clippy -D clean, 19 moq-video tests pass
(incl. H.265 roundtrips), sort/taplo/fmt clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant