Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 2 additions & 129 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 8 additions & 5 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,11 +88,14 @@ moq-cli publish --url https://relay.example.com --broadcast cam.hang capture --n
moq-cli publish --url https://relay.example.com --broadcast cam.hang capture --no-video
```

Video capture uses the platform backend (avfoundation on macOS, v4l2 on Linux,
dshow on Windows) and picks a hardware encoder (`h264_videotoolbox` /
`h264_nvenc` / `h264_vaapi`) when one is present, falling back to software
(`libx264`); force either with `--hardware` / `--software`. Audio capture uses
cpal (CoreAudio / WASAPI / ALSA) and encodes Opus.
Video capture uses a native per-platform backend (AVFoundation on macOS, V4L2 on
Linux, Media Foundation on Windows) and picks a hardware H.264 encoder
(VideoToolbox on macOS, NVENC on Linux NVIDIA) when one is present, falling back
to the built-in software encoder (openh264); force either with `--hardware` /
`--software`. `--camera` takes a bare integer as a device index, otherwise a
device path (Linux) or name (a friendly-name substring on Windows, the
AVFoundation `uniqueID` on macOS). Audio capture uses cpal (CoreAudio / WASAPI /
ALSA) and encodes Opus.

Alternatively, pipe an external FFmpeg process as MPEG-TS:

Expand Down
7 changes: 4 additions & 3 deletions rs/moq-video/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ objc2-screen-capture-kit = "0.3.2"
objc2-video-toolbox = "0.3.2"

[target.'cfg(target_os="windows")'.dependencies]
# Windows still uses nokhwa (Media Foundation) until a native MF capture lands;
# `decoding` handles its MJPEG/YUYV -> RGBA conversion.
nokhwa = { version = "0.10.11", default-features = false, features = ["input-msmf", "decoding"] }
# Native Media Foundation webcam capture (replaces nokhwa). An `IMFSourceReader`
# with its video processor enabled converts the camera's native format to NV12,
# which we deinterleave to I420. Links only system import libs, no vendor SDK.
windows = { version = "0.62", features = ["Win32_Foundation", "Win32_System_Com", "Win32_Media_MediaFoundation"] }
36 changes: 27 additions & 9 deletions rs/moq-video/DESIGN-native-codecs.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
# Native H.264 codecs for moq-video (drop ffmpeg)

Status: **phases 2-5 implemented** (openh264, VideoToolbox, NVENC, capture swap +
ffmpeg removal). VAAPI (phase 6) is a separate follow-up PR. See "As built"
at the bottom for where the implementation diverged from this plan.
ffmpeg removal). Capture is now native on all three platforms (AVFoundation /
ScreenCaptureKit on macOS, V4L2 on Linux, Media Foundation on Windows), so
**nokhwa is fully removed**. VAAPI (phase 6) stays parked until cros-codecs
builds against modern libva. See "As built" at the bottom for where the
implementation diverged from this plan.

## Goal

Expand Down Expand Up @@ -329,10 +332,15 @@ Where the implementation differs from the plan above:
negotiated resolution, so capture frames already match the encoder; `encode_rgba`
now requires input dims == encoder dims (it errors otherwise) instead of
rescaling. This dropped swscale entirely.
- **Capture (nokhwa) yields RGBA.** `Camera::read` decodes each frame to RGBA via
`decode_image::<RgbAFormat>`, uniformly handling MJPEG/YUYV/NV12 sources. Device
string is now "bare integer = index, else path/name" (the avfoundation `:none` /
dshow `video=` specifics are gone).
- **Capture is per-platform native (nokhwa fully removed).** Each platform's
`Camera::read` yields a `Frame` the encoder can take: macOS hands VideoToolbox a
zero-copy `CVPixelBuffer` surface, Linux V4L2 and Windows Media Foundation hand
the software/NVENC path a CPU `I420`. macOS AVFoundation/ScreenCaptureKit and
Linux V4L2 (YUYV resampled, MJPEG via `zune-jpeg`) landed first; Windows uses an
`IMFSourceReader` with its video processor enabled to coerce the camera's native
format to NV12, which we deinterleave to I420 (`I420::from_nv12`). The device
string is uniform: "bare integer = index, else path/name" (a friendly-name
substring on Windows).
- **`Error` slimmed.** The ffmpeg-specific variants (`Ffmpeg`, `NoCaptureBackend`,
`NoVideoStream`) and the `From<ffmpeg_next::Error>` impl are removed; capture and
encode failures now flow through `Error::Codec(anyhow)`.
Expand All @@ -350,6 +358,15 @@ Where the implementation differs from the plan above:
default - feature unification here may need tweaking; (2) the flat input-buffer
`write` matches NVENC's pitch (safe only for 64-aligned widths; we warn otherwise);
(3) forced-IDR via `picture_type` with picture-type-decision enabled.
- **Windows Media Foundation capture is UNVERIFIED on hardware.** The `windows`
0.62 FFI is fully type-checked against the `x86_64-pc-windows-msvc` target (the
whole crate can't cross-compile from the dev Mac because openh264's vendored C++
build needs MSVC, but a scratch crate confirms every Media Foundation call), and
the COM/refcount lifecycle is reviewed. Still needs a real Windows + webcam run
to confirm: (1) the source reader's video processor actually delivers NV12 for
common cameras (MJPEG/YUY2 sources); (2) `IMF2DBuffer::ContiguousCopyTo` yields
unpadded NV12 so the I420 deinterleave is correct; (3) the on-demand
open/`source.Shutdown()` cycle releases the camera (LED off) and reopens cleanly.

### VAAPI removed (cros-codecs is not buildable on modern libva)

Expand Down Expand Up @@ -406,8 +423,9 @@ can carry NVENC and use it only where the driver is present. Upstream the
because its SDK crate has no macOS bindings (see `rs/justfile`'s `ci` recipe).
- Still needed on real hardware: NVENC encode validation on a Linux+GPU box (pitch
alignment, forced-IDR); only synthetic-frame software encode is tested here.
- Live camera run (capture needs camera/screen TCC permission, which a headless or
agent-spawned process can't obtain; run `moq-cli ... capture` from a user terminal).
- `doc/bin/cli.md`: the `capture` device-string semantics changed (index-or-path).
- Live camera run (capture needs camera/screen permission, which a headless or
agent-spawned process can't obtain; run `moq-cli ... capture` from a user
terminal). On Windows this is also where the Media Foundation path gets its
first real exercise (see the unverified note above).
- Consider reusing NVENC input/output buffers across frames (currently allocated
per frame to sidestep the self-referential Session borrow).
Loading