Export pipeline speed

[richiemcilroy]

Implement GPU NV12 export path and optimize H264 encoder settings for faster video export.

The primary goal is to leverage the GPU for RGBA to NV12 conversion, drastically reducing CPU readback data and eliminating CPU-based swscale. Additionally, encoder settings are tuned for export throughput (e.g., realtime=false for VideoToolbox, tune=hq for NVENC) instead of recording-optimized low latency. This PR also includes necessary pipeline adjustments like flush_pipeline_nv12 and robust RGBA fallback handling to ensure correctness.

---

 

Greptile Summary

This PR introduces a GPU-accelerated NV12 export pipeline to replace the previous RGBA-based export path, reducing GPU readback data by ~62.5% and eliminating CPU-based swscale conversion. It also tunes H264 encoder settings for export quality/throughput (e.g., realtime=false for VideoToolbox, tune=hq for NVENC) separately from recording-optimized low-latency settings.

• GPU NV12 pipeline: New render_video_to_channel_nv12 function and export_nv12 method that perform RGBA-to-NV12 conversion on the GPU via compute shader, with robust RGBA fallback when GPU conversion fails
• Pipelined rendering: render() and render_nv12() now return Option<Frame> to support double-buffered readback pipelining; new render_immediate() wraps the old blocking behavior for callers that need synchronous frame output
• Frame prefetching: Overlaps next-frame decoding with current-frame rendering via tokio::join! for better pipeline utilization
• Arc-wrapped frame data: RenderedFrame.data and Nv12RenderedFrame.data changed from Vec<u8> to Arc<Vec<u8>> to reduce cloning costs
• Encoder optimizations: Per-encoder export settings (VideoToolbox, NVENC, QSV, AMF, MF, libx264) with with_export_settings() builder, reusable frame encoding via queue_frame_reusable, and with_external_conversion() to skip internal swscale
• Decoder threading: Software decode now uses both FF_THREAD_FRAME and FF_THREAD_SLICE with CPU-count-aware thread limits
• Clippy fixes: Collapsed nested if-let chains across test crates, derived Default for AudioConfig
• Potential dimension mismatch: The aligned output_size (4-aligned width) used for the encoder may not match the renderer's actual output dimensions when resolutions aren't multiples of 4, since self.resolution_base (unaligned) is passed to the renderer
• Test compilation issue: ensure_nv12_data_passthrough_for_nv12_format test constructs Nv12RenderedFrame with Vec<u8> but the field now expects Arc<Vec<u8>>

Confidence Score: 3/5

• This PR has a test compilation error and a potential dimension mismatch bug for non-standard resolutions, but the core NV12 pipeline logic is sound for common resolutions.
• Score of 3 reflects: (1) a definite compilation error in the test suite where Vec<u8> is used where Arc<Vec<u8>> is needed, and (2) a potential dimension mismatch between the NV12-aligned encoder dimensions and the renderer's actual output dimensions when width is even but not a multiple of 4. The core pipeline changes (pipelined readback, prefetching, Arc-wrapped data, encoder settings) are well-structured and the RGBA fallback path provides safety for edge cases.
• crates/export/src/mp4.rs needs attention for both the test compilation fix and the dimension mismatch between aligned output_size and unaligned resolution_base passed to the renderer.

Important Files Changed

Filename	Overview
crates/export/src/mp4.rs	Major changes to switch export pipeline to NV12 GPU path. Contains a dimension mismatch bug between renderer output and encoder expectations for non-4-aligned widths, and a test compilation error from Arc<Vec<u8>> type change.
crates/rendering/src/lib.rs	Adds render_video_to_channel_nv12, decode_segment_frames_with_retry helper, prefetch-while-rendering optimization, and render_immediate wrapper. Pipeline signature changes from direct frame return to Option<RenderedFrame> for pipelined readback. Well-structured refactor.
crates/rendering/src/frame_pipeline.rs	Changes RenderedFrame.data and Nv12RenderedFrame.data from Vec<u8> to Arc<Vec<u8>> for cheaper cloning. Adds bind group caching using pointer-based texture identity. Pipelined readback now returns Option instead of blocking. Adds Nv12RenderedFrame helper methods.
crates/enc-ffmpeg/src/video/h264.rs	Adds is_export flag to distinguish recording vs export encoder settings. Export mode enables quality-optimized settings per-encoder: VideoToolbox (realtime=false, profile=main), NVENC (preset=p5, tune=hq), QSV (preset=medium), AMF (quality=quality), libx264 (veryfast default). Adds queue_frame_reusable method for zero-alloc frame encoding.
apps/desktop/src-tauri/src/export.rs	Updates callers to use render_immediate (which handles the new Option return from pipelined render). Removes a 200ms sleep before export start.
apps/desktop/src-tauri/src/frame_ws.rs	Changes WSFrame.data from Vec<u8> to Arc<Vec<u8>> to match the rendering crate's change. Uses Arc::try_unwrap with clone fallback when ownership is needed. Collapses nested if-let per clippy lint.
crates/video-decode/src/ffmpeg.rs	Improves software decoder threading: uses both FF_THREAD_FRAME and FF_THREAD_SLICE, adjusts thread count to use clamp instead of hard-coded values, better scaling with CPU count.

Flowchart

flowchart TD
    A[render_video_to_channel_nv12] --> B{Decode Frame}
    B -->|Prefetch next frame| C[tokio::join! decode + render]
    B -->|No prefetch| D[render_nv12]
    C --> D
    D --> E{GPU NV12 Converter}
    E -->|Success| F[Pipelined NV12 Readback]
    E -->|Width not 4-aligned or GPU fail| G[RGBA Fallback via finish_encoder]
    F --> H[Return previous NV12 frame]
    G --> I[Nv12RenderedFrame with format=Rgba]
    H --> J[mpsc channel to encoder thread]
    I --> J
    J --> K[ensure_nv12_data]
    K -->|format=Nv12| L[Pass through Arc data]
    K -->|format=Rgba| M[CPU swscale RGBA to NV12]
    L --> N[fill_nv12_frame]
    M --> N
    N --> O[queue_frame_reusable]
    O --> P[H264 Encoder with export settings]
    P --> Q[MP4 Output]

Loading

_{Last reviewed commit: 5db2514}

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[tembo]

This output_size alignment is only applied on the encoder side. The renderer still derives its own output size from resolution_base/project, and RgbaToNv12Converter::submit_conversion returns false when width isn’t divisible by 4. In that case you’ll fall back to RGBA frames whose dimensions can differ from the aligned VideoInfo/ffmpeg frame size, which looks like it could silently corrupt output (truncated copy) when raw_output_size.0 % 4 != 0.

Suggested change

	let output_size = ((raw_output_size.0 + 3) & !3, (raw_output_size.1 + 1) & !1);
	if output_size != raw_output_size {
	info!(
	raw_width = raw_output_size.0,
	raw_height = raw_output_size.1,
	aligned_width = output_size.0,
	aligned_height = output_size.1,
	"Aligned output dimensions for NV12 GPU path"
	);
	}
	info!(
	width = output_size.0,
	height = output_size.1,
	"Using GPU NV12 export path (reduced readback + no CPU swscale)"
	);
	let output_size = raw_output_size;
	info!(
	width = output_size.0,
	height = output_size.1,
	"Exporting with NV12 pipeline (GPU when possible, CPU fallback otherwise)"
	);

[tembo]

Minor perf note: frame.into_data() will almost always clone here because render_video_to_channel_nv12 keeps last_successful_frame (another Arc) for fallback. If the goal is to avoid per-frame copies, it’d be worth carrying Arc<Vec<u8>> through Nv12ExportFrame for the NV12 case and only allocating/cloning when doing the RGBA->NV12 CPU fallback.

[tembo]

Small readability tweak (and avoids the closure):

Suggested change

	std::sync::Arc::try_unwrap(frame.data).unwrap_or_else(|arc| (*arc).clone()),
	std::sync::Arc::unwrap_or_clone(frame.data),

[tembo]

Same here—Arc::unwrap_or_clone reads a bit cleaner.

Suggested change

	Arc::try_unwrap(self.data).unwrap_or_else(|arc| (*arc).clone())
	Arc::unwrap_or_clone(self.data)

[greptile-apps]

_{17 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Test won't compile: type mismatch

Nv12RenderedFrame.data is Arc<Vec<u8>> (changed in this PR at frame_pipeline.rs:357), but data.clone() returns Vec<u8>. This will fail to compile.

Suggested change

	let data = vec![1u8, 2, 3, 4, 5, 6];
	let frame = Nv12RenderedFrame {
	data: data.clone(),
	let data = vec![1u8, 2, 3, 4, 5, 6];
	let frame = Nv12RenderedFrame {
	data: std::sync::Arc::new(data.clone()),
	width: 4,

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/export/src/mp4.rs
Line: 614:616

Comment:
**Test won't compile: type mismatch**

`Nv12RenderedFrame.data` is `Arc<Vec<u8>>` (changed in this PR at `frame_pipeline.rs:357`), but `data.clone()` returns `Vec<u8>`. This will fail to compile.

```suggestion
        let data = vec![1u8, 2, 3, 4, 5, 6];
        let frame = Nv12RenderedFrame {
            data: std::sync::Arc::new(data.clone()),
            width: 4,
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

Dimension mismatch between renderer and encoder

The Nv12ExportFrame uses the NV12-aligned output_size (4-aligned width, 2-aligned height from line 67), but the renderer receives self.resolution_base (line 340) and internally computes output dimensions via get_output_size() which only aligns to multiples of 2 (see lib.rs:1434).

When the rendered frame width is even but not a multiple of 4 (e.g., 1278, 1918), the GPU NV12 path falls back to RGBA-to-NV12 CPU conversion using the frame's actual dimensions. But here the Nv12ExportFrame declares the aligned output_size dimensions. fill_nv12_frame then reads the NV12 data with the wrong stride, producing garbled video.

Consider either:

1. Passing the aligned output_size as resolution_base to the renderer so it produces frames matching the encoder's expected dimensions, or
2. Using the frame's actual dimensions (frame.width, frame.height) instead of output_size for the Nv12ExportFrame.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/export/src/mp4.rs
Line: 279:287

Comment:
**Dimension mismatch between renderer and encoder**

The `Nv12ExportFrame` uses the NV12-aligned `output_size` (4-aligned width, 2-aligned height from line 67), but the renderer receives `self.resolution_base` (line 340) and internally computes output dimensions via `get_output_size()` which only aligns to multiples of 2 (see `lib.rs:1434`).

When the rendered frame width is even but not a multiple of 4 (e.g., 1278, 1918), the GPU NV12 path falls back to RGBA-to-NV12 CPU conversion using the frame's actual dimensions. But here the `Nv12ExportFrame` declares the aligned `output_size` dimensions. `fill_nv12_frame` then reads the NV12 data with the wrong stride, producing garbled video.

Consider either:
1. Passing the aligned `output_size` as `resolution_base` to the renderer so it produces frames matching the encoder's expected dimensions, or
2. Using the frame's actual dimensions (`frame.width`, `frame.height`) instead of `output_size` for the `Nv12ExportFrame`.

How can I resolve this? If you propose a fix, please make it concise.

[tembo]

bg0/bg1 are built from the same source_view + nv12_buffer + params, and then selected via readback_idx (which is the readback buffer index, not a bind-group variant). This looks redundant/overly complex.

Consider caching a single BindGroup (and pass.set_bind_group(0, bind_group, &[])), or if the intent is to pre-cache variants, key them off the source texture instead of the readback index.