Fix silent frame drop and missing trailing audio in Apple VideoToolbox encoder

[ruccho]

Summary

Two bugs in the Apple VideoToolbox implementation.

1. Silent frame drop and ignored encode status in the VT output callback
handle_video_encode_output (a) fully ignored _status (the encode result), so encode failures went undetected, and (b) silently dropped encoded H.264 frames when tx.try_send returned Full (bounded channel, capacity 32). A dropped P-frame breaks references of subsequent frames and produces block noise; frames flushed via complete_frames on Drop could also be lost.

2. No EOS drain in the audio encoder, dropping trailing samples
The fill_complex_buffer loop terminates once input is consumed, so completed packets remaining inside the AudioConverter plus any sub-1024-sample remainder are never drained at end of input. Every session lost the last few tens of milliseconds of audio.

Fix

Video: propagate backpressure to push instead of dropping at the callback

The callback must neither drop encoded frames (corrupts the P-frame reference chain) nor block (it runs on VideoToolbox's internal queue, which complete_frames() waits on). So flow control is moved to the producer:

• Keep a bounded output channel (capacity 32).
• In push, reserve a channel slot up front via reserve_owned().await before doing any encode work. When the consumer falls behind and the channel is full, this suspends push, propagating backpressure up the pipeline so the C# DroppingChannelInput drops input frames rather than the encoder corrupting output.
• Hand the reserved OwnedPermit to that frame's encode callback via VideoToolbox's per-frame source_frame_ref_con. The callback completes the send through the permit, which can neither block nor drop. Encode failures are propagated as Err through the channel; frames the encoder itself drops (kVTEncodeInfo_FrameDropped) release the permit.
• Error/retry paths (including the iOS-background kVTInvalidSessionErr retry) reclaim the permit so the reserved slot is never leaked.

This closes the earlier concern that an unbounded channel provided no upstream backpressure: the encoded-output channel is now hard-bounded, and when full it throttles the producer (which the input-side dropping stage absorbs) instead of growing memory.

The audio channel intentionally stays unbounded: audio is low-bandwidth, dropping audio packets would create irrecoverable gaps (there is no keyframe/eviction equivalent), packets are produced synchronously inside push at input rate, and the end-of-stream drain runs from Drop where an await is not available. It is drained continuously by the consumer, so it does not grow in practice.

Audio: drain the converter at end of stream

Added an EOS flag to fill_complex_buffer: on input exhaustion it returns "0 packets + noErr" to signal end of stream to the converter (the standard flush). Added drain(), invoked from Drop for AudioToolboxEncoderInput (input Drop = EOS signal), to flush the completed packets and the final partial packet.

Verification (macOS)

• cargo check -p unienc_apple_vt / cargo clippy: no new warnings (the pre-existing runtime dead-code and metal/mod.rs Arc warnings are unrelated).
• cargo test -p unienc_apple_vt -p unienc_common: all pass.
• cargo test -p unienc --test integration_test (real mp4 E2E, exercising push → encode → bounded channel + backpressure → pull → mux): pass, no deadlock. ffprobe on the output shows all 10 pushed video frames present and ~474 AAC packets (≈10.03s, trailing audio retained); A/V durations match.

Note: the Android/Windows/Linux/WebCodecs implementations are out of scope for this PR.

🤖 Generated with Claude Code