fix(ffmpeg): drop non-primary audio streams to support iPhone spatial video#91
Open
yangxin317 wants to merge 1 commit into
Open
fix(ffmpeg): drop non-primary audio streams to support iPhone spatial video#91yangxin317 wants to merge 1 commit into
yangxin317 wants to merge 1 commit into
Conversation
… video
iPhone 15 Pro+ records Spatial Video with two audio streams: a regular
stereo AAC track plus an Apple Spatial Audio Codec (apac) track. ffmpeg
has no codec tag for `apac` in the mp4 container, so a stream-copy that
maps all audio streams during shot segmentation fails with:
[mp4] Could not find tag for codec none in stream FireRedTeam#2,
codec not currently supported in container
[out#0/segment] Could not write header (incorrect codec parameters?)
The downstream LLM then receives the ffmpeg error text as the clip
description and refuses to proceed, so the entire workflow stalls right
after the first user message.
Restrict the implicit audio mapping to the first audio stream
(`0:a:0?`) so the apac stream is dropped during segmentation. The `?`
suffix keeps the mapping optional for media without audio.
Repro: upload an iPhone 15/16 Pro Spatial Video .mov and run any
workflow that triggers `split_shots`.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
iPhone 15/16 Pro shoots Spatial Video (.mov) with two audio streams: a regular AAC stereo track plus an Apple Spatial Audio Codec (
apac) track. ffmpeg has no codec tag forapacin the mp4 container, so a stream-copy that maps every audio stream during shot segmentation fails:The downstream LLM then receives the ffmpeg error text as the clip description and refuses to proceed, so the entire workflow stalls right after the very first user message.
Fix
Restrict the implicit audio mapping in
SAFE_MAP_ARGSto the first audio stream (-map 0:a:0?). The?suffix keeps the mapping optional for media that has no audio, preserving previous behavior on those files.Test plan
apacaudio stream at index 2)ffmpeg ... -f segment ...produces valid mp4 segments andsplit_shotscompletessplit_shots→understand_clips) progresses past the failure pointAffected files
src/open_storyline/utils/ffmpeg_utils.py(1 line)