Skip to content

Commit 614c93b

Browse files
svcnvidia-nemo-cisuiyoubithomasdhcclaude
authored
Port FFmpeg video codec support updates to main (#2217) (#2222)
* Remove libopenh&libx264 and Add vp9 (#1930) * fix: remove libopenh264 and replace with libx264 * fix: install libx264 * fix: drop libx264 support * Enhance video transcoding support by adding `libvpx-vp9` as a CPU fallback encoder alongside `h264_nvenc`. Update installation instructions and verification steps to reflect the new encoder options. Modify `ClipTranscodingStage` to validate encoder selection and handle encoding options for both encoders. Update relevant documentation and examples to guide users on using the new encoder. * Minor update * refine docs * refine docs * add back openlibh264 and inform user to install themself if needed * docs format --------- (cherry picked from commit 79cacb4) * Update FFmpeg codec support without PyAV source-build path (#1959) Port the non-PyAV parts of #1959 onto latest main: - Keep the strict FFmpeg allowlist installer and update its default tag to n8.1.2. - Add the opt-in h264/hevc/av1 software decoder and libopenh264 installer. - Improve video metadata error handling when ffprobe cannot open NVDEC-only codecs in CPU-only stages. - Update video docs, tutorial preflight checks, and decoder tests for the optional software codec path. - Make video benchmark entries install the opt-in h264 support at runtime and use libopenh264 explicitly. The Docker image still follows latest main's #2203 behavior: it does not bake in system FFmpeg, does not force a PyAV source build, and leaves Dockerfile, pyproject.toml, and uv.lock unchanged. (cherry picked from commit 8055f35) * fix(video): avoid PyAV motion decode log callback deadlock * comments addressed * install video codecs for benchmark image build --------- Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Signed-off-by: Ao Tang <aot@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Ao Tang <aot@nvidia.com> Co-authored-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 780aca6 commit 614c93b

17 files changed

Lines changed: 769 additions & 96 deletions

File tree

benchmarking/Dockerfile

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,9 @@ RUN apt-get update \
2727
&& apt-get clean \
2828
&& rm -rf /var/lib/apt/lists/*
2929

30+
# Add video benchmark codec support to the benchmark image.
31+
RUN bash /opt/Curator/docker/common/install_h264_support.sh --with-libopenh264
32+
3033
# Ensure all extras and dependency groups (including test) are synced
3134
RUN cd /opt/Curator \
3235
&& uv sync --extra all --all-groups \

benchmarking/nightly-benchmark.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -877,6 +877,7 @@ entries:
877877
--video-dir={dataset:videos,mp4}
878878
--model-dir={dataset:videos_model_weights,files}
879879
--video-limit=1000
880+
--transcode-encoder=libopenh264
880881
timeout_s: 800
881882
sink_data:
882883
- name: slack
@@ -900,6 +901,7 @@ entries:
900901
--video-dir={dataset:videos,mp4}
901902
--model-dir={dataset:videos_model_weights,files}
902903
--video-limit=1000
904+
--transcode-encoder=libopenh264
903905
timeout_s: 800
904906
sink_data:
905907
- name: slack
@@ -923,6 +925,7 @@ entries:
923925
--no-generate-embeddings
924926
--video-dir={dataset:videos,mp4}
925927
--video-limit=1000
928+
--transcode-encoder=libopenh264
926929
timeout_s: 400
927930
sink_data:
928931
- name: slack
@@ -947,6 +950,7 @@ entries:
947950
--no-generate-embeddings
948951
--video-dir={dataset:videos,mp4}
949952
--video-limit=1000
953+
--transcode-encoder=libopenh264
950954
timeout_s: 800
951955
sink_data:
952956
- name: slack
@@ -974,6 +978,7 @@ entries:
974978
--no-generate-embeddings
975979
--enhance-captions
976980
--video-limit=256
981+
--transcode-encoder=libopenh264
977982
timeout_s: 1800
978983
sink_data:
979984
- name: slack
@@ -1000,6 +1005,7 @@ entries:
10001005
--no-generate-embeddings
10011006
--enhance-captions
10021007
--video-limit=256
1008+
--transcode-encoder=libopenh264
10031009
timeout_s: 1800
10041010
sink_data:
10051011
- name: slack
@@ -1089,6 +1095,7 @@ entries:
10891095
--aesthetic-threshold=3.5
10901096
--transnetv2-frame-decoder-mode ffmpeg_cpu
10911097
--video-limit=1000
1098+
--transcode-encoder=libopenh264
10921099
timeout_s: 800
10931100
sink_data:
10941101
- name: slack
@@ -1116,6 +1123,7 @@ entries:
11161123
--aesthetic-threshold=3.5
11171124
--transnetv2-frame-decoder-mode ffmpeg_cpu
11181125
--video-limit=1000
1126+
--transcode-encoder=libopenh264
11191127
timeout_s: 1800
11201128
sink_data:
11211129
- name: slack

benchmarking/tools/ci_benchmark_launcher.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ apt-get update -qq && apt-get install -y --no-install-recommends lynx
2828
if [[ "${ENTRY_NAME}" == audio_* ]]; then
2929
apt-get install -y --no-install-recommends ffmpeg
3030
elif [[ "${ENTRY_NAME}" == video_* ]]; then
31-
bash /opt/Curator/docker/common/install_ffmpeg.sh
31+
bash /opt/Curator/docker/common/install_h264_support.sh --with-libopenh264
3232
fi
3333

3434
cd /opt/Curator

docker/common/install_ffmpeg.sh

Lines changed: 45 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -31,62 +31,73 @@ done
3131
export DEBIAN_FRONTEND=noninteractive
3232
apt-get update
3333
apt-get install -y \
34-
libcrypt-dev \
3534
autoconf \
3635
automake \
3736
build-essential \
37+
ca-certificates \
3838
cmake \
39-
libaom-dev \
40-
libass-dev \
41-
libdav1d-dev \
42-
libdrm-dev \
43-
libfreetype6-dev \
44-
libgnutls28-dev \
39+
libcrypt-dev \
4540
libnuma-dev \
46-
libopenh264-dev \
4741
libtool \
48-
libva-dev \
49-
libvorbis-dev \
5042
libvpx-dev \
51-
libwebp-dev \
5243
nasm \
5344
pkg-config \
54-
vainfo \
5545
wget \
5646
yasm \
5747
zlib1g-dev
5848

59-
# Install NVCODEC
49+
# Install nv-codec-headers (NVENC + NVDEC bridge to the NVIDIA driver)
6050
wget -O /tmp/nv-codec-headers.tar.gz https://github.com/FFmpeg/nv-codec-headers/releases/download/n${NVCODEC_VERSION}/nv-codec-headers-${NVCODEC_VERSION}.tar.gz
6151
tar xzvf /tmp/nv-codec-headers.tar.gz -C /tmp/
6252
cd /tmp/nv-codec-headers-${NVCODEC_VERSION}
6353
make
6454
make install
6555

66-
# Install FFMPEG
67-
wget -O /tmp/ffmpeg-snapshot.tar.bz2 https://www.ffmpeg.org/releases/ffmpeg-${FFMPEG_VERSION}.tar.bz2
56+
# Build FFmpeg ${FFMPEG_VERSION} from the upstream release tarball:
57+
# - --disable-everything strips ALL components by default (encoders,
58+
# decoders, muxers, demuxers, parsers, bsfs, hwaccels, filters, protocols)
59+
# and we re-enable only what's needed.
60+
# - --enable-version3 selects LGPLv3+.
61+
# - Encoders: only NVENC (h264/hevc/av1) and libvpx-vp9 + rawvideo.
62+
# - Decoders: NVDEC variants for h264/hevc/av1/vp9 + software vp8/vp9 +
63+
# mpeg1/2/4 + libvpx_vp9 + rawvideo. NO software h264/hevc/av1.
64+
# - Shared-linked: installs libav*.so to /usr/local/lib for command-line
65+
# tools and any optional source-built consumers.
66+
cd /tmp
67+
wget -O /tmp/ffmpeg-snapshot.tar.bz2 "https://www.ffmpeg.org/releases/ffmpeg-${FFMPEG_VERSION}.tar.bz2"
6868
tar xjvf /tmp/ffmpeg-snapshot.tar.bz2 -C /tmp/
69-
cd /tmp/ffmpeg-${FFMPEG_VERSION}
70-
PATH="/usr/local/cuda/bin:$PATH" ./configure \
71-
--prefix=/usr/local \
72-
--enable-nonfree \
73-
--enable-cuda-nvcc \
74-
--enable-libnpp \
75-
--enable-libopenh264 \
76-
--enable-libaom \
77-
--enable-libdav1d \
78-
--enable-libvorbis \
79-
--enable-libvpx \
80-
--enable-libwebp \
81-
--enable-vaapi \
82-
--extra-cflags=-I/usr/local/cuda/include \
83-
--extra-ldflags=-L/usr/local/cuda/lib64 \
84-
--extra-libs=-lpthread \
85-
--extra-libs=-lm \
86-
--disable-static \
69+
cd "/tmp/ffmpeg-${FFMPEG_VERSION}"
70+
PKG_CONFIG_PATH="/usr/local/lib/pkgconfig" ./configure \
71+
--prefix="/usr/local" \
8772
--enable-shared \
73+
--disable-static \
74+
--extra-cflags="-I/usr/local/cuda/include" \
75+
--extra-ldflags="-L/usr/local/cuda/lib64" \
76+
--extra-libs="-lpthread -lm" \
77+
--ld="g++" \
78+
--enable-version3 \
79+
--disable-everything \
80+
--disable-network \
8881
--disable-doc \
89-
--disable-debug
82+
--disable-ffplay \
83+
--disable-vaapi \
84+
--disable-vdpau \
85+
--disable-dxva2 \
86+
--disable-libdrm \
87+
--enable-encoder=rawvideo,libvpx_vp9,h264_nvenc,hevc_nvenc,av1_nvenc \
88+
--enable-decoder=rawvideo,libvpx_vp9,vp9,vp8,h264_cuvid,hevc_cuvid,av1_cuvid,mpeg1video,mpeg2video,mpeg4 \
89+
--enable-muxer=mp4,rawvideo,image2pipe \
90+
--enable-demuxer=mov,mp4,m4a,3gp,3g2,mj2,avi,matroska,webm,image2,image2pipe \
91+
--enable-parser=h264,hevc,av1,vp8,vp9 \
92+
--enable-bsf=h264_mp4toannexb,hevc_mp4toannexb \
93+
--enable-protocol=file,pipe \
94+
--enable-filter=scale,format,null,copy \
95+
--enable-libvpx \
96+
--enable-cuda \
97+
--enable-cuvid \
98+
--enable-nvdec \
99+
--enable-nvenc \
100+
--enable-ffnvcodec
90101
make -j$(nproc)
91102
make install
92103
ldconfig
Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
#!/bin/bash
2+
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
# Opt-in installer that adds software h264/hevc/av1 decoder support to a
17+
# Curator container by building FFmpeg from source with those decoders added
18+
# to the strict allowlist.
19+
#
20+
# The release Curator image may omit system ffmpeg entirely. When installed
21+
# with install_ffmpeg.sh, Curator's strict FFmpeg build routes h264/hevc/av1
22+
# decode through NVDEC only. That breaks ffprobe-based metadata extraction in
23+
# CPU-only Ray actors (VideoReader, ClipWriter), which is why this opt-in
24+
# exists.
25+
#
26+
# Run inside a running container, e.g.:
27+
# docker exec <container> bash /opt/Curator/docker/common/install_h264_support.sh
28+
#
29+
# Behaviour:
30+
# - Builds FFmpeg from the upstream release tarball, same version as install_ffmpeg.sh,
31+
# with the existing allowlist plus --enable-decoder=h264,hevc,av1.
32+
# - Optionally also enables the libopenh264 software h264 ENCODER (Cisco's
33+
# free-license OpenH264 binary; opt-in via --with-libopenh264).
34+
# - Replaces /usr/local/bin/{ffmpeg,ffprobe} in place.
35+
# - Default stays LGPLv3 (only FFmpeg-internal native decoders); with
36+
# --with-libopenh264 the resulting binary additionally links Cisco's
37+
# OpenH264 (BSD-2-Clause; see https://www.openh264.org/BINARY_LICENSE.txt).
38+
# - Takes ~5-10 min.
39+
#
40+
# Keep FFMPEG_VERSION and NVCODEC_VERSION in sync with docker/common/install_ffmpeg.sh.
41+
42+
set -euo pipefail
43+
44+
FFMPEG_VERSION=8.0.1
45+
NVCODEC_VERSION=12.1.14.0
46+
WITH_LIBOPENH264=0
47+
48+
usage() {
49+
cat <<'EOF'
50+
Usage: install_h264_support.sh [--with-libopenh264] [--FFMPEG_VERSION=<version>]
51+
[--NVCODEC_VERSION=<ver>]
52+
53+
Builds FFmpeg from source with software h264/hevc/av1 decoders enabled.
54+
55+
Options:
56+
--with-libopenh264 Also enable the libopenh264 software h264 ENCODER
57+
(Cisco's OpenH264 binary; required by Curator's
58+
--transcode-encoder=libopenh264 path).
59+
--FFMPEG_VERSION=<version> FFmpeg upstream release version (default: 8.0.1).
60+
--NVCODEC_VERSION=<ver> nv-codec-headers release version (default: 12.1.14.0).
61+
-h, --help Show this help.
62+
63+
License notice:
64+
Default mode enables only FFmpeg's internal h264/hevc/av1 decoders (LGPL).
65+
With --with-libopenh264 the build additionally links Cisco's OpenH264
66+
binary (BSD-2-Clause + Cisco-distributed binary license; see
67+
https://www.openh264.org/BINARY_LICENSE.txt). By running this script you
68+
are responsible for any license obligations the resulting binaries impose
69+
on your distribution.
70+
EOF
71+
}
72+
73+
for arg in "$@"; do
74+
case $arg in
75+
--with-libopenh264) WITH_LIBOPENH264=1 ;;
76+
--FFMPEG_VERSION=?*) FFMPEG_VERSION="${arg#*=}" ;;
77+
--NVCODEC_VERSION=?*) NVCODEC_VERSION="${arg#*=}" ;;
78+
-h|--help) usage; exit 0 ;;
79+
*) echo "Unknown argument: $arg" >&2; usage >&2; exit 2 ;;
80+
esac
81+
done
82+
83+
if [ "$(id -u)" -ne 0 ]; then
84+
echo "ERROR: must be run as root inside the container." >&2
85+
exit 1
86+
fi
87+
88+
echo "==> install_h264_support.sh: rebuilding ffmpeg ${FFMPEG_VERSION}"
89+
echo " Decoders added: h264, hevc, av1 (software)"
90+
if [ "$WITH_LIBOPENH264" -eq 1 ]; then
91+
echo " Encoder added: libopenh264 (Cisco OpenH264 binary)"
92+
fi
93+
echo " NOTE: This expands the container's codec footprint beyond Curator's"
94+
echo " default strict FFmpeg allowlist. License obligations of the resulting binaries"
95+
echo " are the user's responsibility."
96+
echo
97+
98+
export DEBIAN_FRONTEND=noninteractive
99+
apt-get update
100+
apt_packages=(
101+
autoconf automake build-essential ca-certificates cmake
102+
libcrypt-dev libnuma-dev libtool libvpx-dev nasm pkg-config
103+
wget yasm zlib1g-dev
104+
)
105+
if [ "$WITH_LIBOPENH264" -eq 1 ]; then
106+
apt_packages+=(libopenh264-dev)
107+
fi
108+
apt-get install -y "${apt_packages[@]}"
109+
110+
if [ ! -f /usr/local/include/ffnvcodec/dynlink_loader.h ]; then
111+
wget -O /tmp/nv-codec-headers.tar.gz \
112+
"https://github.com/FFmpeg/nv-codec-headers/releases/download/n${NVCODEC_VERSION}/nv-codec-headers-${NVCODEC_VERSION}.tar.gz"
113+
tar xzf /tmp/nv-codec-headers.tar.gz -C /tmp/
114+
(cd "/tmp/nv-codec-headers-${NVCODEC_VERSION}" && make && make install)
115+
fi
116+
117+
cd /tmp
118+
rm -rf "ffmpeg-${FFMPEG_VERSION}" ffmpeg-snapshot.tar.bz2
119+
wget -O /tmp/ffmpeg-snapshot.tar.bz2 \
120+
"https://www.ffmpeg.org/releases/ffmpeg-${FFMPEG_VERSION}.tar.bz2"
121+
tar xjvf /tmp/ffmpeg-snapshot.tar.bz2 -C /tmp/
122+
cd "/tmp/ffmpeg-${FFMPEG_VERSION}"
123+
124+
# Configure mirrors install_ffmpeg.sh exactly, with the decoder allowlist
125+
# extended to include software h264/hevc/av1, and optionally the libopenh264
126+
# software h264 encoder.
127+
encoder_list="rawvideo,libvpx_vp9,h264_nvenc,hevc_nvenc,av1_nvenc"
128+
extra_configure_flags=()
129+
if [ "$WITH_LIBOPENH264" -eq 1 ]; then
130+
encoder_list="${encoder_list},libopenh264"
131+
extra_configure_flags+=(--enable-libopenh264)
132+
fi
133+
134+
PKG_CONFIG_PATH="/usr/local/lib/pkgconfig" ./configure \
135+
--prefix="/usr/local" \
136+
--enable-shared \
137+
--disable-static \
138+
--extra-cflags="-I/usr/local/cuda/include" \
139+
--extra-ldflags="-L/usr/local/cuda/lib64" \
140+
--extra-libs="-lpthread -lm" \
141+
--ld="g++" \
142+
--enable-version3 \
143+
--disable-everything \
144+
--disable-network \
145+
--disable-doc \
146+
--disable-ffplay \
147+
--disable-vaapi \
148+
--disable-vdpau \
149+
--disable-dxva2 \
150+
--disable-libdrm \
151+
--enable-encoder="${encoder_list}" \
152+
--enable-decoder=rawvideo,libvpx_vp9,vp9,vp8,h264_cuvid,hevc_cuvid,av1_cuvid,mpeg1video,mpeg2video,mpeg4,h264,hevc,av1 \
153+
--enable-muxer=mp4,rawvideo,image2pipe \
154+
--enable-demuxer=mov,mp4,m4a,3gp,3g2,mj2,avi,matroska,webm,image2,image2pipe \
155+
--enable-parser=h264,hevc,av1,vp8,vp9 \
156+
--enable-bsf=h264_mp4toannexb,hevc_mp4toannexb \
157+
--enable-protocol=file,pipe \
158+
--enable-filter=scale,format,null,copy \
159+
--enable-libvpx \
160+
--enable-cuda \
161+
--enable-cuvid \
162+
--enable-nvdec \
163+
--enable-nvenc \
164+
--enable-ffnvcodec \
165+
"${extra_configure_flags[@]}"
166+
make -j"$(nproc)"
167+
make install
168+
ldconfig
169+
170+
cd /
171+
rm -rf /tmp/ffmpeg* /tmp/nv-codec-headers*
172+
echo
173+
if [ "$WITH_LIBOPENH264" -eq 1 ]; then
174+
echo "==> Done. /usr/local/bin/{ffmpeg,ffprobe} now include software h264/hevc/av1"
175+
echo " decoders and the libopenh264 software h264 encoder."
176+
else
177+
echo "==> Done. /usr/local/bin/{ffmpeg,ffprobe} now include software h264/hevc/av1 decoders."
178+
fi

0 commit comments

Comments
 (0)