[Bug]: nvv4l2decoder/nvv4l2h264enc SIGSEGV with nvidia-container-toolkit 1.19.1-1 on WSL2 (fix: downgrade to 1.18.0-1)

[levipereira]

Describe the bug

DeepStream's nvv4l2decoder and nvv4l2h264enc GStreamer elements crash with a SIGSEGV inside a container when using nvidia-container-toolkit 1.19.1-1. The decoder/encoder fail to set the CUDA GPU ID control via ioctl, then segfault during pipeline PREROLL.

Downgrading the toolkit to 1.18.0-1 resolves the issue completely.

To Reproduce

1. Set up DeepStream on WSL2 following the official guide: https://docs.nvidia.com/metropolis/deepstream/9.0/text/DS_on_WSL2.html
2. With nvidia-container-toolkit 1.19.1-1 installed on the host, start the DeepStream container.
3. Inside the container, run a simple transcode pipeline:

gst-launch-1.0 filesrc location=sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=filedump.mp4 -v

4. The pipeline fails during PREROLL with:

Error while setting IOCTL
Invalid control
S_EXT_CTRLS for CUDA_GPU_ID failed
Caught SIGSEGV
#0  0x00007d12439b84fd in poll () at /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007d1243bce8ce in ??? () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007d1243b6ff47 in g_main_loop_run () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00005e3384da8d68 in ??? ()
#4  0x00007d12438c71ca in ??? () at /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007d12438c728b in __libc_start_main () at /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00005e3384da61f5 in ??? ()

Expected behavior

The pipeline should run to completion and produce filedump.mp4, as it does with nvidia-container-toolkit 1.18.0-1.

Fix / Workaround

Downgrading the container toolkit to 1.18.0-1 resolves it:

sudo apt-get install -y --allow-downgrades \
  nvidia-container-toolkit=1.18.0-1 \
  nvidia-container-toolkit-base=1.18.0-1 \
  libnvidia-container-tools=1.18.0-1 \
  libnvidia-container1=1.18.0-1

Environment:

• nvidia-container-toolkit version: 1.19.1-1 (broken) / 1.18.0-1 (working)
• NVIDIA Driver Version: 610.43.02 (KMD 610.47, CUDA UMD 13.3)
• Host OS: Ubuntu 24.04.4 LTS (WSL2)
• Kernel Version: 6.6.87.2-microsoft-standard-WSL2
• Container Runtime Version: Docker 29.5.3 (build d1c06ef)
• CPU Architecture: x86_64
• GPU Model(s): NVIDIA GeForce RTX 5080 (16 GB)
• CUDA Version: 13.3 (UMD)

Information to attach:

Output of nvidia-smi:

Wed Jun  3 20:38:45 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.43.02              KMD Version: 610.47        CUDA UMD Version: 13.3     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5080        On  |   00000000:01:00.0  On |                  N/A |
|  0%   44C    P0             52W /  360W |    1308MiB /  16303MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Full container log (gst-launch -v):

gst-launch-1.0 filesrc location=sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=filedump.mp4 -v
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is PREROLLING ...
Redistribute latency...
Error while setting IOCTL
Invalid control
S_EXT_CTRLS for CUDA_GPU_ID failed
Error while setting IOCTL
Invalid control
S_EXT_CTRLS for CUDA_GPU_ID failed
Caught SIGSEGV
#0  0x00007d12439b84fd in poll () at /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007d1243bce8ce in ??? () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x00007d1243b6ff47 in g_main_loop_run () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3  0x00005e3384da8d68 in ??? ()
#4  0x00007d12438c71ca in ??? () at /usr/lib/x86_64-linux-gnu/libc.so.6
#5  0x00007d12438c728b in __libc_start_main () at /usr/lib/x86_64-linux-gnu/libc.so.6
#6  0x00005e3384da61f5 in ??? ()
Spinning.  Please run 'gdb gst-launch-1.0 213' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

[cdesiniotis]

@levipereira can you capture the contents of /var/run/cdi/nvidia.yaml after installing both 1.18.0 and 1.19.1 of NVIDIA Container Toolkit and see if there are any notable differences?

[levipereira]

@cdesiniotis

Output

1.18.0

root@Levi-PC:~#cat /var/run/cdi/nvidia.yaml
root@Levi-PC:~#cat: /var/run/cdi/nvidia.yaml: No such file or directory
root@Levi-PC:/var/run# find . |grep nvidia
root@Levi-PC:/var/run# find . |grep cdi

root@Levi-PC:~# cat /var/run/cdi/nvidia.yaml
cat: /var/run/cdi/nvidia.yaml: No such file or directory
root@Levi-PC:~# cat  /etc/nvidia-container-runtime/config.toml
#accept-nvidia-visible-devices-as-volume-mounts = false
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
disable-require = false
supported-driver-capabilities = "compat32,compute,display,graphics,ngx,utility,video"
#swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
#debug = "/var/log/nvidia-container-toolkit.log"
environment = []
#ldcache = "/etc/ld.so.cache"
ldconfig = "@/sbin/ldconfig.real"
load-kmods = true
#no-cgroups = false
#path = "/usr/bin/nvidia-container-cli"
#root = "/run/nvidia/driver"
#user = "root:video"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
log-level = "info"
mode = "auto"
runtimes = ["runc", "crun"]

[nvidia-container-runtime.modes]

[nvidia-container-runtime.modes.cdi]
annotation-prefixes = ["cdi.k8s.io/"]
default-kind = "nvidia.com/gpu"
spec-dirs = ["/etc/cdi", "/var/run/cdi"]

[nvidia-container-runtime.modes.csv]
mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

[nvidia-container-runtime.modes.legacy]
cuda-compat-mode = "ldconfig"

[nvidia-container-runtime-hook]
path = "nvidia-container-runtime-hook"
skip-mode-detection = false

[nvidia-ctk]
path = "nvidia-ctk

1.19.1

root@Levi-PC:~#  cat  /etc/nvidia-container-runtime/config.toml
#accept-nvidia-visible-devices-as-volume-mounts = false
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
disable-require = false
supported-driver-capabilities = "compat32,compute,display,graphics,ngx,utility,video"
#swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
#debug = "/var/log/nvidia-container-toolkit.log"
environment = []
#ldcache = "/etc/ld.so.cache"
ldconfig = "@/sbin/ldconfig.real"
load-kmods = true
#no-cgroups = false
#path = "/usr/bin/nvidia-container-cli"
#root = "/run/nvidia/driver"
#user = "root:video"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
log-level = "info"
mode = "auto"
runtimes = ["runc", "crun"]

[nvidia-container-runtime.modes]

[nvidia-container-runtime.modes.cdi]
annotation-prefixes = ["cdi.k8s.io/"]
default-kind = "nvidia.com/gpu"
spec-dirs = ["/etc/cdi", "/var/run/cdi"]

[nvidia-container-runtime.modes.csv]
mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

[nvidia-container-runtime.modes.legacy]
cuda-compat-mode = "ldconfig"

[nvidia-container-runtime-hook]
path = "nvidia-container-runtime-hook"
skip-mode-detection = false

[nvidia-ctk]
path = "nvidia-ctk"

root@Levi-PC:/var/run# cat /var/run/cdi/nvidia.yaml
---
cdiVersion: 0.3.0
kind: nvidia.com/gpu
devices:
    - name: all
      containerEdits:
        deviceNodes:
            - path: /dev/dxg
              major: 10
              minor: 125
              fileMode: 438
              permissions: rwm
containerEdits:
    env:
        - NVIDIA_VISIBLE_DEVICES=void
    hooks:
        - hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
          args:
            - nvidia-cdi-hook
            - create-symlinks
            - --link
            - /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/nvidia-smi::/usr/bin/nvidia-smi
          env:
            - NVIDIA_CTK_DEBUG=false
        - hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
          args:
            - nvidia-cdi-hook
            - update-ldcache
            - --folder
            - /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b
            - --folder
            - /usr/lib/wsl/lib
          env:
            - NVIDIA_CTK_DEBUG=false
    mounts:
        - hostPath: /usr/lib/wsl/lib/libdxcore.so
          containerPath: /usr/lib/wsl/lib/libdxcore.so
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libcuda.so.1.1
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libcuda.so.1.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libcuda_loader.so
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libcuda_loader.so
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvdxgdmal.so.1
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvdxgdmal.so.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvidia-ml.so.1
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvidia-ml.so.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvidia-ml_loader.so
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvidia-ml_loader.so
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvidia-ptxjitcompiler.so.1
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/libnvidia-ptxjitcompiler.so.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/nvcubins.bin
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/nvcubins.bin
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/nvidia-smi
          containerPath: /usr/lib/wsl/drivers/nv_dispi.inf_amd64_c8bc842500fab35b/nvidia-smi
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate

[cdesiniotis]

I was able to reproduce this issue. The root cause is the libnvidia-encode.so and libnvcuvid.so libraries are missing from the generated CDI spec. These libraries are required for video encode / decode. I manually added these libraries and the deepstream container worked as expected.