[Bug]: CDI v0.5.0 specs without UUID device names break device plugin in CDI mode

[aditighag]

Describe the bug
nvidia-container-toolkit generates CDI specs with version to 0.5.0 that only contain a single management.nvidia.com/gpu=all device entry - no UUID or index-based entries are generated.

This causes failures when the device plugin operates in CDI mode, as it annotates pods with specific GPU UUIDs (management.nvidia.com/gpu=GPU-xxxx) that cannot be resolved from the gpu=all-only spec.

Additional notes: I came across this related bug - #1239. But removing runtimeClassName doesn't resolve the issue. I tried setting to runtimeClassName to nvidia or nvidia-cdi as well (although the issue does mention that this shouldn't be required with the native generation of CDI spec).

To Reproduce
Install a kuberntes cluster with component versions noted below, and deploy a test pod.

  apiVersion: v1
  kind: Pod
  metadata:
    name: test-cdi-device-plugin
    namespace: cdi-test
  spec:
    tolerations:
    - key: nvidia.com/gpu
      value: "true"
      effect: NoSchedule
    restartPolicy: Never
    containers:
    - name: gpu-test
      image: nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04
      command: ["sh", "-c", "nvidia-smi -L"]
      resources:
        limits:
          nvidia.com/gpu: 1

The container never starts with error -

  Normal   Pulling    12s   kubelet            spec.containers{gpu-test}: Pulling image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04"
  Normal   Pulled     8s    kubelet            spec.containers{gpu-test}: Successfully pulled image "nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04" in 4.301s (4.301s including waiting). Image size: 139357825 bytes.
  Normal   Created    8s    kubelet            spec.containers{gpu-test}: Container created
  Warning  Failed     8s    kubelet            spec.containers{gpu-test}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-67fa9265-2b87-d8ce-e708-13e93416062c

Observations
The nvidia-container-toolkit pods generate CDI specs on certain platforms withhostPath fields in device nodes, which bumps the CDI spec version to 0.5.0. These 0.5.0 specs only contain a single management.nvidia.com/gpu=all device entry - no UUID or index-based entries are generated.

Snippet of the (failing case) `0.5.0` CDI spec

---

cdiVersion: 0.5.0
kind: management.nvidia.com/gpu
devices:
- name: all
containerEdits:
deviceNodes:
- path: /dev/nvidia-modeset
hostPath: /dev/nvidia-modeset
- path: /dev/nvidia-uvm
hostPath: /dev/nvidia-uvm
- path: /dev/nvidia-uvm-tools
hostPath: /dev/nvidia-uvm-tools
- path: /dev/nvidia0
hostPath: /dev/nvidia0
- path: /dev/nvidia1
hostPath: /dev/nvidia1
- path: /dev/nvidia2
hostPath: /dev/nvidia2
- path: /dev/nvidia3
hostPath: /dev/nvidia3
- path: /dev/nvidia4
hostPath: /dev/nvidia4
- path: /dev/nvidia5
hostPath: /dev/nvidia5
- path: /dev/nvidia6
hostPath: /dev/nvidia6
- path: /dev/nvidia7
hostPath: /dev/nvidia7
- path: /dev/nvidiactl
hostPath: /dev/nvidiactl
- path: /dev/nvidia-caps/nvidia-cap0
hostPath: /dev/nvidia-caps/nvidia-cap0
- path: /dev/nvidia-caps/nvidia-cap1
hostPath: /dev/nvidia-caps/nvidia-cap1
- path: /dev/nvidia-caps/nvidia-cap2
hostPath: /dev/nvidia-caps/nvidia-cap2

Snippet of (working case) `0.3.0` CDI spec

---

cdiVersion: 0.3.0
kind: management.nvidia.com/gpu
devices:
- name: GPU-1da763bb-efa9-5bb1-0740-7ad1ffe7e78e
containerEdits:
deviceNodes:
- path: /dev/nvidia7
major: 195
minor: 7
fileMode: 438
permissions: rwm
- path: /dev/dri/card8
major: 226
minor: 8
fileMode: 432
permissions: rwm
gid: 44
- path: /dev/dri/renderD136
major: 226
minor: 136
fileMode: 432
permissions: rwm
gid: 992
hooks:
- hookName: createContainer
path: /usr/local/nvidia/toolkit/nvidia-cdi-hook
args:
- nvidia-cdi-hook
- create-symlinks
- --link
- ../card8::/dev/dri/by-path/pci-0000:63:00.0-card
- --link
- ../renderD136::/dev/dri/by-path/pci-0000:63:00.0-render
env:
- NVIDIA_CTK_DEBUG=false
- name: GPU-39b958b0-8067-e3d6-aced-7e130721f922
containerEdits:
deviceNodes:
- path: /dev/nvidia0
major: 195
fileMode: 438
permissions: rwm
- path: /dev/dri/card0
major: 226
fileMode: 432
permissions: rwm
gid: 44
- path: /dev/dri/renderD129
major: 226

Expected behavior
The test pod requesting GPU devices is succsessfully completed with logs -

$ k logs test-cdi-device-plugin -n cdi-test
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-06714eaa-5321-72be-2b23-479ee6b6221a)

Environment (please provide the following information):

• nvidia-container-toolkit version: v1.18.1
• NVIDIA Driver Version: 580.126.20
• Host OS: Ubuntu 24.04.4 LTS
• Kernel Version: 6.8.0-1046-nvidia
• Container Runtime Version: bundled with v1.35.2+rke2r1
• CPU Architecture x86_64 and arm64
• GPU Model(s): GH100, GB300

If applicable, also provide:

• Kubernetes Distro and Version: K8s v1.35.2+rke2r1

• NVIDIA GPU Operator version: v25.10.1

• Container logs

  Warning  Failed   3s    kubelet  spec.containers{gpu-test}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-fd24a321-803d-5ba4-0747-2f56f62bb66a

[cdesiniotis]

@aditighag can you share the logs from the k8s-device-plugin pod? In particular, I am interested in what the device-list-strategies configuration is set to.

[georgiastuart]

We had the same problem, also on Ubuntu 24.04 nodes with GH200 GPUs. DEVICE_LIST_STRATEGY was listed as envvar. I got it working with the CDI v0.5.0 spec by manually changing it to cdi-annotations:

In the nvidia operator helm values.yaml (v26.3.2)

devicePlugin:
  env:
    - name: PASS_DEVICE_SPECS
      value: "true"
    - name: FAIL_ON_INIT_ERROR
      value: "true"
    - name: DEVICE_LIST_STRATEGY
      value: cdi-annotations
    - name: DEVICE_ID_STRATEGY
      value: uuid
    - name: NVIDIA_VISIBLE_DEVICES
      value: all
    - name: NVIDIA_DRIVER_CAPABILITIES
      value: all

[cdesiniotis]

@georgiastuart I would recommend not setting this envvar yourself in the helm values, and instead, rely on the defaults set by GPU Operator. For reference, our default values.yaml has devicePlugin.env as an empty list: https://github.com/NVIDIA/gpu-operator/blob/9b198ba801ee9f1754dea0d74d85384659bea1c9/deployments/gpu-operator/values.yaml#L246. When cdi.enabled=true in ClusterPolicy (which is the default), the GPU Operator will ensure that the relevant CDI device list strategies are configured with the plugin.

[georgiastuart]

Except that appears to be the bug. Enabling CDI didn't produce a working environment for the device plugin.

[georgiastuart]

To clarify, the bug occurred when I didn't set devicePlugin.env in values.yaml and produced the incompatible DEVICE_LIST_STRATEGY. I set the full environment since I wasn't sure if it would just clobber the whole thing if not. Changing DEVICE_LIST_STRATEGY to cdi-annotations is a bandaid but not a real fix for the underlying issue.

[aditighag]

@aditighag can you share the logs from the k8s-device-plugin pod? In particular, I am interested in what the device-list-strategies configuration is set to.

Here you go -

$ k logs nvidia-device-plugin-daemonset-jqknc -n gpu-operator | grep -i -C5 deviceListStrategy
Defaulted container "nvidia-device-plugin" out of: nvidia-device-plugin, toolkit-validation (init)
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",
--
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/local/nvidia/toolkit/nvidia-cdi-hook",

[cdesiniotis]

Except that appears to be the bug. Enabling CDI didn't produce a working environment for the device plugin.

@georgiastuart could you raise a new issue against https://github.com/NVIDIA/gpu-operator with details on the issue you are facing? This may not be related to the original issue reported here. The default helm values should work.

@aditighag deviceListStrategy being set to volume-mounts is likely the culprit here. I assume you have this configured in your ClusterPolicy CR? Could you instead set this field to cdi-cri, cdi-annotations, volume-mounts?

[aditighag]

@aditighag deviceListStrategy being set to volume-mounts is likely the culprit here. I assume you have this configured in your ClusterPolicy CR? Could you instead set this field to cdi-cri, cdi-annotations, volume-mounts?

I tried setting the field to the values you suggested, and I'm still seeing the same error -

$ kubectl get clusterpolicy -o json | jq '.items[0].spec.devicePlugin.env[] | select(.name ==
  "DEVICE_LIST_STRATEGY")'
{
  "name": "DEVICE_LIST_STRATEGY",
  "value": "cdi-cri,cdi-annotations,volume-mounts"
}

 $ k logs nvidia-device-plugin-daemonset-5xtzn -n gpu-operator | grep -i -C5 deviceliststrategy
Defaulted container "nvidia-device-plugin" out of: nvidia-device-plugin, toolkit-validation (init)
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "deviceDiscoveryStrategy": "auto",
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "cdi-cri",
        "cdi-annotations",
        "volume-mounts"
      ],
      "deviceIDStrategy": "uuid",

$ k describe pod test-cdi-device-plugin -n cdi-test
:
  Warning  Failed   38s   kubelet  spec.containers{gpu-test}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-769a7250-979c-aa16-2916-092dded30475

[cdesiniotis]

Can you share the exact GPU Operator install procedure you are following? Note, this is what is documented for RKE2 https://docs.rke2.io/add-ons/gpu_operators#operator-installation (see v25.10.x tab). Can you share what your toolkit and cdi settings are in ClusterPolicy? Is nvidia configured as the default runtime with containerd?

[aditighag]

GPU operator is installed using helm.

Here is the info you asked for -

# kubectl get clusterpolicy -o json | jq '.items[0].spec.toolkit'
{
  "enabled": true,
  "env": [
    {
      "name": "ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED",
      "value": "false"
    },
    {
      "name": "ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS",
      "value": "true"
    },
    {
      "name": "CONTAINERD_CONFIG",
      "value": "/var/lib/rancher/rke2/agent/etc/containerd/unused--nvidia-config.toml"
    },
    {
      "name": "CONTAINERD_SOCKET",
      "value": "/run/k3s/containerd/containerd.sock"
    },
    {
      "name": "CONTAINERD_RUNTIME_CLASS",
      "value": "nvidia"
    }
  ],
  "image": "container-toolkit",
  "imagePullPolicy": "IfNotPresent",
  "installDir": "/usr/local/nvidia",
  "repository": "nvcr.io/nvidia/k8s",
  "version": "v1.19.0"
}
# kubectl get clusterpolicy -o json | jq '.items[0].spec.cdi'
{
  "default": false,
  "enabled": true
}

Is nvidia configured as the default runtime with containerd?

yes, see -

# grep -A2 'default_runtime_name' /var/lib/rancher/rke2/agent/etc/containerd/config.toml
  default_runtime_name = "nvidia"

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]

[aditighag]

@cdesiniotis 👋 Please let me know if you need anything else to triage the issue.
I also wanted to mention that upgrading toolkit to v1.19.1 leads to v0.3.0 CDI spec generation that don't have this issue.

What's the expected behavior here - is the v0.5.0 CDI spec supposed to have UUID device names?
Or

This causes failures when the device plugin operates in CDI mode, as it annotates pods with specific GPU UUIDs (management.nvidia.com/gpu=GPU-xxxx) that cannot be resolved from the gpu=all-only spec.

Is the device plugin not expected to use GPU UUIDs in the CDI mode?