docker: nvidia container runtime (`--gpus all`) broken on NixOS 25.05

[Interpause]

Nixpkgs version

• Stable (25.05)

Describe the bug

Continuation from my comment #371527 (comment). Currently, I am stuck on 24.11 because I am unable to use docker --gpus all with the following error on 25.05:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

This is with hardware.nvidia-container-toolkit.enable = false and virtualisation.docker.enableNvidia = true. CDI device method (--device nvidia.com/gpu=all) doesn't work when nvidia-container-toolkit is disabled (of course), but still works fine if both are set to true.

Despite CDI working, this is still a huge issue as many Docker Compose files (i.e., Kasm Workspaces, Immich), still rely on the Nvidia Container Runtime method of declaring GPU mounting.

While modifying those is possible (in Immich's case), and really troublesome (in Kasm's case), the main showstopper is VS Code Devcontainers, which also still uses the Nvidia Container Runtime instead of CDI method and is really hard to modify to use CDI. This will probably remain a problem unless Docker transparently changes --devices all to refer to CDI as a form of deprecated backwards support, or somehow a miracle happens and developers start updating to use CDI everywhere.

The following might be relevant (I was trying to see if I could solve it myself):

• virtualisation/docker: fix nvidia container wrapper #407290
• libnvidia-container: include binaries from driver package #372320
• Nvidia Container Runtime version was 1.15.0-rc.3 and part of nvidia-docker in 24.11. It is now part of nvidia-container-toolkit/tools and version 1.17.6.

Steps to reproduce

See above.

Expected behaviour

Should be able to launch a container using --gpus all.

Screenshots

No response

Relevant log output

Additional context

No response

System metadata

24.11:

• system: "x86_64-linux"
• host os: Linux 6.12.34, NixOS, 24.11 (Vicuna), 24.11.718912.1f426f65ac4e
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.24.14
• channels(root): "nixos-24.11"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

25.05:

• system: "x86_64-linux"
• host os: Linux 6.12.34, NixOS, 25.05 (Warbler), 25.05.804570.c7ab75210cb8
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.28.3
• channels(root): "nixos-25.05"
• nixpkgs: /nix/store/aj312mhdmvjxjhky3cxxmxa13jx4mzh6-nixos-25.05/nixos

Notify maintainers

@ereslibre

---

Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)

I assert that this issue is relevant for Nixpkgs

• I assert that this is a bug and not a support request.
• I assert that this is not a duplicate of an existing issue.
• I assert that I have read the NixOS Code of Conduct and agree to abide by it.

Is this issue important to you?

Add a 👍 reaction to issues you find important.

[ereslibre]

Hello @Interpause,

I found some time yesterday to debug and inspect the issue. I will write a more detailed response later tonight/tomorrow tops, and hopefully open a PR along that time too. Right now:

❯ docker run --rm --runtime=nvidia --gpus=all -it ubuntu:latest nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-c475e08b-0cc5-f5aa-4326-99699429b449)
GPU 1: NVIDIA GeForce RTX 2080 SUPER (UUID: GPU-5cca1a6f-7cee-b649-40f0-2d3ecb0aa207)

❯ docker run --rm --device=nvidia.com/gpu=all -it ubuntu:latest nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-c475e08b-0cc5-f5aa-4326-99699429b449)
GPU 1: NVIDIA GeForce RTX 2080 SUPER (UUID: GPU-5cca1a6f-7cee-b649-40f0-2d3ecb0aa207)

[ereslibre]

Hello! This is the raw change I have at this time, both in my homelab settings and in nixpkgs. Please note that this is the extremely raw version that I am using to debug and create what will be the final nixpkgs PR, so take that into account.

• Homelab: https://github.com/ereslibre/homelab/compare/fix-gpus-argument-with-cdi
• nixpkgs: master...ereslibre:fix-gpus-argument-with-cdi

Things will move around; I also have to clean up the nixpkgs changeset so that it's ready for a PR, but I wanted to share what I have right now in case you want to check.

With this:

❯ docker run --rm --gpus all -it ubuntu:latest nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-c475e08b-0cc5-f5aa-4326-99699429b449)
GPU 1: NVIDIA GeForce RTX 2080 SUPER (UUID: GPU-5cca1a6f-7cee-b649-40f0-2d3ecb0aa207)

❯ docker run --rm --device=nvidia.com/gpu=all -it ubuntu:latest nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-c475e08b-0cc5-f5aa-4326-99699429b449)
GPU 1: NVIDIA GeForce RTX 2080 SUPER (UUID: GPU-5cca1a6f-7cee-b649-40f0-2d3ecb0aa207)

[ereslibre]

#421088 is ready for review and will fix this problem :)

[jgus]

Is this expected to be fixed in 25.11? I've just updated (and rebooted, etc) and --device=nvidia.com/gpu=all still works for me, but --gpus=all does not:

[root@d1:/etc/nixos]# nvidia-smi -L
GPU 0: Quadro P4000 (UUID: GPU-35f1dd5f-a7af-1980-58e4-61bec60811dd)
GPU 1: Quadro P400 (UUID: GPU-818986fd-d17e-ec14-11af-fdf2968d3e75)

[root@d1:/etc/nixos]# docker run --rm --device=nvidia.com/gpu=all -it ubuntu:latest nvidia-smi -L
GPU 0: Quadro P4000 (UUID: GPU-35f1dd5f-a7af-1980-58e4-61bec60811dd)
GPU 1: Quadro P400 (UUID: GPU-818986fd-d17e-ec14-11af-fdf2968d3e75)

[root@d1:/etc/nixos]# docker run --rm --gpus=all -it ubuntu:latest nvidia-smi -L
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

Run 'docker run --help' for more information

Is there some additional configuration or something I'm missing? Here's the (I believe) relevant lines of my config:

  nixpkgs.config.allowUnfree = true;
  nixpkgs.config.cudaSupport = true;
  services.xserver.videoDrivers = [ "nvidia" ];
  hardware.graphics.enable = true;
  hardware.nvidia.open = false;
  hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.stable;
  hardware.nvidia-container-toolkit.enable = true;
  virtualisation.docker.daemon.settings.features.cdi = true;

[ereslibre]

@jgus Yes, it is supported in 25.11. For the --gpus switch to be available you need to enable the enableNvidia support:

virtualisation.docker.enableNvidia = true;
hardware.graphics.enable32Bit = true;

However, in your settings there are things that are redundant (they don't harm though)

nixpkgs.config.allowUnfree = true;
nixpkgs.config.cudaSupport = true;
services.xserver.videoDrivers = [ "nvidia" ];  # redundant
hardware.graphics.enable = true;
hardware.nvidia.open = false;
hardware.nvidia.package = config.boot.kernelPackages.nvidiaPackages.stable;
hardware.nvidia-container-toolkit.enable = true;
virtualisation.docker.daemon.settings.features.cdi = true; # redundant

[jgus]

Thanks for the follow-up @ereslibre!

I didn't have virtualisation.docker.enableNvidia = true; because it warns me with this:

evaluation warning: You have set virtualisation.docker.enableNvidia. This option is deprecated, please set hardware.nvidia-container-toolkit.enable instead.

...and I do have hardware.nvidia-container-toolkit.enable set. Should that be _un_deprecated now?

If I do set that (and the 32-bit option) anyway, it does build (with that warning) but then when I try to run docker with --gpus=all I get this:

[root@d1:/etc/nixos]# docker run --rm --gpus=all -it ubuntu:latest nvidia-smi -L
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: /nix/store/vr9ngm7z1lv431vj2j9d7bmjmdp4byz1-nvidia-container-toolkit-1.18.0-tools/bin/nvidia-container-runtime did not terminate successfully: exit status 2: ERROR: init 250 result=11malloc(): corrupted top size
SIGABRT: abort
PC=0x7fe106ec5aac m=4 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 gp=0xc000002380 m=4 mp=0xc000049808 [syscall]:
runtime.cgocall(0x6d1760, 0xc000506228)
        runtime/cgocall.go:167 +0x4b fp=0xc000506200 sp=0xc0005061c8 pc=0x485e2b
github.com/NVIDIA/nvidia-container-toolkit/internal/lookup._cgo_cmalloc(0x19)
        _cgo_gotypes.go:150 +0x45 fp=0xc000506228 sp=0xc000506200 pc=0x5982a5
github.com/NVIDIA/nvidia-container-toolkit/internal/lookup._Cfunc_CString({0x7a05f9, 0x18})
        _cgo_gotypes.go:65 +0x45 fp=0xc000506260 sp=0xc000506228 pc=0x597f05
github.com/NVIDIA/nvidia-container-toolkit/internal/lookup.dlopenLocator.Locate({{{{0x801910, 0xc00006e480}, {0x798de8, 0x1}, {0xc0001202c0, 0xb, 0xb}, 0x7b00a0, 0x0, 0x0}, ...}}, ...)
        github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/dlopen.go:34 +0x52 fp=0xc000506368 sp=0xc000506260 pc=0x598332
github.com/NVIDIA/nvidia-container-toolkit/internal/lookup.(*dlopenLocator).Locate(0x67?, {0x7a05f9?, 0x0?})
        <autogenerated>:1 +0x9e fp=0xc000506458 sp=0xc000506368 pc=0x598fbe
github.com/NVIDIA/nvidia-container-toolkit/internal/lookup.first.Locate({0xc0002226c0?, 0x486525?, 0xc00021db70?}, {0x7a05f9, 0x18})
        github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/merge.go:42 +0x8b fp=0xc000506538 sp=0xc000506458 pc=0x596b6b
github.com/NVIDIA/nvidia-container-toolkit/internal/lookup.(*first).Locate(0x7a05f9?, {0x7a05f9?, 0x77e2c0?})
        <autogenerated>:1 +0x45 fp=0xc000506570 sp=0xc000506538 pc=0x598cc5
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.(*mounts).Mounts(0xc0002387d0)
        github.com/NVIDIA/nvidia-container-toolkit/internal/discover/mounts.go:65 +0x205 fp=0xc0005067c0 sp=0xc000506570 pc=0x669ba5
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.(*cache).Mounts(0xc000130b00)
        github.com/NVIDIA/nvidia-container-toolkit/internal/discover/cache.go:88 +0x95 fp=0xc000506820 sp=0xc0005067c0 pc=0x660215
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.list.Mounts({0xc000220360?, 0x2, 0xc000506900?})
        github.com/NVIDIA/nvidia-container-toolkit/internal/discover/list.go:76 +0xcf fp=0xc0005068c0 sp=0xc000506820 pc=0x668c4f
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.(*list).Mounts(0x7630c0?)
        <autogenerated>:1 +0x2f fp=0xc0005068e8 sp=0xc0005068c0 pc=0x66cc0f
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.graphicsDriverLibraries.Mounts({{0x800990, 0xc00022a2d0}, {0x801910, 0xc00006e480}, {0x7ff620, 0xc000222780}})
        github.com/NVIDIA/nvidia-container-toolkit/internal/discover/graphics.go:176 +0x62 fp=0xc0005069d8 sp=0xc0005068e8 pc=0x663482
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.(*graphicsDriverLibraries).Mounts(0x20?)
        <autogenerated>:1 +0x59 fp=0xc000506a48 sp=0xc0005069d8 pc=0x66c9d9
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.list.Mounts({0xc000222880?, 0x3, 0x1?})
        github.com/NVIDIA/nvidia-container-toolkit/internal/discover/list.go:76 +0xcf fp=0xc000506ae8 sp=0xc000506a48 pc=0x668c4f
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.(*list).Mounts(0x0?)
        <autogenerated>:1 +0x2f fp=0xc000506b10 sp=0xc000506ae8 pc=0x66cc0f
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.list.Mounts({0xc000222c80?, 0x3, 0xc000506bf0?})
        github.com/NVIDIA/nvidia-container-toolkit/internal/discover/list.go:76 +0xcf fp=0xc000506bb0 sp=0xc000506b10 pc=0x668c4f
github.com/NVIDIA/nvidia-container-toolkit/internal/discover.(*list).Mounts(0x769da0?)
        <autogenerated>:1 +0x2f fp=0xc000506bd8 sp=0xc000506bb0 pc=0x66cc0f
github.com/NVIDIA/nvidia-container-toolkit/internal/edits.FromDiscoverer({0x800990, 0xc00022a450})
        github.com/NVIDIA/nvidia-container-toolkit/internal/edits/edits.go:63 +0x128 fp=0xc000506f00 sp=0xc000506bd8 pc=0x671848
github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi.(*nvmllib).GetCommonEdits(0xc000506f38?)
        github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi/lib-nvml.go:44 +0x65 fp=0xc000506f48 sp=0xc000506f00 pc=0x6a97c5
github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi.(*wrapper).GetCommonEdits(0x3?)
        github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi/wrapper.go:95 +0x1f fp=0xc000506fa8 sp=0xc000506f48 pc=0x6affbf
github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi.(*wrapper).GetSpec(0xc0002383c0, {0xc00021c410?, 0x799170?, 0xc00021c420?})
        github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi/wrapper.go:61 +0xb6 fp=0xc0005071b0 sp=0xc000506fa8 pc=0x6afab6
github.com/NVIDIA/nvidia-container-toolkit/internal/modifier.newAutomaticCDISpecModifier({0x801910, 0xc00006e480}, 0xc000148540, {0xc00021c3c0, 0x1, 0x1})
        github.com/NVIDIA/nvidia-container-toolkit/internal/modifier/cdi.go:200 +0x95b fp=0xc000507670 sp=0xc0005071b0 pc=0x6b743b
github.com/NVIDIA/nvidia-container-toolkit/internal/modifier.NewCDIModifier({0x801910, 0xc00006e480}, 0xc000148540, {{0x801910, 0xc00006e480}, 0x0, 0xc00021a1b0, 0x0, {0xc000236008, 0xa, ...}, ...}, ...)
        github.com/NVIDIA/nvidia-container-toolkit/internal/modifier/cdi.go:68 +0x6bd fp=0xc000507970 sp=0xc000507670 pc=0x6b5e7d
github.com/NVIDIA/nvidia-container-toolkit/internal/runtime.newModeModifier({0x801910?, 0xc00006e480?}, {0x799a7c?, 0x7736c0?}, 0x4?, {{0x801910, 0xc00006e480}, 0x0, 0xc00021a1b0, 0x0, ...})
        github.com/NVIDIA/nvidia-container-toolkit/internal/runtime/runtime_factory.go:112 +0x1fa fp=0xc000507a38 sp=0xc000507970 pc=0x6bf81a
github.com/NVIDIA/nvidia-container-toolkit/internal/runtime.newSpecModifier({0x801910, 0xc00006e480}, 0xc000148540, {0x800c68?, 0xc0000124c8?}, 0xc000130900)
        github.com/NVIDIA/nvidia-container-toolkit/internal/runtime/runtime_factory.go:73 +0x105 fp=0xc000507cb0 sp=0xc000507a38 pc=0x6becc5
github.com/NVIDIA/nvidia-container-toolkit/internal/runtime.newNVIDIAContainerRuntime({0x801910, 0xc00006e480}, 0xc000148540, {0xc000016300, 0x10, 0x10}, 0xc000130900)
        github.com/NVIDIA/nvidia-container-toolkit/internal/runtime/runtime_factory.go:50 +0x22d fp=0xc000507d48 sp=0xc000507cb0 pc=0x6bea8d
github.com/NVIDIA/nvidia-container-toolkit/internal/runtime.rt.Run({0xc00006e480?, {0x0?, 0xc000130600?}}, {0xc000016300, 0x10, 0x10})
        github.com/NVIDIA/nvidia-container-toolkit/internal/runtime/runtime.go:82 +0x5bb fp=0xc000507ee0 sp=0xc000507d48 pc=0x6bdf3b
github.com/NVIDIA/nvidia-container-toolkit/internal/runtime.(*rt).Run(0x0?, {0xc000016300?, 0xa55a78?, 0xc000002380?})
        <autogenerated>:1 +0x47 fp=0xc000507f20 sp=0xc000507ee0 pc=0x6c0ce7
main.main()
        github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-container-runtime/main.go:11 +0x3b fp=0xc000507f50 sp=0xc000507f20 pc=0x6c0e1b
runtime.main()
        runtime/proc.go:285 +0x29d fp=0xc000507fe0 sp=0xc000507f50 pc=0x4553dd
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x48f501

goroutine 2 gp=0xc000002e00 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc000042fa8 sp=0xc000042f88 pc=0x48854e
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.forcegchelper()
        runtime/proc.go:373 +0xb3 fp=0xc000042fe0 sp=0xc000042fa8 pc=0x455713
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x48f501
created by runtime.init.7 in goroutine 1
        runtime/proc.go:361 +0x1a

goroutine 3 gp=0xc000003340 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc000043780 sp=0xc000043760 pc=0x48854e
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.bgsweep(0xc00006c000)
        runtime/mgcsweep.go:323 +0xdf fp=0xc0000437c8 sp=0xc000043780 pc=0x44033f
runtime.gcenable.gowrap1()
        runtime/mgc.go:212 +0x25 fp=0xc0000437e0 sp=0xc0000437c8 pc=0x434545
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc0000437e8 sp=0xc0000437e0 pc=0x48f501
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:212 +0x66

goroutine 4 gp=0xc000003500 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x7fcd68?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc000043f78 sp=0xc000043f58 pc=0x48854e
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.(*scavengerState).park(0xa6da60)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc000043fa8 sp=0xc000043f78 pc=0x43ddc9
runtime.bgscavenge(0xc00006c000)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc000043fc8 sp=0xc000043fa8 pc=0x43e379
runtime.gcenable.gowrap2()
        runtime/mgc.go:213 +0x25 fp=0xc000043fe0 sp=0xc000043fc8 pc=0x4344e5
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc000043fe8 sp=0xc000043fe0 pc=0x48f501
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:213 +0xa5

goroutine 5 gp=0xc000003dc0 m=nil [finalizer wait]:
runtime.gopark(0x4646b5?, 0x42bee5?, 0xb8?, 0x1?, 0xc000002380?)
        runtime/proc.go:460 +0xce fp=0xc000042620 sp=0xc000042600 pc=0x48854e
runtime.runFinalizers()
        runtime/mfinal.go:210 +0x107 fp=0xc0000427e0 sp=0xc000042620 pc=0x433447
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc0000427e8 sp=0xc0000427e0 pc=0x48f501
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:172 +0x3d

goroutine 6 gp=0xc000148380 m=nil [cleanup wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc000044768 sp=0xc000044748 pc=0x48854e
runtime.goparkunlock(...)
        runtime/proc.go:466
runtime.(*cleanupQueue).dequeue(0xa6dc80)
        runtime/mcleanup.go:439 +0xc5 fp=0xc0000447a0 sp=0xc000044768 pc=0x430665
runtime.runCleanups()
        runtime/mcleanup.go:635 +0x45 fp=0xc0000447e0 sp=0xc0000447a0 pc=0x430d25
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc0000447e8 sp=0xc0000447e0 pc=0x48f501
created by runtime.(*cleanupQueue).createGs in goroutine 1
        runtime/mcleanup.go:589 +0xa5

goroutine 18 gp=0xc0002481c0 m=nil [GC worker (idle)]:
runtime.gopark(0x364fb8720f6?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc00003e738 sp=0xc00003e718 pc=0x48854e
runtime.gcBgMarkWorker(0xc000234230)
        runtime/mgc.go:1463 +0xeb fp=0xc00003e7c8 sp=0xc00003e738 pc=0x436a2b
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x25 fp=0xc00003e7e0 sp=0xc00003e7c8 pc=0x436905
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc00003e7e8 sp=0xc00003e7e0 pc=0x48f501
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1373 +0x105

goroutine 19 gp=0xc000248380 m=nil [GC worker (idle)]:
runtime.gopark(0x364fb8f8069?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc00003ef38 sp=0xc00003ef18 pc=0x48854e
runtime.gcBgMarkWorker(0xc000234230)
        runtime/mgc.go:1463 +0xeb fp=0xc00003efc8 sp=0xc00003ef38 pc=0x436a2b
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x25 fp=0xc00003efe0 sp=0xc00003efc8 pc=0x436905
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc00003efe8 sp=0xc00003efe0 pc=0x48f501
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1373 +0x105

goroutine 20 gp=0xc000248540 m=nil [GC worker (idle)]:
runtime.gopark(0x364fb90606d?, 0xc000208600?, 0x1c?, 0xa?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc000055f38 sp=0xc000055f18 pc=0x48854e
runtime.gcBgMarkWorker(0xc000234230)
        runtime/mgc.go:1463 +0xeb fp=0xc000055fc8 sp=0xc000055f38 pc=0x436a2b
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x25 fp=0xc000055fe0 sp=0xc000055fc8 pc=0x436905
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc000055fe8 sp=0xc000055fe0 pc=0x48f501
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1373 +0x105

goroutine 21 gp=0xc000248700 m=nil [GC worker (idle)]:
runtime.gopark(0x364fb8e40af?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:460 +0xce fp=0xc00003ff38 sp=0xc00003ff18 pc=0x48854e
runtime.gcBgMarkWorker(0xc000234230)
        runtime/mgc.go:1463 +0xeb fp=0xc00003ffc8 sp=0xc00003ff38 pc=0x436a2b
runtime.gcBgMarkStartWorkers.gowrap1()
        runtime/mgc.go:1373 +0x25 fp=0xc00003ffe0 sp=0xc00003ffc8 pc=0x436905
runtime.goexit({})
        runtime/asm_amd64.s:1693 +0x1 fp=0xc00003ffe8 sp=0xc00003ffe0 pc=0x48f501
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1373 +0x105

rax    0x0
rbx    0x2aa9a
rcx    0x7fe106ec5aac
rdx    0x6
rdi    0x2aa97
rsi    0x2aa9a
rbp    0x7fe0bf19cb50
rsp    0x7fe0bf19cb10
r8     0x0
r9     0x0
r10    0x8
r11    0x246
r12    0x7fe0bf19d6c0
r13    0x7fe0bc024000
r14    0x6
r15    0x7fe0bf19cc60
rip    0x7fe106ec5aac
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
: unknown

Run 'docker run --help' for more information

...which is progress of a sort? But, any idea what I'm looking at here now Something else I might be missing?

[ereslibre]

@jgus yes, you did it right. This crash has to do with the nvidia-container-toolkit but you have everything correctly configured, we have to see if #467116 fixes it. If not, we have to dig a bit more.

[ereslibre]

Feel free to open an issue for the crash @jgus if you have some minutes, it would be great to handle that separately from this issue.

[ereslibre]

and I do have hardware.nvidia-container-toolkit.enable set. Should that be _un_deprecated now?

@jgus hardware.nvidia-container-toolkit.enable allows to use --device=nvidia.com/gpu=all and specific devices. virtualisation.docker.enableNvidia and virtualisation.podman.enableNvidia enable the --gpus flag, but they are the old version of referring to GPU's; CDI (--device with hardware.nvidia-container-toolkit.enable) is the preferred option as it allows not only to forward GPU's to containers, but any kind of hardware.

[ereslibre]

@jgus Just tried and the segfault is still there when using --gpus, even with #467116... I'll try to check when I have some time.