[setup-nvidia] disable nvidia-container-cli CUDA require check to unblock new CUDA minor versions

[Gasoonjia]

What

Append -e NVIDIA_DISABLE_REQUIRE=1 to GPU_FLAG in the setup-nvidia action.

Why

GPU jobs that pull a container for a newer CUDA minor version fail at container init with exit 125 before any work runs:

docker: Error response from daemon: failed to create task for container: ...
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=13.2,
please update your driver to a newer version, or use an earlier cuda container

The nvidia-container-cli prestart hook strict-compares the image's NVIDIA_REQUIRE_CUDA (e.g. cuda>=13.2) against the host driver's advertised max CUDA. The fleet driver pinned in this action is 580.65.06 (R580 / CUDA 13.0), so any cuda13.1/cuda13.2 image is rejected.

This is a false negative: CUDA minor-version compatibility guarantees that a container built with a newer minor toolkit runs on an older driver of the same major version (e.g. a cuda13.2 image on an R580/CUDA-13.0 driver). pytorch/pytorch already relies on exactly this — it runs CUDA 13.2 GPU tests on the 580.82.07 (CUDA 13.0) driver — it just doesn't hit this gate because its CI images don't carry the strict NVIDIA_REQUIRE_CUDA label that the almalinux-builder images do.

Scope / blast radius

GPU_FLAG is only set for NVIDIA GPU jobs, so this affects GPU CI only. Trade-off: a genuine major-version mismatch (e.g. a cuda14 image on a CUDA-13 driver) will no longer fail fast at init — it'll surface later as a runtime error instead.

Alternative considered: bump the pinned driver to one that advertises ≥13.2. Rejected for now because the current R580 line (incl. 580.82.07) still advertises CUDA 13.0, so a bump wouldn't clear the cuda>=13.2 gate, and it's a fleet-wide change.

Motivation / repro

pytorch/executorch adding 13.2 to its cuda.yml build matrix, which consumes linux_job_v2.yml (and thus this action's default driver + --gpus all).

[vercel]

@Gasoonjia is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

[huydhn]

Please add a test for this in https://github.com/pytorch/test-infra/blob/main/.github/workflows/test-setup-nvidia.yml to confirm that this works as expected.

[huydhn]

@claude Is there any backward compatibility issue with this change? We need this to work not only on newer CUDA 13.2 but older CUDA too

[claude]

Claude finished @huydhn's task in 43s —— View job

---

Backward compatibility analysis

Short answer: No — this is safe for older CUDA images, and is effectively a no-op for them.

NVIDIA_DISABLE_REQUIRE=1 only suppresses one thing: the nvidia-container-cli prestart assertion that strict-compares the image's NVIDIA_REQUIRE_CUDA label (e.g. cuda>=13.2) against the host driver's advertised max CUDA. It does not change anything about how the driver is loaded, what GPU devices/capabilities get mounted, or how CUDA runs inside the container. It's purely a gate-removal, not a behavior change.

Here's the case breakdown for the R580 / CUDA 13.0 driver currently pinned in this action (580.65.06):

Image's CUDA requirement	Without the flag (today)	With the flag (this PR)
Older / equal (e.g. cuda>=12.x, cuda>=13.0)	Check passes, container runs	Check skipped, container runs identically
Newer minor, same major (e.g. cuda>=13.2)	Check fails → exit 125 at init ❌	Check skipped, container runs ✅ (minor-version compat holds)
Newer major (e.g. cuda>=14)	Check fails fast at init	Check skipped → surfaces later as a runtime error

The key point for your concern: for older CUDA the preflight check already passes, so disabling it doesn't remove any working functionality — those jobs behave exactly as before. The flag only takes effect in cases where the check would have blocked the container, which is precisely the new-minor-version scenario we want to unblock.

The only real trade-off (already noted in the PR description) is the last row: a genuine major-version mismatch will no longer fail fast at container init — it'll surface later as a CUDA runtime error instead. That's a worse error message in an already-broken configuration, not a regression for any currently-working CUDA version.

Note: NVIDIA_DISABLE_REQUIRE is a per-container env var honored by libnvidia-container / nvidia-container-toolkit; setting it for a container that doesn't carry a strict NVIDIA_REQUIRE_CUDA label (most pytorch CI images) has no effect at all.

This addresses @huydhn's earlier request to confirm it works for older CUDA too — and his suggestion to add a test in test-setup-nvidia.yml is a good way to lock this in: a matrix entry running both an older CUDA image and a newer-minor (cuda13.2) image would demonstrate both that old images still pass and that new-minor images are no longer blocked. Happy to draft that if you'd like — just say the word.
· branch cuda-disable-nvidia-require

[huydhn]

I don't know why but Vercel is not happy, but please add a test to update the PR and I can try again

[atalman]

Hi @Gasoonjia this will have rather big effect: These reusable workflows are consumed org-wide (pytorch/pytorch,
executorch, vision, audio, ao, binary validation, etc.), so in principle every Linux job routing through them inherits the change.

1. Can we come up with some valid Test strategy before deploying it ?
2. We can perhaps commit this to separate branch and point executorch to this branch to see if this change resolve the executorch issue ?

[atalman]

See the comment about testing