Skip to content

Pin GPU tests to Volta-compatible cuDNN#227

Closed
ChrisRackauckas-Claude wants to merge 3 commits into
SciML:mainfrom
ChrisRackauckas-Claude:codex/fix-gpu-convolution-test
Closed

Pin GPU tests to Volta-compatible cuDNN#227
ChrisRackauckas-Claude wants to merge 3 commits into
SciML:mainfrom
ChrisRackauckas-Claude:codex/fix-gpu-convolution-test

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Ignore this PR until reviewed by @ChrisRackauckas.

Follow-up to #226.

Root cause

The failure is runner-dependent rather than caused by #226:

  • the matching green main GPU job ran on a T4;
  • failing main and pre-Use SciMLTesting public API docs QA #226 jobs ran on V100s;
  • all resolved CUDA 6.2.1, cuDNN.jl 6.2.1, and CUDNN_jll 9.24.0;
  • NVIDIA removed Volta support in cuDNN 9.11, while the generic gpu runner label can select V100 (compute capability 7.0) hosts.

The unsupported stack fails convolution with CUDNN_STATUS_EXECUTION_FAILED_CUDART.

Change

  • Pin the test-only CUDNN_jll dependency to 9.10.0, the latest registered pre-9.11 release.
  • Select CUDA runtime 12.9 through CUDA.jl's documented CUDA_Runtime_jll preference so the matching cuDNN artifact is available.
  • Run the GPU group on Julia 1.10, the package's minimum supported Julia release. CUDNN_jll 9.10 requires CUDA.jl 5.8 through registered compat, and that stack emits unsupported sm_80 PTX instructions when Julia 1.12 targets the V100's sm_70 architecture.
  • Assert both the cuDNN version boundary and a functional CUDA+cuDNN device in GPU mode.

This preserves the complete GPU suite; no test is skipped, disabled, or weakened. Current Julia remains covered by Core and QA, and the hard functional check prevents a GPU job from passing with zero layer tests.

These constraints apply to the test target/environment and do not constrain package users.

Local validation

  • GROUP=Core julia +1.12 --project -e 'using Pkg; Pkg.test()'
    • Utils: 13/13 passed
    • Layers: 1538/1538 passed
    • exit 0
  • GROUP=GPU julia +1.10 --project -e 'using Pkg; Pkg.test()'
    • resolved and precompiled CUDA 12.9 runtime, CUDNN_jll 9.10.0+0, cuDNN.jl 1.4.4, and CUDA.jl 5.8.5
    • Utils: 9 passed and the intentional @test cuda_testing() device guard failed on this CPU-only host
    • the self-hosted GPU CI job is required for passing device execution
  • julia +1.12 -m Runic --check --diff .
  • git diff --check

CI evidence

The prior revision ran on an actual V100: the functional guard passed and Utils passed 15/15, proving the pinned runtime/cuDNN artifact loaded. Its original cuDNN execution error disappeared. It then exposed a separate Julia 1.12/CUDA.jl 5.8 PTX failure (Modifier '.NaN' requires .target sm_80 or higher) while compiling for sm_70, which is why the GPU group now uses Julia 1.10.

The final CI result is valid only if the GPU Layers test count is nonzero.

Copy link
Copy Markdown
Contributor Author

Investigation/validation scratchpad:

  1. Reproduced the same 36 convolution errors on clean post-merge main using a V100 runner.
  2. Found a pre-Use SciMLTesting public API docs QA #226 V100 job with the identical failure and a green T4 job using the same package versions.
  3. Traced the hardware boundary to cuDNN 9.11 dropping Volta support.
  4. Kept the repository's GPU coverage intact and constrained only its test target to CUDNN_jll 9.10.0.
  5. Ran Core locally (13 Utils + 1538 Layers passed), GPU dependency/import validation (8 Utils passed; no local GPU), Runic, and diff checks.
  6. Waiting on the self-hosted GPU job for the decisive V100/T4 hardware result.

Commit: 0f295fe

Copy link
Copy Markdown
Contributor Author

Follow-up after inspecting the first GPU CI log:

The initial pin-only job was green on T4, but it was not accepted as validation because Layers ran 0 tests. CUDA had auto-selected runtime 13.0, for which CUDNN_jll 9.10 has no artifact; cuDNN became non-functional and MLDataDevices disabled the CUDA device.

Commit 434511e adds the documented CUDA 12.9 runtime preference and a hard GPU-group assertion that CUDA+cuDNN must be functional. This prevents a GPU job from ever passing with zero device coverage.

Revised local evidence:

  • GROUP=GPU Pkg.test() selected the CUDA 12.9 runtime and matching cuDNN 9.10 artifact, then failed explicitly at @test cuda_testing() on this non-GPU host (9 other Utils assertions passed). This is the intended guard behavior locally.
  • GROUP=Core Pkg.test(): Utils 13/13 and Layers 1538/1538 passed; exit 0.
  • Runic and git diff --check pass.

The new CI run must show nonzero GPU Layers tests before this fix is considered validated.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude force-pushed the codex/fix-gpu-convolution-test branch from 434511e to a98d9c1 Compare July 12, 2026 08:24
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor Author

V100 follow-up from the previous revision:

  • The job ran on self-hosted-gpu-v100-...-87tnz, passed the functional CUDA/cuDNN guard, and passed Utils 15/15. This confirms the CUDA 12.9 + CUDNN_jll 9.10 artifact was genuinely functional rather than producing a zero-test green job.
  • Layers reached 630 passes before failing with PTX assembly errors: Modifier '.NaN' requires .target sm_80 or higher while targeting the V100's sm_70. The original CUDNN_STATUS_EXECUTION_FAILED_CUDART convolution failure is gone.
  • Exact CUDNN_jll 9.10 compat necessarily resolves CUDA.jl 5.8.5. On Julia 1.12 that older compiler stack cannot target this V100 correctly, so commit a98d9c1 moves only the GPU group to exact Julia 1.10, which is the package's minimum supported Julia version. Core and QA retain current-Julia coverage.

Local GROUP=GPU with Julia 1.10 resolved/precompiled the intended stack, passed 9 utility assertions, and failed only the explicit device guard on this CPU-only host. Fresh hardware CI is now running; I will require a nonzero Layers count before treating it as valid.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor Author

Final CI verification is green on commit a98d9c1.

The GPU job ran on the exact affected Volta host class: self-hosted-gpu-v100-8vcpu-16gb-zjbnn-runner-cbjtm.

Resolved GPU stack from the job log:

  • Julia 1.10
  • CUDA.jl 5.8.5
  • CUDA_Runtime_jll 0.19.2+0 with the repository's CUDA 12.9 preference
  • cuDNN.jl 1.4.4
  • CUDNN_jll 9.10.0+0

Observed test output:

  • Utils Tests: 15/15 passed
  • Layers Tests: 1204/1204 passed
  • Testing DeepEquilibriumNetworks tests passed

This is 1,219 passing assertions in GPU mode with a nonzero Layers count. The former V100 CUDNN_STATUS_EXECUTION_FAILED_CUDART failure and the Julia 1.12/sm_70 PTX failure are absent.

All other required checks also pass: current and LTS Core, current QA, downgrade, Runic, Runic suggestions, spelling, documentation, and matrix detection. The branch is rebased on current upstream main and has no local or remote divergence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants