Pin GPU tests to Volta-compatible cuDNN#227
Conversation
|
Investigation/validation scratchpad:
Commit: 0f295fe |
|
Follow-up after inspecting the first GPU CI log: The initial pin-only job was green on T4, but it was not accepted as validation because Layers ran 0 tests. CUDA had auto-selected runtime 13.0, for which CUDNN_jll 9.10 has no artifact; cuDNN became non-functional and MLDataDevices disabled the CUDA device. Commit 434511e adds the documented CUDA 12.9 runtime preference and a hard GPU-group assertion that CUDA+cuDNN must be functional. This prevents a GPU job from ever passing with zero device coverage. Revised local evidence:
The new CI run must show nonzero GPU Layers tests before this fix is considered validated. |
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
434511e to
a98d9c1
Compare
|
V100 follow-up from the previous revision:
Local |
|
Final CI verification is green on commit a98d9c1. The GPU job ran on the exact affected Volta host class: Resolved GPU stack from the job log:
Observed test output:
This is 1,219 passing assertions in GPU mode with a nonzero Layers count. The former V100 All other required checks also pass: current and LTS Core, current QA, downgrade, Runic, Runic suggestions, spelling, documentation, and matrix detection. The branch is rebased on current upstream main and has no local or remote divergence. |
Ignore this PR until reviewed by @ChrisRackauckas.
Follow-up to #226.
Root cause
The failure is runner-dependent rather than caused by #226:
gpurunner label can select V100 (compute capability 7.0) hosts.The unsupported stack fails convolution with
CUDNN_STATUS_EXECUTION_FAILED_CUDART.Change
CUDNN_jlldependency to 9.10.0, the latest registered pre-9.11 release.CUDA_Runtime_jllpreference so the matching cuDNN artifact is available.This preserves the complete GPU suite; no test is skipped, disabled, or weakened. Current Julia remains covered by Core and QA, and the hard functional check prevents a GPU job from passing with zero layer tests.
These constraints apply to the test target/environment and do not constrain package users.
Local validation
GROUP=Core julia +1.12 --project -e 'using Pkg; Pkg.test()'GROUP=GPU julia +1.10 --project -e 'using Pkg; Pkg.test()'@test cuda_testing()device guard failed on this CPU-only hostjulia +1.12 -m Runic --check --diff .git diff --checkCI evidence
The prior revision ran on an actual V100: the functional guard passed and Utils passed 15/15, proving the pinned runtime/cuDNN artifact loaded. Its original cuDNN execution error disappeared. It then exposed a separate Julia 1.12/CUDA.jl 5.8 PTX failure (
Modifier '.NaN' requires .target sm_80 or higher) while compiling for sm_70, which is why the GPU group now uses Julia 1.10.The final CI result is valid only if the GPU Layers test count is nonzero.