nvidia-driver version configurable#4845
Conversation
Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com>
| # ----------------------------GPU CONFIGURATION-------------------------------- | ||
| # NVIDIA GPU driver configuration for compute nodes | ||
| # nvidia_driver_version: NVIDIA driver version to install on GPU nodes | ||
| # Default: "580.159.04" if not specified |
There was a problem hiding this comment.
please provide 'why' 580.159.04 was chosen as the default.
There was a problem hiding this comment.
When CUDA was installed without specifying a version, the GPU was not functioning properly on the Belton nodes. After explicitly installing CUDA version 580.159.04, the GPU worked correctly on both x86 and ARM architectures. Based on this successful validation, we have set 580.159.04 as the default CUDA version.
Latest version of cuda we are seeing this version
In this repo url https://developer.download.nvidia.com/compute/cuda/repos/rhel10/x86_64/
In the same repo we have multiple minor version of cuda driver
cuda-compat-13-0-580.65.06-1.el10.x86_64.rpm --this version is not working for H100
cuda-compat-13-0-580.82.07-1.el10.x86_64.rpm
cuda-compat-13-0-580.95.05-1.el10.x86_64.rpm
cuda-compat-13-0-580.105.08-1.el10.x86_64.rpm
cuda-compat-13-0-580.126.09-1.el10.x86_64.rpm
cuda-compat-13-0-580.159.04-1.el10.x86_64.rpm this version is working for both on ARM node and x86_64 nodes.
There was a problem hiding this comment.
Please can you answer the 'why' as a comment directly in the file. I like this because it means if I review this file in future, on a system where it is installed I immediately know why this value is set.
It might be something like:
# at time of testing, this is the most recent driver version that works for both AArch64 and x86_64
There was a problem hiding this comment.
I can imagine that admin's will want to put (slightly?) different version for AArch64 or x86_64. Can you provide two driver version entries one for x86_64, and one for AArch64?
There was a problem hiding this comment.
The request to support architecture-specific NVIDIA driver versions has been implemented. Two independent configuration parameters, nvidia_driver_version_x86_64 and nvidia_driver_version_aarch64, have been introduced within the slurm_cluster section of omnia_config.yml. During node provisioning, the install_cuda_driver.sh script determines the system architecture using uname -m and applies the corresponding driver version. Both parameters are optional and default to 580.159.04 when not explicitly configured. The input validation schema and upgrade transformation template have also been updated accordingly.
Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com>
snarthan
left a comment
snarthan
left a comment
There was a problem hiding this comment.
Fix the upgrade input also wrt this. and dont make that variable mandatory
Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com>
The install_cuda_driver.sh script previously used a hardcoded driver version. Add nvidia_driver_version_x86_64 and nvidia_driver_version_aarch64 to omnia_config.yml, with the script selecting the correct version at runtime based on uname -m. Defaults to 580.159.04 for both architectures. Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com>
Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com>
Signed-off-by: Nagachandan-P <Nagachandan.p@dell.com>
4 participants
Description of the Solution
configurablle nvidia version is supported from now, user can give nvidia driver version