Skip to content

[Build] Install the NVIDIA driver and CUDA toolkit from the distribution package manager using NVIDIA local repo packages instead of the run file installers.#3207

Open
gmarciani wants to merge 2 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3160/install-nvidia-0619-1
Open

[Build] Install the NVIDIA driver and CUDA toolkit from the distribution package manager using NVIDIA local repo packages instead of the run file installers.#3207
gmarciani wants to merge 2 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3160/install-nvidia-0619-1

Conversation

@gmarciani

@gmarciani gmarciani commented Jun 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

THIS PR CANNOT BE MERGED UNTIL WE PUBLISH THE REFERENCED ARTIFACTS TO OUR PUBLIC BUCKETS

Description of changes

Migrate the NVIDIA driver and CUDA toolkit installation from the .run runfile installers to NVIDIA local-repo packages (.rpm/.deb) installed through the distribution package manager.

This is the approach suggested both by NVIDIA and AWS. Compared to the original run file approach, it is also more robust against kernel bumps.

As part of this change we also:

  1. converted the installation of cuda toolkit from recipe to resource.
  2. remove the nvidia local repos to save space in the AMI. This is safe because if the user must change the nvidia software versions they should download different local repos in any case.
  3. install fabric manager only if it is not already installed (this fixes a pre-existing bug in our code, where we used to install fabric manager even if it was already installed)

Tests

  1. Unit test (added/updated) included in this PR check
  2. Build image on all supported OSs (with DevSettings pointing to NVIDIA local repos from Nvidia website because artifacts are not published to our bucket yet)
  3. Successful integ tests:
test-suites:
  basic:
    test_essential_features.py::test_essential_features:
      dimensions:
        - regions: ["us-east-1"]
          instances: ["g4dn.xlarge"]
          oss: ["alinux2023","ubuntu2204"]
          schedulers: ["slurm"]
  health_checks:
    test_gpu_health_checks.py::test_cluster_with_gpu_health_checks:
      dimensions:
        - regions: ["us-east-1"]
          instances: ["g4dn.xlarge"]
          oss: ["alinux2023","ubuntu2204"]
          schedulers: ["slurm"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@gmarciani gmarciani changed the title [Build] Install the NVIDIA driver and CUDA toolkit from the distribut… [Build] Install the NVIDIA driver and CUDA toolkit from the distribution package manager using NVIDIA local repo packages instead of the run file installers. Jun 19, 2026
@codecov

codecov Bot commented Jun 19, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.28%. Comparing base (e5dd95b) to head (0c30d55).
⚠️ Report is 4 commits behind head on develop.

Additional details and impacted files 
@@           Coverage Diff            @@
##           develop    #3207   +/-   ##
========================================
  Coverage    77.28%   77.28%           
========================================
  Files           24       24           
  Lines         2545     2545           
========================================
  Hits          1967     1967           
  Misses         578      578
Flag Coverage Δ
unittests 77.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gmarciani gmarciani added the 3.x label Jun 19, 2026
@gmarciani gmarciani force-pushed the wip/mgiacomo/3160/install-nvidia-0619-1 branch 5 times, most recently from d02a66a to 3430c6d Compare June 22, 2026 01:14
@gmarciani gmarciani force-pushed the wip/mgiacomo/3160/install-nvidia-0619-1 branch 4 times, most recently from a02f557 to dd20cd8 Compare June 24, 2026 19:56
@gmarciani gmarciani marked this pull request as ready for review June 25, 2026 13:20
@gmarciani gmarciani requested review from a team as code owners June 25, 2026 13:20
@gmarciani gmarciani force-pushed the wip/mgiacomo/3160/install-nvidia-0619-1 branch 4 times, most recently from d1f9aad to fc01172 Compare June 26, 2026 12:21
hanwen-cluster
hanwen-cluster reviewed Jun 29, 2026
View reviewed changes
Comment thread cookbooks/aws-parallelcluster-platform/libraries/nvidia.rb
Comment thread ...ks/aws-parallelcluster-platform/resources/nvidia_cuda/partial/_nvidia_cuda_install_debian.rb Outdated
Comment thread cookbooks/aws-parallelcluster-platform/resources/nvidia_cuda/partial/_nvidia_cuda_common.rb
@gmarciani
[Build] Install the NVIDIA driver and CUDA toolkit from the distribut…
369bc3c 
…ion package manager using NVIDIA local repo packages instead of the run file installers.
@gmarciani gmarciani force-pushed the wip/mgiacomo/3160/install-nvidia-0619-1 branch from fc01172 to 31c30e5 Compare July 2, 2026 11:03
@gmarciani
[Nvidia] Simplify nvidia_cuda and nvidia_driver resource by using the…
ba0378c 
… chef built-in resource package rather than platform specific logic.
@gmarciani gmarciani force-pushed the wip/mgiacomo/3160/install-nvidia-0619-1 branch from 31c30e5 to ba0378c Compare July 2, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@himani2411 himani2411 himani2411 left review comments
@hanwen-cluster hanwen-cluster hanwen-cluster approved these changes

Assignees

No one assigned

Labels

3.x

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gmarciani @himani2411 @hanwen-cluster