[Build] Install the NVIDIA driver and CUDA toolkit from the distribution package manager using NVIDIA local repo packages instead of the run file installers.

[gmarciani]

THIS PR CANNOT BE MERGED UNTIL WE PUBLISH THE REFERENCED ARTIFACTS TO OUR PUBLIC BUCKETS

Description of changes

Migrate the NVIDIA driver and CUDA toolkit installation from the .run runfile installers to NVIDIA local-repo packages (.rpm/.deb) installed through the distribution package manager.

This is the approach suggested both by NVIDIA and AWS. Compared to the original run file approach, it is also more robust against kernel bumps.

As part of this change we also:

1. converted the installation of cuda toolkit from recipe to resource.
2. remove the nvidia local repos to save space in the AMI. This is safe because if the user must change the nvidia software versions they should download different local repos in any case.
3. install fabric manager only if it is not already installed (this fixes a pre-existing bug in our code, where we used to install fabric manager even if it was already installed)

Tests

1. Unit test (added/updated) included in this PR check
2. Build image on all supported OSs (with DevSettings pointing to NVIDIA local repos from Nvidia website because artifacts are not published to our bucket yet)
3. Successful integ tests:

test-suites:
  basic:
    test_essential_features.py::test_essential_features:
      dimensions:
        - regions: ["us-east-1"]
          instances: ["g4dn.xlarge"]
          oss: ["alinux2023","ubuntu2204"]
          schedulers: ["slurm"]
  health_checks:
    test_gpu_health_checks.py::test_cluster_with_gpu_health_checks:
      dimensions:
        - regions: ["us-east-1"]
          instances: ["g4dn.xlarge"]
          oss: ["alinux2023","ubuntu2204"]
          schedulers: ["slurm"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.28%. Comparing base (e5dd95b) to head (0c30d55).
⚠️ Report is 4 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3207   +/-   ##
========================================
  Coverage    77.28%   77.28%           
========================================
  Files           24       24           
  Lines         2545     2545           
========================================
  Hits          1967     1967           
  Misses         578      578           

Flag	Coverage Δ
unittests	77.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.