Commit 9aa3972
committed
fix: skip fabric manager on arm64 (Grace-Blackwell GB200/GB300)
arm64 cuda nodes are the Grace-Blackwell SKUs (GB200/GB300), which have no
node-local fabric manager -- the NVLink fabric is managed out-of-band via IMEX.
The driver container should neither install nor bundle fabricmanager there.
Today device_init() runs the fabricmanager package installer for every cuda
node. On arm64 it computes NVIDIA_FM_ARCH from $ARCH (uname -m = "aarch64")
and then only remaps to "sbsa" when the value equals "arm64" -- which never
matches -- so it looks for /opt/gpu/fabricmanager-linux-aarch64-<ver>/... which
does not exist (the tarball download.sh bundles is named "sbsa"). The installer
exits 127; installGPUDriverImage retries the whole container until CSE hits its
~780s timeout and exits 84, so ensureKubelet never runs and the node never joins
the cluster. Surfaced by the GB300 managed-driver (--gpu-driver Install) e2e.
Fix: gate fabric manager to x86_64 only.
- install.sh: run fm_run_package_installer.sh only when ARCH == x86_64; drop the
dead arm64->sbsa remap.
- download.sh: download/bundle the fabricmanager tarball only when
TARGETARCH == amd64, so the arm64 image no longer ships an unused tarball.1 parent b62994f commit 9aa3972
2 files changed
Lines changed: 13 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
52 | | - | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
53 | 56 | | |
54 | 57 | | |
55 | 58 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
170 | 170 | | |
171 | 171 | | |
172 | 172 | | |
| |||
0 commit comments