Skip to content

Commit 9aa3972

Browse files
committed
fix: skip fabric manager on arm64 (Grace-Blackwell GB200/GB300)
arm64 cuda nodes are the Grace-Blackwell SKUs (GB200/GB300), which have no node-local fabric manager -- the NVLink fabric is managed out-of-band via IMEX. The driver container should neither install nor bundle fabricmanager there. Today device_init() runs the fabricmanager package installer for every cuda node. On arm64 it computes NVIDIA_FM_ARCH from $ARCH (uname -m = "aarch64") and then only remaps to "sbsa" when the value equals "arm64" -- which never matches -- so it looks for /opt/gpu/fabricmanager-linux-aarch64-<ver>/... which does not exist (the tarball download.sh bundles is named "sbsa"). The installer exits 127; installGPUDriverImage retries the whole container until CSE hits its ~780s timeout and exits 84, so ensureKubelet never runs and the node never joins the cluster. Surfaced by the GB300 managed-driver (--gpu-driver Install) e2e. Fix: gate fabric manager to x86_64 only. - install.sh: run fm_run_package_installer.sh only when ARCH == x86_64; drop the dead arm64->sbsa remap. - download.sh: download/bundle the fabricmanager tarball only when TARGETARCH == amd64, so the arm64 image no longer ships an unused tarball.
1 parent b62994f commit 9aa3972

2 files changed

Lines changed: 13 additions & 10 deletions

File tree

download.sh

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,11 @@ install_fabric_manager () {
4848
mv /opt/gpu/fm_run_package_installer.sh /opt/gpu/fabricmanager-linux-${NVIDIA_FM_ARCH}-${DRIVER_VERSION}/sbin/fm_run_package_installer.sh
4949
}
5050

51-
if [[ "${DRIVER_KIND}" == "cuda" ]]; then
52-
# download fabricmanager for nvlink based systems, e.g. multi instance gpu vms.
51+
if [[ "${DRIVER_KIND}" == "cuda" && "${TARGETARCH}" == "amd64" ]]; then
52+
# download fabricmanager for nvlink based systems, e.g. multi instance gpu vms (x86_64 only).
53+
# arm64 cuda nodes are the Grace-Blackwell SKUs (GB200/GB300), which have no node-local
54+
# fabric manager -- their NVLink fabric is managed out-of-band via IMEX -- so we neither
55+
# install (see install.sh device_init) nor bundle it here.
5356
install_fabric_manager
5457
fi
5558

install.sh

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -159,14 +159,14 @@ device_init() {
159159
cp -r /usr/bin/lib64/lib64/* "/usr/lib/${ARCH}-linux-gnu/"
160160
nvidia-smi
161161

162-
# install fabricmanager for nvlink based systems
163-
if [[ "${DRIVER_KIND}" == "cuda" ]]; then
164-
NVIDIA_FM_ARCH=$ARCH
165-
if [ "$NVIDIA_FM_ARCH" = "arm64" ]; then
166-
# NVIDIA uses the name "SBSA" for ARM64 platforms for the fabric manager. See https://en.wikipedia.org/wiki/Server_Base_System_Architecture
167-
NVIDIA_FM_ARCH="sbsa"
168-
fi
169-
bash /opt/gpu/fabricmanager-linux-${NVIDIA_FM_ARCH}-${DRIVER_VERSION}/sbin/fm_run_package_installer.sh
162+
# install fabricmanager for nvlink based systems (x86_64 only).
163+
# arm64 cuda nodes are the Grace-Blackwell SKUs (GB200/GB300), which have NO
164+
# node-local fabric manager -- their NVLink fabric is managed out-of-band via
165+
# IMEX. Running fm_run_package_installer.sh there is both wrong and fails (the
166+
# bundled tarball is named "sbsa", not uname -m's "aarch64"), stalling CSE past
167+
# its timeout so the node never starts kubelet. So gate FM to x86_64.
168+
if [[ "${DRIVER_KIND}" == "cuda" && "${ARCH}" == "x86_64" ]]; then
169+
bash /opt/gpu/fabricmanager-linux-${ARCH}-${DRIVER_VERSION}/sbin/fm_run_package_installer.sh
170170
fi
171171

172172
mkdir -p /etc/containerd/config.d

0 commit comments

Comments
 (0)