fix(nvidia-smi): exit gracefully instead of panicking on errors (#206)#210
Merged
benamizilberNvidia merged 1 commit intoJun 3, 2026
Merged
Conversation
The fake nvidia-smi panicked on essentially every error path, surfacing a Go stack trace in user pods. Now errors are accumulated rather than panicked: - topology fetch / status / decode failures are fatal-for-devices: print `no devices found` plus the accumulated errors to stderr and exit 1, instead of rendering a bogus table. - a bad RUNAI_NUM_OF_GPUS keeps the default portion (no longer zeroes the reported memory) and readProcessName failures only blank one column — both non-fatal; the table still renders with the errors logged to stderr. - add the missing topology response body Close. Also drop the two os.Setenv(TOPOLOGY_CM_*) calls: this binary has fetched topology over a hardcoded HTTP URL since inception and never reads those env vars — dead boilerplate.
benamizilberNvidia
deleted the
bzilber/RUN-39984-nvidia-smi-error-handling
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The fake
nvidia-smi(cmd/nvidia-smi/main.go)panic()d on essentially every error path, so failures surfaced in user pods as a cryptic Go stack trace instead of a useful message. This replaces the panics with error accumulation and a graceful exit.Behavior
http.Get/ non-2xx status / JSON decodeno devices found+ accumulated errors to stderr, exit 1 (no bogus table)RUNAI_NUM_OF_GPUS1.0); table still renders, error logged to stderrreadProcessName(/proc/1/cmdline)Key fixes:
NODE_NAMEcase) no longer reaches the JSON decoder and panics.RUNAI_NUM_OF_GPUSis parsed into a temp var, so it no longer zeroes reported memory.Close.os.Setenv(TOPOLOGY_CM_*)calls — this binary has fetched topology over a hardcoded HTTP URL since inception and never reads those env vars (dead boilerplate; confirmed via history back to the original commit).Happy-path output is unchanged, so the existing nvidia-smi e2e specs (
device_plugin_test.go,e2e_test.go) continue to hold.Closes RUN-39984. Fixes #206.