Xinference 部署在GB10上启动之后显示xinference.device_utils 13176 ERROR Fail to get GPU info: Not Supported

[Rvelamen]

System Info / 系統信息

uname -a

Linux thinkstationpgx-30b2 6.17.0-1014-nvidia #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 17 19:01:40 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux

nvidia-smi:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.142                Driver Version: 580.142        CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   38C    P8              4W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Log:

xinference-local --host 0.0.0.0 --port 13002
/root/miniconda3/envs/xinference/lib/python3.10/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/root/miniconda3/envs/xinference/lib/python3.10/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-05-08 14:34:30,050 xinference.core.supervisor 13176 INFO     Xinference supervisor 0.0.0.0:41641 started
2026-05-08 14:34:30,202 xinference.core.worker 13176 INFO     Starting metrics export server at 0.0.0.0:None
2026-05-08 14:34:30,203 xinference.core.worker 13176 INFO     Checking metrics export server...
2026-05-08 14:34:30,948 xinference.core.worker 13176 INFO     Metrics server is started at: http://0.0.0.0:39467
2026-05-08 14:34:30,948 xinference.core.worker 13176 INFO     Purge cache directory: /root/.xinference/cache
2026-05-08 14:34:30,948 xinference.core.utils 13176 INFO     Remove empty directory: /root/.xinference/cache/v2
2026-05-08 14:34:30,951 xinference.core.worker 13176 INFO     Connected to supervisor as a fresh worker
2026-05-08 14:34:30,964 xinference.core.worker 13176 INFO     Xinference worker 0.0.0.0:41641 started
2026-05-08 14:34:30,966 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:34:36,785 xinference.api.restful_api 13135 INFO     Starting Xinference at endpoint: http://0.0.0.0:13002
2026-05-08 14:34:36,853 uvicorn.error 13135 INFO     Uvicorn running on http://0.0.0.0:13002 (Press CTRL+C to quit)
2026-05-08 14:34:45,985 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:35:01,002 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:35:16,023 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:35:31,042 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:35:46,062 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:36:01,081 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:36:16,099 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:36:31,119 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported
2026-05-08 14:36:46,139 xinference.device_utils 13176 ERROR    Fail to get GPU info: Not Supported

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

Name: xinference
Version: 2.7.0
Summary: Model Serving Made Easy
Home-page: https://github.com/xorbitsai/inference
Author: Qin Xuye
Author-email: qinxuye@xprobe.io
License: Apache License 2.0

The command used to start Xinference / 用以启动 xinference 的命令

xinference-local --host 0.0.0.0 --port 13002

Reproduction / 复现过程

xinference-local --host 0.0.0.0 --port 13002

Expected behavior / 期待表现

正常使用GPU

[qinxuye]

实际上能使用吗？目前可能获取状态有问题，但不影响使用。

[tbraun96]

Disclosure: I maintain Atlas, answering generically here.

That Fail to get GPU info: Not Supported from device_utils.py:13176 is almost certainly Xinference calling the legacy nvmlDeviceGetMemoryInfo on a GB10. The GB10 doesn't expose the v1 NVML memory-info namespace because of its unified-memory layout, so the call returns NVML_ERROR_NOT_SUPPORTED. HAMi had the same root cause and just got a fix backported:

Project-HAMi/HAMi#1841

For Xinference, the device-probe path needs the v2 variant first with a fallback:

# xinference/device_utils.py, around the failing call
try:
    info = pynvml.nvmlDeviceGetMemoryInfo_v2(handle)
except (pynvml.NVMLError_NotSupported, AttributeError):
    info = pynvml.nvmlDeviceGetMemoryInfo(handle)

That covers GB10 + still works on Hopper/Ada where v1 is fine.

If you need an actual inference target running on the Spark while this gets fixed upstream, Atlas's GB10 recipe set is at Avarok-Cybersecurity/atlas-recipes. OpenAI-compatible API on :8888, so whatever was fronting Xinference for chat/completions can be pointed at it without changes. The qwen3.5-35b-a3b-nvfp4 recipe is a sane starting baseline (~133 tok/s decode at c=1 on a single Spark).

[Rvelamen]

声明：我是Atlas 的维护者，在这里我的回答仅代表个人观点。

几乎可以肯定，这Fail to get GPU info: Not Supported是device_utils.py:13176Xinference 在 GB10 上调用旧版方法的结果nvmlDeviceGetMemoryInfo。由于 GB10 采用的是统一内存布局，因此它没有公开 v1 NVML 内存信息命名空间，所以调用返回了NVML_ERROR_NOT_SUPPORTED错误。HAMi 也存在同样的根本原因，并且刚刚收到一个向后移植的修复程序：

Project-HAMi/HAMi#1841

对于 Xinference，设备探测路径首先需要 v2 版本，并有一个备用版本：

xinference/device_utils.py, around the failing call

try:
info = pynvml.nvmlDeviceGetMemoryInfo_v2(handle)
except (pynvml.NVMLError_NotSupported, AttributeError):
info = pynvml.nvmlDeviceGetMemoryInfo(handle)
这涵盖了 GB10，并且在 Hopper/Ada 上仍然有效，v1 版本也能正常运行。

如果上游修复此问题期间，您需要在 Spark 上运行实际的推理目标，Atlas 的 GB10 配方集位于 [此处应填写 Atlas 的 GB10 配方集地址] Avarok-Cybersecurity/atlas-recipes。OpenAI 兼容的 API 位于 [:8888此处应填写 OpenAI 的 API 地址]，因此之前用于聊天/补全的 Xinference 前端程序无需更改即可指向它。qwen3.5-35b-a3b-nvfp4 配方是一个合理的初始基准（在单个 Spark 上，c=1 时解码速度约为 133 tok/s）。

Great！

[qinxuye]

Disclosure: I maintain Atlas, answering generically here.

That Fail to get GPU info: Not Supported from device_utils.py:13176 is almost certainly Xinference calling the legacy nvmlDeviceGetMemoryInfo on a GB10. The GB10 doesn't expose the v1 NVML memory-info namespace because of its unified-memory layout, so the call returns NVML_ERROR_NOT_SUPPORTED. HAMi had the same root cause and just got a fix backported:

Project-HAMi/HAMi#1841

For Xinference, the device-probe path needs the v2 variant first with a fallback:

xinference/device_utils.py, around the failing call

try:
info = pynvml.nvmlDeviceGetMemoryInfo_v2(handle)
except (pynvml.NVMLError_NotSupported, AttributeError):
info = pynvml.nvmlDeviceGetMemoryInfo(handle)
That covers GB10 + still works on Hopper/Ada where v1 is fine.

If you need an actual inference target running on the Spark while this gets fixed upstream, Atlas's GB10 recipe set is at Avarok-Cybersecurity/atlas-recipes. OpenAI-compatible API on :8888, so whatever was fronting Xinference for chat/completions can be pointed at it without changes. The qwen3.5-35b-a3b-nvfp4 recipe is a sane starting baseline (~133 tok/s decode at c=1 on a single Spark).

Are you willing to submit a PR to address this?