Fix auto device falling back when a provider fails to load#1674
Conversation
When device='auto' and a high-priority execution provider (e.g. CUDA on Linux without libcuda.so installed) is not available, createInferenceSession would throw instead of trying the next provider in the list. The fallback now lives in constructSessions, where the device selection context is available. Each provider is tried individually. On failure a warning is logged and the next provider is attempted. createInferenceSession itself is unchanged. Closes huggingface#1642.
nico-martin
left a comment
There was a problem hiding this comment.
Hi @Suh0161
Thanks fo looking into this. I think it would be a great improvement for device: "auto".
This matches the current device: "auto" behavior, because the candidates are already ordered by preference. But that order is static, and session creation only proves that a provider works, not that it is fastest for the user's hardware or model.
Should auto mean "first preferred provider that starts", or should it try to choose the best provider for the actual hardware/model? I would not block this fix on that, but the intended behavior should be documented.
This PR also overlaps conceptually with PR #1646, which proposes exporting deviceToExecutionProviders and getSupportedDevices so downstream callers can implement their own fallback logic. If both PRs move forward, it would be useful to align the semantics: getSupportedDevices() / deviceToExecutionProviders("auto") should describe the same priority order that this internal fallback uses, and documentation should make clear whether callers should rely on that order as a performance preference or only as a capability fallback order.
| // When device='auto', try each execution provider individually so that a failing | ||
| // accelerator (e.g. CUDA not installed) falls back cleanly to the next one. | ||
| // For any explicit device the caller already knows what they want — no fallback. | ||
| const isAuto = |
There was a problem hiding this comment.
The isAuto check re-reads the raw option/config value instead of using the resolved selectedDevice already computed in getSession and returned in session_config.device:
Please base this on the resolved device, for example session_config.device === DEVICE_TYPES.auto, or return selectedDevice explicitly from getSession and use that.
Closes #1642
Fix: auto device falls back gracefully when a provider fails to load
When
device: 'auto'is used and a high-priority execution provider (e.g. CUDA on Linux withoutlibcuda.soinstalled) is unavailable,createInferenceSessionwould throw instead of trying the next provider in the list.The fallback now lives in
constructSessionsinsession.js, where the device selection context is available. Each provider is tried individually — on failure a warning is logged and the next one is attempted.createInferenceSessionitself is unchanged.