Skip to content

Fix auto device falling back when a provider fails to load#1674

Open
Suh0161 wants to merge 1 commit into
huggingface:mainfrom
Suh0161:fix/auto-device-fallback
Open

Fix auto device falling back when a provider fails to load#1674
Suh0161 wants to merge 1 commit into
huggingface:mainfrom
Suh0161:fix/auto-device-fallback

Conversation

@Suh0161

@Suh0161 Suh0161 commented May 4, 2026

Copy link
Copy Markdown

Closes #1642

Fix: auto device falls back gracefully when a provider fails to load

When device: 'auto' is used and a high-priority execution provider (e.g. CUDA on Linux without libcuda.so installed) is unavailable, createInferenceSession would throw instead of trying the next provider in the list.

The fallback now lives in constructSessions in session.js, where the device selection context is available. Each provider is tried individually — on failure a warning is logged and the next one is attempted. createInferenceSession itself is unchanged.

// Before: hard crash on Linux x64 without libcuda.so
// After: warns and automatically falls back to the next available provider
await pipeline('text-generation', 'onnx-community/LFM2-1.2B-ONNX', { device: 'auto' });

When device='auto' and a high-priority execution provider (e.g. CUDA on
Linux without libcuda.so installed) is not available, createInferenceSession
would throw instead of trying the next provider in the list.

The fallback now lives in constructSessions, where the device selection
context is available. Each provider is tried individually. On failure a
warning is logged and the next provider is attempted. createInferenceSession
itself is unchanged.

Closes huggingface#1642.
@nico-martin nico-martin self-assigned this Jun 8, 2026

@nico-martin nico-martin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Suh0161

Thanks fo looking into this. I think it would be a great improvement for device: "auto".

This matches the current device: "auto" behavior, because the candidates are already ordered by preference. But that order is static, and session creation only proves that a provider works, not that it is fastest for the user's hardware or model.

Should auto mean "first preferred provider that starts", or should it try to choose the best provider for the actual hardware/model? I would not block this fix on that, but the intended behavior should be documented.

This PR also overlaps conceptually with PR #1646, which proposes exporting deviceToExecutionProviders and getSupportedDevices so downstream callers can implement their own fallback logic. If both PRs move forward, it would be useful to align the semantics: getSupportedDevices() / deviceToExecutionProviders("auto") should describe the same priority order that this internal fallback uses, and documentation should make clear whether callers should rely on that order as a performance preference or only as a capability fallback order.

// When device='auto', try each execution provider individually so that a failing
// accelerator (e.g. CUDA not installed) falls back cleanly to the next one.
// For any explicit device the caller already knows what they want — no fallback.
const isAuto =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The isAuto check re-reads the raw option/config value instead of using the resolved selectedDevice already computed in getSession and returned in session_config.device:

Please base this on the resolved device, for example session_config.device === DEVICE_TYPES.auto, or return selectedDevice explicitly from getSession and use that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto device on Linux x64 fails hard when CUDA shared library is unavailable instead of falling back

2 participants