Eval bug: using -devd CUDA0 on draft-dspark causes model to crash

[wallentri88]

Name and Version

latest master

Operating systems

Linux

GGML backends

CUDA

Hardware

nvidia gpus

Models

deepseek flash 0731

Problem description & steps to reproduce

load deepseek with tensor split, then load dspark model with -devd CUDA0

First Bad Commit

No response

Relevant log output

ggml/src/ggml-backend.cpp:898: pre-allocated tensor (output.weight) in a buffer (CUDA8) that cannot run the operation (NONE)

#1  0x000072260eb38683 in ggml_print_backtrace ()
#2  0x000072260eb3882b in ggml_abort ()
#3  0x000072260eb5216c in ggml_backend_sched_backend_id_from_cur(ggml_backend_sched*, ggml_tensor*)
#4  0x000072260eb53c84 in ggml_backend_sched_split_graph ()
#5  0x000072260ece3ad9 in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*)
#6  0x000072260ece3c0a in llama_context::resolve_fused_ops(llama_memory_context_i const*, unsigned int)::{lambda(llm_fused_op_probe const&, bool&)#1}::operator()(llm_fused_op_probe const&, bool&) const ()
#7  0x000072260ece3eaf in llama_context::resolve_fused_ops(llama_memory_context_i const*, unsigned int)
#8  0x000072260ece455c in llama_context::sched_reserve()
#9  0x000072260ece7449 in llama_context::llama_context(llama_model const&, llama_context_params)
#10 0x000072260ece8799 in llama_init_from_model ()
#11 0x000072260f250d5c in common_speculative_init_result::common_speculative_init_result(common_params&, llama_model*, llama_context*)
#12 0x000072260f251036 in common_speculative_init_from_params(common_params&, llama_model*, llama_context*) 

[FlareP1]

I get the same problem but only if the draft model is exclusively on its own device. If I add a single layer of the main model onto the device that the draft model is on it runs. However this results in a massive slow down of the output. I have 2x3090 + 96 GB and want to run the main model on 1x3090 + RAM and the dspark on CUDA1 but get the same error.

This fails.
.\llama-server.exe -m "%MODEL%" -ts 24 --n-cpu-moe 38 -fa on --ctx-size 65536 --parallel 1 --batch-size 2048 --ubatch-size 512 --kv-unified --temp 1.0 --top-p 0.95 --min-p 0.00 --metrics --jinja -lv 4 --mmap --fit off --model-draft "%DRAFTM%" --spec-type draft-dspark --spec-draft-n-max 5 -devd CUDA1

This works but is very slow because the main model is split across devices.
.\llama-server.exe -m "%MODEL%" -ts 24,1 --n-cpu-moe 38 -fa on --ctx-size 65536 --parallel 1 --batch-size 2048 --ubatch-size 512 --kv-unified --temp 1.0 --top-p 0.95 --min-p 0.00 --metrics --jinja -lv 4 --mmap --fit off --model-draft "%DRAFTM%" --spec-type draft-dspark --spec-draft-n-max 5 -devd CUDA1

[zack969969]

Following up on #26475 — I hit the same crash class (GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) in ggml_backend_sched_split_graph, via resolve_fused_ops -> graph_reserve -> sched_reserve), but on an 8-GPU tensor-split setup rather than 2. I tried the workaround from @FlareP1's comment (giving the draft's device a slice of the main model instead of isolating it) plus a few other combinations, and none of them avoided the crash — wanted to share the data points in case it helps narrow down the root cause (possibly related to the graph-node-count instability @ggerganov mentioned in #26369).

Environment

• 8x Tesla V100-SXM2-32GB, no NVLink (PCIe only)
• CUDA 11.5 (nvcc), driver 580.173.02
• llama.cpp commit 221f0f6356efe2260023208365705ec5d5a7c8f5 (2026-08-02), build: version: 363 (221f0f6), CUDA backend, CMAKE_CUDA_ARCHITECTURES=70
• Target model: DeepSeek-V4-Flash-0731, unsloth UD-Q8_K_XL GGUF (~162GB, 8-shard tensor-split)
• Draft model: DSpark support GGUF converted locally via convert_hf_to_gguf.py --remote --dspark from deepseek-ai/DeepSeek-V4-Flash-DSpark (verified byte-identical to am17an/DeepseekV4-Flash-20260731-DSpark, so not a conversion issue)

What I tried (all crash identically)

1. Baseline: main model split across all 8 GPUs, draft with --spec-draft-ngl 999 and no explicit --spec-draft-device

   -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999
   

2. Draft pinned to a device that already holds a full share of the main model (mirroring the "shared device" workaround from Eval bug: using -devd CUDA0 on draft-dspark causes model to crash #26475)

   -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA7
   

3. Main model restricted to CUDA1-7 (7-way split), draft isolated on CUDA0, smaller context

   -dev CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7 -ngl 999 --tensor-split 1,1,1,1,1,1,1 \
   --spec-draft-ngl 999 --spec-draft-device CUDA0 -c 65536
   

4. Baseline + --fit off (to rule out the auto-fit probing pass — this did remove the common_fit_params/common_get_device_memory_data_impl frames from the backtrace, confirming --fit off skips that pass, but the crash still occurs in the real (non-probing) context init)

   -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA0 \
   --fit off --batch-size 2048 --ubatch-size 512 --parallel 1
   

5. Small deliberate overlap, mirroring @FlareP1's exact "give the draft's device a small slice" pattern (-ts 24,1 style) scaled to 8 devices

   -ngl 999 --tensor-split 1,8,8,8,8,8,8,8 --spec-draft-ngl 999 --spec-draft-device CUDA0 \
   --fit off --batch-size 2048 --ubatch-size 512 --parallel 1
   

All five produce the identical stack:

ggml/src/ggml-backend.cpp: GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
#3  ggml_backend_sched_split_graph ()
#4  llama_context::graph_reserve(...)
#5  llama_context::resolve_fused_ops(...)
#6  llama_context::sched_reserve()
#7  llama_context::llama_context(...)

Takeaway

The device-overlap workaround that helped @FlareP1 on a 2-GPU setup doesn't scale to 8 GPUs — the crash reproduces regardless of overlap ratio or --fit state. This is consistent with @ggerganov's note in #26369 about the DSpark graph having a variable node count across batch sizes (397 @ bs=512 vs 417 @ bs=1) — more devices likely means more graph splits, making it easier to exceed GGML_SCHED_MAX_SPLIT_INPUTS even with the overlap trick. Happy to test patches or provide more logs if useful.

[relic664]

Following up on #26475 — I hit the same crash class (GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) in ggml_backend_sched_split_graph, via resolve_fused_ops -> graph_reserve -> sched_reserve), but on an 8-GPU tensor-split setup rather than 2. I tried the workaround from @FlareP1's comment (giving the draft's device a slice of the main model instead of isolating it) plus a few other combinations, and none of them avoided the crash — wanted to share the data points in case it helps narrow down the root cause (possibly related to the graph-node-count instability @ggerganov mentioned in #26369).

Environment

* 8x Tesla V100-SXM2-32GB, no NVLink (PCIe only)

* CUDA 11.5 (nvcc), driver 580.173.02

* llama.cpp commit `221f0f6356efe2260023208365705ec5d5a7c8f5` (2026-08-02), build: `version: 363 (221f0f6)`, CUDA backend, `CMAKE_CUDA_ARCHITECTURES=70`

* Target model: DeepSeek-V4-Flash-0731, unsloth UD-Q8_K_XL GGUF (~162GB, 8-shard tensor-split)

* Draft model: DSpark support GGUF converted locally via `convert_hf_to_gguf.py --remote --dspark` from `deepseek-ai/DeepSeek-V4-Flash-DSpark` (verified byte-identical to `am17an/DeepseekV4-Flash-20260731-DSpark`, so not a conversion issue)

What I tried (all crash identically)

1. Baseline: main model split across all 8 GPUs, draft with `--spec-draft-ngl 999` and no explicit `--spec-draft-device`
   ```
   -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999
   ```

2. Draft pinned to a device that **already holds a full share of the main model** (mirroring the "shared device" workaround from [Eval bug: using -devd CUDA0 on draft-dspark causes model to crash #26475](https://github.com/ggml-org/llama.cpp/issues/26475))
   ```
   -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA7
   ```

3. Main model restricted to CUDA1-7 (7-way split), draft isolated on CUDA0, smaller context
   ```
   -dev CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7 -ngl 999 --tensor-split 1,1,1,1,1,1,1 \
   --spec-draft-ngl 999 --spec-draft-device CUDA0 -c 65536
   ```

4. Baseline + `--fit off` (to rule out the auto-fit probing pass — this did remove the `common_fit_params`/`common_get_device_memory_data_impl` frames from the backtrace, confirming `--fit off` skips that pass, but the crash still occurs in the real (non-probing) context init)
   ```
   -ngl 999 --tensor-split 1,1,1,1,1,1,1,1 --spec-draft-ngl 999 --spec-draft-device CUDA0 \
   --fit off --batch-size 2048 --ubatch-size 512 --parallel 1
   ```

5. Small deliberate overlap, mirroring [@FlareP1](https://github.com/FlareP1)'s exact "give the draft's device a small slice" pattern (`-ts 24,1` style) scaled to 8 devices
   ```
   -ngl 999 --tensor-split 1,8,8,8,8,8,8,8 --spec-draft-ngl 999 --spec-draft-device CUDA0 \
   --fit off --batch-size 2048 --ubatch-size 512 --parallel 1
   ```

All five produce the identical stack:

ggml/src/ggml-backend.cpp: GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed
#3  ggml_backend_sched_split_graph ()
#4  llama_context::graph_reserve(...)
#5  llama_context::resolve_fused_ops(...)
#6  llama_context::sched_reserve()
#7  llama_context::llama_context(...)

Takeaway

The device-overlap workaround that helped @FlareP1 on a 2-GPU setup doesn't scale to 8 GPUs — the crash reproduces regardless of overlap ratio or --fit state. This is consistent with @ggerganov's note in #26369 about the DSpark graph having a variable node count across batch sizes (397 @ bs=512 vs 417 @ bs=1) — more devices likely means more graph splits, making it easier to exceed GGML_SCHED_MAX_SPLIT_INPUTS even with the overlap trick. Happy to test patches or provide more logs if useful.

The workaround for now is as described in #25784 (comment) I believe. You need to compile from source using CMAKE flags -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 -DCMAKE_CXX_FLAGS="-DGGML_SCHED_MAX_SPLIT_INPUTS=128". Note that this resolves the GGML_ASSERT specifically. I have two blackwell GPUs and am able to run when using a build compiled with those flags. I also specify -devd CUDA1 without issue, but perhaps there's an issue with 8 GPUs specifically.

[nomonkeynodeal]

-DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 -DCMAKE_CXX_FLAGS="-DGGML_SCHED_MAX_SPLIT_INPUTS=128"

Worked for my 4 GPU split.