Router (server) stops forwarding parent CLI flags (`--parallel`, `--cache-type-*`, `--flash-attn`, `-ngl`) to spawned child instances — regression between b9641 and b9692

[jhsmith409]

Summary

In router mode (--models-preset / --models-max), the parent llama-server no longer forwards most of its own command-line flags to the per-model child instances it spawns. Flags that are only set on the parent's CLI (not derivable from the preset .ini) silently disappear from the child's argument list, so children fall back to defaults.

The most damaging case is --parallel: its default changed to -1 = auto, which resolves to n_parallel = 4 on a multi-core host. A child sized for a single slot then allocates ~4× the KV cache and OOMs.

• Good: server-cuda-b9641 (also the floating tag prior to this) — parent CLI flags are forwarded to children.
• Bad: server-cuda-b9692 — parent CLI flags are dropped; only preset-derived keys reach the child.

Environment

• Image: ghcr.io/ggml-org/llama.cpp:server-cuda-b9692 (CUDA 12.8.1)
• 2× NVIDIA Quadro P5000 (Pascal), router mode, --models-max 1
• Parent command line (note --parallel 1, --cache-type-k/v q8_0, --flash-attn on, --n-gpu-layers 999):

/app/llama-server --models-preset /models.ini --models-max 1 --parallel 1 \
  --n-gpu-layers 999 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --jinja --metrics --host 0.0.0.0 --port 8080 --threads 40 --numa distribute

Reproduction

1. Run a router (--models-preset preset.ini --models-max 1) with --parallel 1 (and --cache-type-k/v q8_0 --flash-attn on -ngl 999) on the parent CLI.
2. Request any model whose ctx-size was tuned to fit at exactly one slot.
3. The child is spawned without --parallel, defaults to n_parallel = 4, and OOMs on the KV cache.

Observed: child spawn argument lists differ between versions

b9692 — the child's spawning server instance with args: list is missing --parallel, --cache-type-k, --cache-type-v, --flash-attn, --n-gpu-layers (only preset-derived keys survive):

load:   /app/llama-server
load:   --host 127.0.0.1
load:   --port 57803
load:   --alias granite4.1:30b
load:   --ctx-size 79872
load:   --model /models/granite-4.1-30b-UD-Q4_K_XL.gguf
load:   --main-gpu 0
load:   --split-mode layer
load:   --tensor-split 1,1

Child then logs the fallback and OOMs:

I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 10296.00 MiB on device 0: cudaMalloc failed: out of memory
E llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache

b9641 — same preset, same parent CLI — the child does receive --parallel 1 (and --cache-type-k/v, --flash-attn on, --n-gpu-layers 999):

load:   /app/llama-server
load:   --host 127.0.0.1
load:   --jinja
load:   --metrics
load:   --numa distribute
load:   --port 59289
load:   --alias granite4.1:30b
load:   --ctx-size 79872
load:   --cache-type-k q8_0
load:   --cache-type-v q8_0
load:   --flash-attn on
load:   --model /models/granite-4.1-30b-UD-Q4_K_XL.gguf
load:   --main-gpu 0
load:   --n-gpu-layers 999
load:   --parallel 1
load:   --split-mode layer
load:   --threads 40
load:   --tensor-split 1,1

The b9641 child never logs n_parallel is set to auto (because --parallel 1 was passed), loads cleanly, and serves requests.

Expected

Parent CLI flags that configure spawned children (--parallel, --cache-type-k/v, --flash-attn, --n-gpu-layers, --threads, --numa, …) should continue to be forwarded to child instances, as they were in b9641 — or the change in forwarding behavior should be documented, with a preset-level equivalent for each.

Workaround

Moving --parallel 1 from the parent CLI into the preset [*] section as parallel = 1 does forward to children on b9692 (preset-derived keys still propagate). But the same isn't ergonomic for every dropped CLI flag, and it's a behavior change from b9641 either way.

Likely cause

Somewhere in the b9641…b9692 range; the router rework server: (router) add model management API (#23976) is the most likely candidate that touches how child argument lists are constructed.

[jhsmith409]

Likely the same root cause as #24735 ("build b9688 no respect context size"), which is also router mode (its log shows cmd_child_to_router:ready): there the child ignores --ctx-size 120000 and runs at the 4096 default, with b9688 named as the first bad commit — squarely in the b9680→b9692 window. Same regression, different flag (--ctx-size there, --parallel/--cache-type-*/--flash-attn/-ngl here).

#24735 was closed without a root-cause comment, so the underlying flag-forwarding bug appears unaddressed. Cross-linking in case it helps narrow the first bad commit.

[jhsmith409]

Fix submitted in #24763 — the router rework #23976 dropped the final_presets[*].merge(base_preset) pass in server_models::load_models() that gave the parent's CLI args highest precedence (present in b9641, gone in b9692). The PR restores it.

[jhsmith409]

I put the original code back in PR #24763, but that PR was rejected as being AI written code. It was a restoration of the original code from the repo. ???

Claude did diagnose the issue and wrote up the description, but the code came from the repo itself. Why close it for that?