Skip to content

Commit dc9f50d

Browse files
committed
fix(gallery): raise arch-router scoring output cap via parallel:64
Scoring decodes the whole prompt+candidate in a single llama_decode and reads one logit row per candidate token. The vendored llama.cpp server caps causal output rows at n_parallel, so the default of 1 aborts with GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route labels. Set options: [parallel:64] on both arch-router quant entries to lift the cap; kv_unified (the grpc-server default) keeps the full context per sequence, so this does not split the KV cache. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>
1 parent a3054a2 commit dc9f50d

1 file changed

Lines changed: 11 additions & 0 deletions

File tree

gallery/index.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24475,6 +24475,14 @@
2447524475
# concurrent generation traffic; see model_config.go validation).
2447624476
known_usecases:
2447724477
- score
24478+
# Scoring decodes the whole prompt+candidate in one llama_decode and reads
24479+
# a logit row per candidate token. The llama.cpp server caps the causal
24480+
# output rows at n_parallel, so the default (1) aborts with
24481+
# GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max). Raise it to cover
24482+
# multi-token route labels; kv_unified (the grpc-server default) keeps the
24483+
# full context per sequence, so this does not split the KV cache.
24484+
options:
24485+
- parallel:64
2447824486
parameters:
2447924487
model: Arch-Router-1.5B.Q4_K_M.gguf
2448024488
files:
@@ -24486,6 +24494,9 @@
2448624494
overrides:
2448724495
known_usecases:
2448824496
- score
24497+
# See the q4 entry: lift the scoring output-row cap above the default 1.
24498+
options:
24499+
- parallel:64
2448924500
parameters:
2449024501
model: Arch-Router-1.5B.Q8_0.gguf
2449124502
files:

0 commit comments

Comments
 (0)