Skip to content

Commit 6e1dbae

Browse files
localai-botmudler
localai-bot
and
mudler
authored
feat(llama-cpp): expose 12 missing common_params via options[] (#9814)
The llama.cpp backend already accepts a free-form options: array in the model config that maps to common_params fields, but a coverage audit against upstream pin 7f3f843c flagged 12 user-visible knobs that were neither set via the typed proto fields nor reachable via options:. Wire them up under the existing if/else chain in params_parse, before the speculative section. Each new option follows the file's prevailing patterns (try/catch around numeric parses, the same true/1/yes/on bool form used elsewhere, hardware_concurrency() fallback for thread counts, mirror of draft_override_tensor for override_tensor). Top-level / batching / IO: - n_ubatch (alias ubatch) -- physical batch size; was previously force-aliased to n_batch at line 482, blocking embedding/rerank workloads that need independent control - threads_batch (alias n_threads_batch) -- main-model batch threads; mirrors the existing draft_threads_batch - direct_io (alias use_direct_io) -- O_DIRECT model loads - verbosity -- llama.cpp log threshold (line 479 had this commented out) - override_tensor (alias tensor_buft_overrides) -- per-tensor buffer overrides for the main model; mirrors draft_override_tensor Embedding / multimodal: - pooling_type (alias pooling) -- mean/cls/last/rank/none; previously only auto-flipped to RANK for rerankers - embd_normalize (alias embedding_normalize) -- and the embedding handler now reads params_base.embd_normalize instead of a hardcoded 2 at the previous embd_normalize literal in Embedding() - mmproj_use_gpu (alias mmproj_offload) -- mmproj on CPU vs GPU - image_min_tokens / image_max_tokens -- per-image vision token budget Reasoning surface (the audit-focus three; LocalAI's existing ReasoningConfig.DisableReasoning only feeds the per-request chat_template_kwargs.enable_thinking and does not touch any of these): - reasoning_format -- none/auto/deepseek/deepseek-legacy parser - enable_reasoning (alias reasoning_budget) -- -1/0/>0 thinking budget - prefill_assistant -- trailing-assistant-message prefill toggle All 14 referenced fields exist on both the upstream pin and the turboquant fork's common.h, so no LOCALAI_LEGACY_LLAMA_CPP_SPEC guard is needed. Docs: extend model-configuration.md with new "Reasoning Models", "Multimodal Backend Options", "Embedding & Reranking Backend Options", and "Other Backend Tuning Options" subsections; also refresh the Speculative Type Values table to show the new dash-separated canonical names alongside the underscore aliases LocalAI still accepts. Assisted-by: claude-code:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
1 parent 53bdb18 commit 6e1dbae

2 files changed

Lines changed: 188 additions & 13 deletions

File tree

‎backend/cpp/llama-cpp/grpc-server.cpp‎

Lines changed: 133 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -688,6 +688,136 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
688688
// If conversion fails, keep default value (8)
689689
}
690690
}
691+
692+
// --- physical batch size (upstream -ub / --ubatch-size) ---
693+
// Note: line ~482 already aliases n_ubatch to n_batch as a default; this
694+
// option lets users decouple the two (useful for embeddings/rerank).
695+
} else if (!strcmp(optname, "n_ubatch") || !strcmp(optname, "ubatch")) {
696+
if (optval != NULL) {
697+
try { params.n_ubatch = std::stoi(optval_str); } catch (...) {}
698+
}
699+
700+
// --- main-model batch threads (upstream -tb / --threads-batch) ---
701+
} else if (!strcmp(optname, "threads_batch") || !strcmp(optname, "n_threads_batch")) {
702+
if (optval != NULL) {
703+
try {
704+
int n = std::stoi(optval_str);
705+
if (n <= 0) n = (int)std::thread::hardware_concurrency();
706+
params.cpuparams_batch.n_threads = n;
707+
} catch (...) {}
708+
}
709+
710+
// --- pooling type for embeddings (upstream --pooling) ---
711+
} else if (!strcmp(optname, "pooling_type") || !strcmp(optname, "pooling")) {
712+
if (optval != NULL) {
713+
if (optval_str == "none") params.pooling_type = LLAMA_POOLING_TYPE_NONE;
714+
else if (optval_str == "mean") params.pooling_type = LLAMA_POOLING_TYPE_MEAN;
715+
else if (optval_str == "cls") params.pooling_type = LLAMA_POOLING_TYPE_CLS;
716+
else if (optval_str == "last") params.pooling_type = LLAMA_POOLING_TYPE_LAST;
717+
else if (optval_str == "rank") params.pooling_type = LLAMA_POOLING_TYPE_RANK;
718+
// unknown values silently leave UNSPECIFIED (auto-detect)
719+
}
720+
721+
// --- llama log verbosity threshold (upstream -lv / --verbosity) ---
722+
} else if (!strcmp(optname, "verbosity")) {
723+
if (optval != NULL) {
724+
try { params.verbosity = std::stoi(optval_str); } catch (...) {}
725+
}
726+
727+
// --- O_DIRECT model loading (upstream --direct-io) ---
728+
} else if (!strcmp(optname, "direct_io") || !strcmp(optname, "use_direct_io")) {
729+
if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
730+
params.use_direct_io = true;
731+
} else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
732+
params.use_direct_io = false;
733+
}
734+
735+
// --- embedding normalization (upstream --embd-normalize) ---
736+
// -1 none, 0 max-abs, 1 taxicab, 2 L2 (default), >2 p-norm
737+
} else if (!strcmp(optname, "embd_normalize") || !strcmp(optname, "embedding_normalize")) {
738+
if (optval != NULL) {
739+
try { params.embd_normalize = std::stoi(optval_str); } catch (...) {}
740+
}
741+
742+
// --- reasoning parser (upstream --reasoning-format) ---
743+
// Picks the parser for <think> blocks emitted by reasoning models.
744+
// none / auto / deepseek / deepseek-legacy
745+
} else if (!strcmp(optname, "reasoning_format")) {
746+
if (optval != NULL) {
747+
if (optval_str == "none") params.reasoning_format = COMMON_REASONING_FORMAT_NONE;
748+
else if (optval_str == "auto") params.reasoning_format = COMMON_REASONING_FORMAT_AUTO;
749+
else if (optval_str == "deepseek") params.reasoning_format = COMMON_REASONING_FORMAT_DEEPSEEK;
750+
else if (optval_str == "deepseek-legacy" || optval_str == "deepseek_legacy")
751+
params.reasoning_format = COMMON_REASONING_FORMAT_DEEPSEEK_LEGACY;
752+
// unknown values silently keep the upstream default (DEEPSEEK)
753+
}
754+
755+
// --- reasoning budget (upstream --reasoning-budget) ---
756+
// -1 unlimited, 0 disabled, >0 token budget for thinking blocks.
757+
// Distinct from per-request `enable_thinking` (chat_template_kwargs).
758+
} else if (!strcmp(optname, "enable_reasoning") || !strcmp(optname, "reasoning_budget")) {
759+
if (optval != NULL) {
760+
try { params.enable_reasoning = std::stoi(optval_str); } catch (...) {}
761+
}
762+
763+
// --- prefill assistant turn (upstream --no-prefill-assistant) ---
764+
} else if (!strcmp(optname, "prefill_assistant")) {
765+
if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
766+
params.prefill_assistant = true;
767+
} else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
768+
params.prefill_assistant = false;
769+
}
770+
771+
// --- mmproj GPU offload (upstream --no-mmproj-offload, inverted) ---
772+
} else if (!strcmp(optname, "mmproj_use_gpu") || !strcmp(optname, "mmproj_offload")) {
773+
if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
774+
params.mmproj_use_gpu = true;
775+
} else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
776+
params.mmproj_use_gpu = false;
777+
}
778+
779+
// --- per-image vision token budget (upstream --image-min/max-tokens) ---
780+
} else if (!strcmp(optname, "image_min_tokens")) {
781+
if (optval != NULL) {
782+
try { params.image_min_tokens = std::stoi(optval_str); } catch (...) {}
783+
}
784+
} else if (!strcmp(optname, "image_max_tokens")) {
785+
if (optval != NULL) {
786+
try { params.image_max_tokens = std::stoi(optval_str); } catch (...) {}
787+
}
788+
789+
// --- main-model tensor buffer overrides (upstream --override-tensor) ---
790+
// Format: <tensor regex>=<buffer type>,<tensor regex>=<buffer type>,...
791+
// Mirrors the existing `draft_override_tensor` parser below.
792+
} else if (!strcmp(optname, "override_tensor") || !strcmp(optname, "tensor_buft_overrides")) {
793+
ggml_backend_load_all();
794+
std::map<std::string, ggml_backend_buffer_type_t> buft_list;
795+
for (size_t i = 0; i < ggml_backend_dev_count(); ++i) {
796+
auto * dev = ggml_backend_dev_get(i);
797+
auto * buft = ggml_backend_dev_buffer_type(dev);
798+
if (buft) {
799+
buft_list[ggml_backend_buft_name(buft)] = buft;
800+
}
801+
}
802+
static std::list<std::string> override_names;
803+
std::string cur;
804+
auto flush = [&](const std::string & spec) {
805+
auto pos = spec.find('=');
806+
if (pos == std::string::npos) return;
807+
const std::string name = spec.substr(0, pos);
808+
const std::string type = spec.substr(pos + 1);
809+
auto it = buft_list.find(type);
810+
if (it == buft_list.end()) return; // unknown buffer type: ignore
811+
override_names.push_back(name);
812+
params.tensor_buft_overrides.push_back(
813+
{override_names.back().c_str(), it->second});
814+
};
815+
for (char c : optval_str) {
816+
if (c == ',') { if (!cur.empty()) { flush(cur); cur.clear(); } }
817+
else { cur.push_back(c); }
818+
}
819+
if (!cur.empty()) flush(cur);
820+
691821
// Speculative decoding options
692822
} else if (!strcmp(optname, "spec_type") || !strcmp(optname, "speculative_type")) {
693823
#ifdef LOCALAI_LEGACY_LLAMA_CPP_SPEC
@@ -2808,7 +2938,9 @@ class BackendServiceImpl final : public backend::Backend::Service {
28082938
}
28092939
}
28102940

2811-
int embd_normalize = 2; // default to Euclidean/L2 norm
2941+
// Honor the load-time embd_normalize set via options:embd_normalize.
2942+
// -1 none, 0 max-abs, 1 taxicab, 2 L2 (default), >2 p-norm.
2943+
int embd_normalize = params_base.embd_normalize;
28122944
// create and queue the task
28132945
auto rd = ctx_server.get_response_reader();
28142946
{

‎docs/content/advanced/model-configuration.md‎

Lines changed: 55 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -316,23 +316,66 @@ These are set via the `options:` array in the model configuration (format: `key:
316316

317317
#### Speculative Type Values
318318

319-
| Type | Description |
320-
|------|-------------|
321-
| `none` | No speculative decoding (default) |
322-
| `draft` | Draft model-based speculation (auto-set when `draft_model` is configured) |
323-
| `eagle3` | EAGLE3 draft model architecture |
324-
| `ngram_simple` | Simple self-speculative using token history |
325-
| `ngram_map_k` | N-gram with key-only map |
326-
| `ngram_map_k4v` | N-gram with keys and 4 m-gram values |
327-
| `ngram_mod` | Modified n-gram speculation |
328-
| `ngram_cache` | 3-level n-gram cache |
329-
330-
Multiple types can be chained by passing a comma-separated list to `spec_type` (e.g. `spec_type:ngram_simple,ngram_mod`). The runtime tries them in order and accepts the first proposal that meets the acceptance criteria.
319+
The canonical names match upstream llama.cpp (dash-separated). For backward compatibility LocalAI also accepts the underscore-separated forms and the bare `draft` / `eagle3` aliases.
320+
321+
| Type | Aliases accepted | Description |
322+
|------|------------------|-------------|
323+
| `none` | | No speculative decoding (default) |
324+
| `draft-simple` | `draft`, `draft_simple` | Draft model-based speculation (auto-set when `draft_model` is configured) |
325+
| `draft-eagle3` | `eagle3`, `draft_eagle3` | EAGLE3 draft model architecture |
326+
| `ngram-simple` | `ngram_simple` | Simple self-speculative using token history |
327+
| `ngram-map-k` | `ngram_map_k` | N-gram with key-only map |
328+
| `ngram-map-k4v` | `ngram_map_k4v` | N-gram with keys and 4 m-gram values |
329+
| `ngram-mod` | `ngram_mod` | Modified n-gram speculation |
330+
| `ngram-cache` | `ngram_cache` | 3-level n-gram cache |
331+
332+
Multiple types can be chained by passing a comma-separated list to `spec_type` (e.g. `spec_type:ngram-simple,ngram-mod`). The runtime tries them in order and accepts the first proposal that meets the acceptance criteria.
331333

332334
{{% notice note %}}
333335
Speculative decoding is automatically disabled when multimodal models (with `mmproj`) are active. The `n_draft` parameter can also be overridden per-request.
334336
{{% /notice %}}
335337

338+
### Reasoning Models (DeepSeek-R1, Qwen3, etc.)
339+
340+
These load-time options control how the backend parses `<think>` reasoning blocks and how much budget the model is allowed for thinking. They are set per model via the `options:` array.
341+
342+
| Option | Type | Default | Description |
343+
|--------|------|---------|-------------|
344+
| `reasoning_format` | string | `deepseek` | Parser for reasoning/thinking blocks. One of `none`, `auto`, `deepseek`, `deepseek-legacy` (alias `deepseek_legacy`). |
345+
| `enable_reasoning` / `reasoning_budget` | int | `-1` | Reasoning budget in tokens: `-1` unlimited, `0` disabled, `>0` token cap for the thinking section. |
346+
| `prefill_assistant` | bool | `true` | When `false`, the trailing assistant message is not pre-filled by the chat template. |
347+
348+
{{% notice note %}}
349+
This is the load-time reasoning configuration. The orthogonal per-request `enable_thinking` chat-template kwarg (set via the YAML `reasoning.disable` field) toggles thinking on/off per call without restarting the model.
350+
{{% /notice %}}
351+
352+
### Multimodal Backend Options
353+
354+
| Option | Type | Default | Description |
355+
|--------|------|---------|-------------|
356+
| `mmproj_use_gpu` / `mmproj_offload` | bool | `true` | Set `false` to keep the multimodal projector on CPU (saves VRAM at cost of speed). |
357+
| `image_min_tokens` | int | `-1` | Minimum vision tokens per image. `-1` keeps the model default. |
358+
| `image_max_tokens` | int | `-1` | Maximum vision tokens per image. `-1` keeps the model default. |
359+
360+
### Embedding & Reranking Backend Options
361+
362+
| Option | Type | Default | Description |
363+
|--------|------|---------|-------------|
364+
| `pooling_type` / `pooling` | string | auto | Pooling strategy for embeddings: `none`, `mean`, `cls`, `last`, `rank`. Reranking automatically uses `rank`. |
365+
| `embd_normalize` / `embedding_normalize` | int | `2` | Normalization: `-1` none, `0` max-abs, `1` taxicab, `2` Euclidean (L2), `>2` p-norm. |
366+
367+
### Other Backend Tuning Options
368+
369+
These llama.cpp options are passed through the `options:` array.
370+
371+
| Option | Type | Default | Description |
372+
|--------|------|---------|-------------|
373+
| `n_ubatch` / `ubatch` | int | same as `batch` | Physical batch size. Decouple from `n_batch` when an embedding/rerank workload needs a different value. |
374+
| `threads_batch` / `n_threads_batch` | int | same as `threads` | Threads used during prompt processing. `<= 0` means `hardware_concurrency()`. |
375+
| `direct_io` / `use_direct_io` | bool | `false` | Open the model with `O_DIRECT` (faster cold loads on NVMe; ignored if not supported). |
376+
| `verbosity` | int | `3` | llama.cpp internal log verbosity threshold. Higher = more verbose. |
377+
| `override_tensor` / `tensor_buft_overrides` | string | "" | Per-tensor buffer-type overrides for the main model. Format: `<tensor regex>=<buffer type>,<tensor regex>=<buffer type>,...`. Mirrors the existing `draft_override_tensor` syntax for the draft model. |
378+
336379
### Prompt Caching
337380

338381
| Field | Type | Description |

0 commit comments

Comments
 (0)