feat(vllm-cpp): wire the full engine config surface through engine_args

[localai-bot]

What

The vllm-cpp backend could configure four of the engine's knobs — block size, KV block count, max sequence length, max concurrent sequences — out of a config surface that is considerably larger. Speculative decoding, prefix caching, the chunked-prefill token budget, the scheduling policy and the external KV connector (LMCache) were reachable from vllm.cpp's own HTTP server and from nothing LocalAI could write in a model config.

Part of that gap was the C ABI itself, which carried strictly less than EngineParams does. That is fixed upstream in vllm.cpp ABI v9 (merged to main as eec09bed, which this PR pins). The rest was here: the backend parsed a flat options: list with five recognised keys and had no way to express a nested JSON document at all.

Configuration

Config now goes through engine_args: — the same map the vLLM and SGLang backends already take — with keys spelled as vLLM's own CLI flags, so a speculative_config or kv_transfer_config block written for vLLM works verbatim:

name: qwen36-dflash
backend: vllm-cpp
context_size: 8192
engine_args:
  max_num_seqs: 32
  max_num_batched_tokens: 8192
  enable_prefix_caching: true
  scheduling_policy: lpm
  speculative_config:
    method: dflash
    model: z-lab/Qwen3.6-27B-DFlash
    num_speculative_tokens: 4
  kv_transfer_config:
    kv_connector: LMCacheConnector
    kv_role: kv_both
    kv_connector_extra_config: {host: 127.0.0.1, port: 65432}

The options: list keeps working and now reads every key too, so no existing config breaks; engine_args wins where both set the same key.

All three speculative methods the engine supports are reachable: mtp (draft head inside the target checkpoint, safetensors only), dflash (separate draft checkpoint, model: required), and ngram (draft-free).

Two details worth reviewer attention

enable_prefix_caching: false maps to the ABI tri-state force-OFF (2), not 0. 0 means "let the model capability decide", and dense architectures default the cache on — collapsing the two would silently enable it against an explicit false.

cSamplingParams grows the ABI v8 logits-processor tail. LocalAI installs no processor, but the C side reads those fields off the pointer we hand it, so a Go struct that stopped at StructuredJSONObject (120 bytes vs C's 136) would have had the engine read past our allocation and call whatever sat there. Latent only because the old v5 ABI gate refused to load a v8 library.

Importer

A vllm-cpp import of a HuggingFace repo now probes config.json and writes speculative_config: {method: mtp} into the generated engine_args when the checkpoint declares an MTP head — the safetensors analogue of the llama-cpp importer's GGUF header probe. An explicit speculative_config is never overwritten, and every probe failure is non-fatal.

Two asymmetries versus the llama.cpp hook, both deliberate:

• DFlash draft repositories are refused with a warning rather than auto-configured. A drafter cannot serve alone, and the target/draft pairing is not derivable from either repo in isolation.
• The llama-cpp importer stops applying spec_type:draft-mtp when the chosen backend is vllm-cpp. Those are llama.cpp option keys vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source outright because the mtp.* draft tensors do not survive GGUF conversion. This was a pre-existing bug — a GGUF import with the vllm-cpp preference emitted dead llama.cpp options.

Docs

docs/content/features/text-generation.md gains a vllm.cpp section covering the engine_args table, all three speculative methods, LMCache, and the legacy options: list. The backend previously had no documentation page at all.

Testing

• make lint clean.
• core/config, core/gallery/importers, backend/go/vllm-cpp all green. 9 new importer specs, 11 new engine_args specs, plus updated struct-offset assertions for the v9 / v8 layouts.
• Built a CPU-only libvllm.so at the pinned commit and confirmed purego binds all 19 symbols with vllm_abi_version() returning 9. That check is now a spec: set VLLM_CPP_LIBRARY and it runs without needing model weights, so a future pin bump with a stale struct mirror fails in CI rather than at a user's first load.
• Upstream: 31/31 capi test cases on the merged vllm.cpp tree.

Not verified: a real inference end-to-end. The new knobs are checked at the ABI-handshake and unit level, not against a live generation with weights loaded.

🤖 Generated with Claude Code