|
| 1 | +# Working on the vLLM Backend |
| 2 | + |
| 3 | +The vLLM backend lives at `backend/python/vllm/backend.py` (async gRPC) and the multimodal variant at `backend/python/vllm-omni/backend.py` (sync gRPC). Both wrap vLLM's `AsyncLLMEngine` / `Omni` and translate the LocalAI gRPC `PredictOptions` into vLLM `SamplingParams` + outputs into `Reply.chat_deltas`. |
| 4 | + |
| 5 | +This file captures the non-obvious bits — most of the bring-up was a single PR (`feat/vllm-parity`) and the things below are easy to get wrong. |
| 6 | + |
| 7 | +## Tool calling and reasoning use vLLM's *native* parsers |
| 8 | + |
| 9 | +Do not write regex-based tool-call extractors for vLLM. vLLM ships: |
| 10 | + |
| 11 | +- `vllm.tool_parsers.ToolParserManager` — 50+ registered parsers (`hermes`, `llama3_json`, `llama4_pythonic`, `mistral`, `qwen3_xml`, `deepseek_v3`, `granite4`, `openai`, `kimi_k2`, `glm45`, …) |
| 12 | +- `vllm.reasoning.ReasoningParserManager` — 25+ registered parsers (`deepseek_r1`, `qwen3`, `mistral`, `gemma4`, …) |
| 13 | + |
| 14 | +Both can be used standalone: instantiate with a tokenizer, call `extract_tool_calls(text, request=None)` / `extract_reasoning(text, request=None)`. The backend stores the parser *classes* on `self.tool_parser_cls` / `self.reasoning_parser_cls` at LoadModel time and instantiates them per request. |
| 15 | + |
| 16 | +**Selection:** vLLM does *not* auto-detect parsers from model name — neither does the LocalAI backend. The user (or `core/config/hooks_vllm.go`) must pick one and pass it via `Options[]`: |
| 17 | + |
| 18 | +```yaml |
| 19 | +options: |
| 20 | + - tool_parser:hermes |
| 21 | + - reasoning_parser:qwen3 |
| 22 | +``` |
| 23 | +
|
| 24 | +Auto-defaults for known model families live in `core/config/parser_defaults.json` and are applied: |
| 25 | +- at gallery import time by `core/gallery/importers/vllm.go` |
| 26 | +- at model load time by the `vllm` / `vllm-omni` backend hook in `core/config/hooks_vllm.go` |
| 27 | + |
| 28 | +User-supplied `tool_parser:`/`reasoning_parser:` in the config wins over defaults — the hook checks for existing entries before appending. |
| 29 | + |
| 30 | +**When to update `parser_defaults.json`:** any time vLLM ships a new tool or reasoning parser, or you onboard a new model family that LocalAI users will pull from HuggingFace. The file is keyed by *family pattern* matched against `normalizeModelID(cfg.Model)` (lowercase, org-prefix stripped, `_`→`-`). Patterns are checked **longest-first** — keep `qwen3.5` before `qwen3`, `llama-3.3` before `llama-3`, etc., or the wrong family wins. Add a covering test in `core/config/hooks_test.go`. |
| 31 | + |
| 32 | +**Sister file — `core/config/inference_defaults.json`:** same pattern but for sampling parameters (temperature, top_p, top_k, min_p, repeat_penalty, presence_penalty). Loaded by `core/config/inference_defaults.go` and applied by `ApplyInferenceDefaults()`. The schema is `map[string]float64` only — *strings don't fit*, which is why parser defaults needed their own JSON file. The inference file is **auto-generated from unsloth** via `go generate ./core/config/` (see `core/config/gen_inference_defaults/`) — don't hand-edit it; instead update the upstream source or regenerate. Both files share `normalizeModelID()` and the longest-first pattern ordering. |
| 33 | + |
| 34 | +**Constructor compatibility gotcha:** the abstract `ToolParser.__init__` accepts `tools=`, but several concrete parsers (Hermes2ProToolParser, etc.) override `__init__` and *only* accept `tokenizer`. Always: |
| 35 | + |
| 36 | +```python |
| 37 | +try: |
| 38 | + tp = self.tool_parser_cls(self.tokenizer, tools=tools) |
| 39 | +except TypeError: |
| 40 | + tp = self.tool_parser_cls(self.tokenizer) |
| 41 | +``` |
| 42 | + |
| 43 | +## ChatDelta is the streaming contract |
| 44 | + |
| 45 | +The Go side (`core/backend/llm.go`, `pkg/functions/chat_deltas.go`) consumes `Reply.chat_deltas` to assemble the OpenAI response. For tool calls to surface in `chat/completions`, the Python backend **must** populate `Reply.chat_deltas[].tool_calls` with `ToolCallDelta{index, id, name, arguments}`. Returning the raw `<tool_call>...</tool_call>` text in `Reply.message` is *not* enough — the Go regex fallback exists for llama.cpp, not for vllm. |
| 46 | + |
| 47 | +Same story for `reasoning_content` — emit it on `ChatDelta.reasoning_content`, not as part of `content`. |
| 48 | + |
| 49 | +## Message conversion to chat templates |
| 50 | + |
| 51 | +`tokenizer.apply_chat_template()` expects a list of dicts, not proto Messages. The shared helper in `backend/python/common/vllm_utils.py` (`messages_to_dicts`) handles the mapping including: |
| 52 | + |
| 53 | +- `tool_call_id` and `name` for `role="tool"` messages |
| 54 | +- `tool_calls` JSON-string field → parsed Python list for `role="assistant"` |
| 55 | +- `reasoning_content` for thinking models |
| 56 | + |
| 57 | +Pass `tools=json.loads(request.Tools)` and (when `request.Metadata.get("enable_thinking") == "true"`) `enable_thinking=True` to `apply_chat_template`. Wrap in `try/except TypeError` because not every tokenizer template accepts those kwargs. |
| 58 | + |
| 59 | +## CPU support and the SIMD/library minefield |
| 60 | + |
| 61 | +vLLM publishes prebuilt CPU wheels at `https://github.com/vllm-project/vllm/releases/...`. The pin lives in `backend/python/vllm/requirements-cpu-after.txt`. |
| 62 | + |
| 63 | +**Version compatibility — important:** newer vllm CPU wheels (≥ 0.15) declare `torch==2.10.0+cpu` as a hard dep, but `torch==2.10.0` only exists on the PyTorch test channel and pulls in an incompatible `torchvision`. Stay on **`vllm 0.14.1+cpu` + `torch 2.9.1+cpu`** until both upstream catch up. Bumping requires verifying torchvision/torchaudio match. |
| 64 | + |
| 65 | +`requirements-cpu.txt` uses `--extra-index-url https://download.pytorch.org/whl/cpu`. `install.sh` adds `--index-strategy=unsafe-best-match` for the `cpu` profile so uv resolves transformers/vllm from PyPI while pulling torch from the PyTorch index. |
| 66 | + |
| 67 | +**SIMD baseline:** the prebuilt CPU wheel is compiled with AVX-512 VNNI/BF16. On a CPU without those instructions, importing `vllm.model_executor.models.registry` SIGILLs at `_run_in_subprocess` time during model inspection. There is no runtime flag to disable it. Workarounds: |
| 68 | + |
| 69 | +1. **Run on a host with the right SIMD baseline** (default — fast) |
| 70 | +2. **Build from source** with `FROM_SOURCE=true` env var. Plumbing exists end-to-end: |
| 71 | + - `install.sh` hides `requirements-cpu-after.txt`, runs `installRequirements` for the base deps, then clones vllm and `VLLM_TARGET_DEVICE=cpu uv pip install --no-deps .` |
| 72 | + - `backend/Dockerfile.python` declares `ARG FROM_SOURCE` + `ENV FROM_SOURCE` |
| 73 | + - `Makefile` `docker-build-backend` macro forwards `--build-arg FROM_SOURCE=$(FROM_SOURCE)` when set |
| 74 | + - Source build takes 30–50 minutes — too slow for per-PR CI but fine for local. |
| 75 | + |
| 76 | +**Runtime shared libraries:** vLLM's `vllm._C` extension `dlopen`s `libnuma.so.1` at import time. If missing, the C extension silently fails and `torch.ops._C_utils.init_cpu_threads_env` is never registered → `EngineCore` crashes on `init_device` with: |
| 77 | + |
| 78 | +``` |
| 79 | +AttributeError: '_OpNamespace' '_C_utils' object has no attribute 'init_cpu_threads_env' |
| 80 | +``` |
| 81 | +
|
| 82 | +`backend/python/vllm/package.sh` bundles `libnuma.so.1` and `libgomp.so.1` into `${BACKEND}/lib/`, which `libbackend.sh` adds to `LD_LIBRARY_PATH` at run time. The builder stage in `backend/Dockerfile.python` installs `libnuma1`/`libgomp1` so package.sh has something to copy. Do *not* assume the production host has these — backend images are `FROM scratch`. |
| 83 | +
|
| 84 | +## Backend hook system (`core/config/backend_hooks.go`) |
| 85 | +
|
| 86 | +Per-backend defaults that used to be hardcoded in `ModelConfig.Prepare()` now live in `core/config/hooks_*.go` files and self-register via `init()`: |
| 87 | +
|
| 88 | +- `hooks_llamacpp.go` → GGUF metadata parsing, context size, GPU layers, jinja template |
| 89 | +- `hooks_vllm.go` → tool/reasoning parser auto-selection from `parser_defaults.json` |
| 90 | +
|
| 91 | +Hook keys: |
| 92 | +- `"llama-cpp"`, `"vllm"`, `"vllm-omni"`, … — backend-specific |
| 93 | +- `""` — runs only when `cfg.Backend` is empty (auto-detect case) |
| 94 | +- `"*"` — global catch-all, runs for every backend before specific hooks |
| 95 | +
|
| 96 | +Multiple hooks per key are supported and run in registration order. Adding a new backend default: |
| 97 | +
|
| 98 | +```go |
| 99 | +// core/config/hooks_<backend>.go |
| 100 | +func init() { |
| 101 | + RegisterBackendHook("<backend>", myDefaults) |
| 102 | +} |
| 103 | +func myDefaults(cfg *ModelConfig, modelPath string) { |
| 104 | + // only fill in fields the user didn't set |
| 105 | +} |
| 106 | +``` |
| 107 | + |
| 108 | +## The `Messages.ToProto()` fields you need to set |
| 109 | + |
| 110 | +`core/schema/message.go:ToProto()` must serialize: |
| 111 | +- `ToolCallID` → `proto.Message.ToolCallId` (for `role="tool"` messages — links result back to the call) |
| 112 | +- `Reasoning` → `proto.Message.ReasoningContent` |
| 113 | +- `ToolCalls` → `proto.Message.ToolCalls` (JSON-encoded string) |
| 114 | + |
| 115 | +These were originally not serialized and tool-calling conversations broke silently — the C++ llama.cpp backend reads them but always got empty strings. Any new field added to `schema.Message` *and* `proto.Message` needs a matching line in `ToProto()`. |
0 commit comments