feat(serve): qmd serve — shared model server, continuing #511

[brettdavies]

qmd serve: shared model server, continuing #511

Continuation note: Three of the four commits in this PR are @jaylfc's work from #511, cherry-picked with -x (preserving his Author: line and recording the source SHAs). His PR has been open since April; this PR rebases it onto current main, applies the changes he's been iterating on, and adds two small things on top: a low-vram passthrough that piggybacks on the engine flag from #662, and three UX touches to make the combination discoverable. Net new from me is one commit (~17 lines). If you'd rather take #511 and have me re-open a follow-up against it, I can do that. Wanted to put the rebased + stacked version in front of you first since #511 hasn't moved.

Why

Multiple QMD clients each cold-load the embed, generate, and rerank models into their own process. On a 16 GB device running three agents, that's 3× the model memory; on ARM64 / headless servers, each client has to compile node-llama-cpp against whatever GPU drivers are installed locally.

qmd serve centralises the three models behind an HTTP server. Clients (CLIs in LXC containers, agents on separate hosts) talk to a single warm instance via the RemoteLLM class introduced in this PR, activated by QMD_REMOTE_URL=http://host:7832. The remote client implements the same LLM interface as LlamaCpp, so it slots in transparently anywhere the SDK accepts an LLM.

Use cases the original PR called out:

• Multi-agent setups sharing one embedding server.
• LXC/Docker containers reaching a host-level GPU/NPU without compiling node-llama-cpp inside each container.
• ARM64 / headless servers: bypass local llama.cpp build entirely.
• ARM SBCs (Orange Pi RK3588 with NPU) where RAM is scarce.

I've been running this fork on an RTX 3090 Ti alongside an Ollama Gemma 4 26B; the low-vram combination (this PR combined with the engine flag below) made the three-model fit possible.

What this PR does

1. qmd serve: long-running HTTP server. Two backends:

• local (default): loads GGUF models via node-llama-cpp.
• ollama: proxies to any Ollama-compatible REST endpoint (Ollama itself, rkllama on RK3588 NPU, etc).

1. HTTP endpoints: /embed, /embed-batch, /rerank, /expand, /tokenize, /vsearch, /health, /status, /collections, /search, /browse.
2. RemoteLLM (src/llm-remote.ts): LLM-interface implementation that POSTs to the server. Auto-activates via QMD_REMOTE_URL (or --remote-url <url> per-invocation).
3. --low-vram passthrough on serve: picks up the engine flag from feat(llm): add --low-vram to share a GPU with another large model #662. qmd serve --low-vram loads one heavy model at a time so the serve process can coexist with a large local Ollama on the same GPU. Warns rather than silently ignoring when combined with --backend ollama (ollama owns its own model lifecycle).
4. Startup log + help text surface the low-vram mode and the model lifecycle (resident vs on-demand) so an operator can see at a glance what the server will load.

Endpoint	Method	Purpose
/embed, /embed-batch	POST	embedding (single + batch)
/rerank	POST	document reranking
/expand	POST	query expansion (lex/vec/hyde)
/tokenize	POST	token count
/health	GET	server status + loaded models
/status	GET	index health (doc counts, embedding status)
/collections	GET	collection list
/search?q=…	GET	FTS5 keyword search
/vsearch	POST	vector similarity search
/browse	GET	paginated chunk listing

Default bind is 127.0.0.1; --bind 0.0.0.0 exposes to network. Body size capped at 50 MB. Strict input validation on every endpoint.

Relationship to other open PRs

This PR is stacked on the engine --low-vram PR (filed alongside as #662). That PR adds lowVram to LlamaCppConfig and surfaces it as a global flag/env. The serve passthrough in this PR is then trivial: qmd serve --low-vram just sets QMD_LOW_VRAM=1, the LocalBackend's LlamaCpp constructor reads it, and the heavy-model lifecycle is handled in the engine. No serve-specific concurrency code.

This PR also overlaps in problem space with #608 ("Daemon-aware CLI fast-path"). Both keep models warm across requests. The honest comparison:

	qmd mcp --http --daemon (+ #608)	qmd serve (this PR)
Protocol	MCP over Streamable HTTP	Plain REST
What it exposes	High-level tools: query, get, multi_get	Low-level model primitives + index endpoints
Who calls it	MCP agents + same-machine CLI (after #608)	RemoteLLM qmd clients (multi-host, LXC)
Default bind	localhost (PID-file discovery)	designed for 0.0.0.0
Backend swap	local node-llama-cpp only	local or ollama-compat proxy

There's a real question (open below) about whether these should remain two separate daemons or whether the model-primitive endpoints + RemoteLLM should land as an extension of the MCP daemon instead. The substance of @jaylfc's work (the primitive endpoints, the RemoteLLM client, the ollama proxy backend) survives either shape; only the top-level serve subcommand vs mcp --http --daemon choice changes.

Architecture

c9d6c27  Brett    feat(llm): add low-vram mode that loads one heavy model at a time   ← #662
7a1bda7  @jaylfc   feat: remote model server (qmd serve) and client support
34519c3  @jaylfc   feat(serve): add /vsearch endpoint for semantic vector search
87d35e2  @jaylfc   refactor: rename rkllama backend to ollama-compat — generic naming
d4590fa  Brett    feat(serve): surface low-vram in serve help, startup log, ollama warning
fbfcdcd  Brett    fix(remote): route store layer through getDefaultLLM so QMD_REMOTE_URL reaches RemoteLLM

The top commit (c9d6c27) is #662, which this stacks on. The middle three are @jaylfc's feat/remote-llm-provider-clean cherry-picked verbatim onto current main (preserving his Author: field + recording the source SHA in each commit message). One conflict resolution was needed beyond mechanical re-application: b5f156c "Improve qmd diagnostics and embed resilience" landed on main while this branch was being rebased, refactoring the embedding error-handling into tryEmbedChunk / recordFailure / retryFailedChunks. That supersedes @jaylfc's manual 3-attempt retry loop, so the rebase takes main's structure on those hunks; less code, integrates with the new failure tracker.

My next commit (d4590fa) is 17 lines: serve help text, startup log labels (resident / on-demand), and a warning when --low-vram combines with --backend ollama.

Follow-up commit (fbfcdcd): completing the store-layer wiring

Caught while running qmd query against the daemon on a VRAM-constrained box: the CLI sets setDefaultLLM(new RemoteLLM(...)) correctly, but the store layer hardcoded getDefaultLlamaCpp() at every embed / expandQuery / rerank call site (src/store.ts:84/3562/3737/3783). The polymorphic accessor was set and then bypassed one layer deeper. On a box with enough headroom the bypass is invisible: clients silently pay for two LLM backends. With Ollama co-resident (~21 GB on a 24 GB GPU), the rerank stage fails with Failed to create any rerank context.

The follow-up commit:

• Routes the four store call sites through getDefaultLLM() and widens their llmOverride?: LlamaCpp params to LLM. Store.llm widens to LLM to match the documented "Can be LlamaCpp or RemoteLLM" comment.
• Guards cli/qmd.ts:getStore so it skips the eager setDefaultLlamaCpp(new LlamaCpp(...)) when QMD_REMOTE_URL is set. The --remote-url flag mirrors into the env so one source of truth gates the guard.
• Adds optional embedModelName / generateModelName / rerankModelName readonly accessors to the LLM interface so existing consumers keep their ?? DEFAULT_* fallbacks without casting.
• Replaces the buggy lazy require("./llm-remote.js") in getDefaultLLM (which fails under vitest ESM resolution) with a static import. No runtime cycle: llm-remote.ts only type-imports from llm.ts.
• Adds a requireLlamaCpp(llm, op) narrow helper for indexing paths (generateEmbeddings, embedding fingerprint adoption) that genuinely need the local engine. It accepts a real LlamaCpp unconditionally, refuses with a clear message when the active LLM is RemoteLLM, and casts through for duck-typed test fixtures so existing mocks keep working.

Test additions:

• New test/store-remote-llm.test.ts (6 tests): covers both llmOverride and getDefaultLLM() routing for expandQuery and rerank, plus the Store adapter routing through Store.llm. All pass under both Node/vitest and Bun.
• LlamaCpp Integration describe block in test/store.test.ts now opportunistically routes through qmd serve: probes http://127.0.0.1:7832/health in beforeAll, calls setDefaultLLM(new RemoteLLM(...)) when reachable, falls back to local LlamaCpp when not. The 3 rerank tests in that block which previously failed under VRAM pressure now pass via the daemon, turning the integration suite into an end-to-end regression check for the remote path.
• Test environment hermetics: vitest.config.ts and src/test-preload.ts both unset QMD_REMOTE_URL so a developer's shell-set env var doesn't accidentally route 200+ unrelated tests through a real daemon (or fail them with ECONNREFUSED when one isn't running).

Tests

Engine-level concurrency tests for the low-vram mode live in #662 (test/llm-low-vram.test.ts, 6 tests, all pass against a fake llama backend so they run without GPU). The store-layer wiring fix in fbfcdcd adds 6 new tests in test/store-remote-llm.test.ts and converts the existing LlamaCpp Integration describe block in test/store.test.ts into an opportunistic remote regression check (see the follow-up section above).

Full suite on this branch with qmd serve running locally: 880/903 pass. The 23 remaining failures: 14 LLM-class tests in test/llm.test.ts and test/mcp.test.ts that exercise the LlamaCpp class directly (not through the store) and fail under VRAM pressure (Failed to create any rerank context), and 9 CLI tests that fail on Node 26 due to DEP0205 deprecation warnings leaking to stderr. Both families reproduce against bare upstream/main in the same environment.

tsc --noEmit clean.

Backwards compatibility

• New serve subcommand and RemoteLLM class. Purely additive.
• QMD_REMOTE_URL and --remote-url are new (renamed from QMD_SERVER / --server in @jaylfc's earlier draft per a naming discussion; see the rename commit for the rationale). Nothing in main reads them.
• The existing qmd query path is unchanged: with no QMD_REMOTE_URL set, behaviour is identical to today.

Things I deliberately didn't do

• Auth. Bind defaults to 127.0.0.1; if you want network exposure (--bind 0.0.0.0) it's behind a firewall / Tailscale / etc. Adding token auth or mTLS is a separate concern.
• Unify with the MCP daemon. Real question worth maintainer input; see open questions below.
• Replace --backend rkllama with anything more specific. @jaylfc's rename to ollama-compat is the right generic name; specific NPU implementations like rkllama remain compatible via that interface.

Open questions for you

1. Should this and qmd mcp --http --daemon be one process or two? They're both long-running HTTP servers that hold models warm. The differences (MCP protocol vs REST, agent tools vs model primitives, localhost vs network) are real but orthogonal: one daemon could expose both surfaces. If you'd prefer a single canonical daemon, the substance of this PR (primitive endpoints + RemoteLLM client + ollama proxy backend) could land as an extension of the MCP daemon instead of a new serve subcommand. Happy to reshape if that's the call.
2. /v1/ versioning, like Daemon-aware CLI fast-path: ~4× speedup for qmd query #608 introduced? This PR uses unversioned routes (/embed, /rerank, etc.) matching @jaylfc's original. If you want /v1/embed etc. for consistency with Daemon-aware CLI fast-path: ~4× speedup for qmd query #608's /v1/search, easy rename.
3. Authoring credit. Three commits are @jaylfc's verbatim with (cherry picked from commit …) trailers. If you'd rather take feat: remote model server (qmd serve) for shared inference across clients #511 directly instead of this rebased + stacked version, I'm fine to close this and re-open a smaller PR against feat: remote model server (qmd serve) for shared inference across clients #511's branch with just my two additions.