Skip to content

Commit e41dca4

Browse files
authored
feat: RFC-011 -- local LLM backend selection (llama-cpp-python + onnxruntime-genai) (#104)
* docs: add RFC-011 for local LLM backend selection (llama-cpp-python + onnxruntime-genai) Introduce a config-managed local_backend field under llm.provider: local that lets users select between llama-cpp-python (current default) and onnxruntime-genai as the in-process inference engine. Key design decisions: - Internal LocalBackend protocol keeps the change isolated to local_provider.py - Zero caller impact — no change to generate() or any of its 7 call sites - onnxruntime-genai is an optional dependency (pip install zettelforge[local-onnx]) - Execution provider (cpu, cuda, rocm, dml) passes through extra config - Both backends share the same LocalProvider dispatcher and registry entry Related: RFC-002 (Universal LLM Provider Interface) * feat: implement RFC-011 — local LLM backend selection via config (llama-cpp-python + onnxruntime-genai) Implements Phase 1 of RFC-011: adds local_backend config field under llm.provider: local that selects between llama-cpp-python (default) and onnxruntime-genai as the in-process inference engine. Changes: - Extract LlamaCppBackend class from LocalProvider (preserves existing behavior) - Add OnnxGenAIBackend class with lazy SDK load and full generate() support - Refactor LocalProvider to dispatch on backend parameter via _get_impl() - Add local_backend field to LLMConfig dataclass + __repr__ - Add ZETTELFORGE_LLM_LOCAL_BACKEND env override in _apply_env() - Forward local_backend through _provider_kwargs() in llm_client.py - Add local-onnx and local-all optional extras to pyproject.toml - Update config.default.yaml with local_backend documentation and examples - Update test suite: +15 new tests for backend dispatching, all backends, and backward-compat _llm setter path - Update RFC-011 with Decision and Implementation Status sections RFC: docs/rfcs/RFC-011-local-llm-backend-config.md PR: #104 * style: fix ruff format in local_provider.py - Collapse single-line string expressions (ruff auto-format) - Collapse single-line AttributeError messages (ruff auto-format) CI fix: lint check (ruff format) was failing on 4 formatting issues. * fix: backend-aware model/filename defaults in _provider_kwargs (RFC-011) Bug: When local_backend: onnxruntime-genai was configured but llm.model contained an Ollama tag (e.g., the default qwen3.5:9b), _provider_kwargs unconditionally substituted DEFAULT_LLM_MODEL (a GGUF HuggingFace repo) and DEFAULT_LLM_FILENAME (a .gguf file) regardless of which backend was selected. The ONNX backend would receive a GGUF model reference. Fix: _provider_kwargs now inspects local_backend before choosing fallback defaults. When backend is onnxruntime-genai, it uses ONNX-appropriate defaults (microsoft/Phi-3-mini-4k-instruct-onnx, .onnx filename). When backend is llama-cpp-python (default), existing GGUF defaults apply. Also adds _DEFAULT_ONNX_MODEL and _DEFAULT_ONNX_FILENAME constants to llm_client.py for cross-referencing. * style: fix ruff format in llm_client.py Collapsed nested ternary expression that ruff format flagged.
1 parent 16d4a45 commit e41dca4

7 files changed

Lines changed: 985 additions & 45 deletions

File tree

config.default.yaml

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -146,21 +146,34 @@ embedding:
146146
# extraction, and RAG synthesis.
147147
#
148148
# Provider selects the backend registered in zettelforge.llm_providers:
149-
# local — in-process llama-cpp-python (requires zettelforge[local])
149+
# local — in-process LLM inference (see local_backend below; requires zettelforge[local] or zettelforge[local-onnx])
150150
# ollama — Ollama HTTP API
151151
# mock — test-only canned responses
152152
# openai_compat — any /v1/chat/completions endpoint (Phase 2, upcoming)
153153
# anthropic — Anthropic native SDK (Phase 3, upcoming)
154154
#
155+
# local_backend selects the in-process inference engine when provider=local:
156+
# llama-cpp-python — GGUF models via llama-cpp-python (default; requires zettelforge[local])
157+
# onnxruntime-genai — ONNX models via ONNX Runtime GenAI (requires zettelforge[local-onnx])
158+
#
155159
# Examples:
156160
# # Default — Ollama on localhost
157161
# provider: ollama
158162
# model: qwen3.5:9b
159163
#
160164
# # In-process GGUF (fully offline)
161165
# provider: local
166+
# local_backend: llama-cpp-python
162167
# model: Qwen/Qwen2.5-3B-Instruct-GGUF
163168
#
169+
# # In-process ONNX (AMD GPU via ROCm)
170+
# provider: local
171+
# local_backend: onnxruntime-genai
172+
# model: microsoft/Phi-3-mini-4k-instruct-onnx
173+
# extra:
174+
# filename: phi3-mini-4k-instruct-q4.onnx
175+
# provider: rocm # cpu, cuda, dml, rocm, openvino, coreml
176+
#
164177
# api_key supports ${ENV_VAR} references — never commit raw API keys.
165178
# api_key: ${OPENAI_API_KEY}
166179
#
@@ -177,10 +190,11 @@ embedding:
177190
# ZETTELFORGE_LLM_PROVIDER=ollama
178191
# ZETTELFORGE_LLM_MODEL=qwen2.5:7b
179192
# ZETTELFORGE_LLM_URL=http://gpu-box:11434
180-
# ZETTELFORGE_LLM_API_KEY=sk-...
193+
# ZETTELFORGE_LLM_API_KEY=***
181194
# ZETTELFORGE_LLM_TIMEOUT=60
182195
# ZETTELFORGE_LLM_MAX_RETRIES=2
183196
# ZETTELFORGE_LLM_FALLBACK=ollama
197+
# ZETTELFORGE_LLM_LOCAL_BACKEND=onnxruntime-genai # only when provider=local
184198
#
185199
llm:
186200
provider: ollama
@@ -191,6 +205,7 @@ llm:
191205
timeout: 60.0
192206
max_retries: 2
193207
fallback: ""
208+
local_backend: llama-cpp-python # used when provider=local (RFC-011)
194209
extra: {}
195210

196211

0 commit comments

Comments
 (0)