Commit e41dca4
authored
feat: RFC-011 -- local LLM backend selection (llama-cpp-python + onnxruntime-genai) (#104)
* docs: add RFC-011 for local LLM backend selection (llama-cpp-python + onnxruntime-genai)
Introduce a config-managed local_backend field under llm.provider: local
that lets users select between llama-cpp-python (current default) and
onnxruntime-genai as the in-process inference engine.
Key design decisions:
- Internal LocalBackend protocol keeps the change isolated to local_provider.py
- Zero caller impact — no change to generate() or any of its 7 call sites
- onnxruntime-genai is an optional dependency (pip install zettelforge[local-onnx])
- Execution provider (cpu, cuda, rocm, dml) passes through extra config
- Both backends share the same LocalProvider dispatcher and registry entry
Related: RFC-002 (Universal LLM Provider Interface)
* feat: implement RFC-011 — local LLM backend selection via config (llama-cpp-python + onnxruntime-genai)
Implements Phase 1 of RFC-011: adds local_backend config field under
llm.provider: local that selects between llama-cpp-python (default)
and onnxruntime-genai as the in-process inference engine.
Changes:
- Extract LlamaCppBackend class from LocalProvider (preserves existing behavior)
- Add OnnxGenAIBackend class with lazy SDK load and full generate() support
- Refactor LocalProvider to dispatch on backend parameter via _get_impl()
- Add local_backend field to LLMConfig dataclass + __repr__
- Add ZETTELFORGE_LLM_LOCAL_BACKEND env override in _apply_env()
- Forward local_backend through _provider_kwargs() in llm_client.py
- Add local-onnx and local-all optional extras to pyproject.toml
- Update config.default.yaml with local_backend documentation and examples
- Update test suite: +15 new tests for backend dispatching, all backends,
and backward-compat _llm setter path
- Update RFC-011 with Decision and Implementation Status sections
RFC: docs/rfcs/RFC-011-local-llm-backend-config.md
PR: #104
* style: fix ruff format in local_provider.py
- Collapse single-line string expressions (ruff auto-format)
- Collapse single-line AttributeError messages (ruff auto-format)
CI fix: lint check (ruff format) was failing on 4 formatting issues.
* fix: backend-aware model/filename defaults in _provider_kwargs (RFC-011)
Bug: When local_backend: onnxruntime-genai was configured but llm.model
contained an Ollama tag (e.g., the default qwen3.5:9b), _provider_kwargs
unconditionally substituted DEFAULT_LLM_MODEL (a GGUF HuggingFace repo)
and DEFAULT_LLM_FILENAME (a .gguf file) regardless of which backend was
selected. The ONNX backend would receive a GGUF model reference.
Fix: _provider_kwargs now inspects local_backend before choosing fallback
defaults. When backend is onnxruntime-genai, it uses ONNX-appropriate
defaults (microsoft/Phi-3-mini-4k-instruct-onnx, .onnx filename).
When backend is llama-cpp-python (default), existing GGUF defaults apply.
Also adds _DEFAULT_ONNX_MODEL and _DEFAULT_ONNX_FILENAME constants to
llm_client.py for cross-referencing.
* style: fix ruff format in llm_client.py
Collapsed nested ternary expression that ruff format flagged.1 parent 16d4a45 commit e41dca4
7 files changed
Lines changed: 985 additions & 45 deletions
File tree
- docs/rfcs
- src/zettelforge
- llm_providers
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | | - | |
| 149 | + | |
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
155 | 159 | | |
156 | 160 | | |
157 | 161 | | |
158 | 162 | | |
159 | 163 | | |
160 | 164 | | |
161 | 165 | | |
| 166 | + | |
162 | 167 | | |
163 | 168 | | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
164 | 177 | | |
165 | 178 | | |
166 | 179 | | |
| |||
177 | 190 | | |
178 | 191 | | |
179 | 192 | | |
180 | | - | |
| 193 | + | |
181 | 194 | | |
182 | 195 | | |
183 | 196 | | |
| 197 | + | |
184 | 198 | | |
185 | 199 | | |
186 | 200 | | |
| |||
191 | 205 | | |
192 | 206 | | |
193 | 207 | | |
| 208 | + | |
194 | 209 | | |
195 | 210 | | |
196 | 211 | | |
| |||
0 commit comments