Skip to content

Commit 018d186

Browse files
author
Test User
committed
docs(adf): add research and design documents for LLM client auto-configuration Refs #2671
1 parent 51bcad8 commit 018d186

2 files changed

Lines changed: 349 additions & 0 deletions

File tree

‎.docs/adf/2671/design-llm-client-wiring.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
# Revised Design: Full LLM Stack -- Ollama + OpenRouter with Capability Routing
2+
3+
**Status**: Draft (revised per user: all 4 items in scope)
4+
**Research Doc**: `.docs/adf/2671/research-llm-client-wiring.md`
5+
**Author**: opencode
6+
**Date**: 2026-06-14
7+
8+
---
9+
10+
## Overview
11+
12+
Wire both Ollama (local, free, fast) and OpenRouter (cloud, free tier) into RLM, with capability-based routing so simple tasks hit cheap Ollama and complex tasks hit OpenRouter. This unblocks `rlm_query`, the deterministic-rlm-review skill, and the full RLM query loop.
13+
14+
### Providers
15+
16+
| Provider | Model | Cost | Latency | Capabilities |
17+
|----------|-------|------|---------|-------------|
18+
| Ollama local | `gemma3:270m` (cheapest auto-detected) | Free | ~100ms | FastThinking, CodeGeneration, Explanation, Documentation |
19+
| OpenRouter | `meta-llama/llama-3.2-3b-instruct:free` | Free | ~500ms | DeepThinking, CodeGeneration, CodeReview, Architecture |
20+
21+
### Routing strategy
22+
23+
`CapabilityFirst` (`RouterStrategy::QualityFirst`): each `rlm_query` prompt is keyword-matched against provider capabilities. Simple code-gen goes to Ollama. Security audits, architecture reviews go to OpenRouter. Falls back to cheapest if no match.
24+
25+
### Scope (all 4 items included)
26+
27+
1. **Multi-provider routing**: `RouterBridgeLlmClient` with `CapabilityFirst` strategy
28+
2. **OpenRouter**: API key from 1Password, free tier model
29+
3. **genai crate**: Still skipped (redundant; direct Ollama + OpenRouter clients exist)
30+
4. **deterministic-rlm-review**: Unblocked -- capability routing selects the right provider per reviewer role
31+
32+
## Updated Architecture
33+
34+
```
35+
RLM startup
36+
│
37+
├── auto_configure_llm()
38+
│ │
39+
│ ├── env overrides: RLM_PROVIDER, RLM_MODEL, RLM_ROUTER_STRATEGY
40+
│ │
41+
│ ├── Build OllamaClient (local)
42+
│ │ auto-detect cheapest chat model via /api/tags
43+
│ │ → gemma3:270m (292MB)
44+
│ │
45+
│ ├── Build OpenRouterClient (cloud)
46+
│ │ API key from 1Password or env var
47+
│ │ → meta-llama/llama-3.2-3b-instruct:free
48+
│ │
49+
│ └── Wrap in RouterBridgeLlmClient
50+
│ strategy: CapabilityFirst (QualityFirst)
51+
│ register both providers with capability profiles
52+
│
53+
└── set_llm_client(router_bridge)
54+
↓
55+
rlm_query("audit for SQL injection...") → keyword "security"
56+
→ RouterBridgeLlmClient::resolve_client()
57+
→ "security" ∉ Ollama caps, ∈ OpenRouter caps → OpenRouter
58+
→ POST api.openrouter.ai/v1/chat/completions
59+
→ real LLM response
60+
```
61+
62+
## Provider capability matrix
63+
64+
```
65+
Prompt keywords → Capability mapping:
66+
67+
"audit security vulnerability exploit injection" → DeepThinking → OpenRouter
68+
"review code correctness edge case pattern" → CodeReview → OpenRouter
69+
"analyse hot path allocation performance" → FastThinking → Ollama
70+
"evaluate API design breaking change" → Architecture → OpenRouter
71+
"implement function write code generate" → CodeGeneration → Ollama (cheaper)
72+
"explain document describe" → Explanation → Ollama
73+
```
74+
75+
## File changes
76+
77+
| File | Change |
78+
|------|--------|
79+
| `Cargo.toml` | Add `terraphim_config` dep behind `llm` feature; add `ollama` feature to `terraphim_service` |
80+
| `rlm.rs` | Rewrite `auto_configure_llm()`: build dual clients, wrap in RouterBridgeLlmClient |
81+
| `main.rs` | Call `auto_configure_llm()` in `run()` |
82+
83+
## Implementation notes
84+
85+
### Building the OpenRouter client
86+
87+
The `build_llm_from_role()` factory needs `role.extra["llm_provider"] = "openrouter"` and `role.extra["llm_model"] = "meta-llama/llama-3.2-3b-instruct:free"`. The API key must be available in `role.extra["llm_api_key"]` or via the `OPENROUTER_API_KEY` env var (which `build_openrouter_from_role` reads).
88+
89+
### Building the router bridge
90+
91+
`build_llm_from_role()` already has code that constructs `RouterBridgeLlmClient` when `role.llm_router_enabled = true` (`llm.rs:200-231`). We just need to set the right fields on the role:
92+
93+
```rust
94+
role.llm_router_enabled = true;
95+
role.llm_router_config = Some(LlmRouterConfig {
96+
enabled: true,
97+
strategy: RouterStrategy::QualityFirst,
98+
mode: RouterMode::Library,
99+
..Default::default()
100+
});
101+
```
102+
103+
This triggers the code path that:
104+
1. Builds both Ollama and OpenRouter clients
105+
2. Wraps them in `RouterBridgeLlmClient` with `strategy_from_config(&RouterStrategy::QualityFirst)` (= `CapabilityFirst`)
106+
3. Returns the bridge as `Arc<dyn LlmClient>`
107+
108+
### API key retrieval
109+
110+
From 1Password at startup: `op read "op://OdiloVault/OpenRouterTesting-api-key/password"`. Cache in env var to avoid repeated 1Password calls.
111+
112+
## Test strategy
113+
114+
| Test | Purpose |
115+
|------|---------|
116+
| `test_auto_configure_dual_providers` | Both Ollama and OpenRouter clients built |
117+
| `test_routing_codegen_to_ollama` | "Write a function" → Ollama (CodeGeneration, cheaper) |
118+
| `test_routing_security_to_openrouter` | "Audit for SQL injection" → OpenRouter (DeepThinking) |
119+
| `test_rlm_query_returns_real_response` | `rlm_query` returns non-stub text |
120+
| `test_ollama_unreachable_falls_back` | OpenRouter used when Ollama down |

‎.docs/adf/2671/research-llm-client-wiring.md‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# Research Document: Wire LLM Client into RLM with Smart Cheapest-Model Routing
2+
3+
**Status**: Draft
4+
**Author**: opencode
5+
**Date**: 2026-06-14
6+
7+
## Executive Summary
8+
9+
`terraphim_rlm`'s `LlmBridge` correctly returns `LlmNotConfigured` when no LLM client is injected. The existing `terraphim_service::llm::build_llm_from_role()` factory can construct clients (Ollama, OpenRouter, genai) but requires a `terraphim_config::Role`. The orchestrator has `CostFirst` routing at the provider level but not at the model level. We need to: (1) add `terraphim_config` as an optional dependency, (2) build a role config from environment/runtime detection, (3) select the cheapest available model within each provider, and (4) inject the client into RLM.
10+
11+
## Essential Questions Check
12+
13+
| Question | Answer | Evidence |
14+
|----------|--------|----------|
15+
| Energising? | Yes | Unblocks `rlm_query`, the deterministic-rlm-review skill, and the full RLM query loop |
16+
| Leverages strengths? | Yes | Reuses existing `build_llm_from_role()`, `RouterBridgeLlmClient`, Ollama provider |
17+
| Meets real need? | Yes | `rlm_query` currently returns `LlmNotConfigured`; without this, RLM cannot do LLM-based work |
18+
19+
**Proceed**: Yes (3/3 YES)
20+
21+
## Problem Statement
22+
23+
### Description
24+
RLM's LLM bridge has an injection point (`set_llm_client()`) but no auto-configuration mechanism. The CLI binary starts without an LLM client, making the `query` command always return `LlmNotConfigured`. The deterministic-rlm-review skill and the full RLM query loop both require real LLM calls.
25+
26+
### Impact
27+
- `rlm_query` non-functional
28+
- Deterministic-rlm-review skill cannot spawn reviewers
29+
- RLM query loop cannot do recursive LLM work
30+
- Users must manually configure providers
31+
32+
### Success Criteria
33+
1. `terraphim_rlm query` returns real LLM responses, not `LlmNotConfigured`
34+
2. Smart routing selects fastest/cheapest available model automatically
35+
3. No new external dependencies beyond what's already in the workspace
36+
4. Falls back gracefully when no providers are available
37+
38+
## Current State Analysis
39+
40+
### Existing Implementation
41+
42+
| Component | Location | Purpose |
43+
|-----------|----------|---------|
44+
| `LlmBridge` | `crates/terraphim_rlm/src/llm_bridge.rs` | VM-to-host LLM gateway with `set_llm_client()` |
45+
| `TerraphimRlm::set_llm_client()` | `crates/terraphim_rlm/src/rlm.rs:877` | Injection point for `Arc<dyn LlmClient>` |
46+
| `LlmClient` trait | `terraphim_service::llm` (registry) | Canonical LLM abstraction (`chat_completion`, `summarize`, `list_models`) |
47+
| `OllamaClient` | `terraphim_service::llm` (private) | Full Ollama implementation with `/api/chat`, `/api/tags` |
48+
| `build_llm_from_role()` | `terraphim_service::llm` (public) | Factory that builds `Arc<dyn LlmClient>` from `Role` config |
49+
| `RouterBridgeLlmClient` | `terraphim_service::llm::bridge` | Capability-based routing across providers with `CostFirst`/`QualityFirst`/`Balanced`/`Static` |
50+
| `RouterStrategy::CostFirst` | `terraphim_config::llm_router` | Picks provider with lowest `CostLevel` (Cheap < Moderate < Expensive) |
51+
| `Role` | `terraphim_config::Role` | Holds `llm_enabled`, `extra` map with `llm_provider`, `llm_model`, `ollama_base_url`, etc. |
52+
53+
### Data Flow (current)
54+
```
55+
CLI → TerraphimRlm::new(config) → LlmBridge::new() (no client)
56+
↓
57+
rlm.query(prompt)
58+
↓
59+
LlmBridge::query() → LlmNotConfigured
60+
```
61+
62+
### Integration Points
63+
- `TerraphimRlm::set_llm_client(client: Arc<dyn LlmClient>)` -- mutable access required
64+
- `build_llm_from_role(role: &Role) -> Option<Arc<dyn LlmClient>>` -- needs `terraphim_config::Role`
65+
- Ollama `/api/tags` -- returns `{ "models": [{ "name": "...", "size": 12345, ... }] }`
66+
- OpenRouter API key -- `OPENROUTER_API_KEY` env var
67+
68+
### Local Ollama models (available)
69+
70+
| Model | Size | Chat-capable? |
71+
|-------|------|---------------|
72+
| `gemma3:270m` | 292 MB | Yes (fastest) |
73+
| `all-minilm:latest` | 46 MB | No (embedding) |
74+
| `deepseek-coder:1.3b` | 776 MB | Yes |
75+
| `gemma2:2b` | 1.6 GB | Yes |
76+
| `llama3.2:3b` | 2.0 GB | Yes |
77+
| `qwen3:4b` | 2.5 GB | Yes |
78+
| `qwen2.5-coder:latest` | 4.7 GB | Yes |
79+
| `qwen3:8b` | 5.2 GB | Yes |
80+
81+
## Constraints
82+
83+
### Vital Few (Essentialism)
84+
85+
| Constraint | Why It's Vital | Evidence |
86+
|------------|----------------|----------|
87+
| Must use `build_llm_from_role()` | Only public factory for `Arc<dyn LlmClient>`; `OllamaClient` is private | Source: `terraphim_service::llm.rs:78` |
88+
| Must add `terraphim_config` as dep | `Role` type required by factory | Not currently in RLM deps (dev-only) |
89+
| Must select cheapest model at runtime | Statically listing models is fragile; user may pull/unpull models | Ollama `/api/tags` returns sizes |
90+
91+
### Eliminated from Scope
92+
93+
| Eliminated Item | Why Eliminated |
94+
|-----------------|----------------|
95+
| Writing own `LlmClient` impl from scratch | `OllamaClient` already exists and is battle-tested |
96+
| Multi-provider routing in `RouterBridgeLlmClient` | Single Ollama provider is sufficient; routing adds complexity for no gain right now |
97+
| OpenRouter support in this PR | Needs API key; Ollama is local and free |
98+
| genai provider support | Adds `genai` crate dependency; not needed for cheapest path |
99+
| Model quality-based selection | User asked for cheapest/fastest, not best quality |
100+
101+
## Dependencies
102+
103+
### Internal Dependencies
104+
105+
| Dependency | Impact | Risk |
106+
|------------|--------|------|
107+
| `terraphim_config` (new dep) | Must add as optional dep behind `llm` feature | Low -- same feature gate as `terraphim_service` |
108+
| `terraphim_service` (existing) | Used via `build_llm_from_role()` | Low -- already a dep behind `llm` feature |
109+
110+
### External Dependencies
111+
112+
| Dependency | Version | Risk | Alternative |
113+
|------------|---------|------|-------------|
114+
| Ollama daemon | local | Must be running for auto-detect to work | Falls back to `LlmNotConfigured` |
115+
| `reqwest` | workspace | Used by `terraphim_service` internally; needed for `/api/tags` query to find cheapest model | Could skip cheapest detection if model is hardcoded |
116+
117+
## Risks and Unknowns
118+
119+
### Known Risks
120+
121+
| Risk | Likelihood | Impact | Mitigation |
122+
|------|------------|--------|------------|
123+
| `build_llm_from_role` fails silently if role config incomplete | Medium | Medium | Test with real config; validate before injecting |
124+
| Ollama `/api/tags` unreachable | Low | Low | Fallback to config-specified or default model |
125+
| Non-chat models selected (e.g. `all-minilm`) | Low | Medium | Filter to chat-capable models only (heuristic: min 100MB, exclude embedding models) |
126+
| `terraphim_config` transitively already present | Medium | Low | Cargo resolves correctly either way |
127+
128+
### Open Questions
129+
1. Should we filter to chat-capable models only, or let user handle via config? -- We should filter; non-chat models fail at inference time.
130+
2. Should env vars (`RLM_PROVIDER`, `RLM_MODEL`) override auto-detection? -- Yes, for explicit control.
131+
132+
### Assumptions Explicitly Stated
133+
134+
| Assumption | Basis | Risk if Wrong | Verified? |
135+
|------------|-------|---------------|-----------|
136+
| Ollama models with `gemma`/`qwen`/`llama` prefix are chat-capable | Model family naming convention | Non-chat model selected, inference error | No |
137+
| `all-minilm` can be excluded by name pattern | Known embedding model | Similar non-chat models missed | No |
138+
| `terraphim_config` is already a transitive dep through `terraphim_service` | Common Rust pattern | Need explicit dep declaration | Will verify |
139+
| `build_llm_from_role()` is available when `terraphim_service` feature `ollama` is enabled | Cargo feature resolution | Function not found at compile time | Must verify |
140+
141+
### Multiple Interpretations Considered
142+
143+
| Interpretation | Implications | Why Chosen/Rejected |
144+
|----------------|--------------|---------------------|
145+
| Hardcode `gemma3:270m` as model | Simple, no runtime detection | Rejected: fragile if model pulled/renamed |
146+
| Query `/api/tags` at RLM startup, pick smallest model | Dynamic, always uses best available | Chosen: smart, resilient |
147+
| Use `RouterBridgeLlmClient` with multiple models as "providers" | Multi-model routing | Rejected: adds complexity, models aren't separate providers |
148+
| Let user configure via `RLM_MODEL=gemma3:270m` env var | Explicit, predictable | Chosen as fallback alongside auto-detection |
149+
150+
## Research Findings
151+
152+
### Key Insights
153+
154+
1. **`build_llm_from_role()` is the gateway**: It reads `role.extra["llm_provider"]` and `role.extra["llm_model"]` to construct clients. Setting these to `"ollama"` and any valid model name produces a working `Arc<dyn LlmClient>`.
155+
156+
2. **`CostFirst` routing at provider level, not model level**: The existing `RouterBridgeLlmClient` routes between PROVIDERS (Ollama vs OpenRouter) using cost levels. Within a single provider, model selection is static. For intra-provider model selection, we need custom logic.
157+
158+
3. **Ollama `/api/tags` returns sizes**: The API response includes `size` (bytes) per model. We can query this, find the smallest chat-capable model, and use it.
159+
160+
4. **Cheapest chat models**: `gemma3:270m` (292MB), `deepseek-coder:1.3b` (776MB), `gemma2:2b` (1.6GB). `all-minilm` (46MB) should be excluded as embedding-only.
161+
162+
5. **RLM CLI already uses `RlmConfig::default()`**: The CLI binary has no provider configuration. We add auto-detection in `run()` after RLM construction.
163+
164+
### Technical Spikes Needed
165+
166+
| Spike | Purpose | Estimated Effort |
167+
|-------|---------|------------------|
168+
| Verify `terraphim_config` compiles as dep of `terraphim_rlm` | Confirm no version conflicts | 5 min |
169+
170+
## Recommendations
171+
172+
### Proceed/No-Proceed
173+
**Proceed**. Risk is low, reuse of existing infrastructure is high, user need is clear.
174+
175+
### Implementation Approach (3 tiers)
176+
177+
**Tier 1 -- Env var override (always respected)**:
178+
- `RLM_PROVIDER=ollama` + `RLM_MODEL=gemma3:270m` → explicit control
179+
- `OPENROUTER_API_KEY` present → auto-select OpenRouter with cheapest model
180+
181+
**Tier 2 -- Ollama auto-detection (if env vars absent)**:
182+
- `GET http://127.0.0.1:11434/api/tags` → parse models
183+
- Filter to chat-capable (exclude embedding/vision-only models)
184+
- Select model with smallest `size` field
185+
- If Ollama unreachable, fall through
186+
187+
**Tier 3 -- Hardcoded fallback**:
188+
- `gemma3:270m` as last resort default (guaranteed chat-capable)
189+
190+
### Files to modify
191+
192+
| File | Change |
193+
|------|--------|
194+
| `crates/terraphim_rlm/Cargo.toml` | Add `terraphim_config = { path = "../terraphim_config", optional = true }`; add to `llm` feature |
195+
| `crates/terraphim_rlm/src/rlm.rs` | Add `auto_configure_llm()` method with 3-tier detection |
196+
| `crates/terraphim_rlm/src/main.rs` | Call `auto_configure_llm()` after `TerraphimRlm::new()` |
197+
| `crates/terraphim_rlm/src/error.rs` | No changes (reuse existing `LlmNotConfigured`) |
198+
199+
## Appendix
200+
201+
### Ollama `/api/tags` response format
202+
```json
203+
{
204+
"models": [
205+
{ "name": "gemma3:270m", "model": "gemma3:270m", "size": 306651136, "digest": "...", "details": { "parent_model": "", "format": "gguf", "family": "gemma3", "parameter_size": "268.19M", "quantization_level": "Q8_0" } },
206+
{ "name": "gemma2:2b", "model": "gemma2:2b", "size": 1675000000, "digest": "...", "details": { "family": "gemma2", "parameter_size": "2.6B" } }
207+
]
208+
}
209+
```
210+
211+
### Key: non-chat models to exclude
212+
- `all-minilm` -- embedding model (Bert architecture)
213+
- `nomic-embed-text` -- embedding model
214+
- Any model with `embed` in name
215+
216+
### Key: `build_llm_from_role()` minimal invocation
217+
```rust
218+
let mut extra = AHashMap::new();
219+
extra.insert("llm_provider".to_string(), serde_json::Value::String("ollama".to_string()));
220+
extra.insert("llm_model".to_string(), serde_json::Value::String("gemma3:270m".to_string()));
221+
222+
let role = Role {
223+
name: "rlm-auto".to_string(),
224+
llm_enabled: true,
225+
extra,
226+
..Default::default() // NOTE: some fields may need explicit values
227+
};
228+
let client = terraphim_service::llm::build_llm_from_role(&role);
229+
```

0 commit comments

Comments
 (0)