|
| 1 | +# Free-Tier Local Models for Test Generation |
| 2 | + |
| 3 | +Run AQE's test generation on a **free local model first**, with an automatic |
| 4 | +**repair loop**, and only pay for a cloud model when you choose to. Most routine |
| 5 | +test generation can be handled by a small local model at **$0** — you reserve |
| 6 | +paid models for the hard cases. |
| 7 | + |
| 8 | +> **Status:** opt-in, off by default. Enabling it changes nothing for anyone who |
| 9 | +> doesn't turn it on. Today the free tier runs **local-only with a repair loop** |
| 10 | +> (no automatic escalation to paid models yet). |
| 11 | +
|
| 12 | +This is AQE's adaptation of the "cheap-first, repair, escalate" economics |
| 13 | +demonstrated by [Darwin Mode](https://github.com/ruvnet/agent-harness-generator): |
| 14 | +a feedback/repair loop roughly doubled a fixed cheap model's bug-fix rate |
| 15 | +(7.7% → 15.3%) **without retraining anything** — the software around the model |
| 16 | +did the work. |
| 17 | + |
| 18 | +--- |
| 19 | + |
| 20 | +## What you get |
| 21 | + |
| 22 | +- **Cheap-first generation** — test generation tries your configured free/local |
| 23 | + model before any paid LLM call. |
| 24 | +- **Repair loop (D8)** — if the model's first output isn't a valid test (no |
| 25 | + assertion, etc.), AQE feeds the failure back and asks it to fix it, up to a |
| 26 | + configurable number of retries, *before* falling back. |
| 27 | +- **Safe fallback** — if the local model can't produce a valid test, AQE |
| 28 | + silently falls back to the normal generation path. You never get a worse |
| 29 | + result than today. |
| 30 | +- **Self-learning (D9)** — when wired to routing feedback, every cheap-vs-fallback |
| 31 | + outcome is recorded so routing confidence improves over time. |
| 32 | + |
| 33 | +--- |
| 34 | + |
| 35 | +## Quick start (local Ollama) |
| 36 | + |
| 37 | +1. **Install Ollama** and pull a small coding model: |
| 38 | + |
| 39 | + ```bash |
| 40 | + # on your machine (or the Docker host) |
| 41 | + ollama pull qwen3:8b |
| 42 | + ``` |
| 43 | + |
| 44 | + `qwen3:8b` (~5 GB) is the recommended default — it was the most productive |
| 45 | + local worker in our benchmarks and fits an 8 GB machine. `qwen3:30b-a3b` is |
| 46 | + faster if you have the RAM; `gemma3/4` work but were slower for this task. |
| 47 | + |
| 48 | +2. **Enable the opt-in** (either env var or config): |
| 49 | + |
| 50 | + ```bash |
| 51 | + export AQE_FREE_TIER=1 # turn it on |
| 52 | + export AQE_FREE_TIER_MODEL=qwen3:8b # optional; this is the default |
| 53 | + ``` |
| 54 | + |
| 55 | +3. **Generate tests as usual.** AQE now tries the local model first: |
| 56 | + |
| 57 | + ```bash |
| 58 | + claude "Use qe-test-architect to generate tests for src/services/Add.ts" |
| 59 | + ``` |
| 60 | + |
| 61 | + You'll see a log line: `Free-tier local test generation enabled (model=qwen3:8b, repair-only, no escalation)`. |
| 62 | + |
| 63 | +### Running in a container (Docker Desktop / dev container) |
| 64 | + |
| 65 | +If AQE runs in a container and Ollama runs on the host, point the free tier at |
| 66 | +the host gateway (this is the default for the `local-ollama` provider): |
| 67 | + |
| 68 | +``` |
| 69 | +http://host.docker.internal:11434/v1 |
| 70 | +``` |
| 71 | + |
| 72 | +--- |
| 73 | + |
| 74 | +## Choosing a provider |
| 75 | + |
| 76 | +The free tier speaks the OpenAI-compatible `/v1/chat/completions` API, so it |
| 77 | +works with several backends. Configure programmatically via `defaultFreeTierLadder()` |
| 78 | +and rebinding the bottom (`local`) tier: |
| 79 | + |
| 80 | +```ts |
| 81 | +import { defaultFreeTierLadder } from 'agentic-qe/routing/free-tier'; |
| 82 | + |
| 83 | +// 1) Local Ollama (default) — $0, private |
| 84 | +const ladder = defaultFreeTierLadder('qwen3:8b'); |
| 85 | + |
| 86 | +// 2) Cloud Ollama (ollama.com) — key from env, never stored |
| 87 | +ladder.bindings.local = { |
| 88 | + provider: 'free-tier', |
| 89 | + config: { kind: 'cloud-ollama', model: 'qwen3:8b', apiKeyEnv: 'OLLAMA_API_KEY' }, |
| 90 | +}; |
| 91 | + |
| 92 | +// 3) OpenRouter free models |
| 93 | +ladder.bindings.local = { |
| 94 | + provider: 'free-tier', |
| 95 | + config: { kind: 'openrouter', model: 'mistralai/devstral-small:free', apiKeyEnv: 'OPENROUTER_API_KEY' }, |
| 96 | +}; |
| 97 | + |
| 98 | +// 4) Any OpenAI-compatible endpoint (Groq, vLLM, LM Studio, llama.cpp, …) |
| 99 | +ladder.bindings.local = { |
| 100 | + provider: 'free-tier', |
| 101 | + config: { kind: 'openai-compatible', model: 'llama-3.3-70b', |
| 102 | + baseUrl: 'https://api.groq.com/openai/v1', apiKeyEnv: 'GROQ_API_KEY' }, |
| 103 | +}; |
| 104 | +``` |
| 105 | + |
| 106 | +| Provider kind | `baseUrl` default | API key | |
| 107 | +|---|---|---| |
| 108 | +| `local-ollama` | `http://host.docker.internal:11434/v1` | none | |
| 109 | +| `cloud-ollama` | `https://ollama.com/v1` | `OLLAMA_API_KEY` (or custom `apiKeyEnv`) | |
| 110 | +| `openrouter` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` | |
| 111 | +| `openai-compatible` | *(you must set `baseUrl`)* | optional `apiKeyEnv` | |
| 112 | + |
| 113 | +**Secrets:** API keys are read from the named environment variable at run time |
| 114 | +and **never stored** in any config file — the same policy as AQE's main LLM |
| 115 | +router. |
| 116 | + |
| 117 | +--- |
| 118 | + |
| 119 | +## Configuration reference |
| 120 | + |
| 121 | +When constructing the test-generation coordinator (or via project config): |
| 122 | + |
| 123 | +| Option | Env var | Default | Meaning | |
| 124 | +|---|---|---|---| |
| 125 | +| `enableFreeTier` | `AQE_FREE_TIER=1` | `false` | Turn the free tier on | |
| 126 | +| `freeTierModel` | `AQE_FREE_TIER_MODEL` | `qwen3:8b` | Local model id | |
| 127 | +| `freeTierRepairAttempts` | — | `1` | Same-tier repair retries before fallback | |
| 128 | + |
| 129 | +--- |
| 130 | + |
| 131 | +## How it works |
| 132 | + |
| 133 | +1. AQE reads the source file and asks the local model for tests. |
| 134 | +2. An **objective check** verifies the output is a real test (contains a |
| 135 | + `test`/`it`/`describe` block *and* an `expect`/`assert`). |
| 136 | +3. On failure, the **repair loop** feeds the rejection reason back and retries |
| 137 | + the same local model (up to `freeTierRepairAttempts`). |
| 138 | +4. If a valid test is produced, AQE returns it and **skips the paid path**. |
| 139 | +5. Otherwise AQE **falls back** to the normal generation path unchanged. |
| 140 | + |
| 141 | +For the design and measured results, see |
| 142 | +[Darwin-for-QE action lane](../metaharness/06-darwin-qe-self-learning-action-lane.md). |
| 143 | + |
| 144 | +--- |
| 145 | + |
| 146 | +## Troubleshooting |
| 147 | + |
| 148 | +- **No effect / paid path still used:** confirm `AQE_FREE_TIER=1` is set in the |
| 149 | + environment AQE runs in, and that the model responds: |
| 150 | + `curl http://localhost:11434/api/tags` (or `host.docker.internal` in a container). |
| 151 | +- **Empty output from reasoning models:** qwen3/gemma split a `reasoning` channel |
| 152 | + from `content`; AQE reads `content` and allows a generous token budget, so this |
| 153 | + is handled — but a very small model on a large file may still fail verification |
| 154 | + and fall back. Try a wider model (`qwen3:30b-a3b`) or a smaller source file. |
| 155 | +- **Slow first call:** the first request loads the model into memory; subsequent |
| 156 | + calls are much faster. |
0 commit comments