Skip to content

Commit 14f3884

Browse files
committed
docs: document local model benchmark profiles
Document bounded local OpenAI-compatible provider behavior and keep benchmark profile comparisons separate between neutral, generic-static, and regression modes. Expose the review toggles and bounded local model setting in the public configuration docs. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Anders Heimer <anders.heimer@est.tech>
1 parent 37c3617 commit 14f3884

4 files changed

Lines changed: 27 additions & 0 deletions

File tree

docs/benchmarking.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,22 @@ The tool prints a summary to the console:
6060
Detailed results are written to `benchmark_results.json` in the current
6161
directory, including the AI judge's explanation for each finding.
6262

63+
## Local Model Benchmark Profiles
64+
65+
When benchmarking a bounded local OpenAI-compatible model, report these
66+
profiles separately:
67+
68+
| Profile | `enable_static_bug_seeds` | `enable_targeted_bug_pattern_prescan` | Purpose |
69+
|---------|---------------------------|---------------------------------------|---------|
70+
| Neutral | `false` | `false` | Model-quality runs without regression aids. |
71+
| Generic static | `true` | `false` | Measures opt-in deterministic diff-local bug-pattern detectors without the LLM prescan. |
72+
| Regression | `true` | `true` | Tracks regression behavior with all opt-in regression aids enabled. |
73+
74+
Static seeds are opt-in generic bug-pattern detectors. For example, the skb
75+
fragment seed looks for diff-local skb fragment append/growth sites that lack an
76+
apparent `MAX_SKB_FRAGS`-style capacity guard; it is not keyed to a specific
77+
benchmark commit. Keep seeded and neutral results separate in benchmark reports.
78+
6379
## Re-evaluating existing results
6480

6581
If you have already run ingestion and reviews but want to re-score with

docs/configuration.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,8 @@ Optional array of additional git remotes to track.
154154
| `email_policy_path` | string | `"email_policy.toml"` | Path to the email policy file. |
155155
| `max_total_tokens` | integer | `5000000` | Maximum cumulative uncached tokens (input + output) per review. Cached tokens are excluded. Set to 0 to disable. |
156156
| `max_total_output_tokens` | integer | `500000` | Maximum cumulative output tokens per review. Set to 0 to disable. |
157+
| `enable_static_bug_seeds` | bool | `false` | Enable deterministic proof-required bug-pattern seeds for local regression profiles. Not intended for neutral production review. |
158+
| `enable_targeted_bug_pattern_prescan` | bool | `false` | Enable the LLM-based diff prescan for selected high-signal kernel bug-pattern classes. The prescan is keyword-gated, capped, and should stay disabled for neutral production review while evaluating that mode. |
157159

158160
## email_policy.toml
159161

docs/examples/Settings.openai-compat.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,6 @@ model = "gemma-xyz"
77
base_url = "http://localhost:8080/v1"
88
context_window_size = 8192
99
max_tokens = 2048
10+
# Set true only for local/Ollama-style models that need bounded host-side
11+
# exploration, tool, prompt, and finalization controls.
12+
bounded_local_model = false

docs/llm-providers.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,3 +258,9 @@ cp docs/examples/Settings.openai-compat.toml Settings.toml
258258
```
259259

260260
Adjust `base_url` to point to your provider's endpoint.
261+
262+
For local/Ollama-style models that need host-side controls, set
263+
`[ai.openai_compat] bounded_local_model = true`. This enables bounded
264+
exploration, finalization-only schemas, Todo tool filtering, prompt preflight
265+
margins, and minimal fallback. It is not enabled merely by choosing
266+
`provider = "openai-compatible"`.

0 commit comments

Comments
 (0)