Skip to content

Commit 79948a4

Browse files
authored
Render the gate terminal and PR comment as native docs, not screenshots (#140)
Two docs-accuracy/polish fixes: 1. **Native renders on the regression-gate page.** It was the one page with heavy embedded images next to otherwise-native code blocks and tables. Both were text anyway, so they now render natively: the mvn terminal session is a titled code block (matching every other snippet on the page), and the eval-gate PR comment is a Docusaurus note admonition with a native metric/value table. Sharp at any zoom, theme-aware by construction. Removes three now-unused PNGs. 2. **Fix a stale claim in the overview.** The 'For AI agents' section claimed every page has an 'Open in ChatGPT / Claude' footer action; that was dropped in favor of a 'View as Markdown' link. Corrected the text to match the actual footer.
1 parent bed9261 commit 79948a4

5 files changed

Lines changed: 32 additions & 12 deletions

File tree

docs/docs/evaluation/regression-gate.md

Lines changed: 31 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,24 @@ title: Regression gate (server-free)
77

88
import Tabs from '@theme/Tabs';
99
import TabItem from '@theme/TabItem';
10-
import ThemedImage from '@theme/ThemedImage';
1110

1211
Run your evals as a test and fail the build when quality drops. You commit a baseline next to your test, and on every run the gate compares the fresh result against it and throws on a real regression. No server, no account, no API key for the gate itself. The failing test is the gate, and it fires the same way locally and in CI.
1312

1413
This is eval-driven development: a quality change shows up as a red build on the PR that caused it, the same place a broken unit test does.
1514

16-
![The eval gate as a JUnit test: a clean run passes, a quality drop fails with the regressed cases, and re-running with the update flag re-baselines](/img/regression-gate.png)
15+
```text title="The gate is a JUnit test: green until quality drops"
16+
$ mvn test -Dtest=RegressionGateExampleTest
17+
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
18+
[INFO] BUILD SUCCESS
19+
20+
# a change drops answer quality...
21+
$ mvn test -Dtest=RegressionGateExampleTest
22+
[ERROR] Eval gate FAILED: pass rate 100% -> 75% (-25 pts).
23+
Regressed cases: item-2 Exact Match 1.0 -> 0.0
24+
To accept: re-run with DOKIMOS_UPDATE_BASELINE=true and commit the baseline.
25+
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
26+
[INFO] BUILD FAILURE
27+
```
1728

1829
## Quickstart
1930

@@ -273,15 +284,24 @@ eval-gate:
273284

274285
The `if: always()` on the report step is the load-bearing part. The gate writes a per-baseline verdict JSON under `target/dokimos` *before* it throws, so the report step posts the sticky PR comment after a failing build. Without `always()`, the one run you most want explained would post nothing. The action renders every verdict file in the directory, so one job can gate several baselines. The comment shows the pass-rate move and the regressed cases, and updates in place on each push instead of stacking up.
275286

276-
<div style={{maxWidth: '560px'}}>
277-
<ThemedImage
278-
alt="The eval gate's comment on a pull request: a failing run posts the pass-rate move, the significance flag, and the regressed cases"
279-
sources={{
280-
light: '/img/eval-gate-pr-comment-light.png',
281-
dark: '/img/eval-gate-pr-comment-dark.png',
282-
}}
283-
/>
284-
</div>
287+
:::note The comment the gate posts on your PR
288+
289+
**❌ Eval gate failed**
290+
291+
| metric | value |
292+
| --- | --- |
293+
| pass rate | 100% → 75% (-25 pts) |
294+
| significant | false |
295+
| regressed cases | 1 |
296+
| improved cases | 0 |
297+
| added / removed | 0 / 0 |
298+
| pairing | positional |
299+
300+
**Regressed cases**
301+
302+
- row item-2: Exact Match 1.0→0.0
303+
304+
:::
285305

286306
**Not on GitHub?** A failing `mvn test` is the gate on every runner: GitLab, Jenkins, Gradle, local. The verdict JSON lands under `target/dokimos`, one file per baseline (named for the baseline stem), if you want to render it yourself.
287307

docs/docs/overview.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ See the [server overview](./server/overview) for how the pieces fit together.
171171

172172
## For AI agents
173173

174-
Point a coding agent at the machine-readable docs. [llms.txt](https://dokimos.dev/llms.txt) indexes the documentation, and [llms-full.txt](https://dokimos.dev/llms-full.txt) is the whole thing in one file. Every page also has a markdown version and an "Open in ChatGPT / Claude" action in its footer.
174+
Point a coding agent at the machine-readable docs. [llms.txt](https://dokimos.dev/llms.txt) indexes the documentation, and [llms-full.txt](https://dokimos.dev/llms-full.txt) is the whole thing in one file. Every page also has a Markdown version, linked from its footer under "For AI agents".
175175

176176
## What's next
177177

-65.2 KB
Binary file not shown.
-65.4 KB
Binary file not shown.
-516 KB
Binary file not shown.

0 commit comments

Comments
 (0)