ci: gate page-level text retrieval against a committed baseline

The regression step compared baseline.json against itself, so it always
passed and gated nothing. Replace it with a real gate a stock CPU runner
can execute.

scripts/eval_retrieval_ci.py runs the hybrid text retriever (bge-m3 on CPU,
BM25, RRF) over the committed rag_corpus snapshot and scores nDCG@5 /
recall@10 / MRR, gated against data/eval/baseline_retrieval.json at a 5%
threshold.

Metrics are scored at page granularity: rag_corpus is the demo corpus and is
periodically re-baked by the docling chunker (ADR 0017/0021), which renumbers
the ::cN chunk suffix, so the v3 golden's chunk-level labels drift out of sync
with it while the page each points at does not. Projecting both sides to
paper::pN coarsens the existing human labels rather than authoring new ones,
the same re-chunk-robustness reasoning as ADR 0019's answer_correctness.

The visual (GPU) and generation/judge (API) legs stay in the manual or
scheduled full eval against baseline.json. Docs updated to match (evals.md,
CONTRIBUTING, results.md).

Author: NorthernLightx

.github/workflows/ci.yml

@@ -44,13 +44,23 @@ jobs:
       - name: Pytest (unit)
         run: uv run pytest -v --cov=src --cov-report=term-missing
 
-      - name: Eval regression gate (vs committed baseline)
-        # Compares the committed baseline against itself as a smoke test until
-        # a self-hosted runner can execute the live eval stack. When a fresh
-        # run is fetched from artifacts, point --candidate at it. Threshold:
-        # 5% (any metric dropping more than this fails CI).
+      - name: Cache bge-m3 weights
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/huggingface
+          key: hf-bge-m3
+
+      - name: Eval regression gate (CPU page-level text retrieval)
+        # Runs the hybrid text retriever (bge-m3 on CPU + BM25 + RRF) over the
+        # committed rag_corpus snapshot and gates page-level retrieval metrics
+        # against data/eval/baseline_retrieval.json at a 5% threshold. The visual
+        # (GPU) and generation/judge (API) legs run in the manual/scheduled full
+        # eval, not here — see docs/evals.md "Regression gate".
         run: |
+          uv run python -m scripts.eval_retrieval_ci \
+            --output data/eval/runs/retrieval-ci.json
           uv run python -m scripts.check_regression \
-            --baseline data/eval/baseline.json \
-            --candidate data/eval/baseline.json \
+            --baseline data/eval/baseline_retrieval.json \
+            --candidate data/eval/runs/retrieval-ci.json \
+            --metrics ndcg_at_5 recall_at_10 mrr \
             --threshold 0.05

CONTRIBUTING.md

@@ -26,10 +26,11 @@ uv run ruff check .
 uv run ruff format --check .
 uv run mypy src tests scripts
 uv run pytest -v --cov=src --cov-report=term-missing
+uv run python -m scripts.eval_retrieval_ci --output data/eval/runs/retrieval-ci.json
 uv run python -m scripts.check_regression \
-    --baseline data/eval/baseline.json \
-    --candidate data/eval/baseline.json \
-    --threshold 0.05
+    --baseline data/eval/baseline_retrieval.json \
+    --candidate data/eval/runs/retrieval-ci.json \
+    --metrics ndcg_at_5 recall_at_10 mrr --threshold 0.05
 ```
 
 ## Scripts layout