Skip to content

Commit 5eedb33

Browse files
DumbrisPaperclip-Paperclip
Dumbris
and
Paperclip-Paperclip
committed
ci(065): eval.yml regression gate — D2 security (blocking) + D1 retrieval (MCP-742)
Spec 065 / C1 (FR-009, US3/P2). Add `.github/workflows/eval.yml` running both Spec-065 evaluations as a regression gate over the frozen datasets: - security-d2 (blocking): provenance/license guard (FR-007/CN-005) → cmd/scan-eval ×3 → mcp-eval SecurityScorer. Thresholds --fpr-ceiling 0.10 --recall-floor 0.05 (the sensitive-data detector measures recall ≈0.10 on this corpus; scorer defaults of 0.80 would always fail). Sourced in one place pending the MCP-815 baseline `security.gate` block so gate and baseline never drift. - retrieval-d1: boots mcpproxy over snapshot-servers.config.json, waits for index readiness, runs the RetrievalScorer with baseline+tolerance. Report-only on PRs (npx/uvx fetch flake), blocking on the nightly schedule. Shared D2 logic in scripts/eval-ci-smoke.sh (CI == local). Reports upload as artifacts, never committed (CN-003, guarded). mcp-eval checked out at a pinned public ref. Verified locally: full D2 gate PASS (P=0.667 R=0.100 FPR=0.043), actionlint clean. Related #555 datasets; implements MCP-742 (Gate-2 plan rev 2 accepted). Co-Authored-By: Paperclip <noreply@paperclip.ing>
1 parent b749dbd commit 5eedb33

3 files changed

Lines changed: 323 additions & 0 deletions

File tree

‎.github/workflows/eval.yml‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
name: Eval (Spec 065 regression gate)
2+
3+
# Spec 065 / C1 (FR-009, US3/P2): regression gate over the frozen Spec-065
4+
# datasets. Two independent jobs so a network flake in D1 (retrieval) never
5+
# masks or blocks the deterministic D2 (security) gate.
6+
#
7+
# security-d2 — Go + Python only, no live upstreams. HARD gate (blocking).
8+
# retrieval-d1 — needs a live mcpproxy serving 7 reference servers; network
9+
# dependent. Report-only on PRs, blocking on the nightly
10+
# schedule (promote to PR-blocking after a green soak — see the
11+
# plan on MCP-742). Reports are uploaded as artifacts, never
12+
# committed (CN-003).
13+
#
14+
# mcp-eval (smart-mcp-proxy/mcp-eval) is a separate PUBLIC repo, checked out at a
15+
# pinned ref — no token needed.
16+
17+
on:
18+
pull_request:
19+
paths:
20+
- "cmd/scan-eval/**"
21+
- "internal/security/**"
22+
- "specs/065-evaluation-foundation/datasets/**"
23+
- "scripts/eval-ci-smoke.sh"
24+
- ".github/workflows/eval.yml"
25+
workflow_dispatch: {}
26+
schedule:
27+
# Nightly soak (02:30 UTC) — exercises D1 against live upstreams.
28+
- cron: "30 2 * * *"
29+
30+
permissions:
31+
contents: read
32+
33+
env:
34+
MCP_EVAL_REF: "76df3a47e1480bfde2433b4f19df19312c985963" # SecurityScorer (B3) merge — pin for reproducibility
35+
PYTHON_VERSION: "3.11.13"
36+
37+
jobs:
38+
security-d2:
39+
name: Security regression gate (D2)
40+
runs-on: ubuntu-latest
41+
steps:
42+
- name: Checkout mcpproxy-go
43+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
44+
45+
- name: Set up Go
46+
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
47+
with:
48+
go-version: "1.25"
49+
cache: true
50+
51+
- name: Checkout mcp-eval (public, pinned)
52+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
53+
with:
54+
repository: smart-mcp-proxy/mcp-eval
55+
ref: ${{ env.MCP_EVAL_REF }}
56+
path: mcp-eval
57+
58+
- name: Set up uv + Python
59+
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
60+
with:
61+
python-version: ${{ env.PYTHON_VERSION }}
62+
63+
- name: Sync mcp-eval environment
64+
working-directory: mcp-eval
65+
run: uv sync
66+
67+
- name: Run D2 security gate
68+
env:
69+
MCP_EVAL_DIR: ${{ github.workspace }}/mcp-eval
70+
OUT_DIR: reports/security
71+
run: bash scripts/eval-ci-smoke.sh
72+
73+
- name: Upload D2 reports
74+
if: always()
75+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
76+
with:
77+
name: eval-security-d2
78+
path: reports/security/
79+
retention-days: 14
80+
if-no-files-found: ignore
81+
82+
- name: Assert reports are not committed (CN-003)
83+
if: always()
84+
run: |
85+
tracked="$(git ls-files reports/ || true)"
86+
if [ -n "$tracked" ]; then
87+
echo "::error::Eval reports must never be committed (CN-003). Tracked under reports/:"
88+
echo "$tracked"
89+
exit 1
90+
fi
91+
echo "OK: no eval reports are tracked by git."
92+
93+
retrieval-d1:
94+
name: Retrieval regression gate (D1)
95+
runs-on: ubuntu-latest
96+
# Report-only on PRs (D1 depends on npx/uvx package fetches — a known flake
97+
# source); blocking on the nightly schedule. Promote to PR-blocking after a
98+
# green soak (plan on MCP-742).
99+
continue-on-error: ${{ github.event_name == 'pull_request' }}
100+
steps:
101+
- name: Checkout mcpproxy-go
102+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
103+
104+
- name: Set up Go
105+
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
106+
with:
107+
go-version: "1.25"
108+
cache: true
109+
110+
- name: Set up Node.js (npx-launched servers)
111+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
112+
with:
113+
node-version: "22"
114+
115+
- name: Checkout mcp-eval (public, pinned)
116+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
117+
with:
118+
repository: smart-mcp-proxy/mcp-eval
119+
ref: ${{ env.MCP_EVAL_REF }}
120+
path: mcp-eval
121+
122+
- name: Set up uv + Python (uvx-launched servers + mcp-eval)
123+
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
124+
with:
125+
python-version: ${{ env.PYTHON_VERSION }}
126+
127+
- name: Sync mcp-eval environment
128+
working-directory: mcp-eval
129+
run: uv sync
130+
131+
- name: Build mcpproxy
132+
run: go build -o mcpproxy ./cmd/mcpproxy
133+
134+
- name: Start mcpproxy (7 reference servers)
135+
run: |
136+
./mcpproxy serve \
137+
--config specs/065-evaluation-foundation/datasets/snapshot-servers.config.json \
138+
--data-dir "$RUNNER_TEMP/eval" \
139+
--listen 127.0.0.1:8092 \
140+
--log-level info > "$RUNNER_TEMP/mcpproxy.log" 2>&1 &
141+
echo "MCPPROXY_PID=$!" >> "$GITHUB_ENV"
142+
143+
- name: Wait for index readiness
144+
run: |
145+
base="http://127.0.0.1:8092"; key="eval-corpus-snapshot"
146+
for i in $(seq 1 60); do
147+
if curl -fsS -H "X-API-Key: $key" "$base/api/v1/status" >/dev/null 2>&1; then
148+
n="$(curl -fsS -H "X-API-Key: $key" "$base/api/v1/index/search?q=file&limit=5" \
149+
| python3 -c 'import sys,json;d=json.load(sys.stdin);print(len(d.get("tools",d.get("results",[]))))' 2>/dev/null || echo 0)"
150+
echo "attempt $i: index search returned $n result(s)"
151+
[ "$n" -ge 1 ] && { echo "Index ready."; exit 0; }
152+
else
153+
echo "attempt $i: server not up yet"
154+
fi
155+
sleep 5
156+
done
157+
echo "::error::mcpproxy index did not become ready in time"
158+
tail -50 "$RUNNER_TEMP/mcpproxy.log" || true
159+
exit 1
160+
161+
- name: Run D1 retrieval gate
162+
working-directory: mcp-eval
163+
env:
164+
DS: ${{ github.workspace }}/specs/065-evaluation-foundation/datasets
165+
run: |
166+
PYTHONPATH=src uv run python -m mcp_eval.cli retrieval \
167+
--corpus "$DS/corpus_v1.tools.json" \
168+
--golden "$DS/retrieval_golden_v1.json" \
169+
--baseline "$DS/baseline_v1.json" \
170+
--tolerance 0.05 \
171+
--runs 1 \
172+
--base-url http://127.0.0.1:8092 \
173+
--api-key eval-corpus-snapshot \
174+
--out-dir "${{ github.workspace }}/reports/retrieval"
175+
176+
- name: Stop mcpproxy
177+
if: always()
178+
run: kill "${MCPPROXY_PID}" 2>/dev/null || true
179+
180+
- name: Upload D1 reports
181+
if: always()
182+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
183+
with:
184+
name: eval-retrieval-d1
185+
path: reports/retrieval/
186+
retention-days: 14
187+
if-no-files-found: ignore

‎scripts/eval-ci-smoke.sh‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
#!/usr/bin/env bash
2+
#
3+
# eval-ci-smoke.sh — Spec 065 / D2 security regression gate (and local smoke).
4+
#
5+
# Runs the deterministic D2 half of the Spec-065 evaluation end to end:
6+
# 1. provenance/license guard over the security corpus (FR-007 / CN-005),
7+
# 2. scan-eval (this repo) N times -> per-run verdict JSON (FR-010 averaging),
8+
# 3. mcp-eval SecurityScorer gate (P/R/F1/FPR per detector, absolute thresholds).
9+
#
10+
# It is the single source of truth shared by `.github/workflows/eval.yml` (Job A)
11+
# and local pre-flight, so the gate logic is proven the same way in both places.
12+
#
13+
# Exit non-zero if the corpus fails the provenance guard, scan-eval fails, or the
14+
# SecurityScorer gate fails (FPR above ceiling / recall below floor).
15+
#
16+
# Config via env (all have CI-friendly defaults; paths may be relative to repo root):
17+
# DATASETS_DIR dataset directory (default: specs/065-evaluation-foundation/datasets)
18+
# MCP_EVAL_DIR checkout of smart-mcp-proxy/mcp-eval. If unset/missing, the
19+
# SecurityScorer step is SKIPPED (Go-only smoke) with a warning.
20+
# OUT_DIR report output dir (default: reports/security) — never committed (CN-003)
21+
# RUNS number of scan-eval runs to average (default: 3)
22+
# FPR_CEILING max allowed per-detector false-positive rate (default: 0.10) # matches MCP-815
23+
# RECALL_FLOOR min allowed detector recall (default: 0.05) # matches MCP-815
24+
#
25+
# Threshold provenance (critical): the SecurityScorer *defaults* are recall-floor
26+
# 0.80, but the production `sensitive-data` detector measures recall ~= 0.10 on
27+
# this corpus (most malicious entries are prompt-injection / tool-poisoning /
28+
# rug-pull — out of scope for a secret/path detector). The gate therefore uses
29+
# the MCP-815 thresholds below, not the defaults. Once MCP-815 lands a
30+
# `security.gate` block in baseline_v1.json, source these from there so the gate
31+
# and the baseline never drift.
32+
set -euo pipefail
33+
34+
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
35+
cd "$REPO_ROOT"
36+
37+
DATASETS_DIR="${DATASETS_DIR:-specs/065-evaluation-foundation/datasets}"
38+
OUT_DIR="${OUT_DIR:-reports/security}"
39+
RUNS="${RUNS:-3}"
40+
FPR_CEILING="${FPR_CEILING:-0.10}" # matches MCP-815
41+
RECALL_FLOOR="${RECALL_FLOOR:-0.05}" # matches MCP-815
42+
CORPUS="${DATASETS_DIR}/security_corpus_v1.json"
43+
44+
ALLOWED_LICENSES="MIT Apache-2.0 BSD-3-Clause CC0-1.0 self-authored"
45+
46+
log() { printf '\n\033[1;34m==>\033[0m %s\n' "$*"; }
47+
48+
[ -f "$CORPUS" ] || { echo "error: corpus not found: $CORPUS" >&2; exit 4; }
49+
50+
log "Provenance / license guard (FR-007 / CN-005) over $CORPUS"
51+
ALLOWED_LICENSES="$ALLOWED_LICENSES" python3 - "$CORPUS" <<'PY'
52+
import json, os, sys
53+
corpus = json.load(open(sys.argv[1]))
54+
entries = corpus["entries"] if isinstance(corpus, dict) else corpus
55+
allowed = set(os.environ["ALLOWED_LICENSES"].split())
56+
bad = []
57+
for e in entries:
58+
prov = e.get("provenance") or {}
59+
lic = prov.get("license")
60+
if not e.get("category"):
61+
bad.append(f'{e.get("id","?")}: missing category')
62+
if not prov.get("source"):
63+
bad.append(f'{e.get("id","?")}: missing provenance.source')
64+
if lic not in allowed:
65+
bad.append(f'{e.get("id","?")}: license {lic!r} not in allowlist {sorted(allowed)}')
66+
if bad:
67+
print("PROVENANCE GUARD FAILED:", file=sys.stderr)
68+
for b in bad:
69+
print(" -", b, file=sys.stderr)
70+
sys.exit(5)
71+
print(f"OK: {len(entries)} entries, all carry category + allowlisted provenance.license")
72+
PY
73+
74+
WORK="$(mktemp -d)"
75+
trap 'rm -rf "$WORK"' EXIT
76+
VERDICT_ARGS=()
77+
log "Running scan-eval x${RUNS} (deterministic detector; N for FR-010 averaging contract)"
78+
for i in $(seq 1 "$RUNS"); do
79+
vf="${WORK}/verdicts_${i}.json"
80+
go run ./cmd/scan-eval --corpus "$CORPUS" --out "$vf"
81+
VERDICT_ARGS+=(--verdicts "$vf")
82+
done
83+
echo "Produced ${RUNS} verdict file(s)."
84+
85+
if [ -z "${MCP_EVAL_DIR:-}" ] || [ ! -d "${MCP_EVAL_DIR:-/nonexistent}" ]; then
86+
echo "::warning::MCP_EVAL_DIR unset or missing — skipping the SecurityScorer gate (Go-only smoke). Set MCP_EVAL_DIR to a smart-mcp-proxy/mcp-eval checkout to run the full gate."
87+
exit 0
88+
fi
89+
90+
mkdir -p "$OUT_DIR"
91+
ABS_CORPUS="$(cd "$(dirname "$CORPUS")" && pwd)/$(basename "$CORPUS")"
92+
ABS_OUT="$(cd "$OUT_DIR" && pwd)"
93+
log "SecurityScorer gate: fpr-ceiling=${FPR_CEILING} recall-floor=${RECALL_FLOOR} (MCP-815 thresholds)"
94+
# mcp-eval is run as a module with PYTHONPATH=src (its console-script entry point
95+
# is not installed by `uv sync`); uv supplies the synced 3.11 interpreter.
96+
( cd "$MCP_EVAL_DIR" && PYTHONPATH=src uv run python -m mcp_eval.cli security \
97+
"${VERDICT_ARGS[@]}" \
98+
--corpus "$ABS_CORPUS" \
99+
--fpr-ceiling "$FPR_CEILING" \
100+
--recall-floor "$RECALL_FLOOR" \
101+
--out-dir "$ABS_OUT" )
102+
103+
log "D2 security gate PASSED — reports in $OUT_DIR"

‎specs/065-evaluation-foundation/datasets/README.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,3 +107,36 @@ text from them is vendored into this repo:
107107
- **`mcp-injection-experiments`** — LICENSE unconfirmed (research.md R-A); where it
108108
inspired a pattern, the corresponding entry was rewritten from scratch and
109109
labeled `self-authored`. The corpus test rejects any entry sourced from these.
110+
111+
## CI regression gate (Spec 065 / C1)
112+
113+
`.github/workflows/eval.yml` runs both evaluations as a regression gate
114+
(FR-009). Two independent jobs keep a network flake in D1 from masking the
115+
deterministic D2 gate:
116+
117+
- **`security-d2` (D2, blocking)** — Go + Python only, no live upstreams.
118+
Provenance/license guard over `security_corpus_v1.json` (FR-007 / CN-005),
119+
then `cmd/scan-eval` ×3 → the mcp-eval `SecurityScorer`. Gate thresholds are
120+
**`--fpr-ceiling 0.10 --recall-floor 0.05`** (not the scorer defaults: the
121+
`sensitive-data` detector measures recall ≈ 0.10 here because most malicious
122+
entries are prompt-injection / tool-poisoning / rug-pull, out of scope for a
123+
secret/path detector). These thresholds will move to a `security.gate` block
124+
in `baseline_v1.json` once MCP-815 lands, so the gate and baseline never drift.
125+
- **`retrieval-d1` (D1)** — boots `mcpproxy serve` over
126+
`snapshot-servers.config.json` (7 reference servers), waits for index
127+
readiness, then runs the mcp-eval `RetrievalScorer` with
128+
`--baseline baseline_v1.json --tolerance 0.05`. **Report-only on PRs**
129+
(npx/uvx fetches are a known flake source), **blocking on the nightly
130+
schedule**; promote to PR-blocking after a green soak.
131+
132+
Both jobs upload HTML/JSON reports as run artifacts and the build **never
133+
commits** them (CN-003). The shared D2 logic lives in `scripts/eval-ci-smoke.sh`
134+
so the gate runs identically in CI and locally:
135+
136+
```bash
137+
# Local D2 smoke (full gate needs a mcp-eval checkout):
138+
MCP_EVAL_DIR=/path/to/mcp-eval bash scripts/eval-ci-smoke.sh
139+
```
140+
141+
mcp-eval (`smart-mcp-proxy/mcp-eval`, public) is checked out at a pinned ref for
142+
reproducibility.

0 commit comments

Comments
 (0)