Skip to content

Commit a1a95f4

Browse files
ci(065): D1 readiness waits for full tool catalog before scoring
The retrieval scorer was running against a partially-indexed instance: the readiness probe passed at the first indexed tool (>=1 search result), so scoring started before all 7 reference servers connected -> Recall@5 measured 0.387 vs baseline threshold 0.631 (false regression). Fix: poll /api/v1/tools until the catalog reaches the near-full count (~45 tools across the 7 servers) and add a short settle for the index build, then score. Verified locally end-to-end on a fully-indexed instance: Recall@1/3/5/10 = 0.418/0.560/0.681/0.791, Gate(recall_at_5) PASS (0.681 vs 0.631) — the baseline is exactly reproducible. actionlint clean. Related #555 datasets; MCP-742. Co-Authored-By: Paperclip <noreply@paperclip.ing>
1 parent 27b07cd commit a1a95f4

1 file changed

Lines changed: 13 additions & 9 deletions

File tree

.github/workflows/eval.yml

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -151,25 +151,29 @@ jobs:
151151
server_pid=$!
152152
trap 'kill "$server_pid" 2>/dev/null || true' EXIT
153153
154+
# Wait for the FULL tool catalog before scoring: the retrieval index is
155+
# built from the connected servers' tools, and scoring a partially
156+
# indexed instance tanks recall (a ≥1-result check fires far too early).
157+
# The 7 reference servers expose ~45 tools; require near-full + a short
158+
# settle for the index build. /api/v1/tools wraps as
159+
# {"success":true,"data":{"tools":[...]}}.
154160
ready=0
161+
expected=44
155162
for i in $(seq 1 60); do
156163
if ! kill -0 "$server_pid" 2>/dev/null; then
157164
echo "::error::mcpproxy process exited during startup"
158165
break
159166
fi
160-
if curl -fsS -H "X-API-Key: $key" "$base/api/v1/status" >/dev/null 2>&1; then
161-
# /api/v1/index/search wraps results as {"success":true,"data":{"results":[...]}}
162-
n="$(curl -fsS -H "X-API-Key: $key" "$base/api/v1/index/search?q=file&limit=5" \
163-
| python3 -c 'import sys,json;d=json.load(sys.stdin);print(len((d.get("data") or {}).get("results", [])))' 2>/dev/null || echo 0)"
164-
echo "attempt $i: index search returned $n result(s)"
165-
[ "$n" -ge 1 ] && { ready=1; echo "Index ready."; break; }
166-
else
167-
echo "attempt $i: server not up yet"
167+
t="$(curl -fsS -H "X-API-Key: $key" "$base/api/v1/tools" \
168+
| python3 -c 'import sys,json;d=json.load(sys.stdin);print(len((d.get("data") or {}).get("tools", [])))' 2>/dev/null || echo 0)"
169+
echo "attempt $i: catalog has $t tool(s)"
170+
if [ "$t" -ge "$expected" ]; then
171+
ready=1; echo "Catalog full ($t tools); settling 8s for index build."; sleep 8; break
168172
fi
169173
sleep 5
170174
done
171175
if [ "$ready" != 1 ]; then
172-
echo "::error::mcpproxy index did not become ready in time"
176+
echo "::error::mcpproxy catalog did not reach ${expected} tools in time"
173177
echo "----- mcpproxy.log (tail) -----"; tail -80 "$RUNNER_TEMP/mcpproxy.log" || true
174178
exit 1
175179
fi

0 commit comments

Comments
 (0)