Skip to content

research.md

Latest commit

 

History

History
33 lines (26 loc) · 5.39 KB

research.md

File metadata and controls

33 lines (26 loc) · 5.39 KB
Raw

Phase 0 Research: Evaluation Foundation (D1+D2)

Grounded in the roadmap research sweep (2026-05-31) + direct recon of internal/index/manager.go, internal/config/types.go, internal/security/detector.go, internal/httpapi/server.go, and ~/repos/mcp-eval. Decisions below resolve every Technical-Context choice.

Decision summary

# Decision Rationale Alternatives rejected
R-01 Retrieval scorer drives the REST GET /api/v1/index/search?q=&limit= endpoint (confirmed at internal/httpapi/server.go:475); returns []SearchResult{Server,Tool,Score,…} Deterministic, no LLM, language-agnostic (Python harness → Go index over HTTP), exercises the real index incl. tokenizer; cheap enough to gate every PR (Principle I) Direct Go package call (couples harness to Go build); MCP retrieve_tools round-trip (adds LLM/agent variance — keep as an optional realism variant, not the gate)
R-02 tool_id = "<server>:<tool>" as the join key between golden set and search results Matches mcpproxy's canonical tool format and SearchResult.Server+.Tool server-only (too coarse); tool-name-only (collides across servers)
R-03 Security scorer bridges via a tiny cmd/scan-eval Go helper in mcpproxy-go that reads the corpus JSON, runs each entry through Detector.Scan + the bundled scanner registry, and emits per-entry verdicts as JSON; the Python SecurityScorer computes P/R/F1/FPR from that Smallest deterministic bridge; no new runtime REST endpoint/attack surface; reuses the exact production detectors; verdict JSON is contamination-safe to diff A REST scan endpoint (new surface, against Security-by-Default); reimplementing detectors in Python (drift from production — invalidates the measurement)
R-04 Golden-set queries: hybrid generation — (a) synthetic per-tool intents via an LLM (ScaleMCP/ToolRet recipe), 3–5 per tool, labeled to that tool; (b) human-verified hard negatives = near-duplicate tools across servers Synthetic gives breadth cheaply; hard negatives test discrimination (the failure mode that matters for profiles/shadowing) Pure manual (doesn't scale); pure synthetic (misses discrimination cases)
R-05 Graded relevance (e.g. 2=primary, 1=acceptable, 0=irrelevant) so nDCG and multi-relevant queries work Real queries often have >1 acceptable tool; binary labels can't score ranking quality Binary-only (can't compute nDCG meaningfully / penalizes valid alternates)
R-06 Frozen corpus = a committed snapshot generated by a documented datasets.py snapshot step pulling current tool descriptions (mcp-eval mcp_servers.json + optional registry pull) into corpus_v1.json CN-002 contamination control; deliberate, versioned refresh (FR-012) Scoring live index (drifts; non-reproducible baselines)
R-07 Vendor only license-clear malicious samples: DVMCP (MIT) + invariant mcp-injection-experiments (confirm LICENSE) + self-authored samples per category; MCPTox/MCP-AttackBench referenced as external benchmarks, not vendored (restrictive license) CN-005/FR-007; keeps the repo redistributable Vendoring MCP-AttackBench (research-only license — can't redistribute)
R-08 Benign corpus = real tool descriptions from the frozen corpus (same source as D1) + deliberately attack-resembling hard negatives Real base rates are tiny → FPR is the dominant metric; hard negatives expose noisy detectors (SC-004) Synthetic-only benign (too easy; hides false positives)
R-09 Metrics: D1 = Recall@{1,3,5,10}, MRR, nDCG@10 (+MAP for multi-relevant); D2 = precision, recall, F1, FPR per detector Industry-standard IR + detection metrics (ToolRet/MCP-Guard); FPR is the quiet-security gate Accuracy-only (misleading on imbalanced corpora)
R-10 Gates: D1 Recall@5 ≥ baseline−tolerance; D2 FPR ≤ ceiling AND recall ≥ floor; both averaged over N runs for any non-deterministic (LLM-judge) detector FR-006/009/010; mean±tolerance avoids flaky gates (blog noted ~15% swings) Single-run gate (flaky); pass-only without FPR ceiling (lets noise through)

Resolved unknowns

  • Driving path (was: direct vs MCP): REST /api/v1/index/search for the gate (R-01); MCP round-trip optional.
  • Python↔Go security bridge (the one real design tension): cmd/scan-eval JSON in/out (R-03) — the only mcpproxy-go code this spec adds, and it's test-tooling, not runtime.
  • Dataset schema: pinned in contracts/ (Phase 1).
  • Seed-corpus licensing: R-07 — DVMCP MIT + self-authored vendored; restricted corpora external-only.
  • Query generation: R-04 hybrid.

Open risks

  • R-A: invariant mcp-injection-experiments has no explicit LICENSE file — confirm before vendoring; if unclear, cite as external + self-author equivalents.
  • R-B: LLM-judge scanners in the registry introduce variance → those detectors scored over N runs; rule-based detectors (sensitive-data, pattern TPA) are deterministic and gate tightly.
  • R-C: synthetic queries can leak the tool name verbatim (trivial retrieval) — generation prompt must paraphrase intent, not name the tool; validated by a held-out manual spot check.
  • R-D: /api/v1/index/search requires the API key — the harness already manages a running mcpproxy + key (its Docker-isolation flow); confirm the search endpoint is reachable with the eval API key.