[v0.5.0] benchmark adapters — competitor MCP tool adapters (Context7, GitMCP, DeepWiki, Ref.tools) (#103)

* agent: guard + eligibility — two-latch competitor guard, manifest-load-time exclusion

Adds require_live_competitor() (issue #87 PLAN-87 section 2.2) alongside
the existing require_live_environment(): two non-secret latches
(BENCHMARK_LIVE_PROVIDERS_ENABLED + a per-target BENCHMARK_LIVE_COMPETITORS
allowlist) for keyless competitors, mirroring the two-condition shape of
keyed providers. Registers "ref"/"context7" in PROVIDER_API_KEY_ENV for the
two keyed competitor adapters. Additive only; the guard's unknown-provider
test (test_adapters.py) is unaffected.

Adds benchmarks/adapters/eligibility.py: per-entry and manifest-level
eligibility screening for the four competitor docs-MCP adapters. Exclusion
is a manifest-load-time refusal (BenchmarkValidationError), never a
per-cell BenchmarkCellFailure, so an ineligible/excluded competitor never
executes a scored cell and never contaminates the correctness/error-rate
denominators (PLAN-87 section 2.4, Codex round-1 finding 1).

* agent: adapters — Context7, GitMCP, DeepWiki, Ref.tools competitor adapters

Four plain-class adapters (issue #87 PLAN-87 section 2.1), each mirroring
python_docs_mcp_adapter.py's proven shape: lazy mcp imports, an
adapter-local _transport_factory seam all transport construction routes
through, asyncio.run(asyncio.wait_for(...)) with the established
BenchmarkCellFailure/timeout/mcp_protocol_crash exception mapping, and the
applicable live-competitor guard as the first statement of run().

- context7_adapter.py: stdio (npx -y @upstash/context7-mcp@3.2.3, default)
  + streamable-HTTP fallback; resolve-library-id -> query-docs flow.
  Keyless mode goes through require_live_competitor("context7"); key mode
  goes through require_live_environment("context7").
- gitmcp_adapter.py: streamable HTTP to https://gitmcp.io/python/cpython;
  search_cpython_documentation -> (search_cpython_code ->
  fetch_generic_url_content) fallback, since doc-search is verified to
  return no stdlib hits (a reportable finding, not an exclusion).
- deepwiki_adapter.py: streamable HTTP to https://mcp.deepwiki.com/mcp;
  single ask_question call, 60s timeout (LLM-generated answers are slower
  and nondeterministic vs. plain retrieval).
- ref_tools_adapter.py: streamable HTTP to https://api.ref.tools/mcp with
  an x-ref-api-key header; ref_search_documentation -> ref_read_url flow.
  Always keyed (require_live_environment("ref")) -- no keyless mode exists.

None of these adapters make a network call in this change: guard refusal
happens before any transport is ever constructed, and no test exercises a
real transport.

* agent: runner — dispatch registry + manifest-load eligibility gate (#87)

Adds one lazily-importing dispatch function + one _ADAPTER_DISPATCH entry
per competitor adapter (the documented extension seam, runner.py:434-444),
plus one _validate_manifest_eligibility() call in run_benchmark's
manifest-loading sequence (the runner has no function literally named
_load_manifest; the call is placed where the manifest is parsed, before
_load_competitors builds cells and before any cell executes).

benchmarks/adapters/__init__.py: additive exports for the four new adapter
classes/results and the eligibility helpers.

* agent: docs — competitor manifest template with pins + terms_check (#87)

docs/benchmarks/competitor-manifest.template.yml: the four competitors
from the methodology's candidate matrix, each with researched pin +
terms_check metadata (clauses quoted from PLAN-87-competitor-adapters.md
section 1) and eligibility.status: conditional (every entry carries at
least one unmet condition -- permission, credits, or a disclosure
requirement). TEMPLATE header per PLAN-87 section 2.5: not a benchmark
result; live-run manifests are maintainer-authored copies. Not placed
under docs/benchmarks/results/ (that path is claim-bearing).

* agent: tests — cover competitor adapters, guard, eligibility (#87)

tests/benchmarks/test_competitor_adapters.py (additive, pinned test file):
scripted-session flow tests per adapter (including GitMCP's
doc-search-empty fallback chain and Ref.tools' search->read URL chaining);
failure-category mapping (tool_failure/timeout/mcp_protocol_crash);
fail-closed guard tests that patch each adapter's _transport_factory seam
plus asyncio.run and socket.socket to prove no transport is ever built on
refusal, across every disabled-env permutation; require_live_competitor
latch semantics in isolation; eligibility/exclusion validation including
the runner's manifest-load-time refusal (BenchmarkValidationError, CLI
exit 2, zero artifacts written), the exclusions: block's byte-for-byte
snapshot survival, and the committed template manifest; and dispatch
registry integration for all four adapter ids. Zero network calls
anywhere in this file.

* agent: fix — wire plan §2.4 defense-in-depth eligibility call into dispatch fns (#87)

Adversarial-verification blocking finding: plan §2.4 requires each of the
four competitor dispatch functions in benchmarks/runner.py to also call
validate_competitor_eligibility (defense in depth; unreachable-for-excluded
by construction) -- none of them did, and the gap was not disclosed in the
PR body's "Why this approach" section. Adds the one-line call, reading
cell.competitor.raw, as the first statement after each dispatch fn's lazy
imports, in _context7_answer/_gitmcp_answer/_deepwiki_answer/_ref_tools_answer.

Also applies three cheap minor findings from the same review pass:
- _patch_no_transport (tests/benchmarks/test_competitor_adapters.py) now
  also patches subprocess.Popen, restoring literal parity with plan §4
  item 3's "subprocess/stdio + socket.socket" patch list for Context7's
  stdio path (functionally already covered via the _transport_factory and
  asyncio.run chokepoints; this closes the checklist gap at zero cost).
- guard.py's module docstring now names Context7's default keyless mode
  alongside GitMCP/DeepWiki as covered by require_live_competitor,
  matching what the adapters/template/PR body already say correctly.
- context7_adapter.py/ref_tools_adapter.py's _transport_factory comments
  no longer imply timeout_seconds is wired into the httpx.AsyncClient
  timeout; it is enforced only by the outer asyncio.wait_for.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* agent: tests — fix CodeQL unnecessary-lambda + CodeRabbit review findings on PR #103

- Replace four trivial zero-arg lambdas (`lambda: X()`) with direct class
  references in _KEYLESS_ADAPTERS/_KEYED_ADAPTERS parametrize tables
  (py/unnecessary-lambda alerts #74-#77). Left the context7-keyed lambda
  (`lambda: Context7Adapter(key_mode=True)`) untouched -- it bakes in a
  keyword argument not in its own signature, so it is a genuine adapter,
  not a pass-through wrapper.
- Escape literal dots in three pytest.raises(match=...) patterns
  ("pin.kind", "terms_check.verdict", "eligibility.status") so the regex
  matches a literal dot instead of "any character" (CodeRabbit 3553908100).
- Set cwd=_REPO_ROOT on the `python -m benchmarks` CLI subprocess test so
  it passes when pytest is invoked from outside the repo root, matching
  the cwd=REPO_ROOT idiom in tests/benchmarks/test_report.py's CLI test
  (CodeRabbit 3553908105).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: ayhammouda

benchmarks/adapters/__init__.py

@@ -11,6 +11,14 @@
 its ``LiveClaudeTokenCounter`` does perform a real, guarded HTTP call to the
 Anthropic count-tokens API, confined to the maintainer-run live phase (see
 that module's docstring).
+
+``context7_adapter.py`` / ``gitmcp_adapter.py`` / ``deepwiki_adapter.py`` /
+``ref_tools_adapter.py`` (issue #87) are the competitor docs-MCP adapters:
+like ``python_docs_mcp_adapter.py`` (issue #86), these model a retrieval
+tool call rather than an LLM provider call, so none of them subclass
+``ProviderAdapter``. ``eligibility.py`` (issue #87) is the manifest-load-time
+eligibility screener that keeps ineligible/excluded competitors out of
+scored cells entirely (see that module's docstring).
 """
 
 from __future__ import annotations
@@ -29,13 +37,23 @@
     build_client_wrapped_envelope,
     count_cell_tokens,
 )
+from benchmarks.adapters.context7_adapter import Context7Adapter, Context7Result
+from benchmarks.adapters.deepwiki_adapter import DeepWikiAdapter, DeepWikiResult
+from benchmarks.adapters.eligibility import (
+    COMPETITOR_ADAPTER_IDS,
+    validate_competitor_eligibility,
+    validate_manifest_eligibility,
+)
+from benchmarks.adapters.gitmcp_adapter import GitMcpAdapter, GitMcpResult
 from benchmarks.adapters.google_adapter import LiveGoogleAdapter, MockGoogleAdapter
 from benchmarks.adapters.guard import (
     LiveExecutionNotImplementedError,
     LiveProviderDisabledError,
+    require_live_competitor,
     require_live_environment,
 )
 from benchmarks.adapters.openai_adapter import LiveOpenAIAdapter, MockOpenAIAdapter
+from benchmarks.adapters.ref_tools_adapter import RefToolsAdapter, RefToolsResult
 from benchmarks.runner import BenchmarkCellFailure
 
 __all__ = [
@@ -47,6 +65,7 @@
     "LiveExecutionNotImplementedError",
     "LiveProviderDisabledError",
     "require_live_environment",
+    "require_live_competitor",
     "MockOpenAIAdapter",
     "LiveOpenAIAdapter",
     "MockGoogleAdapter",
@@ -57,4 +76,15 @@
     "TokenCountResult",
     "build_client_wrapped_envelope",
     "count_cell_tokens",
+    "Context7Adapter",
+    "Context7Result",
+    "GitMcpAdapter",
+    "GitMcpResult",
+    "DeepWikiAdapter",
+    "DeepWikiResult",
+    "RefToolsAdapter",
+    "RefToolsResult",
+    "COMPETITOR_ADAPTER_IDS",
+    "validate_competitor_eligibility",
+    "validate_manifest_eligibility",
 ]