Summary

New contrib evaluator that scans selector-selected payloads for potential secrets using
Yelp detect-secrets, wired through the
detect-secrets-async subprocess-pool runtime.

Registered under the entry point yelp.detect_secrets.

Why this matters for agent workflows

Agent payloads move secrets around constantly without anyone meaning to. Concrete leak surfaces
in selector-selected step payloads:

• LLM tool outputs that echo upstream API responses (auth headers, bearer tokens, session cookies).
• Config dumps and environment snapshots fetched by a code tool.
• Log lines emitted by user-facing tools and fed back into the LLM context.
• Retrieved documents that happened to contain credentials.

Agents chain tool calls, so a leaked token in one step becomes input to the next — that's the
blast radius this evaluator is here to cap. Agent Control's evaluator layer is the natural gate:
it runs after the selector narrows the payload to the relevant field and before the control's
policy decision is committed.

Yelp detect-secrets contributes a battle-tested detector set — AWS keys, GitHub tokens, Basic
auth, private keys, high-entropy blobs, and keyword patterns — with per-request plugin narrowing
when you want fewer false positives on a specific control.

Why a separate async runtime

detect-secrets is synchronous and configures itself via process-global settings, which makes
asyncio.to_thread (theatrical timeouts — the scan keeps running) and a serialising lock (kills
throughput) both poor fits for Agent Control. The external
detect-secrets-async package wraps it in a
bounded pool of long-lived subprocess workers with real timeouts, per-request plugin isolation,
and automatic worker replacement on timeout/crash/cancellation. This PR is the thin Agent Control
adapter on top of that runtime.

What the evaluator does

1. Normalize the selector payload:
   • None → no match
   • str → scanned directly
   • dict / list → deterministic pretty JSON with RFC 6901 pointer mapping
   • int / float / bool → JSON scalar text
2. Filter lines matching exclude_lines_regex (blanked, so line numbers stay stable).
3. Enforce the max_bytes cap on post-filter UTF-8 bytes.
4. Scan via detect-secrets-async using the shared host-level runtime.
5. Map findings into EvaluatorResult:
   • str payloads get line_number.
   • Structured payloads get json_pointer, conservatively truncated at any secret-looking
     segment in the path so a token appearing as a key name never leaks through the pointer.
   • Plaintext, snippets, full matching lines, and upstream hashed_secret are never surfaced.

Example

A control fragment that scans the output field for GitHub and AWS credentials, failing open on
evaluator errors:

{
  "selector": { "path": "output" },
  "evaluator": {
    "name": "yelp.detect_secrets",
    "config": {
      "timeout_ms": 10000,
      "on_error": "allow",
      "enabled_plugins": ["GitHubTokenDetector", "AWSKeyDetector"]
    }
  }
}

For a plain-string payload "github_token = 'ghp_abc...'":

result.matched = True
result.confidence = 1.0
result.metadata = {
    "findings_count": 1,
    "findings": [{"type": "GitHub Token", "line_number": 1}],
    "normalized_payload_type": "str",
    "detect_secrets_version": "1.5.0",
}

For a structured payload {"response": {"headers": {"authorization": "ghp_abc..."}}}:

result.matched = True
result.metadata = {
    "findings_count": 1,
    "findings": [
        {"type": "GitHub Token", "json_pointer": "/response/headers/authorization"}
    ],
    "normalized_payload_type": "dict",
    "detect_secrets_version": "1.5.0",
}

A dict keyed by a secret-looking string reports the safe ancestor instead of leaking the key:

# payload: {"ghp_abc...": {"nested": "safe"}}
# result.metadata["findings"] == [{"type": "GitHub Token", "json_pointer": ""}]

Config

Failure handling

Every failure maps to a stable metadata["failure_mode"]: normalization_error,
payload_too_large, queue_full, queue_timeout, worker_startup_error, worker_timeout,
worker_crash, worker_protocol_error, runtime_error.

on_error controls the fallback in EvaluatorResult:

• allow → matched=False, metadata["fallback_action"]="allow"
• deny → matched=True, metadata["fallback_action"]="deny"

Consumers should branch on metadata["failure_mode"] + metadata["fallback_action"], not on
matched alone, to distinguish a real finding from a fail-closed evaluator failure.

Dependencies

• detect-secrets-async>=0.2.0,<0.3.0 — the
  async subprocess-pool runtime over Yelp detect-secrets.
• google-re2>=1.1 — matches existing Agent Control regex policy.

Validation

make check in evaluators/contrib/detect_secrets:

• 90 tests pass, covering detection, structured-pointer mapping, RE2 exclusion on both string
  and structured payloads, plugin validation (strip + dedup + unknown rejection), max_bytes
  boundary behavior, recursive / NaN / empty-container / unsupported-type normalization paths,
  timeout short-circuit, every failure_mode path × {allow, deny} (16 runtime + 4
  evaluator-layer combinations), FAILURE_MESSAGES drift pin against the ScanFailureCode enum,
  concurrent dispatch on a cached evaluator instance, and entry-point .load() round-trip.
• mypy strict clean.
• ruff check + format clean.
• Coverage 98% (config.py 100%, evaluator.py 98%, normalization.py 98%).