Skip to content

Commit e272a0a

Browse files
committed
refactor(atr): field-aware evaluator with typed rule models + redaction
Addresses @lan17's 2026-04-26 architectural review on PR #170. The previous v0.1 evaluator flattened every input into a single string and scanned every rule's regex against it. @lan17's critique was that ATR is an event/field-aware rule format, not a flat regex scanner, and that the v0.1 design also leaked the raw matched text (including any secret that triggered the rule) through `metadata.matched_text`. ## What v0.2 changes * `models.py` (new) — typed `ATREvent`, `ATRRule`, `ATRCondition`, `RuleMatch` dataclasses. `ATR_FIELDS` is the canonical field vocabulary (`content`, `user_input`, `agent_output`, `tool_name`, `tool_args`, `tool_description`, `tool_response`, `agent_message`, `skill_manifest`). `ATR_CATEGORY_DEFAULT_FIELD` maps each ATR category to its default field for the legacy flat-pattern auto-upgrade path. * `ATREvent.from_agent_control_data()` adapts an Agent Control selector input into a typed event with explicit field semantics. Bare strings land in `content`. Dicts map known field names directly. Aliases (`input`/`output`/`text`/`message`) map to canonical fields. Unknown keys are JSON-serialised into `content` so broad content-targeted rules still have something to scan defensively. * `evaluator.py` (rewritten) — runs each rule condition only against the condition's intended field via `event.get_field(condition.field)`. Conditions with no field value short-circuit without compiling or searching. Rule-level `condition: any|all` is honoured. * `redact.py` (new) — Python port of the upstream ATR `redactMatchedValue()` helper (`agent-threat-rules@2.1.2` `src/redact.ts`). Recognises AWS access keys, GitHub PATs / OAuth tokens, Slack tokens, OpenAI / Anthropic API keys, Bearer credentials, JWTs, and PEM private keys. Every match is run through this before reaching `EvaluatorResult.metadata` — the v0.1 `matched_text` field is intentionally REMOVED and replaced with `redacted_excerpt`. * `_wall_clock_search` (new in `evaluator.py`) — bounds per-condition regex evaluation time with `signal.SIGALRM` (default 50 ms) so a pathological pattern cannot block the evaluator pipeline. Configurable via `ATRConfig.condition_budget_ms`. Falls back to a soft wall-clock check on Windows / worker threads. * `evaluator._normalise_rule` accepts both the modern field-aware `conditions: [{field, operator, value, ...}]` shape AND the legacy flat `patterns: [{pattern, description}]` shape. Legacy patterns are auto-upgraded to conditions targeting the category default field. Existing `rules.json` keeps working without regeneration. * `pyproject.toml` — version bump to `0.2.0` (minor: API change on the metadata field, but no plugin discovery shape change). ## Why this addresses each of @lan17's concrete asks | @lan17's review point | Addressed by | |---|---| | "preserve ATR fields, scan targets, and condition logic in typed rule models" | `models.py` ATRRule + ATRCondition + ATR_FIELDS | | "adapt Agent Control selector output into an explicit ATR event" | `ATREvent.from_agent_control_data` | | "only run each condition against its intended field" | `_evaluate_rule` dispatches via `event.get_field(condition.field)` | | "avoid metadata leakage" / "keep metadata safe" | `redact.py`; `metadata.matched_text` removed | | "put a harder bound around regex evaluation latency" | `_wall_clock_search` + `condition_budget_ms` | | "position complementary to yelp.detect_secrets" | The redacted-excerpt output structure makes the evaluator deliberately non-overlapping with secret scanners; rules only match attack patterns, the redactor only renders the match safely | ## Tests `tests/test_evaluator.py` rewritten on 2026-05-11 to match the new architecture: * Field-aware dispatch tests (field isolation, alias mapping, unknown key fallback). * Redacted-excerpt assertions on every match path — verifies no raw AWS key, GitHub PAT, or matched substring ever appears in `EvaluatorResult.metadata`. * Condition budget config validation. * Backwards-compat tests retained for severity / category / block_on_match / on_error policies, with input shapes updated to dicts where field semantics matter. ``` $ pytest evaluators/contrib/atr/tests/test_evaluator.py -v ============================== 31 passed in 0.21s ============================== ```
1 parent d7ea37d commit e272a0a

7 files changed

Lines changed: 909 additions & 253 deletions

File tree

evaluators/contrib/atr/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "agent-control-evaluator-atr"
3-
version = "0.1.0"
3+
version = "0.2.0"
44
description = "ATR (Agent Threat Rules) evaluator for agent-control"
55
readme = "README.md"
66
requires-python = ">=3.12"
Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,16 @@
11
from .config import ATRConfig
22
from .evaluator import ATREvaluator
3+
from .models import ATR_FIELDS, ATRCondition, ATREvent, ATRRule, RuleMatch
4+
from .redact import redact_matched_value, redact_matched_values
35

4-
__all__ = ["ATREvaluator", "ATRConfig"]
6+
__all__ = [
7+
"ATREvaluator",
8+
"ATRConfig",
9+
"ATREvent",
10+
"ATRRule",
11+
"ATRCondition",
12+
"RuleMatch",
13+
"ATR_FIELDS",
14+
"redact_matched_value",
15+
"redact_matched_values",
16+
]

evaluators/contrib/atr/src/agent_control_evaluator_atr/threat_rules/config.py

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,19 @@ class ATRConfig(EvaluatorConfig):
1010
"""Configuration for ATR (Agent Threat Rules) evaluator.
1111
1212
Attributes:
13-
min_severity: Minimum severity level to match ("low", "medium", "high", "critical")
14-
block_on_match: Whether to set matched=True when a threat is detected
15-
categories: Category filter; empty list means all categories
16-
on_error: Error policy ("allow" = fail-open, "deny" = fail-closed)
13+
min_severity: Minimum severity level to match ("low", "medium", "high", "critical").
14+
block_on_match: Whether to set matched=True when a threat is detected.
15+
categories: Category filter; empty list means all categories.
16+
on_error: Error policy ("allow" = fail-open, "deny" = fail-closed).
17+
condition_budget_ms: Wall-clock budget for each regex condition evaluation,
18+
in milliseconds. Patterns exceeding this budget are skipped with a
19+
warning rather than blocking the evaluator pipeline. Default 50 ms
20+
is generous for any reasonable pattern; the budget only fires on
21+
catastrophic backtracking.
1722
"""
1823

1924
min_severity: Literal["low", "medium", "high", "critical"] = "medium"
2025
block_on_match: bool = True
2126
categories: list[str] = Field(default_factory=list)
2227
on_error: Literal["allow", "deny"] = "allow"
28+
condition_budget_ms: int = Field(default=50, ge=1, le=10_000)

0 commit comments

Comments
 (0)