Commit e272a0a
committed
refactor(atr): field-aware evaluator with typed rule models + redaction
Addresses @lan17's 2026-04-26 architectural review on PR #170.
The previous v0.1 evaluator flattened every input into a single string
and scanned every rule's regex against it. @lan17's critique was that
ATR is an event/field-aware rule format, not a flat regex scanner, and
that the v0.1 design also leaked the raw matched text (including any
secret that triggered the rule) through `metadata.matched_text`.
## What v0.2 changes
* `models.py` (new) — typed `ATREvent`, `ATRRule`, `ATRCondition`,
`RuleMatch` dataclasses. `ATR_FIELDS` is the canonical field
vocabulary (`content`, `user_input`, `agent_output`, `tool_name`,
`tool_args`, `tool_description`, `tool_response`, `agent_message`,
`skill_manifest`). `ATR_CATEGORY_DEFAULT_FIELD` maps each ATR
category to its default field for the legacy flat-pattern auto-upgrade
path.
* `ATREvent.from_agent_control_data()` adapts an Agent Control selector
input into a typed event with explicit field semantics. Bare strings
land in `content`. Dicts map known field names directly. Aliases
(`input`/`output`/`text`/`message`) map to canonical fields. Unknown
keys are JSON-serialised into `content` so broad content-targeted
rules still have something to scan defensively.
* `evaluator.py` (rewritten) — runs each rule condition only against the
condition's intended field via `event.get_field(condition.field)`.
Conditions with no field value short-circuit without compiling or
searching. Rule-level `condition: any|all` is honoured.
* `redact.py` (new) — Python port of the upstream ATR
`redactMatchedValue()` helper (`agent-threat-rules@2.1.2`
`src/redact.ts`). Recognises AWS access keys, GitHub PATs / OAuth
tokens, Slack tokens, OpenAI / Anthropic API keys, Bearer
credentials, JWTs, and PEM private keys. Every match is run through
this before reaching `EvaluatorResult.metadata` — the v0.1
`matched_text` field is intentionally REMOVED and replaced with
`redacted_excerpt`.
* `_wall_clock_search` (new in `evaluator.py`) — bounds per-condition
regex evaluation time with `signal.SIGALRM` (default 50 ms) so a
pathological pattern cannot block the evaluator pipeline. Configurable
via `ATRConfig.condition_budget_ms`. Falls back to a soft wall-clock
check on Windows / worker threads.
* `evaluator._normalise_rule` accepts both the modern field-aware
`conditions: [{field, operator, value, ...}]` shape AND the legacy
flat `patterns: [{pattern, description}]` shape. Legacy patterns are
auto-upgraded to conditions targeting the category default field.
Existing `rules.json` keeps working without regeneration.
* `pyproject.toml` — version bump to `0.2.0` (minor: API change on
the metadata field, but no plugin discovery shape change).
## Why this addresses each of @lan17's concrete asks
| @lan17's review point | Addressed by |
|---|---|
| "preserve ATR fields, scan targets, and condition logic in typed rule models" | `models.py` ATRRule + ATRCondition + ATR_FIELDS |
| "adapt Agent Control selector output into an explicit ATR event" | `ATREvent.from_agent_control_data` |
| "only run each condition against its intended field" | `_evaluate_rule` dispatches via `event.get_field(condition.field)` |
| "avoid metadata leakage" / "keep metadata safe" | `redact.py`; `metadata.matched_text` removed |
| "put a harder bound around regex evaluation latency" | `_wall_clock_search` + `condition_budget_ms` |
| "position complementary to yelp.detect_secrets" | The redacted-excerpt output structure makes the evaluator deliberately non-overlapping with secret scanners; rules only match attack patterns, the redactor only renders the match safely |
## Tests
`tests/test_evaluator.py` rewritten on 2026-05-11 to match the new
architecture:
* Field-aware dispatch tests (field isolation, alias mapping, unknown
key fallback).
* Redacted-excerpt assertions on every match path — verifies no raw
AWS key, GitHub PAT, or matched substring ever appears in
`EvaluatorResult.metadata`.
* Condition budget config validation.
* Backwards-compat tests retained for severity / category /
block_on_match / on_error policies, with input shapes updated to
dicts where field semantics matter.
```
$ pytest evaluators/contrib/atr/tests/test_evaluator.py -v
============================== 31 passed in 0.21s ==============================
```1 parent d7ea37d commit e272a0a
7 files changed
Lines changed: 909 additions & 253 deletions
File tree
- evaluators/contrib/atr
- src/agent_control_evaluator_atr/threat_rules
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
Lines changed: 13 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
3 | 5 | | |
4 | | - | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
Lines changed: 10 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
17 | 22 | | |
18 | 23 | | |
19 | 24 | | |
20 | 25 | | |
21 | 26 | | |
22 | 27 | | |
| 28 | + | |
0 commit comments