Commit d652a56
Victor Valbuena
REFACTOR: Collapse AdversarialBenchmark to RapidResponse pattern
Removes the local factory-construction override and the
adversarial_models constructor parameter. AdversarialBenchmark now
inherits the base Scenario._get_atomic_attacks_async loop and reads its
strategy enum from AttackTechniqueRegistry entries tagged
benchmark_fanout (registered by BenchmarkInitializer in the previous
commit).
What's removed:
- adversarial_models: list[PromptTarget] constructor param + validation.
- _adversarial_configs dict construction in __init__.
- _get_atomic_attacks_async override that built local factories,
iterated models x techniques x datasets, and injected
attack_adversarial_config_override at create-time.
- _infer_labels static method + the entire dedupe/collision-suffix loop
that inferred model labels from target identifiers - replaced by
TargetConfig.registry_name as the canonical label (set explicitly in
ENV_TARGET_CONFIGS, no inference needed).
- _get_benchmarkable_specs and _build_benchmark_strategy as
@staticmethods on the class - replaced by a module-level
_build_benchmark_strategy function. Strategy-class construction never
reads scenario instance state, so the function does not belong to the
class; module-level placement makes the dependency (only the
registry) explicit and the unit-test surface flat.
What's added:
- BENCHMARK_FANOUT_TAG module constant (= "benchmark_fanout") as the
shared contract between BenchmarkInitializer (writes the tag) and
AdversarialBenchmark (reads it).
- _StrategyOnlyMarker sentinel class to satisfy the required
AttackTechniqueSpec.attack_class field when reconstructing minimal
specs for strategy-enum construction. build_strategy_class_from_specs
reads only name + strategy_tags, so the sentinel never reaches a
runtime construction site; the real factory is fetched by name from
the registry at attack-execution time.
- _build_display_group override: extracts the target label from the
fanned f"src__target" technique name so display rolls up per-model.
Falls back to the full name when no __ separator is present.
Where the (technique x target x dataset) permutation now happens:
The pre-collapse override did all three dimensions at scenario runtime
in one nested loop. Post-collapse the permutation is split across two
stages, owned by different layers:
1. Initializer time - BenchmarkInitializer.initialize_async runs the
(technique x adversarial-target) cross-product and registers one
fanned AttackTechniqueFactory per pair into AttackTechniqueRegistry,
tagged benchmark_fanout. Target binding lives on the factory.
2. Scenario runtime - Scenario._get_atomic_attacks_async (inherited,
base class) runs the (fanned-variant x dataset) cross-product,
building one AtomicAttack per pair. The target dimension is already
resolved on the factory at this point.
Net atomic-attack count is unchanged for the same inputs; the change
is which layer owns which dimension. See the AdversarialBenchmark
class docstring for the full explanation.
VERSION bump 1 -> 2:
The atomic_attack_name format changes from
f"{technique}__{model}__{dataset}" (triple-segment, old
override-driven) to f"{technique}__{model}_{dataset}" (double-then-
single-underscore, base-inherited). Cached results from VERSION=1
remain queryable via memory.get_scenario_results(scenario_version=1)
but won't suppress fresh runs with skip_cached=True (the param itself
lands in the next commit). No CHANGELOG file in this repo; this note
will land in the PR description.
Doc notebook (doc/scanner/benchmark.{py,ipynb}) still references the
removed adversarial_models API and will fail at runtime until Commit 9
rewrites it; deferred per plan F7. Not gated by any unit test.
Tests rewritten end-to-end (619 -> 268 lines): see test_adversarial.py
for the four test classes covering metadata, strategy construction,
collapsed init surface, and display grouping.
Wider regression: 1091/1091 pass across scenario+setup+registry;
547/547 pass in backend.1 parent 9ec8234 commit d652a56
2 files changed
Lines changed: 383 additions & 766 deletions
File tree
- pyrit/scenario/scenarios/benchmark
- tests/unit/scenario/benchmark
0 commit comments