Skip to content

Commit d652a56

Browse files
author
Victor Valbuena
committed
REFACTOR: Collapse AdversarialBenchmark to RapidResponse pattern
Removes the local factory-construction override and the adversarial_models constructor parameter. AdversarialBenchmark now inherits the base Scenario._get_atomic_attacks_async loop and reads its strategy enum from AttackTechniqueRegistry entries tagged benchmark_fanout (registered by BenchmarkInitializer in the previous commit). What's removed: - adversarial_models: list[PromptTarget] constructor param + validation. - _adversarial_configs dict construction in __init__. - _get_atomic_attacks_async override that built local factories, iterated models x techniques x datasets, and injected attack_adversarial_config_override at create-time. - _infer_labels static method + the entire dedupe/collision-suffix loop that inferred model labels from target identifiers - replaced by TargetConfig.registry_name as the canonical label (set explicitly in ENV_TARGET_CONFIGS, no inference needed). - _get_benchmarkable_specs and _build_benchmark_strategy as @staticmethods on the class - replaced by a module-level _build_benchmark_strategy function. Strategy-class construction never reads scenario instance state, so the function does not belong to the class; module-level placement makes the dependency (only the registry) explicit and the unit-test surface flat. What's added: - BENCHMARK_FANOUT_TAG module constant (= "benchmark_fanout") as the shared contract between BenchmarkInitializer (writes the tag) and AdversarialBenchmark (reads it). - _StrategyOnlyMarker sentinel class to satisfy the required AttackTechniqueSpec.attack_class field when reconstructing minimal specs for strategy-enum construction. build_strategy_class_from_specs reads only name + strategy_tags, so the sentinel never reaches a runtime construction site; the real factory is fetched by name from the registry at attack-execution time. - _build_display_group override: extracts the target label from the fanned f"src__target" technique name so display rolls up per-model. Falls back to the full name when no __ separator is present. Where the (technique x target x dataset) permutation now happens: The pre-collapse override did all three dimensions at scenario runtime in one nested loop. Post-collapse the permutation is split across two stages, owned by different layers: 1. Initializer time - BenchmarkInitializer.initialize_async runs the (technique x adversarial-target) cross-product and registers one fanned AttackTechniqueFactory per pair into AttackTechniqueRegistry, tagged benchmark_fanout. Target binding lives on the factory. 2. Scenario runtime - Scenario._get_atomic_attacks_async (inherited, base class) runs the (fanned-variant x dataset) cross-product, building one AtomicAttack per pair. The target dimension is already resolved on the factory at this point. Net atomic-attack count is unchanged for the same inputs; the change is which layer owns which dimension. See the AdversarialBenchmark class docstring for the full explanation. VERSION bump 1 -> 2: The atomic_attack_name format changes from f"{technique}__{model}__{dataset}" (triple-segment, old override-driven) to f"{technique}__{model}_{dataset}" (double-then- single-underscore, base-inherited). Cached results from VERSION=1 remain queryable via memory.get_scenario_results(scenario_version=1) but won't suppress fresh runs with skip_cached=True (the param itself lands in the next commit). No CHANGELOG file in this repo; this note will land in the PR description. Doc notebook (doc/scanner/benchmark.{py,ipynb}) still references the removed adversarial_models API and will fail at runtime until Commit 9 rewrites it; deferred per plan F7. Not gated by any unit test. Tests rewritten end-to-end (619 -> 268 lines): see test_adversarial.py for the four test classes covering metadata, strategy construction, collapsed init surface, and display grouping. Wider regression: 1091/1091 pass across scenario+setup+registry; 547/547 pass in backend.
1 parent 9ec8234 commit d652a56

2 files changed

Lines changed: 383 additions & 766 deletions

File tree

0 commit comments

Comments
 (0)