MAINT: Simplifying building atomic attacks#2107
Open
rlundeen2 wants to merge 6 commits into
Open
Conversation
…n point Slim the Scenario base class toward pure orchestration by adding one explicit extension point, `_build_atomic_attacks_async(self, *, context: ScenarioContext)`, and extracting the technique x dataset cross-product into a standalone MatrixAtomicAttackBuilder. ScenarioContext is a frozen dataclass carrying the run-resolved inputs (objective_target, scenario_strategies, dataset_config, memory_labels, include_baseline). A bridge in the legacy `_get_atomic_attacks_async` builds the context from resolved self state and forwards to the new method, so every existing call site keeps working with zero test edits. Migrated AdversarialBenchmark, RedTeamAgent, AdaptiveScenario, Jailbreak, Psychosocial, Scam, and Encoding onto the new method. objective_scorer is treated as scenario configuration (read from self) rather than a universal context field, since scenarios like Psychosocial select scorers per harm category and strategy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolved import-line conflicts in two files, both stemming from main's parameter/registry refactor (Parameter moved to pyrit.models, coerce_value/ validate_param_type became Parameter methods) overlapping this branch's ScenarioContext + MatrixAtomicAttackBuilder refactor: - pyrit/scenario/core/scenario.py: typing import — dropped both `cast` (this branch removed the scoring code that used it) and `get_origin` (main removed the choices-validation block that used it). Merged body uses neither. - pyrit/scenario/scenarios/benchmark/adversarial.py: dropped the deprecated `from pyrit.common import Parameter` (now sourced from pyrit.models.parameter) and the now-unused `AttackScoringConfig` import (scoring moved into the builder). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…narios # Conflicts: # pyrit/scenario/core/scenario.py # pyrit/scenario/scenarios/adaptive/adaptive_scenario.py # pyrit/scenario/scenarios/benchmark/adversarial.py
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
romanlutz
reviewed
Jun 30, 2026
Comment on lines
+32
to
+35
| ``initialize_async`` and handed to ``_build_atomic_attacks_async``. Because the | ||
| method receives a fully-populated context, scenarios no longer depend on the | ||
| "set ``self._x`` in ``initialize_async``, then read it in the override" sequencing | ||
| that historically caused "accessed before initialize_async" bugs. |
Contributor
There was a problem hiding this comment.
I don't know if this class description needs to talk about what it doesn't do 😆 "no longer"
romanlutz
approved these changes
Jun 30, 2026
Describe ScenarioContext, MatrixAtomicAttackBuilder, and the bridge by what they are rather than by what scenarios previously did, and refresh two stale _get_atomic_attacks_async references to the current method name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move seed-group resolution and baseline emission into the base Scenario bridge so custom scenarios stop duplicating _resolve_seed_groups_async, self._seed_groups state, and the include_baseline block. Add seed_groups and seed_groups_by_dataset to ScenarioContext, resolve seeds once via the new _resolve_seed_groups_by_dataset_async hook, and prepend the baseline centrally from the same sample (fixes the ADO 9012 double-sampling inconsistency). Simplify jailbreak, scam, encoding, red_team_agent, adaptive, adversarial, and psychosocial accordingly, and update coupled tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
Scenariobase class bakes in a single construction shape — knowstrategy_class+ oneobjective_scorer+default_dataset_configat__init__, then cross-product techniques × datasets in a default_get_atomic_attacks_async. That shape only fits the trivial scenarios; every important one (Encoding,RedTeamAgent,AdversarialBenchmark,AdaptiveScenario) fully overrides that method, so it inherits base hooks (_get_attack_technique_factories,_build_display_group,_prepare_strategies, the default cross-product, automatic baseline injection) it never uses but must understand and work around — and it has to remember the "self._objective_target/self._dataset_configare only valid afterinitialize_async" footgun. Writing a new scenario means fighting machinery that doesn't generalize (there isn't always a single default scorer, a single baseline, or a single default strategy).This PR
Introduces one explicit extension point,
async _build_atomic_attacks_async(self, *, context: ScenarioContext), whereScenarioContextis a frozen dataclass carrying the resolved run inputs (objective_target,scenario_strategies,dataset_config,memory_labels,include_baseline) — which kills the "beforeinitialize_async" footgun because the method can't be called without a populated context. The reusable technique × dataset (× optional adversarial-target) cross-product moves into a standaloneMatrixAtomicAttackBuilderthat scenarios call (composition) instead of override around (inheritance); scenarios whose construction isn't a cross-product build theirAtomicAttacks directly. All overriding scenarios (Encoding,RedTeamAgent,AdaptiveScenario,Jailbreak,Psychosocial,Scam,AdversarialBenchmark) migrate by the same minimal rename-and-read-from-contextpattern. A bridge keeps the legacy_get_atomic_attacks_asyncentry point working, so this is non-breaking and required zero test edits (fulltests/unit/scenario/green). Deleting the bridge + dead hooks, making_build_atomic_attacks_asyncabstract, and rewriting the scenario authoring docs are deferred to follow-up PRs.Design RFC / full phased plan: https://gist.github.com/rlundeen2/d2fa5f91eb411ba9b1d0bbf98412f700