MAINT: Rapid response Scenario (microsoft#1622)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: rlundeen2

.env_example

@@ -74,6 +74,11 @@ AZURE_OPENAI_GPT4O_UNSAFE_CHAT_KEY2="xxxxx"
 AZURE_OPENAI_GPT4O_UNSAFE_CHAT_MODEL2="deployment-name"
 AZURE_OPENAI_GPT4O_UNSAFE_CHAT_UNDERLYING_MODEL2=""
 
+# Adversarial chat target (used by scenario attack techniques, e.g. role-play, TAP)
+ADVERSARIAL_CHAT_ENDPOINT="https://xxxxx.openai.azure.com/openai/v1"
+ADVERSARIAL_CHAT_KEY="xxxxx"
+ADVERSARIAL_CHAT_MODEL="deployment-name"
+
 AZURE_FOUNDRY_DEEPSEEK_ENDPOINT="https://xxxxx.eastus2.models.ai.azure.com"
 AZURE_FOUNDRY_DEEPSEEK_KEY="xxxxx"
 AZURE_FOUNDRY_DEEPSEEK_MODEL=""

.github/instructions/scenarios.instructions.md

@@ -11,7 +11,7 @@ Scenarios orchestrate multi-attack security testing campaigns. Each scenario gro
 All scenarios inherit from `Scenario` (ABC) and must:
 
 1. **Define `VERSION`** as a class constant (increment on breaking changes)
-2. **Implement four abstract methods:**
+2. **Implement three abstract methods:**
 
 ```python
 class MyScenario(Scenario):
@@ -28,11 +28,12 @@ class MyScenario(Scenario):
     @classmethod
     def default_dataset_config(cls) -> DatasetConfiguration:
         return DatasetConfiguration(dataset_names=["my_dataset"])
-
-    async def _get_atomic_attacks_async(self) -> list[AtomicAttack]:
-        ...
 ```
 
+3. **Optionally override `_get_atomic_attacks_async()`** — the base class provides a default
+   that uses the factory/registry pattern (see "AtomicAttack Construction" below).
+   Only override if your scenario needs custom attack construction logic.
+
 ## Constructor Pattern
 
 ```python
@@ -82,7 +83,7 @@ DatasetConfiguration(
 class MyDatasetConfiguration(DatasetConfiguration):
     def get_seed_groups(self) -> dict[str, list[SeedGroup]]:
         result = super().get_seed_groups()
-        # Filter by selected strategies via self._scenario_composites
+        # Filter by selected strategies via self._scenario_strategies
         return filtered_result
 ```
 
@@ -94,26 +95,66 @@ Options:
 
 ## Strategy Enum
 
-Strategies should be selectable by an axis. E.g. it could be harm category or and attack type, but likely not both or it gets confusing.
+Strategy members should represent **attack techniques** — the *how* of an attack (e.g., prompt sending, role play, TAP).  Datasets control *what* is tested (e.g., harm categories, compliance topics).  Avoid mixing dataset/category selection into the strategy enum; use `DatasetConfiguration` and `--dataset-names` for that axis.
 
 ```python
 class MyStrategy(ScenarioStrategy):
-    ALL = ("all", {"all"})            # Required aggregate
-    EASY = ("easy", {"easy"})
+    ALL = ("all", {"all"})                  # Required aggregate
+    DEFAULT = ("default", {"default"})      # Recommended default aggregate
+    SINGLE_TURN = ("single_turn", {"single_turn"})  # Category aggregate
 
-    Base64 = ("base64", {"easy", "converter"})
-    Crescendo = ("crescendo", {"difficult", "multi_turn"})
+    PromptSending = ("prompt_sending", {"single_turn", "default"})
+    RolePlay = ("role_play", {"single_turn"})
+    ManyShot = ("many_shot", {"multi_turn", "default"})
 
     @classmethod
     def get_aggregate_tags(cls) -> set[str]:
-        return {"all", "easy", "difficult"}
+        return {"all", "default", "single_turn", "multi_turn"}
 ```
 
 - `ALL` aggregate is always required
 - Each member: `NAME = ("string_value", {tag_set})`
 - Aggregates expand to all strategies matching their tag
 
-## AtomicAttack Construction
+### `_build_display_group()` — Result Grouping
+
+Override `_build_display_group()` on the `Scenario` base class to control how attack results are grouped for display:
+
+```python
+def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:
+    # Default: group by technique name (most common)
+    return technique_name
+
+    # Override examples:
+    # Group by dataset/harm category: return seed_group_name
+    # Cross-product: return f"{technique_name}_{seed_group_name}"
+```
+
+Note: `atomic_attack_name` must remain unique per `AtomicAttack` for correct resume behaviour.
+`display_group` controls user-facing aggregation only.
+
+## AtomicAttack Construction — Default Base Class Behaviour
+
+The `Scenario` base class provides a default `_get_atomic_attacks_async()` that uses the
+factory/registry pattern.  Scenarios that register their techniques via `_get_attack_technique_factories()`
+get atomic-attack construction **for free** — no override needed.
+
+The default implementation:
+1. Calls `self._get_attack_technique_factories()` to get name→factory mapping
+2. Iterates over every (technique × dataset) pair from `self._dataset_config`
+3. Calls `factory.create()` with `objective_target` and conditional scorer override
+4. Uses `self._build_display_group()` for user-facing grouping
+5. Builds `AtomicAttack` with unique `atomic_attack_name` = `"{technique}_{dataset}"`
+
+### Customization hooks (no need to override `_get_atomic_attacks_async`):
+- **`_get_attack_technique_factories()`** — override to add/remove/replace factories
+- **`_build_display_group()`** — override to change grouping (default: by technique)
+
+### When to override `_get_atomic_attacks_async`:
+Only override when the scenario **cannot** use the factory/registry pattern — e.g., scenarios
+with custom composite logic, per-strategy converter stacks, or non-standard attack construction.
+
+### Manual AtomicAttack construction (for overrides):
 
 ```python
 AtomicAttack(
@@ -134,7 +175,7 @@ New scenarios must be registered in `pyrit/scenario/__init__.py` as virtual pack
 
 ## Common Review Issues
 
-- Accessing `self._objective_target` or `self._scenario_composites` before `initialize_async()`
+- Accessing `self._objective_target` or `self._scenario_strategies` before `initialize_async()`
 - Forgetting `@apply_defaults` on `__init__`
 - Empty `seed_groups` passed to `AtomicAttack`
 - Missing `VERSION` class constant

doc/code/scenarios/0_scenarios.ipynb

@@ -53,32 +53,42 @@
     "\n",
     "### Required Components\n",
     "\n",
-    "1. **Strategy Enum**: Create a `ScenarioStrategy` enum that defines the available strategies for your scenario.\n",
-    "   - Each enum member is defined as `(value, tags)` where value is a string and tags is a set of strings\n",
+    "1. **Strategy Enum**: Create a `ScenarioStrategy` enum that defines the available attack techniques for your scenario.\n",
+    "   - Each enum member represents an **attack technique** (the *how* of an attack)\n",
+    "   - Each member is defined as `(value, tags)` where value is a string and tags is a set of strings\n",
     "   - Include an `ALL` aggregate strategy that expands to all available strategies\n",
     "   - Optionally override `_prepare_strategies()` for custom composition logic (see `FoundryComposite`)\n",
     "\n",
     "2. **Scenario Class**: Extend `Scenario` and implement these abstract methods:\n",
     "   - `get_strategy_class()`: Return your strategy enum class\n",
     "   - `get_default_strategy()`: Return the default strategy (typically `YourStrategy.ALL`)\n",
-    "   - `_get_atomic_attacks_async()`: Build and return a list of `AtomicAttack` instances\n",
+    "   - The base class provides a default `_get_atomic_attacks_async()` that uses the factory/registry\n",
+    "     pattern. Override it only if your scenario needs custom attack construction logic.\n",
     "\n",
-    "3. **Constructor**: Use `@apply_defaults` decorator and call `super().__init__()` with scenario metadata:\n",
+    "3. **Default Dataset**: Implement `default_dataset_config()` to specify the datasets your scenario uses out of the box.\n",
+    "   - Returns a `DatasetConfiguration` with one or more named datasets (e.g., `DatasetConfiguration(dataset_names=[\"my_dataset\"])`)\n",
+    "   - Users can override this at runtime via `--dataset-names` in the CLI or by passing a custom `dataset_config` programmatically\n",
+    "\n",
+    "4. **Constructor**: Use `@apply_defaults` decorator and call `super().__init__()` with scenario metadata:\n",
     "   - `name`: Descriptive name for your scenario\n",
     "   - `version`: Integer version number\n",
     "   - `strategy_class`: The strategy enum class for this scenario\n",
     "   - `objective_scorer_identifier`: Identifier dict for the scoring mechanism (optional)\n",
     "   - `include_default_baseline`: Whether to include a baseline attack (default: True)\n",
     "   - `scenario_result_id`: Optional ID to resume an existing scenario (optional)\n",
     "\n",
-    "4. **Initialization**: Call `await scenario.initialize_async()` to populate atomic attacks:\n",
+    "5. **Initialization**: Call `await scenario.initialize_async()` to populate atomic attacks:\n",
     "   - `objective_target`: The target system being tested (required)\n",
     "   - `scenario_strategies`: List of strategies to execute (optional, defaults to ALL)\n",
     "   - `max_concurrency`: Number of concurrent operations (default: 1)\n",
     "   - `max_retries`: Number of retry attempts on failure (default: 0)\n",
     "   - `memory_labels`: Optional labels for tracking (optional)\n",
     "\n",
-    "### Example Structure"
+    "### Example Structure\n",
+    "\n",
+    "The simplest approach uses the **factory/registry pattern**: define your strategy,\n",
+    "dataset config, and constructor — the base class handles building atomic attacks\n",
+    "automatically from registered attack techniques."
    ]
   },
   {
@@ -98,12 +108,8 @@
     }
    ],
    "source": [
-    "from typing import Optional\n",
-    "\n",
     "from pyrit.common import apply_defaults\n",
-    "from pyrit.executor.attack import AttackScoringConfig, PromptSendingAttack\n",
     "from pyrit.scenario import (\n",
-    "    AtomicAttack,\n",
     "    DatasetConfiguration,\n",
     "    Scenario,\n",
     "    ScenarioStrategy,\n",
@@ -116,76 +122,56 @@
     "\n",
     "class MyStrategy(ScenarioStrategy):\n",
     "    ALL = (\"all\", {\"all\"})\n",
-    "    StrategyA = (\"strategy_a\", {\"tag1\", \"tag2\"})\n",
-    "    StrategyB = (\"strategy_b\", {\"tag1\"})\n",
+    "    DEFAULT = (\"default\", {\"default\"})\n",
+    "    SINGLE_TURN = (\"single_turn\", {\"single_turn\"})\n",
+    "    # Strategy members represent attack techniques\n",
+    "    PromptSending = (\"prompt_sending\", {\"single_turn\", \"default\"})\n",
+    "    RolePlay = (\"role_play\", {\"single_turn\"})\n",
     "\n",
     "\n",
     "class MyScenario(Scenario):\n",
-    "    version: int = 1\n",
+    "    \"\"\"Quick-check scenario for testing model behavior across harm categories.\"\"\"\n",
+    "\n",
+    "    VERSION: int = 1\n",
     "\n",
-    "    # A strategy definition helps callers define how to run your scenario (e.g. from the scanner CLI)\n",
     "    @classmethod\n",
     "    def get_strategy_class(cls) -> type[ScenarioStrategy]:\n",
     "        return MyStrategy\n",
     "\n",
     "    @classmethod\n",
     "    def get_default_strategy(cls) -> ScenarioStrategy:\n",
-    "        return MyStrategy.ALL\n",
+    "        return MyStrategy.DEFAULT\n",
     "\n",
-    "    # This is the default dataset configuration for this scenario (e.g. prompts to send)\n",
     "    @classmethod\n",
     "    def default_dataset_config(cls) -> DatasetConfiguration:\n",
-    "        return DatasetConfiguration(dataset_names=[\"dataset_name\"])\n",
+    "        return DatasetConfiguration(dataset_names=[\"dataset_name\"], max_dataset_size=4)\n",
     "\n",
     "    @apply_defaults\n",
     "    def __init__(\n",
     "        self,\n",
     "        *,\n",
-    "        objective_scorer: Optional[TrueFalseScorer] = None,\n",
-    "        scenario_result_id: Optional[str] = None,\n",
-    "    ):\n",
-    "        self._objective_scorer = objective_scorer\n",
-    "        self._scorer_config = AttackScoringConfig(objective_scorer=objective_scorer)\n",
+    "        objective_scorer: TrueFalseScorer | None = None,\n",
+    "        scenario_result_id: str | None = None,\n",
+    "    ) -> None:\n",
+    "        self._objective_scorer: TrueFalseScorer = (\n",
+    "            objective_scorer if objective_scorer else self._get_default_objective_scorer()\n",
+    "        )\n",
     "\n",
-    "        # Call parent constructor - note: objective_target is NOT passed here\n",
     "        super().__init__(\n",
-    "            name=\"My Custom Scenario\",\n",
-    "            version=self.version,\n",
-    "            strategy_class=MyStrategy,\n",
-    "            objective_scorer=objective_scorer,\n",
+    "            version=self.VERSION,\n",
+    "            objective_scorer=self._objective_scorer,\n",
+    "            strategy_class=self.get_strategy_class(),\n",
     "            scenario_result_id=scenario_result_id,\n",
     "        )\n",
     "\n",
-    "    async def _get_atomic_attacks_async(self) -> list[AtomicAttack]:\n",
-    "        \"\"\"\n",
-    "        Build atomic attacks based on selected strategies.\n",
-    "\n",
-    "        This method is called by initialize_async() after strategies are prepared.\n",
-    "        Use self._scenario_strategies to access the resolved strategy list.\n",
-    "        \"\"\"\n",
-    "        atomic_attacks = []\n",
-    "\n",
-    "        # objective_target is guaranteed to be non-None by parent class validation\n",
-    "        assert self._objective_target is not None\n",
-    "\n",
-    "        for strategy in self._scenario_strategies:\n",
-    "            # self._dataset_config is set by the parent class\n",
-    "            seed_groups = self._dataset_config.get_all_seed_groups()\n",
-    "\n",
-    "            # Create attack instances based on strategy\n",
-    "            attack = PromptSendingAttack(\n",
-    "                objective_target=self._objective_target,\n",
-    "                attack_scoring_config=self._scorer_config,\n",
-    "            )\n",
-    "            atomic_attacks.append(\n",
-    "                AtomicAttack(\n",
-    "                    atomic_attack_name=strategy.value,\n",
-    "                    attack=attack,\n",
-    "                    seed_groups=seed_groups,  # type: ignore[arg-type]\n",
-    "                    memory_labels=self._memory_labels,\n",
-    "                )\n",
-    "            )\n",
-    "        return atomic_attacks"
+    "    # Optional: override _build_display_group to customize result grouping.\n",
+    "    # Default groups by technique name; override to group by dataset instead:\n",
+    "    def _build_display_group(self, *, technique_name: str, seed_group_name: str) -> str:\n",
+    "        return seed_group_name\n",
+    "\n",
+    "    # No _get_atomic_attacks_async override needed!\n",
+    "    # The base class builds attacks from the (technique x dataset) cross-product\n",
+    "    # using the factory/registry pattern automatically."
    ]
   },
   {