FEAT: Adversarial Benchmark Scenario Refactor

.env_example

@@ -75,10 +75,24 @@ AZURE_OPENAI_GPT4O_UNSAFE_CHAT_MODEL2="deployment-name"
 AZURE_OPENAI_GPT4O_UNSAFE_CHAT_UNDERLYING_MODEL2=""
 
 # Adversarial chat target (used by scenario attack techniques, e.g. role-play, TAP)
+# Default endpoint goes here; specialized ones below
 ADVERSARIAL_CHAT_ENDPOINT="https://xxxxx.openai.azure.com/openai/v1"
 ADVERSARIAL_CHAT_KEY="xxxxx"
 ADVERSARIAL_CHAT_MODEL="deployment-name"
 
+ADVERSARIAL_CHAT_SINGLETURN_ENDPOINT="https://xxxxxx.westus3.inference.ml.azure.com/score"
+ADVERSARIAL_CHAT_SINGLETURN_KEY="xxxxx"
+ADVERSARIAL_CHAT_SINGLETURN_MODEL="deployment-name"
+
+ADVERSARIAL_CHAT_MULTITURN_ENDPOINT="https://xxxxxx.westus3.inference.ml.azure.com/score"
+ADVERSARIAL_CHAT_MULTITURN_KEY="xxxxx"
+ADVERSARIAL_CHAT_MULTITURN_MODEL="deployment-name"
+
+ADVERSARIAL_CHAT_REASONING_ENDPOINT="https://xxxxxx.westus3.inference.ml.azure.com/score"
+ADVERSARIAL_CHAT_REASONING_KEY="xxxxx"
+ADVERSARIAL_CHAT_REASONING_MODEL="deployment-name"
+
+
 # Objective Scorer chat target (used in scorers in scenarios)
 OBJECTIVE_SCORER_CHAT_ENDPOINT="https://xxxxx.openai.azure.com/openai/v1"
 OBJECTIVE_SCORER_CHAT_KEY="xxxxx"

doc/scanner/benchmark.ipynb

@@ -7,7 +7,10 @@
    "source": [
     "# Benchmark Scenarios\n",
     "\n",
-    "Benchmark scenarios are a subset of scenarios that compare the effectiveness of attacks across an axis that varies within the scenario itself. The axis can be many things; currently, the only benchmark variant is the adversarial benchmark, whose axis of change is the adversarial model used in attacks."
+    "Benchmark scenarios compare attack effectiveness across an axis that varies within the scenario\n",
+    "itself. Currently the only benchmark variant is the adversarial benchmark, whose axis of change is\n",
+    "the **adversarial chat helper model** used in attacks. For full configuration options see\n",
+    "`pyrit_scan --help` and the [Scenarios Programming Guide](../code/scenarios/0_scenarios.ipynb)."
    ]
   },
   {
@@ -16,143 +19,97 @@
    "metadata": {},
    "source": [
     "## Adversarial Benchmark\n",
-    "The adversarial benchmarking scenario (`AdversarialBenchmark`) compares the effectiveness of different adversarial models in successfully executing attacks against a target model."
+    "\n",
+    "`AdversarialBenchmark` holds the objective target and dataset constant and varies the adversarial\n",
+    "chat model used to drive multi-turn attacks. Useful for evaluating which adversarial helper\n",
+    "models produce stronger or weaker attack success rates against the same target.\n",
+    "\n",
+    "Adversarial targets are user-provided via the `adversarial_targets` scenario parameter. Each name\n",
+    "must already be registered in `TargetRegistry` — typically by `TargetInitializer` from the\n",
+    "`ADVERSARIAL_CHAT_*` env vars (see `.env_example`). Use `pyrit_scan --list-targets` to see every\n",
+    "target currently registered.\n",
+    "\n",
+    "```bash\n",
+    "pyrit_scan benchmark.adversarial \\\n",
+    "  --initializers target load_default_datasets \\\n",
+    "  --target openai_chat \\\n",
+    "  --adversarial-targets adversarial_chat_singleturn adversarial_chat_multiturn \\\n",
+    "  --max-dataset-size 4\n",
+    "```\n",
+    "\n",
+    "Pass multiple `--adversarial-targets` values to compare across models in a single run.\n",
+    "\n",
+    "**Available strategies:** `light` (default — a quick snapshot using the cheaper techniques),\n",
+    "`single_turn`, `multi_turn`, plus one member per adversarial-capable source technique\n",
+    "(e.g. `red_teaming`, `tap`, `crescendo_simulated`). The `light` aggregate excludes `tap` and\n",
+    "`crescendo_simulated`, which can take hours."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {},
+   "source": [
+    "## Setup"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "2",
+   "id": "3",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n",
-      "Loaded environment file: ./.pyrit/.env\n",
-      "Loaded environment file: ./.pyrit/.env.local\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "8316db039ba1408499df0a2de6c8d6f6",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Executing AdversarialBenchmark:   0%|          | 0/3 [00:00<?, ?attack/s]"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Scenario result id: e01b35d2-c7f8-49bd-aafd-5d44ef7235f4\n",
-      "\n",
-      "\u001b[36m====================================================================================================\u001b[0m\n",
-      "\u001b[1m\u001b[36m                              📊 SCENARIO RESULTS: AdversarialBenchmark                              \u001b[0m\n",
-      "\u001b[36m====================================================================================================\u001b[0m\n",
-      "\n",
-      "\u001b[1m\u001b[36m▼ Scenario Information\u001b[0m\n",
-      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
-      "\u001b[1m  📋 Scenario Details\u001b[0m\n",
-      "\u001b[36m    • Name: AdversarialBenchmark\u001b[0m\n",
-      "\u001b[36m    • Scenario Version: 1\u001b[0m\n",
-      "\u001b[36m    • PyRIT Version: 0.14.0.dev0\u001b[0m\n",
-      "\u001b[36m    • Description:\u001b[0m\n",
-      "\u001b[36m        Benchmarking scenario that compares the attack success rate (ASR) of several different adversarial models.\u001b[0m\n",
-      "\n",
-      "\u001b[1m  🎯 Target Information\u001b[0m\n",
-      "\u001b[36m    • Target Type: OpenAIChatTarget\u001b[0m\n",
-      "\u001b[36m    • Target Model: gpt-4o-japan-nilfilter\u001b[0m\n",
-      "\u001b[36m    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1\u001b[0m\n",
-      "\n",
-      "\u001b[1m  📊 Scorer Information\u001b[0m\n",
-      "\u001b[37m    ▸ Scorer Identifier\u001b[0m\n",
-      "\u001b[36m      • Scorer Type: TrueFalseInverterScorer\u001b[0m\n",
-      "\u001b[36m      • scorer_type: true_false\u001b[0m\n",
-      "\u001b[36m      • score_aggregator: OR_\u001b[0m\n",
-      "\u001b[36m        └─ Composite of 1 scorer(s):\u001b[0m\n",
-      "\u001b[36m            • Scorer Type: SelfAskRefusalScorer\u001b[0m\n",
-      "\u001b[36m            • scorer_type: true_false\u001b[0m\n",
-      "\u001b[36m            • score_aggregator: OR_\u001b[0m\n",
-      "\u001b[36m            • model_name: gpt-4o-japan-nilfilter\u001b[0m\n",
-      "\n",
-      "\u001b[37m    ▸ Performance Metrics\u001b[0m\n",
-      "\u001b[33m      Official evaluation has not been run yet for this specific configuration\u001b[0m\n",
-      "\n",
-      "\u001b[1m\u001b[36m▼ Overall Statistics\u001b[0m\n",
-      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
-      "\u001b[1m  📈 Summary\u001b[0m\n",
-      "\u001b[32m    • Total Strategies: 3\u001b[0m\n",
-      "\u001b[32m    • Total Attack Results: 24\u001b[0m\n",
-      "\u001b[36m    • Overall Success Rate: 25%\u001b[0m\n",
-      "\u001b[32m    • Unique Objectives: 8\u001b[0m\n",
-      "\n",
-      "\u001b[1m\u001b[36m▼ Per-Group Breakdown\u001b[0m\n",
-      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
-      "\n",
-      "\u001b[1m  🔸 Group: gpt-4o-japan-nilfilter\u001b[0m\n",
-      "\u001b[33m    • Number of Results: 24\u001b[0m\n",
-      "\u001b[36m    • Success Rate: 25%\u001b[0m\n",
-      "\n",
-      "\u001b[36m====================================================================================================\u001b[0m\n",
-      "\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from pyrit.output import output_scenario_async\n",
     "from pyrit.prompt_target import OpenAIChatTarget\n",
+    "from pyrit.scenario import DatasetConfiguration\n",
     "from pyrit.scenario.scenarios.benchmark import AdversarialBenchmark\n",
     "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n",
-    "from pyrit.setup.initializers import LoadDefaultDatasets\n",
+    "from pyrit.setup.initializers import LoadDefaultDatasets, ScorerInitializer, TargetInitializer\n",
     "\n",
-    "await initialize_pyrit_async(memory_db_type=IN_MEMORY, initializers=[LoadDefaultDatasets()])  # type: ignore\n",
-    "\n",
-    "# Pass any number of adversarial PromptTarget instances (with chat-target\n",
-    "# capabilities \u2014 multi-turn and editable history) as a list; AdversarialBenchmark\n",
-    "# infers a label for each from its identifier and runs every benchmark-friendly\n",
-    "# attack technique against the objective target with each adversarial model.\n",
-    "adversarial_model = OpenAIChatTarget()\n",
-    "\n",
-    "benchmark_scenario = AdversarialBenchmark(adversarial_models=[adversarial_model])\n",
-    "\n",
-    "await benchmark_scenario.initialize_async(  # type: ignore\n",
-    "    objective_target=OpenAIChatTarget(), max_concurrency=2\n",
+    "await initialize_pyrit_async(  # type: ignore\n",
+    "    memory_db_type=IN_MEMORY,\n",
+    "    initializers=[TargetInitializer(), ScorerInitializer(), LoadDefaultDatasets()],\n",
     ")\n",
     "\n",
-    "baseline_result = await benchmark_scenario.run_async()  # type: ignore\n",
-    "\n",
-    "# Resume handle: re-run with `AdversarialBenchmark(..., scenario_result_id=<this id>)` to pick\n",
-    "# up where this run left off (constructor args must match the original run).\n",
-    "print(f\"Scenario result id: {baseline_result.id}\")\n",
+    "objective_target = OpenAIChatTarget()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataset_config = DatasetConfiguration(dataset_names=[\"harmbench\"], max_dataset_size=4)\n",
     "\n",
+    "scenario = AdversarialBenchmark()\n",
+    "scenario.set_params_from_args(\n",
+    "    args={\"adversarial_targets\": [\"adversarial_chat_singleturn\", \"adversarial_chat_multiturn\"]}\n",
+    ")\n",
+    "await scenario.initialize_async(  # type: ignore\n",
+    "    objective_target=objective_target,\n",
+    "    dataset_config=dataset_config,\n",
+    ")\n",
     "\n",
-    "await output_scenario_async(baseline_result)"
+    "scenario_result = await scenario.run_async()  # type: ignore"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "await output_scenario_async(scenario_result)"
    ]
   }
  ],
  "metadata": {
   "jupytext": {
    "main_language": "python"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.13.13"
   }
  },
  "nbformat": 4,

doc/scanner/benchmark.py

@@ -11,37 +11,69 @@
 # %% [markdown]
 # # Benchmark Scenarios
 #
-# Benchmark scenarios are a subset of scenarios that compare the effectiveness of attacks across an axis that varies within the scenario itself. The axis can be many things; currently, the only benchmark variant is the adversarial benchmark, whose axis of change is the adversarial model used in attacks.
+# Benchmark scenarios compare attack effectiveness across an axis that varies within the scenario
+# itself. Currently the only benchmark variant is the adversarial benchmark, whose axis of change is
+# the **adversarial chat helper model** used in attacks. For full configuration options see
+# `pyrit_scan --help` and the [Scenarios Programming Guide](../code/scenarios/0_scenarios.ipynb).
 
 # %% [markdown]
 # ## Adversarial Benchmark
-# The adversarial benchmarking scenario (`AdversarialBenchmark`) compares the effectiveness of different adversarial models in successfully executing attacks against a target model.
+#
+# `AdversarialBenchmark` holds the objective target and dataset constant and varies the adversarial
+# chat model used to drive multi-turn attacks. Useful for evaluating which adversarial helper
+# models produce stronger or weaker attack success rates against the same target.
+#
+# Adversarial targets are user-provided via the `adversarial_targets` scenario parameter. Each name
+# must already be registered in `TargetRegistry` — typically by `TargetInitializer` from the
+# `ADVERSARIAL_CHAT_*` env vars (see `.env_example`). Use `pyrit_scan --list-targets` to see every
+# target currently registered.
+#
+# ```bash
+# pyrit_scan benchmark.adversarial \
+#   --initializers target load_default_datasets \
+#   --target openai_chat \
+#   --adversarial-targets adversarial_chat_singleturn adversarial_chat_multiturn \
+#   --max-dataset-size 4
+# ```
+#
+# Pass multiple `--adversarial-targets` values to compare across models in a single run.
+#
+# **Available strategies:** `light` (default — a quick snapshot using the cheaper techniques),
+# `single_turn`, `multi_turn`, plus one member per adversarial-capable source technique
+# (e.g. `red_teaming`, `tap`, `crescendo_simulated`). The `light` aggregate excludes `tap` and
+# `crescendo_simulated`, which can take hours.
+
+# %% [markdown]
+# ## Setup
 
 # %%
 from pyrit.output import output_scenario_async
 from pyrit.prompt_target import OpenAIChatTarget
+from pyrit.scenario import DatasetConfiguration
 from pyrit.scenario.scenarios.benchmark import AdversarialBenchmark
 from pyrit.setup import IN_MEMORY, initialize_pyrit_async
-from pyrit.setup.initializers import LoadDefaultDatasets
+from pyrit.setup.initializers import LoadDefaultDatasets, ScorerInitializer, TargetInitializer
 
-await initialize_pyrit_async(memory_db_type=IN_MEMORY, initializers=[LoadDefaultDatasets()])  # type: ignore
+await initialize_pyrit_async(  # type: ignore
+    memory_db_type=IN_MEMORY,
+    initializers=[TargetInitializer(), ScorerInitializer(), LoadDefaultDatasets()],
+)
 
-# Pass any number of adversarial PromptTarget instances (with chat-target
-# capabilities — multi-turn and editable history) as a list; AdversarialBenchmark
-# infers a label for each from its identifier and runs every benchmark-friendly
-# attack technique against the objective target with each adversarial model.
-adversarial_model = OpenAIChatTarget()
+objective_target = OpenAIChatTarget()
 
-benchmark_scenario = AdversarialBenchmark(adversarial_models=[adversarial_model])
+# %%
+dataset_config = DatasetConfiguration(dataset_names=["harmbench"], max_dataset_size=4)
 
-await benchmark_scenario.initialize_async(  # type: ignore
-    objective_target=OpenAIChatTarget(), max_concurrency=2
+scenario = AdversarialBenchmark()
+scenario.set_params_from_args(
+    args={"adversarial_targets": ["adversarial_chat_singleturn", "adversarial_chat_multiturn"]}
+)
+await scenario.initialize_async(  # type: ignore
+    objective_target=objective_target,
+    dataset_config=dataset_config,
 )
 
-baseline_result = await benchmark_scenario.run_async()  # type: ignore
-
-# Resume handle: re-run with `AdversarialBenchmark(..., scenario_result_id=<this id>)` to pick
-# up where this run left off (constructor args must match the original run).
-print(f"Scenario result id: {baseline_result.id}")
+scenario_result = await scenario.run_async()  # type: ignore
 
-await output_scenario_async(baseline_result)
+# %%
+await output_scenario_async(scenario_result)

pyrit/analytics/__init__.py

@@ -4,7 +4,11 @@
 """Analytics module for PyRIT conversation and result analysis."""
 
 from pyrit.analytics.conversation_analytics import ConversationAnalytics
-from pyrit.analytics.result_analysis import AttackStats, analyze_results
+from pyrit.analytics.result_analysis import (
+    AttackStats,
+    analyze_results,
+    get_cached_results_for_technique,
+)
 from pyrit.analytics.text_matching import (
     ApproximateTextMatching,
     ExactTextMatching,
@@ -17,5 +21,6 @@
     "AttackStats",
     "ConversationAnalytics",
     "ExactTextMatching",
+    "get_cached_results_for_technique",
     "TextMatching",
 ]