Merge branch 'main' into users/evekazarian/split-payload-attack-strategy

Author: ek0212

.github/instructions/output.instructions.md

@@ -0,0 +1,67 @@
+---
+applyTo: "pyrit/output/**"
+---
+
+# PyRIT Output Module — Coding & Review Guidelines
+
+For full architecture documentation, usage examples, and extension guides, see [doc/code/output/0_output.py](../../../doc/code/output/0_output.py).
+
+This file covers the rules for **writing and reviewing** code in `pyrit/output/`.
+
+## Critical Rules
+
+### Output goes through the sink — never call `print()` directly
+
+All rendering methods return `str`. The inherited `write_async` calls `render_async` then `_write_async(content)`. No bare `print()` calls anywhere in the output module except inside `StdoutSink`.
+
+When reviewing: reject any `print()` call outside `StdoutSink`.
+
+### Data fetching belongs in leaf classes only
+
+Format classes (`PrettyAttackResultPrinter`, `MarkdownAttackResultPrinter`) must not import or reference `CentralMemory`. Only `*MemoryPrinter` leaf classes do data I/O.
+
+When reviewing: reject any `CentralMemory` import in a non-leaf file (`pretty.py`, `markdown.py`, `json.py`).
+
+### Sinks must use async I/O
+
+Sink implementations must not block the event loop. Use `asyncio.to_thread()` or native async libraries for I/O operations. `FileSink` uses an `asyncio.Lock` to prevent concurrent write races.
+
+When reviewing: reject synchronous `open()`, `write()`, or network calls inside a sink's `write_async`.
+
+## Three-Layer Hierarchy
+
+Every domain follows this structure. Do not mix responsibilities across layers.
+
+| Layer | File | Responsibility | May import CentralMemory? |
+|-------|------|---------------|---------------------------|
+| **Base** | `base.py` | Abstract data-fetching methods + abstract `render_async` | No |
+| **Format** | `pretty.py`, `markdown.py`, `json.py` | Implements `render_async`, returns `str` | No |
+| **Leaf** | Same file as format (e.g., `PrettyAttackResultMemoryPrinter`) | Implements data methods via CentralMemory; forwarding `render_async` | Yes |
+
+### File names = output format
+
+- `pretty.py` — ANSI-colored human-readable
+- `markdown.py` — Markdown
+- `json.py` — structured JSON
+
+### Memory leaf classes must work with zero args
+
+```python
+printer = PrettyAttackResultMemoryPrinter()  # defaults: StdoutSink, matching sub-printers
+await printer.write_async(result)
+```
+
+Pass `sink=` to redirect output. Pass sub-printers only to override defaults.
+
+### Convenience functions live in `helpers.py`
+
+Every new domain printer **must** have a corresponding convenience function added to `helpers.py`. This is the primary entry point most callers use.
+
+```python
+from pyrit.output.helpers import output_attack_async
+await output_attack_async(result, format="pretty")
+```
+
+`helpers.py` resolves `format` → printer class, `sink` → Sink, and calls `write_async`.
+
+When reviewing: if a new domain printer is added without a helper function, request one.

.github/instructions/scenarios.instructions.md

@@ -11,15 +11,15 @@ Scenarios orchestrate multi-attack security testing campaigns. Each scenario gro
 All scenarios inherit from `Scenario` (ABC) and must:
 
 1. **Define `VERSION`** as a class constant (increment on breaking changes)
-2. **Optionally declare `BASELINE_POLICY`** (defaults to `BaselinePolicy.Enabled` — a baseline `PromptSendingAttack` is prepended and callers can opt out per run via `initialize_async(include_baseline=False)`):
-   - `BaselinePolicy.Disabled` — baseline supported but off by default (e.g. `Jailbreak`, where templates dominate the run).
-   - `BaselinePolicy.Forbidden` — baseline is meaningless for this scenario's comparison axis (e.g. `AdversarialBenchmark`, which compares against gold-standard answers). Explicit `include_baseline=True` raises `ValueError`.
+2. **Optionally declare `BASELINE_ATTACK_POLICY`** (defaults to `BaselineAttackPolicy.Enabled` — a baseline `PromptSendingAttack` is prepended and callers can opt out per run via `initialize_async(include_baseline=False)`):
+   - `BaselineAttackPolicy.Disabled` — baseline supported but off by default (e.g. `Jailbreak`, where templates dominate the run).
+   - `BaselineAttackPolicy.Forbidden` — baseline is meaningless for this scenario's comparison axis (e.g. `AdversarialBenchmark`, which compares against gold-standard answers). Explicit `include_baseline=True` raises `ValueError`.
 3. **Implement three abstract methods:**
 
 ```python
 class MyScenario(Scenario):
     VERSION: int = 1
-    BASELINE_POLICY: ClassVar[BaselinePolicy] = BaselinePolicy.Enabled
+    BASELINE_ATTACK_POLICY: ClassVar[BaselineAttackPolicy] = BaselineAttackPolicy.Enabled
 
     @classmethod
     def get_strategy_class(cls) -> type[ScenarioStrategy]:

.github/instructions/style-guide.instructions.md

@@ -293,6 +293,26 @@ def process_items(self, *, items: list[str]) -> list[str]:
         return []
 ```
 
+## Deprecations
+
+When deprecating a public class, function, method, parameter, or module path, use `pyrit.common.deprecation.print_deprecation_message` — never `warnings.warn` directly. It wraps `warnings.warn(..., DeprecationWarning, stacklevel=3)` with a consistent format so filtering still works.
+
+Set `removed_in` to **current version + 2 minor versions** (e.g. `0.14.x` → `removed_in="0.16.0"`). This gives one full release cycle of warning before removal.
+
+```python
+from pyrit.common.deprecation import print_deprecation_message
+
+def old_method(self, *, foo: str) -> None:
+    print_deprecation_message(
+        old_item="MyClass.old_method",
+        new_item="MyClass.new_method",
+        removed_in="0.16.0",
+    )
+    ...
+```
+
+`old_item` / `new_item` accept a class/callable (qualified name is generated) or a string.
+
 ## Pythonic Patterns
 
 ### List Comprehensions

.github/workflows/docs.yml

@@ -55,10 +55,24 @@ jobs:
     - name: Install PyRIT with uv
       run: uv sync --extra all
 
-    # Build the book
+    # LaTeX toolchain needed for the PDF export (jupyter-book --pdf via xelatex).
+    # Mirrors the ReadTheDocs build so CI catches PDF-only issues (e.g. broken
+    # image paths) instead of letting them silently break the RTD build.
+    - name: Install LaTeX (for PDF export)
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y --no-install-recommends \
+          texlive-xetex \
+          texlive-fonts-recommended \
+          texlive-fonts-extra \
+          texlive-plain-generic \
+          texlive-latex-extra \
+          latexmk
+
+    # Build the book (HTML site + PDF export)
     - name: Build the book
       run: |
-        make docs-build
+        make docs-build-all
     # Upload the book's HTML as an artifact
     - name: Upload artifact
       uses: actions/upload-pages-artifact@v3

.pyrit_conf_example

@@ -117,6 +117,20 @@ operation: op_trash_panda
 # Applies only to the pyrit_backend server.
 max_concurrent_scenario_runs: 3
 
+# Custom Initializer Registration (REST API)
+# -------------------------------------------
+# When true, the REST API accepts POST /api/initializers to register custom
+# initializer scripts and DELETE /api/initializers/{name} to remove custom
+# initializers.
+#
+# ⚠️  WARNING: Enabling this allows arbitrary Python code execution on the
+# server via the REST API. Only enable on trusted networks.
+# The pyrit_backend default host is localhost, which limits exposure.
+# If you bind to 0.0.0.0, ensure you are on a trusted network.
+#
+# Default: false
+allow_custom_initializers: false
+
 # Silent Mode
 # -----------
 # If true, suppresses print statements during initialization.

Makefile

@@ -20,14 +20,25 @@ ty:
 # Build the full documentation site:
 # 1. Generate API reference JSON from Python source (griffe)
 # 2. Convert API JSON to MyST markdown pages
-# 3. Build the Jupyter Book site
+# 3. Build the Jupyter Book site (HTML only — fast, no LaTeX needed)
 # 4. Generate RSS feed
 docs-build:
 	uv run python build_scripts/pydoc2json.py pyrit --submodules -o doc/_api/pyrit_all.json
 	uv run python build_scripts/gen_api_md.py
 	cd doc && uv run jupyter-book build --all --html
 	uv run ./build_scripts/generate_rss.py
 
+# Build the full documentation site including the PDF export.
+# Mirrors the ReadTheDocs build (.readthedocs.yaml) so CI catches PDF-only issues
+# such as missing images that the HTML-only build silently ignores.
+# Requires xelatex / latexmk on PATH (texlive-xetex + texlive-fonts-recommended +
+# texlive-plain-generic + latexmk on Ubuntu).
+docs-build-all:
+	uv run python build_scripts/pydoc2json.py pyrit --submodules -o doc/_api/pyrit_all.json
+	uv run python build_scripts/gen_api_md.py
+	cd doc && uv run jupyter-book build --all --html --pdf
+	uv run ./build_scripts/generate_rss.py
+
 # Regenerate only the API reference pages (without building the full site)
 docs-api:
 	uv run python build_scripts/pydoc2json.py pyrit --submodules -o doc/_api/pyrit_all.json

doc/bibliography.md

@@ -5,6 +5,6 @@ All academic papers, research blogs, and technical reports referenced throughout
 :::{dropdown} Citation Keys
 :class: hidden-citations
 
-[@aakanksha2024multilingual; @adversaai2023universal; @andriushchenko2024tense; @anthropic2024manyshot; @aqrawi2024singleturncrescendo; @bethany2024mathprompt; @bryan2025agentictaxonomy; @bullwinkel2024airtlessons; @bullwinkel2025repeng; @bullwinkel2026trigger; @chao2023pair; @chao2024jailbreakbench; @chu2023harmfulqa; @cui2024orbench; @darkbench2025; @derczynski2024garak; @ding2023wolf; @embracethered2024unicode; @embracethered2025sneakybits; @ghosh2025aegis; @haider2024phi3safety; @han2024medsafetybench; @hines2024spotlighting; @ji2023beavertails; @ji2024pkusaferlhf; @jiang2025sosbench; @jones2025computeruse; @kingma2014adam; @li2024flipattack; @li2024saladbench; @li2024wmdp; @lin2023toxicchat; @lopez2024pyrit; @lv2024codechameleon; @mazeika2023tdc; @mazeika2024harmbench; @mckee2024transparency; @mehrotra2023tap; @microsoft2024skeletonkey; @palaskar2025vlsu; @pfohl2024equitymedqa; @promptfoo2025ccp; @robustintelligence2024bypass; @roccia2024promptintel; @rottger2023xstest; @russinovich2024crescendo; @russinovich2025price; @scheuerman2025transphobia; @shayegani2025computeruse; @shen2023donotanything; @sheshadri2024lat; @stok2023ansi; @tang2025multilingual; @tedeschi2024alert; @vantaylor2024socialbias; @vidgen2023simplesafetytests; @vidgen2024ailuminate; @wang2023decodingtrust; @wang2023donotanswer; @wei2023jailbroken; @xie2024sorrybench; @yu2023gptfuzzer; @yuan2023cipherchat; @zeng2024persuasion; @zhang2024cbtbench; @zou2023gcg]
+[@aakanksha2024multilingual; @adversaai2023universal; @andriushchenko2024tense; @anthropic2024manyshot; @aqrawi2024singleturncrescendo; @bethany2024mathprompt; @bhardwaj2023harmfulqa; @bryan2025agentictaxonomy; @bullwinkel2025airtlessons; @bullwinkel2025repeng; @bullwinkel2026trigger; @chao2023pair; @chao2024jailbreakbench; @cui2024orbench; @darkbench2025; @derczynski2024garak; @ding2023wolf; @embracethered2024unicode; @embracethered2025sneakybits; @ghosh2025aegis; @haider2024phi3safety; @han2024medsafetybench; @hines2024spotlighting; @ji2023beavertails; @ji2024pkusaferlhf; @jiang2025sosbench; @jones2025computeruse; @kingma2014adam; @li2024saladbench; @li2024wmdp; @lin2023toxicchat; @liu2024flipattack; @lopez2024pyrit; @lv2024codechameleon; @mazeika2023tdc; @mazeika2024harmbench; @mckee2024transparency; @mehrotra2023tap; @microsoft2024skeletonkey; @palaskar2025vlsu; @pfohl2024equitymedqa; @promptfoo2025ccp; @robustintelligence2024bypass; @roccia2024promptintel; @rottger2023xstest; @russinovich2024crescendo; @russinovich2025price; @scheuerman2025transphobia; @shayegani2025computeruse; @shen2023donotanything; @sheshadri2024lat; @stok2023ansi; @tan2026comicjailbreak; @tang2025multilingual; @tedeschi2024alert; @vantaylor2024socialbias; @vidgen2023simplesafetytests; @vidgen2024ailuminate; @wang2023decodingtrust; @wang2023donotanswer; @wei2023jailbroken; @xie2024sorrybench; @yu2023gptfuzzer; @yuan2023cipherchat; @zeng2024persuasion; @zhang2024cbtbench; @zou2023gcg]
 
 :::

doc/code/auxiliary_attacks/0_auxiliary_attacks.ipynb

@@ -64,7 +64,6 @@
    "source": [
     "from pyrit.executor.attack import (\n",
     "    AttackScoringConfig,\n",
-    "    ConsoleAttackResultPrinter,\n",
     "    PromptSendingAttack,\n",
     ")\n",
     "from pyrit.prompt_target import OpenAIChatTarget\n",
@@ -83,8 +82,7 @@
     "attack = PromptSendingAttack(objective_target=target, attack_scoring_config=scoring_config)\n",
     "result = await attack.execute_async(objective=objective)  # type: ignore\n",
     "\n",
-    "printer = ConsoleAttackResultPrinter()\n",
-    "await printer.print_conversation_async(result=result)  # type: ignore"
+    "await output_attack_async(result)"
    ]
   },
   {
@@ -180,7 +178,7 @@
     ")\n",
     "\n",
     "result = await attack.execute_async(objective=objective)  # type: ignore\n",
-    "await printer.print_result_async(result=result)  # type: ignore"
+    "await output_attack_async(result)"
    ]
   }
  ],

doc/code/auxiliary_attacks/0_auxiliary_attacks.py

@@ -8,29 +8,24 @@
 #       format_version: '1.3'
 #       jupytext_version: 1.17.3
 # ---
-
 # %% [markdown]
 # # Auxiliary Attacks
-
 # %% [markdown]
 # Auxiliary attacks cover a variety of techniques that do not fit into the core PyRIT functionality.
 #
 # These attack pipelines may be useful to run before orchestrating other attacks. For example, we provide an Azure Machine Learning (AML) pipeline for generating suffixes using the greedy coordinate gradient (GCG) [@zou2023gcg] algorithm.
-
 # %% [markdown]
 # ## GCG Suffixes
-
 # %% [markdown]
 # The [GCG demo notebook](1_gcg_azure_ml.ipynb) shows how to create an AML environment and submit a job that generates GCG suffixes, which can be appended to a base prompt to jailbreak a language model. In the example below, we compare the response generated by Phi-3-mini with and without a GCG suffix trained on that model.
 #
 # First, we send a harmful prompt to Phi-3-mini without a GCG suffix. If the environment variables `PHI3_MINI_ENDPOINT` and `PHI3_MINI_KEY` are not set in your .env file, the target will default to the model with `AZURE_ML_MANAGED_ENDPOINT` and `AZURE_ML_MANAGED_KEY`.
-
 # %%
 from pyrit.executor.attack import (
     AttackScoringConfig,
-    ConsoleAttackResultPrinter,
     PromptSendingAttack,
 )
+from pyrit.output import output_attack_async
 from pyrit.prompt_target import OpenAIChatTarget
 from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer
 from pyrit.setup import IN_MEMORY, initialize_pyrit_async
@@ -47,8 +42,7 @@
 attack = PromptSendingAttack(objective_target=target, attack_scoring_config=scoring_config)
 result = await attack.execute_async(objective=objective)  # type: ignore
 
-printer = ConsoleAttackResultPrinter()
-await printer.print_conversation_async(result=result)  # type: ignore
+await output_attack_async(result)
 
 # %% [markdown]
 # Next, let's apply a GCG suffix trained on Phi-3-mini to the base prompt using the `SuffixAppendConverter`.
@@ -73,4 +67,4 @@
 )
 
 result = await attack.execute_async(objective=objective)  # type: ignore
-await printer.print_result_async(result=result)  # type: ignore
+await output_attack_async(result)

doc/code/converters/0_converters.ipynb

@@ -279,7 +279,6 @@
    "source": [
     "from pyrit.executor.attack import (\n",
     "    AttackConverterConfig,\n",
-    "    ConsoleAttackResultPrinter,\n",
     "    PromptSendingAttack,\n",
     ")\n",
     "from pyrit.prompt_converter import StringJoinConverter, VariationConverter\n",
@@ -306,8 +305,7 @@
     "\n",
     "result = await attack.execute_async(objective=objective)  # type: ignore\n",
     "\n",
-    "printer = ConsoleAttackResultPrinter()\n",
-    "await printer.print_conversation_async(result=result)  # type: ignore"
+    "await output_attack_async(result)"
    ]
   },
   {
@@ -370,7 +368,6 @@
    "source": [
     "from pyrit.executor.attack import (\n",
     "    AttackConverterConfig,\n",
-    "    ConsoleAttackResultPrinter,\n",
     "    PromptSendingAttack,\n",
     ")\n",
     "from pyrit.prompt_converter import TranslationConverter\n",
@@ -407,8 +404,7 @@
     "result = await attack.execute_async(objective=objective)  # type: ignore\n",
     "\n",
     "# Print the conversation showing both original and converted values\n",
-    "printer = ConsoleAttackResultPrinter()\n",
-    "await printer.print_conversation_async(result=result)  # type: ignore"
+    "await output_attack_async(result)"
    ]
   }
  ],