Skip to content

Commit 38359ce

Browse files
FEAT deprecate prompt chat (microsoft#1678)
1 parent 957f4e7 commit 38359ce

98 files changed

Lines changed: 1295 additions & 410 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/instructions/converters.instructions.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ from pyrit.identifiers import ComponentIdentifier
6565

6666
For LLM-based converters, also import:
6767
```python
68-
from pyrit.prompt_target import PromptChatTarget
68+
from pyrit.prompt_target import PromptTarget
6969
```
7070

7171
## Constructor Pattern
@@ -77,7 +77,7 @@ from pyrit.common.apply_defaults import apply_defaults
7777

7878
class MyConverter(PromptConverter):
7979
@apply_defaults
80-
def __init__(self, *, target: PromptChatTarget, template: str = "default") -> None:
80+
def __init__(self, *, target: PromptTarget, template: str = "default") -> None:
8181
...
8282
```
8383

.github/instructions/scenarios.instructions.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ class MyScenario(Scenario):
4141
def __init__(
4242
self,
4343
*,
44-
adversarial_chat: PromptChatTarget | None = None,
44+
adversarial_chat: PromptTarget | None = None,
4545
objective_scorer: TrueFalseScorer | None = None,
4646
scenario_result_id: str | None = None,
4747
) -> None:

doc/code/datasets/5_simulated_conversation.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -544,7 +544,7 @@
544544
"| Parameter | Type | Description |\n",
545545
"|-----------|------|-------------|\n",
546546
"| `objective` | `str` | The goal the adversarial chat works toward |\n",
547-
"| `adversarial_chat` | `PromptChatTarget` | The LLM that generates attack prompts (also plays the simulated target) |\n",
547+
"| `adversarial_chat` | `PromptTarget` | The LLM that generates attack prompts (also plays the simulated target). Must declare `supports_multi_turn=True` and `supports_editable_history=True`. |\n",
548548
"| `objective_scorer` | `TrueFalseScorer` | Evaluates whether the final turn achieved the objective |\n",
549549
"| `num_turns` | `int` | Number of conversation turns to generate (default: 3) |\n",
550550
"| `adversarial_chat_system_prompt_path` | `str \\| Path` | System prompt for the adversarial chat role |\n",

doc/code/datasets/5_simulated_conversation.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@
125125
# | Parameter | Type | Description |
126126
# |-----------|------|-------------|
127127
# | `objective` | `str` | The goal the adversarial chat works toward |
128-
# | `adversarial_chat` | `PromptChatTarget` | The LLM that generates attack prompts (also plays the simulated target) |
128+
# | `adversarial_chat` | `PromptTarget` | The LLM that generates attack prompts (also plays the simulated target). Must declare `supports_multi_turn=True` and `supports_editable_history=True`. |
129129
# | `objective_scorer` | `TrueFalseScorer` | Evaluates whether the final turn achieved the objective |
130130
# | `num_turns` | `int` | Number of conversation turns to generate (default: 3) |
131131
# | `adversarial_chat_system_prompt_path` | `str \| Path` | System prompt for the adversarial chat role |

doc/code/executor/attack/2_red_teaming_attack.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1456,7 +1456,7 @@
14561456
"source": [
14571457
"## Other Multi-Turn Attacks\n",
14581458
"\n",
1459-
"The above examples should work using other multi-turn attacks with minimal modification. Check out attacks under `pyrit.executor.attack.multi_turn` for other examples, like Crescendo and Tree of Attacks. These algorithms are always more effective than `RedTeamingAttack`, which is a simple algorithm. However, `RedTeamingAttack` by its nature supports more targets - because it doesn't modify conversation history it can support any `PromptTarget` and not only `PromptChatTargets`."
1459+
" \"The above examples should work using other multi-turn attacks with minimal modification. Check out attacks under `pyrit.executor.attack.multi_turn` for other examples, like Crescendo and Tree of Attacks. These algorithms are always more effective than `RedTeamingAttack`, which is a simple algorithm. However, `RedTeamingAttack` by its nature supports more targets - because it doesn't modify conversation history it can support any `PromptTarget`, not only chat-style targets that declare `supports_multi_turn=True` and `supports_editable_history=True`.\""
14601460
]
14611461
}
14621462
],

doc/code/executor/attack/2_red_teaming_attack.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -314,4 +314,4 @@
314314
# %% [markdown]
315315
# ## Other Multi-Turn Attacks
316316
#
317-
# The above examples should work using other multi-turn attacks with minimal modification. Check out attacks under `pyrit.executor.attack.multi_turn` for other examples, like Crescendo and Tree of Attacks. These algorithms are always more effective than `RedTeamingAttack`, which is a simple algorithm. However, `RedTeamingAttack` by its nature supports more targets - because it doesn't modify conversation history it can support any `PromptTarget` and not only `PromptChatTargets`.
317+
# The above examples should work using other multi-turn attacks with minimal modification. Check out attacks under `pyrit.executor.attack.multi_turn` for other examples, like Crescendo and Tree of Attacks. These algorithms are always more effective than `RedTeamingAttack`, which is a simple algorithm. However, `RedTeamingAttack` by its nature supports more targets - because it doesn't modify conversation history it can support any `PromptTarget`, not only chat-style targets that declare `supports_multi_turn=True` and `supports_editable_history=True`.

doc/code/setup/default_values.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ from pyrit.common.apply_defaults import apply_defaults
2323

2424
class MyConverter(PromptConverter):
2525
@apply_defaults
26-
def __init__(self, *, converter_target: Optional[PromptChatTarget] = None, temperature: Optional[float] = None):
26+
def __init__(self, *, converter_target: Optional[PromptTarget] = None, temperature: Optional[float] = None):
2727
self.converter_target = converter_target
2828
self.temperature = temperature
2929
```

doc/code/targets/0_prompt_targets.md

Lines changed: 81 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,20 +19,93 @@ async def send_prompt_async(self, *, message: Message) -> Message:
1919

2020
A `Message` object is a normalized object with all the information a target will need to send a prompt, including a way to get a history for that prompt (in the cases that also needs to be sent). This is discussed in more depth [here](../memory/3_memory_data_types.md).
2121

22-
## PromptChatTargets vs PromptTargets
22+
## Chat-style targets vs general targets
2323

2424
A `PromptTarget` is a generic place to send a prompt. With PyRIT, the idea is that it will eventually be consumed by an AI application, but that doesn't have to be immediate. For example, you could have a SharePoint target. Everything you send a prompt to is a `PromptTarget`. Many attacks work generically with any `PromptTarget` including `RedTeamingAttack` and `PromptSendingAttack`.
2525

26-
With some algorithms, you want to send a prompt, set a system prompt, and modify conversation history (including PAIR [@chao2023pair], TAP [@mehrotra2023tap], and flip attack [@li2024flipattack]). These often require a `PromptChatTarget`, which implies you can modify a conversation history. `PromptChatTarget` is a subclass of `PromptTarget`.
26+
With some algorithms, you want to send a prompt, set a system prompt, and modify conversation history (including PAIR [@chao2023pair], TAP [@mehrotra2023tap], and flip attack [@li2024flipattack]). These algorithms require a target whose [`TargetCapabilities`](#target-capabilities) declare both `supports_multi_turn=True` and `supports_editable_history=True` — i.e. you can modify a conversation history. Consumers express this requirement via `CHAT_TARGET_REQUIREMENTS` and validate it against `target.configuration` at construction time. See [Target Capabilities](#target-capabilities) below for the full list of capabilities and how they compose into a `TargetConfiguration`.
27+
28+
Note: The previous `PromptChatTarget` class is **deprecated** as of v0.13.0 and will be removed in v0.15.0. Use `PromptTarget` directly with a `TargetConfiguration` declaring `supports_multi_turn=True` and `supports_editable_history=True`. See [Target Capabilities](#target-capabilities) for details.
29+
2730

2831
Here are some examples:
2932

30-
| Example | Is `PromptChatTarget`? | Notes |
31-
|-------------------------------------|---------------------------------------|-------------------------------------------------------------------------------------------------|
32-
| **OpenAIChatTarget** (e.g., GPT-4) | **Yes** (`PromptChatTarget`) | Designed for conversational prompts (system messages, conversation history, etc.). |
33-
| **OpenAIImageTarget** | **No** (not a `PromptChatTarget`) | Used for image generation; does not manage conversation history. |
34-
| **HTTPTarget** | **No** (not a `PromptChatTarget`) | Generic HTTP target. Some apps might allow conversation history, but this target doesn't handle it. |
35-
| **AzureBlobStorageTarget** | **No** (not a `PromptChatTarget`) | Used primarily for storage; not for conversation-based AI. |
33+
| Example | Chat-style target? | Notes |
34+
|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------------------------|
35+
| **OpenAIChatTarget** (e.g., GPT-4) | **Yes** (multi-turn + editable history) | Designed for conversational prompts (system messages, conversation history, etc.). |
36+
| **OpenAIImageTarget** | **No** | Used for image generation; does not manage conversation history. |
37+
| **HTTPTarget** | **No** | Generic HTTP target. Some apps might allow conversation history, but this target doesn't handle it. |
38+
| **AzureBlobStorageTarget** | **No** | Used primarily for storage; not for conversation-based AI. |
39+
40+
## Target Capabilities
41+
42+
Every `PromptTarget` exposes a `TargetConfiguration` (via `target.configuration`) that declares what the target natively supports. This lets attacks, converters, and scorers reason about whether a given target is suitable for a given workflow — and, where possible, adapt automatically when a capability is missing.
43+
44+
A `TargetConfiguration` composes three concerns:
45+
46+
- **`TargetCapabilities`** — an immutable, declarative description of what the target natively supports.
47+
- **`CapabilityHandlingPolicy`** — for each capability that *can* be adapted, whether to `ADAPT` (apply a normalization step to work around the gap) or `RAISE` (fail immediately).
48+
- **`ConversationNormalizationPipeline`** — the ordered set of normalizers derived from the gap between the declared capabilities and the policy.
49+
50+
### Capabilities
51+
52+
`TargetCapabilities` declares the following flags:
53+
54+
| Capability | Meaning |
55+
|----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
56+
| `supports_multi_turn` | The target accepts and uses conversation history (or maintains state externally, e.g. via a WebSocket). |
57+
| `supports_multi_message_pieces` | The target accepts more than one `MessagePiece` in a single request (e.g. text + image in one message). |
58+
| `supports_editable_history` | The conversation history can be modified after the fact. Implies `supports_multi_turn`. Required for attacks that rewrite prior turns. |
59+
| `supports_system_prompt` | The target natively supports a system-role message. |
60+
| `supports_json_output` | The target supports a "json" response format that guarantees valid JSON output. |
61+
| `supports_json_schema` | The target supports constraining output to a caller-provided JSON schema. |
62+
| `input_modalities` | The set of input modality combinations the target accepts (e.g. `{text}`, `{text, image_path}`). |
63+
| `output_modalities` | The set of output modality combinations the target produces. |
64+
65+
Each target class defines defaults; instances can override individual capabilities when they depend on deployment configuration (e.g. `HTTPTarget`, `PlaywrightTarget`).
66+
67+
For well-known underlying models, you can look up a profile with `TargetCapabilities.get_known_capabilities(underlying_model="gpt-4o")`.
68+
69+
### How consumers use capabilities
70+
71+
Components that need a particular capability declare it as a `TargetRequirements` and validate at construction time:
72+
73+
```python
74+
from pyrit.prompt_target import CHAT_TARGET_REQUIREMENTS
75+
76+
CHAT_TARGET_REQUIREMENTS.validate(target=target)
77+
```
78+
79+
`TargetRequirements.validate` collects every missing capability and raises a single `ValueError`. For one-off checks against a single capability you can also call `target.configuration.ensure_can_handle(capability=...)` directly.
80+
81+
### Adapting vs raising
82+
83+
Some capability gaps can be papered over by PyRIT itself. For example, a single-turn target can be made to *appear* multi-turn by flattening the conversation history into a single prompt before sending. The `CapabilityHandlingPolicy` controls this on a per-capability basis:
84+
85+
- `UnsupportedCapabilityBehavior.RAISE` — fail at construction time. This is the safe default.
86+
- `UnsupportedCapabilityBehavior.ADAPT` — run the corresponding normalizer in the conversation pipeline before the target sees the messages.
87+
88+
Non-adaptable capabilities (e.g. `supports_editable_history`) are not represented in the policy at all; requesting them on a target that lacks them always raises.
89+
90+
### Overriding capabilities per instance
91+
92+
For targets whose capabilities depend on deployment (HTTP endpoints, Playwright-driven UIs, custom backends), pass a `TargetConfiguration` to the constructor:
93+
94+
```python
95+
from pyrit.prompt_target.common.target_capabilities import TargetCapabilities
96+
from pyrit.prompt_target.common.target_configuration import TargetConfiguration
97+
98+
config = TargetConfiguration(
99+
capabilities=TargetCapabilities(
100+
supports_multi_turn=True,
101+
supports_editable_history=True,
102+
supports_system_prompt=True,
103+
),
104+
)
105+
target = MyHTTPTarget(custom_configuration=config, ...)
106+
```
107+
108+
The full implementation lives in [`pyrit/prompt_target/common/target_capabilities.py`](https://github.com/microsoft/PyRIT/blob/main/pyrit/prompt_target/common/target_capabilities.py) and [`pyrit/prompt_target/common/target_configuration.py`](https://github.com/microsoft/PyRIT/blob/main/pyrit/prompt_target/common/target_configuration.py). For runnable examples — inspecting capabilities on a real target, comparing known model profiles, and `ADAPT` vs `RAISE` in action — see [Target Capabilities](./6_1_target_capabilities.ipynb).
36109

37110
## Multi-Modal Targets
38111

doc/code/targets/10_3_websocket_copilot_target.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@
8282
"\n",
8383
"The `WebSocketCopilotTarget` supports multi-turn conversations by leveraging Copilot's server-side conversation management. It automatically generates consistent `session_id` and `conversation_id` values for each PyRIT conversation, enabling Copilot to maintain context across multiple turns.\n",
8484
"\n",
85-
"However, this target does not support setting a system prompt nor modifying conversation history. As a result, it cannot be used with attack strategies that require altering prior messages (such as PAIR, TAP, or flip attack) or in contexts where a `PromptChatTarget` is required.\n",
85+
" \"However, this target does not support setting a system prompt nor modifying conversation history. As a result, it cannot be used with attack strategies that require altering prior messages (such as PAIR, TAP, or flip attack) or in contexts where a chat-style target (one that declares `supports_multi_turn=True` and `supports_editable_history=True`) is required.\\n\",\n",
8686
"\n",
8787
"Here is a simple multi-turn conversation example:"
8888
]

doc/code/targets/10_3_websocket_copilot_target.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@
4545
#
4646
# The `WebSocketCopilotTarget` supports multi-turn conversations by leveraging Copilot's server-side conversation management. It automatically generates consistent `session_id` and `conversation_id` values for each PyRIT conversation, enabling Copilot to maintain context across multiple turns.
4747
#
48-
# However, this target does not support setting a system prompt nor modifying conversation history. As a result, it cannot be used with attack strategies that require altering prior messages (such as PAIR, TAP, or flip attack) or in contexts where a `PromptChatTarget` is required.
48+
# However, this target does not support setting a system prompt nor modifying conversation history. As a result, it cannot be used with attack strategies that require altering prior messages (such as PAIR, TAP, or flip attack) or in contexts where a chat-style target (one that declares `supports_multi_turn=True` and `supports_editable_history=True`) is required.
4949
#
5050
# Here is a simple multi-turn conversation example:
5151

0 commit comments

Comments
 (0)