Skip to content

Commit d50caf0

Browse files
rlundeen2Copilot
andauthored
DOC: Scoring Docs Refactor (#1892)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 46d52b1 commit d50caf0

45 files changed

Lines changed: 2201 additions & 3710 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

doc/blog/2025_01_27.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ When examining this request, you may discover that occasionally the Adversarial
8282

8383
[^8]: "PyRIT - SearchReplaceConverter", ../api/pyrit_prompt_converter.md#searchreplaceconverter
8484

85-
[^9]: "PyRIT - True False Scoring", ../code/scoring/2_true_false_scorers.ipynb#true-false-scoring
85+
[^9]: "PyRIT - True False Scoring", ../code/scoring/1_true_false_scorers.ipynb
8686

8787
### Final Thoughts
8888

doc/blog/2026_04_14_scoring_scorers.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ flowchart TB
108108

109109
There are a few different ways to view metrics for specific scoring configurations.
110110

111-
**Directly on a scorer instance:** Call `get_scorer_metrics()` on any scorer object to look up its saved metrics (if they exist), as described at the bottom of the [Scorer Evaluation Identifier](#scorer-evaluation-identifier) section above. See the [scorer metrics notebook](../code/scoring/7_scorer_metrics.ipynb) to try it yourself!
111+
**Directly on a scorer instance:** Call `get_scorer_metrics()` on any scorer object to look up its saved metrics (if they exist), as described at the bottom of the [Scorer Evaluation Identifier](#scorer-evaluation-identifier) section above. See the [scorer metrics notebook](../code/scoring/4_scorer_metrics.ipynb) to try it yourself!
112112

113113
**Automatically in scenario output:** When running scenarios and printing results (i.e., in [pyrit_scan](../scanner/1_pyrit_scan.ipynb) or [pyrit_shell](../scanner/2_pyrit_shell.md)), metrics are automatically fetched and displayed alongside the attack results (as long as the scoring configuration has been evaluated before):
114114

@@ -132,7 +132,7 @@ The framework checks the JSONL registry for an existing entry matching the score
132132

133133
![alt text](2026_04_14_running_evaluation.png)
134134

135-
For the full walkthrough — including running objective and harm evaluations, configuring custom datasets, and comparing results — give the [scorer metrics notebook](../code/scoring/7_scorer_metrics.ipynb) a try!
135+
For the full walkthrough — including running objective and harm evaluations, configuring custom datasets, and comparing results — give the [scorer metrics notebook](../code/scoring/4_scorer_metrics.ipynb) a try!
136136

137137
## Closing Thoughts
138138

doc/code/framework.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ Ways to contribute: Check out our [target docs](./targets/0_prompt_targets.md).
108108

109109
The scoring engine is a component that gives feedback to the attack on what happened with the prompt. This could be as simple as "Was this prompt blocked?" or "Was our objective achieved?"
110110

111-
Ways to contribute: Check out our [scoring docs](./scoring/0_scoring.md). Is there data you want to use to make decisions or analyze?
111+
Ways to contribute: Check out our [scoring docs](./scoring/0_scoring.ipynb). Is there data you want to use to make decisions or analyze?
112112

113113
## Memory
114114

doc/code/memory/5_advanced_memory.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@
172172
"id": "2",
173173
"metadata": {},
174174
"source": [
175-
"Because you have labeled `group1`, you can retrieve these prompts later. For example, you could score them as shown [here](../scoring/6_batch_scorer.ipynb). Or you could resend them as shown below; this script will resend any prompts with the label regardless of modality."
175+
"Because you have labeled `group1`, you can retrieve these prompts later. For example, you could score them as shown [here](../scoring/0_scoring.ipynb#batch-scoring). Or you could resend them as shown below; this script will resend any prompts with the label regardless of modality."
176176
]
177177
},
178178
{

doc/code/memory/5_advanced_memory.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@
6363
await output_attack_async(result)
6464

6565
# %% [markdown]
66-
# Because you have labeled `group1`, you can retrieve these prompts later. For example, you could score them as shown [here](../scoring/6_batch_scorer.ipynb). Or you could resend them as shown below; this script will resend any prompts with the label regardless of modality.
66+
# Because you have labeled `group1`, you can retrieve these prompts later. For example, you could score them as shown [here](../scoring/0_scoring.ipynb#batch-scoring). Or you could resend them as shown below; this script will resend any prompts with the label regardless of modality.
6767

6868
# %%
6969
from pyrit.executor.attack import AttackConverterConfig

0 commit comments

Comments
 (0)