feat: Add BijectionConverter and BijectionAttack (#1903)

[sajisanchu1913-source]

Summary

Implements the Bijection Attack from arXiv:2410.01294 (Haize Labs) into PyRIT.

The attack works by teaching a target LLM a secret character mapping through
demonstration shots, then sending harmful prompts encoded in that mapping to
bypass safety filters. Responses are decoded using the inverse mapping.

Changes

New Files

• pyrit/prompt_converter/bijection_converter.py — generates random letter-to-letter mapping, encodes prompts, decodes responses
• pyrit/executor/attack/single_turn/bijection_attack.py — runs full bijection attack with teaching phase
• tests/unit/prompt_converter/test_bijection_converter.py — 11 unit tests for converter
• tests/unit/executor/test_bijection_attack.py — 5 unit tests for attack
• doc/code/executor/attack/bijection_attack.ipynb — usage notebook

Modified Files

• pyrit/prompt_converter/__init__.py — registered BijectionConverter
• pyrit/executor/attack/single_turn/__init__.py — registered BijectionAttack

How It Works

1. BijectionConverter generates a random secret mapping (e.g. a→q, b→x...)
2. BijectionAttack sends teaching messages to target AI to teach the mapping
3. Harmful prompt is encoded and sent as TASK is '⟪encoded prompt⟫'
4. Response is decoded using inverse mapping
5. Decoded response is scored by the judge

Pattern Followed

• BijectionConverter follows FlipConverter pattern
• BijectionAttack follows FlipAttack pattern

Reference

• Haize Labs implementation: https://github.com/haizelabs/bijection-learning
• Paper: arXiv:2410.01294
• Closes FEAT Bijection #1903

[romanlutz]

This is a great start! There are a few things that need addressing but we're pretty close.

[sajisanchu1913-source]

Hi @romanlutz I've addressed all the review comments:

• Removed @pytest.mark.asyncio decorators
• Fixed init.py alphabetical ordering
• Used patch_central_database fixture in attack tests
• Used MagicMock(spec=PromptTarget) instead of plain MagicMock
• Removed dead num_digits parameter
• Added BijectionType StrEnum for validation
• Used private attributes with underscore prefix
• Added _build_identifier() method
• Fixed teaching shots cap with programmatic cycling
• Fixed alternating user/assistant roles in teaching messages
• Fixed response decoding in _perform_async
• Added BijectionConverter to _request_converters pipeline
• Fixed notebook format and added paired .py jupytext file

Ready for re-review!

[sajisanchu1913-source]

Hi @romanlutz I've addressed the remaining review comments:

• Resolved merge conflicts with upstream/main (kept BidiConverter from main, added BijectionConverter alphabetically)
• Added end-to-end test in TestBijectionAttackEndToEnd that uses MockPromptTarget, returns a cipher-text response, and asserts the result is decoded back to plain text
• Fixed ComponentIdentifier import to use pyrit.models.identifiers

Ready for re-review

[sajisanchu1913-source]

Hi @romanlutz
I've completed the architectural restructure

• BijectionConverter is now an abstract base class
• Added LetterBijectionConverter with fixed_size and seed parameters
• Added DigitBijectionConverter with num_digits and seed parameters
• Added seed parameter for reproducibility across all converters
• Added explicit mapping parameter for replay/deterministic experiments
• BijectionAttack now accepts a bijection_converter instance instead of bijection_type/fixed_size
• Updated all tests to use the concrete subclasses
• All 23 tests passing

Example usage:

# Default letter mapping
attack = BijectionAttack(objective_target=target)

# Custom letter mapping with seed
attack = BijectionAttack(
    objective_target=target,
    bijection_converter=LetterBijectionConverter(fixed_size=5, seed=42),
)

# Digit mapping
attack = BijectionAttack(
    objective_target=target,
    bijection_converter=DigitBijectionConverter(num_digits=10, seed=42),
)

Ready for re-review....

[sajisanchu1913-source]

Hi @romanlutz
For TokenBijectionConverte, since it requires a tokenizer dependency, I'll implement it in a separate follow-up PR to keep this one focused. Let me know if you'd prefer it here instead!

[romanlutz]

I agree @sajisanchu1913-source.

[romanlutz]

Nice‑to‑have follow‑up — tokenizer bijection mode.

The paper (§2) describes a third bijection type beyond letter and digit modes: "tokens from the target model's tokenizer" — each English letter maps to a randomly‑sampled distinct token from the target's vocabulary. The paper notes these complexity parameters (fixed_size for letter mode, ℓ for digit mode, vocab subset for token mode) are what give the attack its scale‑adaptive property, so token mode is meaningful for evaluating frontier models.

Not blocking for this PR — the abstract base class makes adding it later straightforward (TokenBijectionConverter(BijectionConverter) that takes a tokenizer reference). Worth opening as a follow‑up issue once this lands. Or if you want to tackle it yourself you can do that as well, of course.

[romanlutz]

can you open a GH issue for this and note that you'll do that yourself (if you want to!)? Just so that it's tracked.

[sajisanchu1913-source]

Hi @romanlutz addressed the latest review comments:

• Removed duplicate docstring line
• Removed unused SeedPrompt import
• Bumped num_teaching_shots default to 10 per paper spec
• Fixed DigitBijectionConverter. now maps letters to digit
• Added num_digits validation (raises ValueError for values outside 1-4)
• Fixed bijection_attack.py notebook to correct jupytext percent format
• Fixed bijection_attack.ipynb to use new API (LetterBijectionConverter)
• Added test_digit_converter_encodes_letters and test_digit_converter_invalid_num_digits tests

All 25 tests passing. Ready for re-review!

[romanlutz]

I'll make another pass in the morning. Thanks for your patience in addressing all my concerns so far 🙂

[sajisanchu1913-source]

Hi @romanlutz I can see there are merge conflicts. Should I resolve them, or would you prefer to handle it on your end? Happy to do whatever is most helpful!

[sajisanchu1913-source]

Hi @romanlutz just wanted to check in on the merge conflicts, happy to resolve them myself if that would help move things along. Let me know!

[sajisanchu1913-source]

Hi @romanlutz I've resolved the merge conflicts, kept both the bijection citation and the new upstream citations in bibliography.md, fixed the init.py conflict by keeping both DecompositionConverter and DigitBijectionConverter alphabetically, and updated myst.yml to the new executor folder structure. Should be ready to merge now!

[sajisanchu1913-source]

Hi @romanlutz
I've resolved the merge conflicts. One thing to flag I forgot to create a separate branch before starting work on TokenBijectionConverter, so those changes ended up here. I've gone ahead and included it since the architecture was already in place, but happy to revert it out and open a separate PR if you'd prefer to keep this one focused on the original scope. Let me know what works best!

[romanlutz]

I've tried this and it essentially didn't work. The target model didn't seem to learn the mapping and the result was garbage (or at least I couldn't decode it with the provided mapping). Did you have a different experience?

I've tried fixing that in a few different ways and so far not succeeded.

[sajisanchu1913-source]

Hi @romanlutz, I hadn't tested it end-to-end with a real model ,I focused on getting the architecture and unit tests right but didn't validate the actual attack effectiveness. Thanks for trying it!

Given that it's not working, would you prefer I:

1. Remove TokenBijectionConverter from this PR and open a separate issue to investigate the correct approach
2. Keep it here but mark it as experimental with a note in the docstring

Happy to go with whatever keeps this PR on track!

[sajisanchu1913-source]

Hi Roman,

Thank you for taking the time to review my PR. I'd really like to get this feature merged, and I'm happy to put in whatever work is needed to make it meet the project's expectations.

If there are specific issues with the implementation or areas where my approach doesn't align with the project's design, I'd really appreciate your guidance. I'm committed to addressing the feedback and learning from it.

I'm currently a Master's student and this is one of my first open-source feature contributions. Having a feature implementation merged into PyRIT would be a significant milestone for me, both as a learning experience and for my resume. That said, I want to earn the merge by making the implementation as good as it can be.

Thanks again for your time and feedback—I appreciate any suggestions you have, and I'll work through them.

[romanlutz]

I understand, @sajisanchu1913-source .

I've not yet seen an LLM actually respond in a way that the decoding worked.

For example, I tried

a=q, b=m, c=j, d=z, e=t, f=g, g=f, h=k, i=p, j=w, k=l, l=s, m=b,
n=o, o=x, p=n, q=c, r=r, s=y, t=e, u=v, v=h, w=i, x=a, y=d, z=u

and got this response

zqqy bm ksp fxa wdykydy xf c bzxu pltkchv gtrktfs twttks

which maps to the following (inverse):

daas mb hli gox jyshsys og q mdoz ikehqvu ferhegl ejeehl

I need to start seeing a working bijection. I'm not even saying a successful attack, but a working bijection.

We can explore options of restructuring it, or do a detailed comparison to the paper. But something appears to be off.

[sajisanchu1913-source]

Thanks, Roman. I understand the issue now.

I'll revisit my implementation, compare it more closely with the paper and the reference implementation, and focus on getting the model to produce a working bijection before worrying about the attack itself. I'll work on this and update the PR by tomorrow morning with my findings and any changes.

I really appreciate your feedback and the time you've taken to review it. As a Master's student, this is one of my first feature contributions to a large open-source project, so your guidance is incredibly valuable. Thank you!