Skip to content

README.md

Latest commit

 

History

History
94 lines (75 loc) · 7.45 KB

README.md

File metadata and controls

94 lines (75 loc) · 7.45 KB
Raw

Unit tests

The test suite covers all 28 attack modules, both model loaders, the cloud/batch pipeline, and shared infrastructure. Every test runs without a GPU or any model download, the model wrapper is replaced by a mock whose generate() / tokenizer.decode() methods return pre-set strings, so the tests are fast and fully deterministic.

601 test functions across 30 files, authored alongside the implementation with GitHub Copilot (https://github.com/features/copilot) assistance.

Running the tests

# from the root, with the main venv active
pytest tests/ -q

Test file index

Attack modules

File Tests What is covered
test_skeleton_key.py 62 Two-turn flow, SK-accepted / SK-filtered branches, evaluator scoring
test_continuation.py 43 Default prompts, dataset loading (txt + jsonl), prompt-cap, cloze evaluation
test_fitd.py 35 Multi-turn escalation, refusal detection, conversation result serialisation
test_packagehallucination.py 26 Python / Rust / Go modes, real vs hallucinated package detection
test_snowball.py 25 All three question families, regex scorer, correct/snowballed/other labelling
test_lmrc.py 23 Active + all probe modes, per-class detector wiring, prompt-cap behaviour
test_latentinjection.py 22 Active / translation / WHOIS / jailbreak probe variants, cap logic
test_leakreplay.py 22 Cloze + complete modes, all four corpora, mask-stripping, <name> tag removal
test_prompt_transform_attacks.py 21 Shared single-turn logic for FlipAttack, ContextFlooding, EmbeddedInstructionJSON, CharacterStream (parametrised)
test_encoding.py 21 All encoding modes (Base64, rot13, morse, …), chat-template on/off
test_divergence.py 21 Prompt construction, repetition detection, evaluator
test_autodan.py 21 GA loop with mocked scoring, config validation, evaluator
test_crescendo.py 20 Multi-turn with adversarial-JSON generation, backtrack logic, JSON-retry
test_topic.py 18 BFS/DFS tree expansion, controversial/blocked/allowed modes, WordNet mock
test_sata.py 17 Payload loading, synonym substitution, evaluation
test_glitch.py 17 Glitch-token injection, evaluator modes
test_av_spam_scanning.py 17 EICAR / GTUBE / GTPHISH signature embedding, all mode variants
test_apikey.py 17 get-key / leak / multi-turn prompt modes, evaluator
test_misleading.py 16 False-claim prefixes, keyword + model detector modes, prompt_cap
test_flipattack.py 16 Flip variants, behaviours vs dataset_path, evaluator
test_bon.py 15 Round iteration, sigma perturbation, HarmBench-judge mock, ASR threshold
test_pii.py 14 PII extraction prompt construction, evaluation
test_ttft.py 13 TTFT measurement, API loader path, provider config
test_gcg.py 13 Suffix-manager token logic, gradient mock, config validation
test_promptinject.py 12 Rogue-string injection, prompt-cap, evaluator
test_beast_cache.py 2 KV-cache adapter for dynamic-cache and legacy tuple-cache shapes

Infrastructure

File Tests What is covered
test_cloud_units.py 31 AttackSpec, BatchJob, WorkUnit models, InMemoryQueue, dispatcher, worker retry / nack logic, collector, Redis queue (patched)
test_cloud_integration.py 8 Full dispatcher → queue → worker → collector pipeline with run_attack patched, failure/retry flow, CLI subcommands
test_hf_loader.py 7 HFLoader black-box / white-box wrappers, dtype parsing, device handling
test_api_loader.py 6 APILoader request construction, streaming, timeout

Implementation

Mock model wrapper: every attack test builds a lightweight mock with MagicMock:

def _make_wrapper(responses: list[str]) -> MagicMock:
    response_iter = iter(responses)
    tokenizer = MagicMock()
    tokenizer.encode.return_value = torch.zeros(1, 5, dtype=torch.long)
    tokenizer.decode.side_effect = lambda ids, **kw: next(response_iter)
    wrapper = MagicMock()
    wrapper.tokenizer = tokenizer
    wrapper.generate.return_value = torch.zeros(1, 10, dtype=torch.long)
    return wrapper

The wrapper produces a 2-D tensor of zeros, the prompt length is inferred as 5 (the encode length), so the first 5 tokens are treated as the prompt and the remainder as generated output.

Temporary files: dataset files are written to tmp_path (pytest's built-in fixture) so tests are hermetic and leave no state on disk.

Config validation: each attack test includes at least one pytest.raises case that exercises the config validate() method with a bad value, confirming that the runner will reject malformed invocations before touching a model.

Evaluator smoke tests: the evaluator for each attack is instantiated and called with a minimal AttackResult, the test checks that the returned EvaluationResult has the expected fields and that jailbreak_rate is in [0, 1].