Skip to content

test(probes): add unit tests for donotanswer probe module#1825

Open
nuthalapativarun wants to merge 1 commit into
NVIDIA:mainfrom
nuthalapativarun:test/donotanswer-probe-coverage
Open

test(probes): add unit tests for donotanswer probe module#1825
nuthalapativarun wants to merge 1 commit into
NVIDIA:mainfrom
nuthalapativarun:test/donotanswer-probe-coverage

Conversation

@nuthalapativarun
Copy link
Copy Markdown

Add probe-unique tests for garak/probes/donotanswer.py.

Verification

  • python -m pytest tests/probes/test_probes_donotanswer.py -v — all 58 tests pass

What is tested

  • Class generation: All 5 classes are generated from DNA_PROBE_TAGS with the correct title-cased names
  • dna_category attribute: Each probe carries the matching category key string
  • Goal string derivation: goal is "get model to respond as <category with spaces>" for every class
  • Primary detector: All probes use mitigation.MitigationBypass
  • Per-category tags: Spot-checks for OWASP, AVID, and content-safety tags on each category
  • Prompt counts: Each probe loads exactly the right number of lines from its local .txt data file
  • Prompt structure: All prompts are non-empty strings; known duplicate count (1 in InformationHazard) is documented
  • active=False and Tier.UNLISTED: Verified for all 5 probes
  • URI: Points to the Do-Not-Answer arXiv paper (arxiv.org/abs/2308.13387)
  • Language: All probes are lang="en"

No network access is needed — the probes load from local .txt files under garak/data/donotanswer/.

Generic checks (loading, isinstance, etc.) are covered by tests/plugins/test_plugin_load.py.

@jmartin-tech
Copy link
Copy Markdown
Collaborator

PR is not correctly focused, the commit from #1826 is also in the branch.

Add probe-unique tests covering: class generation from DNA_PROBE_TAGS,
local data-loading path, prompt counts, prompt structure (non-empty
strings, known duplicates), per-category goal/tag/detector correctness,
active=False, Tier.UNLISTED, URI and language attributes.

Signed-off-by: Varun Nuthalapati <nuthalapativarun@gmail.com>
@nuthalapativarun nuthalapativarun force-pushed the test/donotanswer-probe-coverage branch from 2d97dd5 to 753c7cd Compare June 2, 2026 15:18
@nuthalapativarun
Copy link
Copy Markdown
Author

Thanks for catching this, @jmartin-tech. The branch has been rebased — the stray fix(attempt) commit (from #1826) has been dropped. The branch now contains only the donotanswer test commit on top of current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants