tests: cover a completed parallel_attempts run#1816
Open
EphraiemSarabamoun wants to merge 1 commit into
Open
Conversation
Add a parametrized test that runs garak through garak.cli.main with --parallel_attempts set to 1 and to 4, against the CPU-only test.Blank generator and the multi-prompt test.Test probe, and asserts the run completes without error. The only existing --parallel_attempts run deliberately expects a ValueError from the max_workers cap, so the path where parallel_attempts is greater than 1 and a run finishes was not covered anywhere. test.Test carries 8 prompts, so the greater than 1 case actually reaches the multiprocessing path in Probe._execute_all. Closes NVIDIA#338 Signed-off-by: Ephraiem Sarabamoun <ephraiemsam16@gmail.com>
jmartin-tech
changed the title
leondz
reviewed
Jun 1, 2026
leondz
reviewed
Jun 1, 2026
| ).split() | ||
| garak.cli.main(args) | ||
|
|
||
| assert _config.system.parallel_attempts == parallel_attempts |
Collaborator
There was a problem hiding this comment.
asserts should have a message describing the desired condition. e.g. "--parallel_atempts cli param should pass through to _config.system"
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a test for the happy path of
parallel_attempts, which issue #338 asks for.The only existing run with
--parallel_attemptsistest_site_yaml_overrides_max_workersintests/test_config.py, and it deliberately expects aValueErrorfrom themax_workerscap. So the case the issue cares about, whereparallel_attemptsis greater than 1 and a run finishes successfully, was not tested anywhere.The new
test_parallel_attempts_run_completesruns garak end to end throughgarak.cli.mainwith--parallel_attemptsset to both 1 and 4. It uses the cheap CPU onlytest.Blankgenerator and thetest.Testprobe with one generation, then asserts the run finishes without raising and that the parallel worker count landed in config as requested. This matches the test conditions in the issue, namely the test generator, one generation, a continuations style probe, andparallel_attemptsin both the serial and parallel cases.I picked
test.Testrather thantest.Blankas the probe on purpose.test.Testcarries 8 prompts, so with one generation the run produces more than one attempt, and the test generator is parallel capable by default. That means the value greater than 1 case actually reaches the multiprocessing path ingarak.probes.base.Probe._execute_allrather than quietly falling through to the serial branch, so the test exercises the thing it claims to.Verification. I installed the package into a fresh venv with
uv pip install -ethe tests extra, then ran only this test by node id. Both parameter cases pass. I also ran black on the changed file and my added lines are clean.