tests: cover a completed parallel_attempts run (issue #338)#1815
Closed
EphraiemSarabamoun wants to merge 1 commit into
Closed
tests: cover a completed parallel_attempts run (issue #338)#1815EphraiemSarabamoun wants to merge 1 commit into
EphraiemSarabamoun wants to merge 1 commit into
Conversation
Add a parametrized test that runs garak end to end with --parallel_attempts set to both 1 and a value greater than 1, against the CPU-only test.Blank generator and the multi-prompt test.Test probe, and asserts the run completes without error. The only existing --parallel_attempts run (test_site_yaml_ overrides_max_workers) deliberately expects a ValueError from the max_workers cap, so the happy path where parallel_attempts > 1 completes a run was not covered. Using test.Test (8 prompts) ensures the >1 case reaches the multiprocessing path in Probe._execute_all. Signed-off-by: Ephraiem Sarabamoun <ephraiemsam16@gmail.com>
EphraiemSarabamoun
force-pushed
the
test/parallel-attempts-run-completes
branch
from
b77991b to
f1472f2
Compare
EphraiemSarabamoun
requested review from
erickgalinkin,
jmartin-tech and
leondz
as code owners
Author
|
Superseded by #1816, which carries the identical 14-line test with clean history against main and the DCO sign-off. This branch's shared history with main was severed by a force-push, which inflated the diff to the whole tree. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a test for the happy path of parallel_attempts, which issue #338 asks for.
The only existing run with --parallel_attempts is test_site_yaml_overrides_max_workers in tests/test_config.py, and it deliberately expects a ValueError from the max_workers cap. So the case the issue cares about, where parallel_attempts is greater than 1 and a run finishes successfully, was not tested anywhere.
The new test_parallel_attempts_run_completes runs garak end to end through garak.cli.main with --parallel_attempts set to both 1 and 4. It uses the cheap CPU only test.Blank generator and the test.Test probe with one generation, then asserts the run finishes without raising and that the parallel worker count landed in config as requested. This matches the test conditions in the issue, namely the test generator, one generation, a continuations style probe, and parallel_attempts in both the serial and parallel cases.
I picked test.Test rather than test.Blank as the probe on purpose. test.Test carries 8 prompts, so with one generation the run produces more than one attempt, and the test generator is parallel capable by default. That means the value greater than 1 case actually reaches the multiprocessing path in garak.probes.base.Probe._execute_all rather than quietly falling through to the serial branch, so the test exercises the thing it claims to.
Verification. I installed the package into a fresh venv with uv pip install -e the tests extra, then ran only this test by node id. Both parameter cases pass. I also ran black on the changed file and my added lines are clean.