chore: Parallelize mutating projects and running the initial test#3415
chore: Parallelize mutating projects and running the initial test#3415richardwerkman wants to merge 20 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR aims to improve overall runtime on large solutions by overlapping the “initial test run” phase with the project mutation phase.
Changes:
- Renamed orchestration entrypoint to
MutateAndTestProjectsAsyncand wired it throughStrykerRunner. - Split initialization into “create inputs” vs “run initial tests”, enabling initial tests and mutation to execute in parallel.
- Adjusted mutator behavior to enrich test-project metadata after initial tests complete, plus updated unit tests for the new flow.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Stryker.Core/Stryker.Core/StrykerRunner.cs | Uses the new orchestrator method that performs parallel initial testing + mutation. |
| src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs | Implements the parallel execution of initial tests and mutation, and applies initial-test results after both complete. |
| src/Stryker.Core/Stryker.Core/Initialisation/ProjectMutator.cs | Moves initial-test enrichment out of MutateProject into a dedicated post-initial-test method. |
| src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs | Splits input creation from running initial tests (now returns a per-input result map). |
| src/Stryker.Core/Stryker.Core.UnitTest/StrykerRunnerTests.cs | Updates mocks/verifications for the renamed orchestrator API. |
| src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectOrchestratorTests.cs | Updates tests to call the new orchestration method and stubs the enrichment call. |
| src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectMutatorTests.cs | Ensures enrichment is invoked explicitly after mutation. |
| src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/InitialisationProcessTests.cs | Updates tests to use the new “inputs then initial tests” API split. |
|
…cess.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…sationProcessTests.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Change method signature to async for proper handling of asynchronous operations.
…tryker-mutator/stryker-net into 3414_parallel_mutating_testing
… into 3414_parallel_mutating_testing
| var mutationTask = Task.FromResult(MutateProjects(options, reporters, inputs).ToList()); | ||
|
|
||
| // Run initial test and mutation in parallel for better performance |
There was a problem hiding this comment.
mutationTask is created via Task.FromResult(MutateProjects(...).ToList()), which executes MutateProjects synchronously before Task.WhenAll is awaited. This means mutation exceptions will be thrown synchronously (leaving initialTestRunTask running but unobserved), and it’s also misleading because the mutation work isn’t represented as a real task. Use a real task (e.g., Task.Run(() => MutateProjects(...).ToList())) or run mutation synchronously without wrapping it and remove the WhenAll pattern.
| var mutationTask = Task.FromResult(MutateProjects(options, reporters, inputs).ToList()); | |
| // Run initial test and mutation in parallel for better performance | |
| var mutationTask = Task.Run(() => MutateProjects(options, reporters, inputs).ToList()); | |
| // Run initial test and mutation in parallel for better performance. |
| var executedTestIds = executedTests.GetIdentifiers().ToHashSet(); | ||
|
|
||
| TestDescriptions = executedTests.IsEveryTest | ||
| ? vsTestDescriptions.ToList() | ||
| : vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList(); |
There was a problem hiding this comment.
executedTests.GetIdentifiers().ToHashSet() is computed even when executedTests.IsEveryTest is true, which means an unnecessary allocation on the common “every test ran” path. Consider only materializing the HashSet in the filtered branch. Also, there appears to be trailing whitespace on these lines, which can fail formatting checks depending on repo settings.
| var executedTestIds = executedTests.GetIdentifiers().ToHashSet(); | |
| TestDescriptions = executedTests.IsEveryTest | |
| ? vsTestDescriptions.ToList() | |
| : vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList(); | |
| if (executedTests.IsEveryTest) | |
| { | |
| TestDescriptions = vsTestDescriptions.ToList(); | |
| } | |
| else | |
| { | |
| var executedTestIds = executedTests.GetIdentifiers().ToHashSet(); | |
| TestDescriptions = vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList(); | |
| } |




I'm expecting a modest performance improvement from this on large projects. After this change we:
This will use up more threads, but reduces the runtime of stryker on multi core systems.
Test results:
I got the initial testing and mutating duration on the integration test project down from 4500ms to 1600ms on a 10 core system.
todo:
closes #3414