Skip to content

chore: Parallelize mutating projects and running the initial test#3415

Open
richardwerkman wants to merge 20 commits into
masterfrom
3414_parallel_mutating_testing
Open

chore: Parallelize mutating projects and running the initial test#3415
richardwerkman wants to merge 20 commits into
masterfrom
3414_parallel_mutating_testing

Conversation

@richardwerkman
Copy link
Copy Markdown
Member

@richardwerkman richardwerkman commented Jan 31, 2026

I'm expecting a modest performance improvement from this on large projects. After this change we:

  • Run the initial test while mutating the syntax trees
  • Mutate syntax trees in parallel, instead of mutate projects in parallel

This will use up more threads, but reduces the runtime of stryker on multi core systems.

Test results:
I got the initial testing and mutating duration on the integration test project down from 4500ms to 1600ms on a 10 core system.

todo:

  • Test on a real world project to measure performance boost
  • Delay writing mutated assembly to after inital test is finished
  • Fix NUnit and MSTest integration tests

closes #3414

Copilot AI review requested due to automatic review settings January 31, 2026 15:39
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to improve overall runtime on large solutions by overlapping the “initial test run” phase with the project mutation phase.

Changes:

  • Renamed orchestration entrypoint to MutateAndTestProjectsAsync and wired it through StrykerRunner.
  • Split initialization into “create inputs” vs “run initial tests”, enabling initial tests and mutation to execute in parallel.
  • Adjusted mutator behavior to enrich test-project metadata after initial tests complete, plus updated unit tests for the new flow.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/Stryker.Core/Stryker.Core/StrykerRunner.cs Uses the new orchestrator method that performs parallel initial testing + mutation.
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Implements the parallel execution of initial tests and mutation, and applies initial-test results after both complete.
src/Stryker.Core/Stryker.Core/Initialisation/ProjectMutator.cs Moves initial-test enrichment out of MutateProject into a dedicated post-initial-test method.
src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs Splits input creation from running initial tests (now returns a per-input result map).
src/Stryker.Core/Stryker.Core.UnitTest/StrykerRunnerTests.cs Updates mocks/verifications for the renamed orchestrator API.
src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectOrchestratorTests.cs Updates tests to call the new orchestration method and stubs the enrichment call.
src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectMutatorTests.cs Ensures enrichment is invoked explicitly after mutation.
src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/InitialisationProcessTests.cs Updates tests to use the new “inputs then initial tests” API split.

Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Copilot AI review requested due to automatic review settings February 3, 2026 20:08
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Comment thread src/Stryker.Core/Stryker.Core/MutationTest/CsharpMutationProcess.cs
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Copilot AI review requested due to automatic review settings February 5, 2026 19:32
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented Feb 5, 2026

Quality Gate Failed Quality Gate failed

Failed conditions
1 New Minor Issues (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Copilot AI review requested due to automatic review settings February 5, 2026 20:14
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Copilot AI review requested due to automatic review settings February 5, 2026 20:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Comment thread src/Stryker.Core/Stryker.Core/MutationTest/CsharpMutationProcess.cs Outdated
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs Outdated
…cess.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 5, 2026 20:39
…sationProcessTests.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Comment thread src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Outdated
Comment thread src/Stryker.Core/Stryker.Core/MutationTest/CsharpMutationProcess.cs Outdated
Copilot AI review requested due to automatic review settings February 7, 2026 16:15
@richardwerkman richardwerkman enabled auto-merge (squash) February 7, 2026 16:17
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Comment thread src/Stryker.Core/Stryker.Core/MutationTest/CsharpMutationProcess.cs
Change method signature to async for proper handling of asynchronous operations.
Copilot AI review requested due to automatic review settings February 9, 2026 12:37
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Comment thread src/Stryker.TestRunner/Results/TestRunResult.cs Outdated
Comment thread src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs Outdated
Copilot AI review requested due to automatic review settings April 10, 2026 14:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Comment on lines +80 to +82
var mutationTask = Task.FromResult(MutateProjects(options, reporters, inputs).ToList());

// Run initial test and mutation in parallel for better performance
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mutationTask is created via Task.FromResult(MutateProjects(...).ToList()), which executes MutateProjects synchronously before Task.WhenAll is awaited. This means mutation exceptions will be thrown synchronously (leaving initialTestRunTask running but unobserved), and it’s also misleading because the mutation work isn’t represented as a real task. Use a real task (e.g., Task.Run(() => MutateProjects(...).ToList())) or run mutation synchronously without wrapping it and remove the WhenAll pattern.

Suggested change
var mutationTask = Task.FromResult(MutateProjects(options, reporters, inputs).ToList());
// Run initial test and mutation in parallel for better performance
var mutationTask = Task.Run(() => MutateProjects(options, reporters, inputs).ToList());
// Run initial test and mutation in parallel for better performance.

Copilot uses AI. Check for mistakes.
Comment on lines +31 to +35
var executedTestIds = executedTests.GetIdentifiers().ToHashSet();

TestDescriptions = executedTests.IsEveryTest
? vsTestDescriptions.ToList()
: vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList();
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

executedTests.GetIdentifiers().ToHashSet() is computed even when executedTests.IsEveryTest is true, which means an unnecessary allocation on the common “every test ran” path. Consider only materializing the HashSet in the filtered branch. Also, there appears to be trailing whitespace on these lines, which can fail formatting checks depending on repo settings.

Suggested change
var executedTestIds = executedTests.GetIdentifiers().ToHashSet();
TestDescriptions = executedTests.IsEveryTest
? vsTestDescriptions.ToList()
: vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList();
if (executedTests.IsEveryTest)
{
TestDescriptions = vsTestDescriptions.ToList();
}
else
{
var executedTestIds = executedTests.GetIdentifiers().ToHashSet();
TestDescriptions = vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList();
}

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve performance by running initial test and mutating in parallel

2 participants