feat: Add test information to mtp result for reporting#3541
Open
richardwerkman wants to merge 7 commits into
Open
feat: Add test information to mtp result for reporting#3541richardwerkman wants to merge 7 commits into
richardwerkman wants to merge 7 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enhances the Microsoft Testing Platform (MTP) test runner integration so discovered tests carry source-location metadata (file/line/type/method), enabling more accurate downstream test reporting in Stryker.
Changes:
- Extend the MTP
TestNodemodel to deserialize location metadata (file/line/type/method) from MTP JSON payloads. - Populate
MtpTestCasewithCodeFilePath,LineNumber, and a derivedFullyQualifiedName. - Add unit tests covering location deserialization and
MtpTestCasemapping; update.github/copilot-instructions.mdcontent/structure.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/Stryker.TestRunner.MicrosoftTestPlatform/Models/TestNode.cs |
Adds optional location fields mapped to MTP JSON property names. |
src/Stryker.TestRunner.MicrosoftTestPlatform/Models/MtpTestCase.cs |
Maps TestNode location data onto ITestCase fields used for reporting. |
src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/MtpTestCaseTests.cs |
Adds tests for the new location mapping and JSON deserialization behavior. |
.github/copilot-instructions.md |
Expands project overview and updates contributor guidance (but introduces a duplicate heading). |
Comment on lines
12
to
+15
| _testNode = testNode; | ||
| CodeFilePath = testNode.LocationFile ?? string.Empty; | ||
| LineNumber = testNode.LocationLineStart ?? 0; | ||
| FullyQualifiedName = BuildFullyQualifiedName(testNode); |
Comment on lines
+13
to
+14
| CodeFilePath = testNode.LocationFile ?? string.Empty; | ||
| LineNumber = testNode.LocationLineStart ?? 0; |
Comment on lines
+48
to
+53
| public void MtpTestCase_WithoutLocationType_FallsBackToUid() | ||
| { | ||
| var testNode = new TestNode("uid-1", "TestMethod1", "action", "discovered", | ||
| LocationFile: "/path/to/TestFile.cs", | ||
| LocationLineStart: 10); | ||
|
|
|
|
||
| ### Running Stryker on itself | ||
|
|
||
| Running Stryker on itself doesn't work as assemblies will be in use. To run Stryker on the stryker codebase, use the official nuget release via `dotnet tool install dotnet-stryker` and then `dotnet stryker`. |
Member
There was a problem hiding this comment.
Should work with the Stryker on Stryker script
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Microsoft Testing Platform (MTP) test-case metadata surfaced to Stryker’s reporting pipeline (file path, line, and a better fully-qualified name), and adjusts TestRunResult to correctly retain test descriptions when a run represents “all tests”. It also expands repository Copilot guidance.
Changes:
- Treat
executedTests.IsEveryTestas “include all discovered descriptions” when constructingTestRunResult.TestDescriptions. - Extend MTP
TestNode/MtpTestCaseto carry location metadata intoITestCasefields used for reporting. - Add unit tests validating MTP location deserialization and
MtpTestCasefield population; update.github/copilot-instructions.mdcontent.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Stryker.TestRunner/Results/TestRunResult.cs | Avoids filtering out all test descriptions when ExecutedTests represents “every test”. |
| src/Stryker.TestRunner.MicrosoftTestPlatform/Models/TestNode.cs | Adds optional JSON-mapped location fields needed for reporting. |
| src/Stryker.TestRunner.MicrosoftTestPlatform/Models/MtpTestCase.cs | Maps MTP node location/type/method into ITestCase properties for downstream consumers. |
| src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/MtpTestCaseTests.cs | Adds coverage for location deserialization and MtpTestCase population behavior. |
| .github/copilot-instructions.md | Expands repo overview and updates local testing guidance/examples. |
Comment on lines
+31
to
+33
| TestDescriptions = executedTests.IsEveryTest | ||
| ? vsTestDescriptions.ToList() | ||
| : vsTestDescriptions.Where(p => executedTests.GetIdentifiers().Contains(p.Id)).ToList(); |
|
|
||
| To test Stryker.NET on a project in a terminal, you can build Stryker.NET and then run the resulting `stryker.dll` on the target project. Run `dotnet <path-to-stryker.dll>` in the root of the project you want to test (adjust path as needed based on your build configuration). | ||
|
|
||
| For example in the `/integrationtest/TargetProjects/NetCore/TargetProject` directory, you can run `dotnet ../../src/Stryker.CLI/bin/Debug/net8.0/stryker.dll` to test Stryker.NET on the target projects. |
Co-authored-by: Copilot <copilot@github.com>
richardwerkman
changed the title
Co-authored-by: Copilot <copilot@github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Microsoft Testing Platform (MTP) support by enriching test case data (file path, line, fully qualified name) so Stryker can include test information in reports, and updates validations accordingly. It also includes a release-script enhancement and expanded Copilot instructions.
Changes:
- Populate MTP test case location/type metadata and add unit tests to verify JSON deserialization + mapping behavior.
- Fix/optimize
TestRunResulttest-description selection to correctly include all tests when the runner reports “every test executed”. - Update integration validations to assert test counts for MTP target projects.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Stryker.TestRunner/Results/TestRunResult.cs | Uses IsEveryTest to include all descriptions; avoids repeated identifier lookups. |
| src/Stryker.TestRunner.MicrosoftTestPlatform/Models/TestNode.cs | Adds optional location metadata fields for MTP JSON payloads. |
| src/Stryker.TestRunner.MicrosoftTestPlatform/Models/MtpTestCase.cs | Maps MTP node location/type to ITestCase properties used for reporting. |
| src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/MtpTestCaseTests.cs | Adds unit tests covering mapping + JSON deserialization of new fields. |
| prepare-release.js | Adds conventional-commit-based version suggestion and release note/tag creation changes. |
| integrationtest/Validation/ValidationProject/ValidateStrykerResults.cs | Asserts non-zero test totals for MTP-based runs. |
| .github/copilot-instructions.md | Expands repository overview and local testing guidance. |
Comment on lines
+31
to
+34
| var executedTestsSet = executedTests.GetIdentifiers().ToHashSet(); | ||
| TestDescriptions = executedTests.IsEveryTest | ||
| ? vsTestDescriptions.ToList() | ||
| : vsTestDescriptions.Where(p => executedTestsSet.Contains(p.Id)).ToList(); |
|
|
||
| For example in the `/integrationtest/TargetProjects/NetCore/TargetProject` directory, you can run `dotnet ../../../../src/Stryker.CLI/Stryker.CLI/bin/Debug/net8.0/Stryker.CLI.dll` to run the locally built Stryker.NET on the target projects. | ||
|
|
||
| Keep in mind the different runmodes of Stryker.NET: |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment on lines
+2
to
+6
| const { promisify } = require('util'); | ||
| const readline = require('readline'); | ||
| const fs = require('fs'); | ||
| const semver = require('semver'); | ||
| const conventionalRecommendedBump = require('conventional-recommended-bump'); |
Comment on lines
+76
to
+83
| const changelogPath = './CHANGELOG.md'; | ||
| const changelog = fs.readFileSync(changelogPath, { encoding: 'UTF-8' }); | ||
| const marker = '<!-- changelog -->'; | ||
| if (!changelog.includes(marker)) { | ||
| throw new Error(`${changelogPath} is missing the '${marker}' insertion marker`); | ||
| } | ||
| }); | ||
| fs.writeFileSync(changelogPath, changelog.replace(marker, `${marker}\n\n${releaseNotes}`), { encoding: 'UTF-8' }); | ||
| } |
Comment on lines
+5
to
+9
| namespace Stryker.TestRunner.MicrosoftTestPlatform.UnitTest; | ||
|
|
||
| [TestClass] | ||
| public class MtpTestCaseTests | ||
| { |
Comment on lines
+31
to
+34
| var executedTestsSet = executedTests.GetIdentifiers().ToHashSet(); | ||
| TestDescriptions = executedTests.IsEveryTest | ||
| ? vsTestDescriptions.ToList() | ||
| : vsTestDescriptions.Where(p => executedTestsSet.Contains(p.Id)).ToList(); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
related to #3094
Also improve copilot-instructions