Unmatched test results are excluded from the Test Run

[ombacke]

Description

When the extension publishes results to a Test Plan, only tests that match a Test Point are included in the Test Run. Tests that do not match (e.g. setup/helper tests with no Test Case ID) are silently excluded from the run entirely. This makes the Test Run an incomplete representation of what was actually executed.

Expected behavior

All test results from the JUnit XML should appear in the Test Run. Unmatched tests should still be published — either as unlinked results or under a catch-all category — so the Test Run reflects the full execution.

Actual behavior

Only matched tests appear in the Test Run. In our case, 4 tests were executed but only 1 was matched to a Test Point, so the Test Run shows 1 result instead of 4.

From the debug log:

Mapped 1 automated tests to test points:
| Test Point | Test Case | Configuration | Automated Test           | TestOutcome |
| 111111     | 11111     | Windows 10    | TC11111_...        | Failed      |

Unmatched test cases:
No matching test point found for 'setup-test-1'
No matching test point found for 'setup-test-2'
No matching test point found for 'TC22222_SomeFeature_SomeScenario'

The resulting Test Run only contains 1 result, even though 4 tests were executed.

Steps to reproduce

1. Configure the extension with testCaseMatchingStrategy: Regex
2. Run a Playwright test suite where some tests have TC IDs and some do not (e.g. setup/auth fixtures)
3. Observe that the Test Run only contains the matched tests

Environment

• Extension version: 0.9.0
• Test framework: Playwright / JUnit XML
• Matching strategy: Regex (^TC(\d+))

Suggested fix

Consider adding an option such as includeUnmatchedResults: true that publishes unmatched tests to the Test Run without linking them to a Test Point. This would give teams full visibility of their test execution while still linking the tests that can be matched.

Additional context

The failTaskOnUnmatchedTestCases: false flag prevents the task from failing, but does not include the unmatched results in the run. These are two separate concerns and it would be helpful to have independent control over both behaviors.

Note: one of the unmatched tests (TC22222) does contain a valid TC ID in the name, but was still excluded because the Test Case did not exist in the configured Test Plan. This appears to be a separate but related issue — it may be worth logging a warning that distinguishes between “no TC ID found” and “TC ID found but not in plan”.

[bryanbcook]

This is by design. As implied by the name of the task PublishTestPlanResults the primary goal of this task is to match your automated tests to Test Cases in the Test Plan. The primary scenario would be for mapping results from an automated regression suite to the Test Plan to report on test coverage against a planned regression. Teams can view progress of their plan using ADO's Test Plan report and they can view the outcome of the automated regressions by viewing the Test Run (e.g. 5 automated tests of the planned 40 test cases failed).

There may be a limitation that prevents test-points and non-matched test results mixed in the same Test Run, which requires some investigation, but in my opinion mixing these concerns diminishes the ability to view pass/fail/skip for the Test Run against the Test Plan.

If you have unmatched automation that you want to track separately, my suggestion would be to use the PublishTestPlanResults@1 to update your Test Plan and use the out of box PublishTestResults@2 task to publish the test automation results. The out of box task also adds a tab in the pipeline's Build summary that allows you to explore the test results directly in the pipeline. The out of the box task requires your test results are in a supported format (typically junit xml or mstest trx). I'd be open to investigating if I can emulate what the out-of-box task does so that you don't need the second task, but I'm fairly confident that tapping into that functionality would create a second Test Run.

it may be worth logging a warning that distinguishes between “no TC ID found” and “TC ID found but not in plan”.

If I understand correctly, what I think you're asking for is a way to differentiate between test results that cannot be matched versus test results that could be matched if the Test Plan was adjusted. The failOnUnmatchedTestCases option only fails the task when the test automation produces zero matches. To accommodate gaps in the Test Plan, there are a few challenges with that ask:

• the task matches against tests in the Test Plan. If the Test Case is not part of the Test Plan, it technically does not exist. The task has absolutely no way to validate that the value found by the regex is even a valid ID. There would need to be a way to find all the test cases regardless of their Test Plan, and this could be thousands of test cases depending on how you carry tests forward (you can preserve test cases between plans or clone them to create new test cases)
• the task supports different matching strategies, not just RegEx. The VSProperty matching strategy uses the class and method names of the tests to create a Test Case identifier. By default, all test results will have a value but won't be found in the Plan, so it's not as simple as suggesting there's a gap just because a regex found a value.

I'm going to mark this as wont-fix at the moment.