Known Test Repository Issue Sometimes Misses Test Failures

[nagilson]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1196220

Build leg reported

Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.BasicTest

Pull Request

dotnet/sdk#51591

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The SSL connection",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

So, I believe there is an issue with the known issue flagger, where it is failing to mark certain failures as known, even if they are a known issue. Please let me know if I am mistaken.

Here's the issue:
Looking at this image, this test was not considered a known issue:
https://github.com/dotnet/sdk/pull/51591/checks?check_run_id=54740778719

I believe that, because it's showing as a test failure, and other similar test failures are counted in the above section:

Furthermore, it appeared to be blocking build-analysis from being green.

Here's such an example of 1 of several 'known issues' that flag this issue:
#5015
dotnet/sdk#51574
#6410

As you can see in, #51574 looks for the string "The SSL connection could not be established" as a ErrorMessage which is supposed to use String.Contains. The test, however, that I think was not flagged as a 'known issue', did indeed have that error message string. It looks like some were flagged, while others were not. Here's the output of that 'unknown' test failure:

Expected command to exit with 0 but it did not.\nFile Name: /Users/runner/work/1/s/artifacts/bin/redist/Release/dotnet/dotnet\nArguments: new console --search --debug:custom-hive /Users/runner/work/1/s/artifacts/tmp/Release/dotnet-new.IntegrationTests/SharedHomeDirectory/20251106211951401\nExit Code: 103\nStdOut:\nWarning: use of 'dotnet new --search' is deprecated. Use 'dotnet new search' instead.\nFor more information, run: \n   dotnet new search -h\n\nSearching for the templates...\nStdErr:\nMatches from template source: NuGet.org\nSearch failed: Failed to update search cache. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)\n\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#103\n [[Full log]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220&view=ms.vss-test-web.build-test-results-tab&runId=33106418&resultId=100166&paneView=debug)

While there may be a confounding variable in that there are now several known issues which are tracking substrings of each other, or may overlap, the whole reason there are multiple in the first place is because the first existing known test repository issue did not flag all of the tests which failed with the known issue.

Could this alternatively be a limitation of:

1. A limit of how many tests can be flagged as a known issue for the same known issue
2. A different 'failing configuration' which causes issues to be missed?
3. A misunderstanding of which strings get searched? Note that I believe the error is also in the 'helix logs'.
4. A failure of my own accord to file the known issue? @dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL Connection C]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/6/2025 10:39:11 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL connection]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/6/2025 10:42:09 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1196423	dotnet/machinelearning	Build	Log	dotnet/machinelearning#7543
1196398	dotnet/aspnetcore	Run build.sh	Log	dotnet/aspnetcore#64262
1196235	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#121356
1196186	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#121417
1196168	dotnet/runtime	Send tests to Helix (Unix)	Log

Build	Definition	Test	Pull Request
1196358	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51558
1196220	dotnet/sdk	Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.CanShowDeprecationMessage_WhenLegacyCommandIsUsed	dotnet/sdk#51591

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
7	7	7

Report

Build	Repository	Step Name	Console log	Pull Request
2991196	dotnet-runtime	Restore and Build Product	Log	#61716
1439772	dotnet/runtime	Build product	Log	dotnet/runtime#128667
1439765	dotnet/runtime	Build product	Log	dotnet/runtime#128721
2986317	dotnet-maui	Run Integration Tests - Build	Log
1437619	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#54469
1437514	dotnet/runtime	Build Tests	Log	dotnet/runtime#128611
2984356	dotnet-aspnetcore	Run build.sh	Log	#61208
2983775	dotnet-maui	Pack	Log
1431562	dotnet/runtime	Build product	Log	dotnet/runtime#126187
2980419	dotnet-performance	Run performance job script	Log
2979126	dotnet-maui	Run Integration Tests - RunOniOS_BlazorDebug	Log
1426533	dotnet/maui	Controls SearchBar,Shape,Slider	Log
2978299	dotnet-aspnetcore	Run eng/scripts/CodeCheck.ps1	Log	#61250
2977978	dotnet-maui	Controls (v18.5) CarouselView	Log
2975853	dotnet-performance	Run performance job script	Log
1420988	dotnet/runtime	Build product	Log	dotnet/runtime#128188
1421640	dotnet/runtime	Send tests to Helix (Unix)	Log
2975167	dotnet-aspnetcore	Run build.sh	Log	#60795
1418956	dotnet/runtime	Build product	Log	dotnet/runtime#125437
1418470	dotnet/maui	Controls (API 30) SoftInput,Stepper,Switch,SwipeView	Log	dotnet/maui#35079
1418565	dotnet/maui	Controls (API 30) Entry	Log	dotnet/maui#35311
1417305	dotnet/maui	Controls TabbedPage,TableView,TimePicker,TitleView,ToolbarItem,Triggers	Log	dotnet/maui#34600
2972989	dotnet-maui	Pack	Log
1415768	dotnet/sdk	🟣 Build	Log	dotnet/sdk#53835
2970359	dotnet-fsharp	build.cmd microbuild (proto -> real -> setup -> sign -> insertion)	Log
2969811	dotnet-maui	Controls (vlatest) SearchBar,Shape,Slider	Log
1412668	dotnet/sdk	🟣 Build	Log	dotnet/sdk#54222

Build	Repository	Test	Pull Request
1451880	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
2992522	dotnet-sdk	Microsoft.NET.Build.Containers.UnitTests.dll.2.WorkItemExecution	#61850
1449092	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1448602	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1446344	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1445761	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1443952	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1443540	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1442703	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1441232	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1436219	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable	dotnet/runtime#128610
1436092	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1431792	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1428400	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1427594	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_ClientCertificates_Test.AutomaticOrManual_DoesntFailRegardlessOfWhetherClientCertsAreAvailable
1427585	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1425179	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1421026	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
1418874	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1418581	dotnet/sdk	dotnet.Tests.dll.9.WorkItemExecution	dotnet/sdk#54288
1418163	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1418156	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
2972404	dotnet-sdk	Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishASingleFileApp.Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishASingleFileApp.It_runs_single_file_apps
2970560	dotnet-sdk	EndToEnd.Tests.GivenFrameworkDependentApps.EndToEnd.Tests.GivenFrameworkDependentApps.ItDoesNotRollForwardToTheLatestVersionOfAspNetCoreApp
1414294	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_SslProtocols_Test.GetAsync_AllowedSSLVersion_Succeeds
2970599	dotnet-sdk	Microsoft.NET.ToolPack.Tests.GivenThatWeWantToPackAToolProjectWithPackagedShim.Microsoft.NET.ToolPack.Tests.GivenThatWeWantToPackAToolProjectWithPackagedShim.Generate_shims_runs_incrementally	#60901
1412581	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	10	54

[nagilson]

I'm still seeing this issue where the string contains match is identical, yet the known issue labeler doesn't think it's a known issue.