Add artifact naming service with template support for consistent naming across extensions#7856
Add artifact naming service with template support for consistent naming across extensions#7856Evangelink wants to merge 112 commits into
Conversation
Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
…ssing imports Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
…ames Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
# Conflicts: # src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs # src/Platform/Microsoft.Testing.Platform/Services/ServiceProviderExtensions.cs
…ct-naming-service
- Remove <root> placeholder (fragile directory traversal) - Fix OperatingSystem.IsWindows/IsLinux/IsMacOS for netstandard2.0 compat (use RuntimeInformation) - Fix KeyValuePair deconstruction for netstandard2.0 compat - Fix Guard.NotNullOrEmpty (non-existent API) - Fix legacy %p pattern handling (move to HangDump caller, not the service) - Fix time format to use hyphens instead of colons (filesystem safe) - Remove unnecessary using directives - Simplify GetOperatingSystemName with ternary expression - Simplify TFM detection (remove hard-coded version map) - Fix tests: use Assert.ThrowsExactly, match constructor signature, update assertions
There was a problem hiding this comment.
Pull request overview
This PR introduces an internal IArtifactNamingService to centralize artifact filename/path templating across Microsoft.Testing.Platform extensions, and migrates HangDump to use it (while keeping legacy %p support).
Changes:
- Added
IArtifactNamingService+ArtifactNamingServiceimplementation with placeholder-based template resolution. - Registered the service in
TestHostBuilderand exposed it viaIServiceProviderextension method. - Updated HangDump to use the service and added/updated unit + integration tests plus documentation.
Show a summary per file
| File | Description |
|---|---|
| test/UnitTests/Microsoft.Testing.Platform.UnitTests/Services/ArtifactNamingServiceTests.cs | Adds unit tests covering placeholder replacement, casing, unknown placeholders, and error cases. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HangDumpTests.cs | Adds integration coverage for HangDump template-based filenames. |
| src/Platform/Microsoft.Testing.Platform/Services/ServiceProviderExtensions.cs | Adds GetArtifactNamingService() accessor. |
| src/Platform/Microsoft.Testing.Platform/Services/IArtifactNamingService.cs | Introduces the internal artifact naming service contract. |
| src/Platform/Microsoft.Testing.Platform/Services/ArtifactNamingService.cs | Implements regex-based placeholder resolution and default replacement sources. |
| src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.cs | Registers ArtifactNamingService into the platform service provider. |
| src/Platform/Microsoft.Testing.Platform/Helpers/System/SystemEnvironment.cs | Implements newly added Version property on the environment wrapper. |
| src/Platform/Microsoft.Testing.Platform/Helpers/System/IEnvironment.cs | Adds Version to the environment abstraction. |
| src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs | Uses IArtifactNamingService for dump filename resolution (then applies legacy %p). |
| src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpExtensions.cs | Wires IArtifactNamingService into HangDumpProcessLifetimeHandler construction. |
| docs/ArtifactNamingService.md | Documents placeholders, custom replacements, and HangDump usage. |
Copilot's findings
- Files reviewed: 11/11 changed files
- Comments generated: 2
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…: Build ID 2970824 (#7973) Co-authored-by: Amaury Levé <amauryleve@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…Parser.ParseBool (#8098) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
…aseAttribute (#8091) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
…tionable TODOs (#8092) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
…informal spec & CI simplification (#8105) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…: Build ID 2972073 (#8107)
…er.MatchFilterPattern + CI fix (#8111) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…LLMEnvironmentDetector research (#8115) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ndArgument (#8116) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n, Mathlib removal, weekly schedule (#8117) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
…ibute() (#8103) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
…UniqueExtension (#8128) Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
# Conflicts: # docs/Changelog.md # eng/Version.Details.xml # eng/Versions.props # global.json
- Use TargetFrameworkAttribute.FrameworkDisplayName before falling back to RuntimeInformation.FrameworkDescription for <tfm> placeholder - Omit <asm> and <tfm> keys from replacements when null instead of using empty string (prevents path traversal with directory templates) - Strengthen subdirectory assertion to verify directory name equals 'HangDump'
| => RoslynString.IsNullOrWhiteSpace(template) | ||
| ? throw new ArgumentException("Template cannot be null, empty, or whitespace.", nameof(template)) | ||
| : replacements is null || replacements.Count == 0 | ||
| ? template | ||
| : TemplateFieldRegex.Replace(template, match => | ||
| { | ||
| string fieldName = match.Groups[1].Value; | ||
| return replacements.TryGetValue(fieldName, out string? value) && value is not null ? value : match.Value; | ||
| }); | ||
|
|
| .Replace("%p", processId); | ||
| finalDumpFileName = Path.GetFullPath(Path.Combine(_configuration.GetTestResultDirectory(), finalDumpFileName)); | ||
|
|
||
| // Ensure the destination directory exists (templates may include directory separators, e.g. <asm>/<pname>). | ||
| Directory.CreateDirectory(Path.GetDirectoryName(finalDumpFileName)!); |
Evangelink
left a comment
There was a problem hiding this comment.
Summary
Workflow: Test Expert Reviewer 🧪
Date: 2026-05-12
Repository: microsoft/testfx
Key Findings
The test suite is in good shape — previous reviews have successfully addressed the major findings (broken TFM regex, weak ContainsSingle assertion). Two minor coverage gaps remain in the unit tests:
-
[Coverage] No test for repeated placeholder substitution (
ArtifactNamingHelperTests.csline 88) —Regex.Replacewith aMatchEvaluatorreplaces all matches globally, but this global-replace behaviour is never explicitly asserted. A template like<pname>_<pname>.dmpwould be the minimal test. -
[Coverage] No test for a no-placeholder template with non-empty replacements (
ArtifactNamingHelperTests.csline 126) — TheNullReplacements/EmptyReplacementstests short-circuit before the regex runs. A plain template like"simple.dmp"with a non-empty dictionary exercises the regex path with zero matches, which is currently untested.
What looks good
- Isolation: No shared mutable state; every test uses local variables only.
- Error paths: All three invalid-template inputs (null, empty, whitespace) are covered with
Assert.ThrowsExactly. - Case-sensitivity: Explicitly tested and correctly asserted.
- Integration tests:
Guid.NewGuid()per-run directories avoid file-system conflicts;Assert.ContainsSingleused for deterministic dump-file detection; TFM regex correctly usesnet[\w.]+to covernet9.0,net10.0,net462, etc. - Platform assertion:
GetOperatingSystemName_ReturnsKnownValueusesAssert.Containsagainst a closed set and explicitly anchors the three known platforms.
Recommendations
Add the two suggested test cases noted above — they are small and will make the coverage complete.
Generated by Test Expert Reviewer 🧪
🧪 Test quality reviewed by Test Expert Reviewer 🧪
|
|
||
| string result = ArtifactNamingHelper.ResolveTemplate(template, replacements); | ||
|
|
||
| Assert.AreEqual("TestAssembly/test-process_12345_net9.0_2025-09-22_13-49-34.0000000.dmp", result); |
There was a problem hiding this comment.
[Coverage] No test verifies that a template with a repeated placeholder replaces all occurrences.
Regex.Replace with a MatchEvaluator (the approach in ArtifactNamingHelper) replaces all non-overlapping matches globally, but this behaviour is not explicitly exercised:
[TestMethod]
public void ResolveTemplate_RepeatedPlaceholder_ReplacesAllOccurrences()
{
string template = "<pname>_<pname>.dmp";
var replacements = new Dictionary<string, string> { ["pname"] = "test-process" };
string result = ArtifactNamingHelper.ResolveTemplate(template, replacements);
Assert.AreEqual("test-process_test-process.dmp", result);
}Impact: If someone inadvertently changed TemplateFieldRegex.Replace to a single-replace helper, no test would catch the regression.
| string result = ArtifactNamingHelper.ResolveTemplate(template, replacements); | ||
|
|
||
| Assert.AreEqual("<pname>_<pid>.dmp", result); | ||
| } |
There was a problem hiding this comment.
[Coverage] Missing test for a template that contains no angle-bracket placeholders when replacements are non-empty.
The existing NullReplacements and EmptyReplacements tests short-circuit at the replacements is null || replacements.Count == 0 guard and never invoke the regex. Passing a plain template with a non-empty dictionary exercises a distinct code path — the regex runs, finds zero matches, and returns the original template unchanged:
[TestMethod]
public void ResolveTemplate_NoPlaceholders_ReturnsTemplateUnchanged()
{
string template = "simple.dmp";
var replacements = new Dictionary<string, string> { ["pname"] = "test-process" };
string result = ArtifactNamingHelper.ResolveTemplate(template, replacements);
Assert.AreEqual("simple.dmp", result);
}Impact: Low — the behaviour is straightforward — but the line TemplateFieldRegex.Replace(template, ...) is not covered when the regex produces no matches.
Evangelink
left a comment
There was a problem hiding this comment.
Summary
Workflow: Expert Code Reviewer 🧠
Date: 2026-05-12
Repository: microsoft/testfx
Key Findings
[Correctness] <asm>/<tfm> placeholder omission creates invalid Windows paths when resolution fails
The latest commit (5f8bb212) correctly replaced the "" (empty-string) fallback for asm/tfm to fix a path-traversal risk (<asm>/ + empty string → root-relative path on Linux). However, the chosen fix — omitting the key from replacements when the value is null — introduces a symmetric correctness problem on Windows: if the entry assembly is null or TargetFrameworkParser can't parse the framework description, the placeholder remains as literal <asm> or <tfm> in the resolved filename. Path.GetFullPath then throws ArgumentException on Windows because < and > are illegal path characters.
Impact: ArgumentException crash on Windows for any user whose --hangdump-filename template uses <asm> or <tfm> and whose entry assembly or TFM resolution returns null. The <os> placeholder is immune because GetOperatingSystemName() always returns a safe string (falling back to "unknown").
Suggestion: Replace the conditional-add pattern with an unconditional fallback of "unknown", which is already the established pattern for <os>:
replacements["asm"] = asmName ?? "unknown";
replacements["tfm"] = tfm ?? "unknown";Positive Observations
- The overall design of
ArtifactNamingHelperas a file-linked,[Embedded]static class is clean and avoids IVT coupling. ResolveTemplatecorrectly preserves unknown placeholders as-is and is case-sensitive by design.- The static
RegexwithRegexOptions.Compiledis appropriately placed. - The integration tests are thorough, covering both the flat template case and the subdirectory case (with
<asm>/), and the unit tests cover all documented edge cases including case-sensitivity and null/empty inputs. - The legacy
%pbackward compatibility viastring.ReplaceafterResolveTemplateis a clean approach.
Overall Assessment
The PR is in very good shape — all previously flagged issues have been addressed. One narrow correctness issue remains in the new commit's null-handling strategy for <asm> and <tfm>.
Generated by Expert Code Reviewer 🧠
🧠 Reviewed by Expert Code Reviewer 🧠
| string? asmName = Assembly.GetEntryAssembly()?.GetName().Name; | ||
| if (asmName is not null) | ||
| { | ||
| replacements["asm"] = asmName; | ||
| } | ||
|
|
||
| string? tfm = TargetFrameworkParser.GetShortTargetFramework(Assembly.GetEntryAssembly()?.GetCustomAttribute<TargetFrameworkAttribute>()?.FrameworkDisplayName) | ||
| ?? TargetFrameworkParser.GetShortTargetFramework(RuntimeInformation.FrameworkDescription); | ||
| if (tfm is not null) | ||
| { | ||
| replacements["tfm"] = tfm; | ||
| } |
There was a problem hiding this comment.
[Correctness] When Assembly.GetEntryAssembly() returns null (or when TargetFrameworkParser fails to parse both FrameworkDisplayName and FrameworkDescription), the asm/tfm keys are intentionally omitted from replacements. If the user's template contains <asm> or <tfm>, ResolveTemplate preserves them as literal text — producing filenames like <asm>_hang.dmp. On Windows, < and > are illegal path characters, so the subsequent Path.GetFullPath call throws ArgumentException.
Impact: Crash on Windows for any user whose template includes <asm> or <tfm> when entry assembly resolution or TFM parsing fails. Contrast with <os>, which always resolves to a safe string via GetOperatingSystemName().
The previous behavior (empty-string fallback) was correctly replaced to prevent the path-traversal bug (<asm>/ → /<pname> on Linux). However, a safe non-empty fallback like "unknown" satisfies both constraints — no path bypass and no invalid characters:
replacements["asm"] = asmName ?? "unknown";
// ...
replacements["tfm"] = tfm ?? "unknown";This also gives consistent behaviour with <os>, which always returns "unknown" rather than omitting its key.
Summary
Add
ArtifactNamingHelper— a reusable static helper that allows extensions to generate consistent artifact file names using template placeholders.Users can specify templates like:
<pname>_<pid>_<time>_hang.dmp→MyTests_12345_2025-09-22_13-49-34.0000000_hang.dmp<pname>_<tfm>_<time>.log→MyTests_net9.0_2025-09-22_13-49-34.0000000.logAvailable Placeholders
<pname>MyTests<pid>12345<os>windows,linux,macos<asm>MyTests<tfm>net9.0<time>2025-09-22_13-49-34.0000000Key Design Decisions
ArtifactNamingHelperis a shared static class, compiled into each extension via file linking (no service registration or IVT required)<pname>, not<PName>)%ppattern handled in the HangDump caller viastring.ReplaceChanges
ArtifactNamingHelperstatic class (file-linked into extensions)HangDumpProcessLifetimeHandlerupdated to use the helperdocs/microsoft.testing.platform/ArtifactNamingHelper.mdNote
This PR revives and fixes the content from #6587 (closed copilot PR), with extensive code review fixes:
netstandard2.0(removedOperatingSystem.IsXxx(),KeyValuePairdeconstruction,[..8]range syntax)Guard.NotNullOrEmptyAPI%phandling<project>placeholder (directory tree traversal)EnvironmentAPI usage (RS0030)Fixes #6586
Related to #5364, #7345, #6648, #4130, #7126, #6778