Skip to content

Add artifact naming service with template support for consistent naming across extensions#7856

Open
Evangelink wants to merge 112 commits into
mainfrom
dev/amauryleve/artifact-naming-service
Open

Add artifact naming service with template support for consistent naming across extensions#7856
Evangelink wants to merge 112 commits into
mainfrom
dev/amauryleve/artifact-naming-service

Conversation

@Evangelink
Copy link
Copy Markdown
Member

@Evangelink Evangelink commented Apr 27, 2026

Summary

Add ArtifactNamingHelper — a reusable static helper that allows extensions to generate consistent artifact file names using template placeholders.

Users can specify templates like:

  • <pname>_<pid>_<time>_hang.dmpMyTests_12345_2025-09-22_13-49-34.0000000_hang.dmp
  • <pname>_<tfm>_<time>.logMyTests_net9.0_2025-09-22_13-49-34.0000000.log

Available Placeholders

Placeholder Description Example
<pname> Process name MyTests
<pid> Process ID 12345
<os> Operating system windows, linux, macos
<asm> Assembly name MyTests
<tfm> Target framework moniker net9.0
<time> Timestamp (high precision, 100ns) 2025-09-22_13-49-34.0000000

Key Design Decisions

  • Static helperArtifactNamingHelper is a shared static class, compiled into each extension via file linking (no service registration or IVT required)
  • Case-sensitive placeholders — use lowercase names (e.g., <pname>, not <PName>)
  • Custom replacements allow callers to override defaults (e.g., when dumping a child process with a different PID)
  • Unknown placeholders are preserved as-is
  • Backward compatibility — legacy %p pattern handled in the HangDump caller via string.Replace
  • Filesystem-safe time format — uses hyphens and underscores instead of colons

Changes

  • New ArtifactNamingHelper static class (file-linked into extensions)
  • HangDumpProcessLifetimeHandler updated to use the helper
  • Unit tests and integration test
  • Documentation in docs/microsoft.testing.platform/ArtifactNamingHelper.md

Note

This PR revives and fixes the content from #6587 (closed copilot PR), with extensive code review fixes:

  • Fixed build for netstandard2.0 (removed OperatingSystem.IsXxx(), KeyValuePair deconstruction, [..8] range syntax)
  • Fixed non-existent Guard.NotNullOrEmpty API
  • Fixed broken legacy %p handling
  • Removed fragile <project> placeholder (directory tree traversal)
  • Fixed banned Environment API usage (RS0030)
  • Fixed all analyzer warnings

Fixes #6586

Related to #5364, #7345, #6648, #4130, #7126, #6778

Copilot AI and others added 16 commits September 22, 2025 12:04
Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
…ssing imports

Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
Co-authored-by: nohwnd <5735905+nohwnd@users.noreply.github.com>
…ames

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
# Conflicts:
#	src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs
#	src/Platform/Microsoft.Testing.Platform/Services/ServiceProviderExtensions.cs
- Remove <root> placeholder (fragile directory traversal)
- Fix OperatingSystem.IsWindows/IsLinux/IsMacOS for netstandard2.0 compat (use RuntimeInformation)
- Fix KeyValuePair deconstruction for netstandard2.0 compat
- Fix Guard.NotNullOrEmpty (non-existent API)
- Fix legacy %p pattern handling (move to HangDump caller, not the service)
- Fix time format to use hyphens instead of colons (filesystem safe)
- Remove unnecessary using directives
- Simplify GetOperatingSystemName with ternary expression
- Simplify TFM detection (remove hard-coded version map)
- Fix tests: use Assert.ThrowsExactly, match constructor signature, update assertions
Copilot AI review requested due to automatic review settings April 27, 2026 06:29
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an internal IArtifactNamingService to centralize artifact filename/path templating across Microsoft.Testing.Platform extensions, and migrates HangDump to use it (while keeping legacy %p support).

Changes:

  • Added IArtifactNamingService + ArtifactNamingService implementation with placeholder-based template resolution.
  • Registered the service in TestHostBuilder and exposed it via IServiceProvider extension method.
  • Updated HangDump to use the service and added/updated unit + integration tests plus documentation.
Show a summary per file
File Description
test/UnitTests/Microsoft.Testing.Platform.UnitTests/Services/ArtifactNamingServiceTests.cs Adds unit tests covering placeholder replacement, casing, unknown placeholders, and error cases.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HangDumpTests.cs Adds integration coverage for HangDump template-based filenames.
src/Platform/Microsoft.Testing.Platform/Services/ServiceProviderExtensions.cs Adds GetArtifactNamingService() accessor.
src/Platform/Microsoft.Testing.Platform/Services/IArtifactNamingService.cs Introduces the internal artifact naming service contract.
src/Platform/Microsoft.Testing.Platform/Services/ArtifactNamingService.cs Implements regex-based placeholder resolution and default replacement sources.
src/Platform/Microsoft.Testing.Platform/Hosts/TestHostBuilder.cs Registers ArtifactNamingService into the platform service provider.
src/Platform/Microsoft.Testing.Platform/Helpers/System/SystemEnvironment.cs Implements newly added Version property on the environment wrapper.
src/Platform/Microsoft.Testing.Platform/Helpers/System/IEnvironment.cs Adds Version to the environment abstraction.
src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpProcessLifetimeHandler.cs Uses IArtifactNamingService for dump filename resolution (then applies legacy %p).
src/Platform/Microsoft.Testing.Extensions.HangDump/HangDumpExtensions.cs Wires IArtifactNamingService into HangDumpProcessLifetimeHandler construction.
docs/ArtifactNamingService.md Documents placeholders, custom replacements, and HangDump usage.

Copilot's findings

  • Files reviewed: 11/11 changed files
  • Comments generated: 2

Comment thread src/Platform/Microsoft.Testing.Platform/Services/ArtifactNamingService.cs Outdated
Comment thread docs/ArtifactNamingService.md Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Services/ArtifactNamingService.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Services/IArtifactNamingService.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Services/ArtifactNamingService.cs Outdated
Comment thread src/Platform/Microsoft.Testing.Platform/Services/ArtifactNamingService.cs Outdated
dependabot Bot and others added 25 commits May 12, 2026 09:37
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…: Build ID 2970824 (#7973)

Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…Parser.ParseBool (#8098)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
…aseAttribute (#8091)

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
…tionable TODOs (#8092)

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
…informal spec & CI simplification (#8105)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…er.MatchFilterPattern + CI fix (#8111)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…LLMEnvironmentDetector research (#8115)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ndArgument (#8116)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n, Mathlib removal, weekly schedule (#8117)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
…ibute() (#8103)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: GitHub Copilot <copilot@github.com>
…UniqueExtension (#8128)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
# Conflicts:
#	docs/Changelog.md
#	eng/Version.Details.xml
#	eng/Versions.props
#	global.json
- Use TargetFrameworkAttribute.FrameworkDisplayName before falling back to
  RuntimeInformation.FrameworkDescription for <tfm> placeholder
- Omit <asm> and <tfm> keys from replacements when null instead of using
  empty string (prevents path traversal with directory templates)
- Strengthen subdirectory assertion to verify directory name equals 'HangDump'
Copilot AI review requested due to automatic review settings May 12, 2026 08:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 7/8 changed files
  • Comments generated: 2

Comment on lines +18 to +27
=> RoslynString.IsNullOrWhiteSpace(template)
? throw new ArgumentException("Template cannot be null, empty, or whitespace.", nameof(template))
: replacements is null || replacements.Count == 0
? template
: TemplateFieldRegex.Replace(template, match =>
{
string fieldName = match.Groups[1].Value;
return replacements.TryGetValue(fieldName, out string? value) && value is not null ? value : match.Value;
});

Comment on lines +337 to +341
.Replace("%p", processId);
finalDumpFileName = Path.GetFullPath(Path.Combine(_configuration.GetTestResultDirectory(), finalDumpFileName));

// Ensure the destination directory exists (templates may include directory separators, e.g. <asm>/<pname>).
Directory.CreateDirectory(Path.GetDirectoryName(finalDumpFileName)!);
Copy link
Copy Markdown
Member Author

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Workflow: Test Expert Reviewer 🧪
Date: 2026-05-12
Repository: microsoft/testfx

Key Findings

The test suite is in good shape — previous reviews have successfully addressed the major findings (broken TFM regex, weak ContainsSingle assertion). Two minor coverage gaps remain in the unit tests:

  1. [Coverage] No test for repeated placeholder substitution (ArtifactNamingHelperTests.cs line 88) — Regex.Replace with a MatchEvaluator replaces all matches globally, but this global-replace behaviour is never explicitly asserted. A template like <pname>_<pname>.dmp would be the minimal test.

  2. [Coverage] No test for a no-placeholder template with non-empty replacements (ArtifactNamingHelperTests.cs line 126) — The NullReplacements/EmptyReplacements tests short-circuit before the regex runs. A plain template like "simple.dmp" with a non-empty dictionary exercises the regex path with zero matches, which is currently untested.

What looks good

  • Isolation: No shared mutable state; every test uses local variables only.
  • Error paths: All three invalid-template inputs (null, empty, whitespace) are covered with Assert.ThrowsExactly.
  • Case-sensitivity: Explicitly tested and correctly asserted.
  • Integration tests: Guid.NewGuid() per-run directories avoid file-system conflicts; Assert.ContainsSingle used for deterministic dump-file detection; TFM regex correctly uses net[\w.]+ to cover net9.0, net10.0, net462, etc.
  • Platform assertion: GetOperatingSystemName_ReturnsKnownValue uses Assert.Contains against a closed set and explicitly anchors the three known platforms.

Recommendations

Add the two suggested test cases noted above — they are small and will make the coverage complete.


Generated by Test Expert Reviewer 🧪

🧪 Test quality reviewed by Test Expert Reviewer 🧪


string result = ArtifactNamingHelper.ResolveTemplate(template, replacements);

Assert.AreEqual("TestAssembly/test-process_12345_net9.0_2025-09-22_13-49-34.0000000.dmp", result);
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Coverage] No test verifies that a template with a repeated placeholder replaces all occurrences.

Regex.Replace with a MatchEvaluator (the approach in ArtifactNamingHelper) replaces all non-overlapping matches globally, but this behaviour is not explicitly exercised:

[TestMethod]
public void ResolveTemplate_RepeatedPlaceholder_ReplacesAllOccurrences()
{
    string template = "<pname>_<pname>.dmp";
    var replacements = new Dictionary<string, string> { ["pname"] = "test-process" };

    string result = ArtifactNamingHelper.ResolveTemplate(template, replacements);

    Assert.AreEqual("test-process_test-process.dmp", result);
}

Impact: If someone inadvertently changed TemplateFieldRegex.Replace to a single-replace helper, no test would catch the regression.

string result = ArtifactNamingHelper.ResolveTemplate(template, replacements);

Assert.AreEqual("<pname>_<pid>.dmp", result);
}
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Coverage] Missing test for a template that contains no angle-bracket placeholders when replacements are non-empty.

The existing NullReplacements and EmptyReplacements tests short-circuit at the replacements is null || replacements.Count == 0 guard and never invoke the regex. Passing a plain template with a non-empty dictionary exercises a distinct code path — the regex runs, finds zero matches, and returns the original template unchanged:

[TestMethod]
public void ResolveTemplate_NoPlaceholders_ReturnsTemplateUnchanged()
{
    string template = "simple.dmp";
    var replacements = new Dictionary<string, string> { ["pname"] = "test-process" };

    string result = ArtifactNamingHelper.ResolveTemplate(template, replacements);

    Assert.AreEqual("simple.dmp", result);
}

Impact: Low — the behaviour is straightforward — but the line TemplateFieldRegex.Replace(template, ...) is not covered when the regex produces no matches.

Copy link
Copy Markdown
Member Author

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Workflow: Expert Code Reviewer 🧠
Date: 2026-05-12
Repository: microsoft/testfx

Key Findings

[Correctness] <asm>/<tfm> placeholder omission creates invalid Windows paths when resolution fails

The latest commit (5f8bb212) correctly replaced the "" (empty-string) fallback for asm/tfm to fix a path-traversal risk (<asm>/ + empty string → root-relative path on Linux). However, the chosen fix — omitting the key from replacements when the value is null — introduces a symmetric correctness problem on Windows: if the entry assembly is null or TargetFrameworkParser can't parse the framework description, the placeholder remains as literal <asm> or <tfm> in the resolved filename. Path.GetFullPath then throws ArgumentException on Windows because < and > are illegal path characters.

Impact: ArgumentException crash on Windows for any user whose --hangdump-filename template uses <asm> or <tfm> and whose entry assembly or TFM resolution returns null. The <os> placeholder is immune because GetOperatingSystemName() always returns a safe string (falling back to "unknown").

Suggestion: Replace the conditional-add pattern with an unconditional fallback of "unknown", which is already the established pattern for <os>:

replacements["asm"] = asmName ?? "unknown";
replacements["tfm"] = tfm ?? "unknown";

Positive Observations

  • The overall design of ArtifactNamingHelper as a file-linked, [Embedded] static class is clean and avoids IVT coupling.
  • ResolveTemplate correctly preserves unknown placeholders as-is and is case-sensitive by design.
  • The static Regex with RegexOptions.Compiled is appropriately placed.
  • The integration tests are thorough, covering both the flat template case and the subdirectory case (with <asm>/), and the unit tests cover all documented edge cases including case-sensitivity and null/empty inputs.
  • The legacy %p backward compatibility via string.Replace after ResolveTemplate is a clean approach.

Overall Assessment

The PR is in very good shape — all previously flagged issues have been addressed. One narrow correctness issue remains in the new commit's null-handling strategy for <asm> and <tfm>.


Generated by Expert Code Reviewer 🧠

🧠 Reviewed by Expert Code Reviewer 🧠

Comment on lines +320 to +331
string? asmName = Assembly.GetEntryAssembly()?.GetName().Name;
if (asmName is not null)
{
replacements["asm"] = asmName;
}

string? tfm = TargetFrameworkParser.GetShortTargetFramework(Assembly.GetEntryAssembly()?.GetCustomAttribute<TargetFrameworkAttribute>()?.FrameworkDisplayName)
?? TargetFrameworkParser.GetShortTargetFramework(RuntimeInformation.FrameworkDescription);
if (tfm is not null)
{
replacements["tfm"] = tfm;
}
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Correctness] When Assembly.GetEntryAssembly() returns null (or when TargetFrameworkParser fails to parse both FrameworkDisplayName and FrameworkDescription), the asm/tfm keys are intentionally omitted from replacements. If the user's template contains <asm> or <tfm>, ResolveTemplate preserves them as literal text — producing filenames like <asm>_hang.dmp. On Windows, < and > are illegal path characters, so the subsequent Path.GetFullPath call throws ArgumentException.

Impact: Crash on Windows for any user whose template includes <asm> or <tfm> when entry assembly resolution or TFM parsing fails. Contrast with <os>, which always resolves to a safe string via GetOperatingSystemName().

The previous behavior (empty-string fallback) was correctly replaced to prevent the path-traversal bug (<asm>//<pname> on Linux). However, a safe non-empty fallback like "unknown" satisfies both constraints — no path bypass and no invalid characters:

replacements["asm"] = asmName ?? "unknown";

// ...

replacements["tfm"] = tfm ?? "unknown";

This also gives consistent behaviour with <os>, which always returns "unknown" rather than omitting its key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add artifact naming service

6 participants