[Test Optimization] Add test.final_status tag (#8091)

## Summary of changes

Add a new test.final_status tag to the final execution span of tests
across NUnit, XUnit, and MsTest frameworks. This tag represents the
adjusted final outcome of a test for CI pipeline result determination,
with values pass, fail, or skip.

Jira: SDTEST-2985

### Core changes:
- Add TestFinalStatus constant to TestTags.cs and FinalStatus property
to TestSpanTags.cs
- Add shared CalculateFinalStatus() helper in Common.cs implementing the
priority logic
- Implement final status tracking in NUnit
(TestOptimizationTestCommand.cs), XUnit (XUnitIntegration.cs), and
MsTest (TestMethodAttributeExecuteIntegration.cs)
- Add ATR budget pre-check methods
(GetRemainingBudget/GetRemainingAtrBudget) for early termination
detection
- Add per-row caching for MsTest parameterized tests to track execution
results independently

### Fix:
- Corrected **test.test_management.attempt_to_fix_passed** tag to only
be set on Attempt-to-Fix (ATF) tests; previously it was incorrectly set
on all retried tests (EFD, ATR, ATF) due to a hardcoded behavior type.

## Reason for change

When retry mechanisms are enabled (ATR, EFD, Attempt to Fix), a single
test can run multiple times with different outcomes. Some intermediate
outcomes are suppressed to avoid failing CI pipelines. Currently, there
is no way to query tests by their final adjusted status to build
monitors and alerts for hard failures on default branches.
The test.final_status tag enables customers to:
- Query tests by their effective CI outcome in Datadog
- Build monitors for tests that are truly failing (not just flaky)
- Distinguish between tests that eventually passed vs. those that
consistently failed
- Track quarantined/disabled tests separately from actual failures
#### Priority logic:
1. Quarantined/disabled tests → always skip (CI never sees actual
result)
2. **ATF tests with any execution failed → fail (test is still flaky,
fix didn't work)**
3. Any execution passed → pass (matches CI behavior: one pass = pipeline
pass)
4. Last retry is skip/inconclusive AND no pass → skip
  5. All executions failed → fail

#### ATF (Attempt to Fix) semantics:
For ATF tests, the goal is to determine if a fix actually resolved a
flaky test. Therefore:
- If **any** execution fails (initial or retry), the test is still flaky
→ `final_status = fail`
- Only if **all** executions pass, the fix worked → `final_status =
pass`
- Skip/inconclusive does **not** count as failure (only actual failures
count)
- `attempt_to_fix_passed` tag is derived from the same logic for
consistency

## Implementation details

#### Shared logic (Common.cs):
- CalculateFinalStatus(anyExecutionPassed, anyExecutionFailed,
isSkippedOrInconclusive, testTags) implements the 5-priority
determination
- Added `anyExecutionFailed` parameter to support ATF-specific behavior

#### NUnit:
- Extended RetryState struct with InitialExecutionPassed,
InitialExecutionFailed, and AnyRetryPassed fields
- Added GetRemainingBudget() to FlakyRetryBehavior for ATR budget
exhaustion pre-check
- Set final_status in ExecuteTest() before span close, handling ATR
early exit and budget exhaustion
- AllAttemptsPassed only clears on actual failure (not skip) for ATF
semantics
  - AttemptToFixPassed derived from anyExecutionFailed for consistency

#### XUnit:
- Extended TestCaseMetadata with InitialExecutionPassed,
InitialExecutionFailed, and AnyRetryPassed fields
- Added unified GetRemainingAtrBudget() handling both v2 and v3 via
Math.Max() strategy
- Updated WriteFinalTagsFromMetadata() with final execution detection
(handles stale TotalExecutions on initial EFD)
  - Track InitialExecutionFailed on exception path for first execution
  - AttemptToFixPassed derived from anyExecutionFailed for consistency

#### MsTest:
- Added per-row caches (InitialExecutionPassedCache,
InitialExecutionFailedCache, AnyRetryPassedCache,
AllAttemptsPassedCache) using ConditionalWeakTable<object,
ConcurrentDictionary<string, bool>> to track parameterized test results
independently
- Added SetFinalStatusIfApplicable() helper with proper final execution
detection
- Added GetRemainingAtrBudget() for budget exhaustion pre-check
- Updated SkipTestMethodExecutor and UnitTestRunnerRunSingleTest*
integrations for pre-execution skip paths
- Track InitialExecutionFailed in both exception and non-exception paths
  - AttemptToFixPassed derived from anyExecutionFailed for consistency

## Test coverage

#### Unit tests (TestFinalStatusTests.cs): 136 tests covering:
- Priority order verification (quarantined/disabled → ATF-fail → pass →
skip → fail)
- Single execution scenarios (pass/fail/skip)
- EFD scenarios (new tests, duration-based retries, slow abort)
- ATR scenarios (initial pass, retry pass, all fail, budget exhaustion)
  - ATF scenarios (any failure → fail, all pass → pass, skip semantics)
- ATF skip semantics: skip does NOT count as failure (5 dedicated tests)
- MsTest-specific: parameterized per-row tracking, class/assembly init
errors, inconclusive/not-runnable
- XUnit-specific: SkipException handling, null metadata
- NUnit-specific: ITR skipped, attribute skipped, inconclusive
- Mixed feature interactions (EFD+ATR, EFD+ATF, ATR+ATF)
- Edge cases: empty strings, case sensitivity, null tags

#### Integration tests:
- Added TestFinalStatus to span ordering in TestingFrameworkEvpTest.cs
for deterministic snapshot verification

## Other details
<!-- Fixes #{issue} -->


<!--  ⚠️ Note:

Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.

MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->

Author: tonyredondo

tracer/src/Datadog.Trace/Ci/Tagging/TestSpanTags.cs

@@ -127,6 +127,9 @@ public TestSpanTags(TestSuiteSpanTags suiteTags, string testName)
     [Tag(TestTags.TestAttemptToFixPassed)]
     public string? AttemptToFixPassed { get; set; }
 
+    [Tag(TestTags.TestFinalStatus)]
+    public string? FinalStatus { get; set; }
+
     [Tag(CapabilitiesTags.LibraryCapabilitiesTestImpactAnalysis)]
     public string? CapabilitiesTestImpactAnalysis { get; set; }
 

tracer/src/Datadog.Trace/Ci/Tags/TestTags.cs

@@ -156,6 +156,21 @@ internal static class TestTags
     /// </summary>
     public const string TestRetryReason = "test.retry_reason";
 
+    /// <summary>
+    /// Retry reason value for Early Flake Detection
+    /// </summary>
+    public const string TestRetryReasonEfd = "efd";
+
+    /// <summary>
+    /// Retry reason value for Auto Test Retries
+    /// </summary>
+    public const string TestRetryReasonAtr = "atr";
+
+    /// <summary>
+    /// Retry reason value for Attempt to Fix (Test Management)
+    /// </summary>
+    public const string TestRetryReasonAttemptToFix = "attempt_to_fix";
+
     /// <summary>
     /// Test is quarantined flag
     /// </summary>
@@ -185,4 +200,9 @@ internal static class TestTags
     /// Test management enabled flag
     /// </summary>
     public const string TestManagementEnabled = "test.test_management.enabled";
+
+    /// <summary>
+    /// Test final status - the adjusted test outcome for CI pipelines
+    /// </summary>
+    public const string TestFinalStatus = "test.final_status";
 }

tracer/src/Datadog.Trace/Ci/Test.cs

@@ -555,8 +555,8 @@ public void Close(TestStatus status, TimeSpan? duration, string? skipReason)
         {
             var retryReasonTag = tags.TestRetryReason switch
             {
-                "efd" => MetricTags.CIVisibilityTestingEventTypeRetryReason.EarlyFlakeDetection,
-                "atr" => MetricTags.CIVisibilityTestingEventTypeRetryReason.AutomaticTestRetry,
+                TestTags.TestRetryReasonEfd => MetricTags.CIVisibilityTestingEventTypeRetryReason.EarlyFlakeDetection,
+                TestTags.TestRetryReasonAtr => MetricTags.CIVisibilityTestingEventTypeRetryReason.AutomaticTestRetry,
                 _ => MetricTags.CIVisibilityTestingEventTypeRetryReason.None
             };
 

tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/Common.cs

@@ -10,6 +10,7 @@
 using System.Threading;
 using Datadog.Trace.Ci;
 using Datadog.Trace.Ci.Net;
+using Datadog.Trace.Ci.Tagging;
 using Datadog.Trace.Ci.Tags;
 using Datadog.Trace.Logging;
 using Datadog.Trace.Util;
@@ -188,7 +189,7 @@ internal static bool SetEarlyFlakeDetectionTestTagsAndAbortReason(Test test, boo
                 if (isRetry)
                 {
                     testTags.TestIsRetry = "true";
-                    testTags.TestRetryReason = "efd";
+                    testTags.TestRetryReason = TestTags.TestRetryReasonEfd;
                 }
                 else
                 {
@@ -211,7 +212,7 @@ internal static bool SetFlakyRetryTags(Test test, bool isRetry)
         {
             var testTags = test.GetTags();
             testTags.TestIsRetry = "true";
-            testTags.TestRetryReason = "atr";
+            testTags.TestRetryReason = TestTags.TestRetryReasonAtr;
         }
 
         return flakyRetryFeature;
@@ -244,7 +245,7 @@ internal static TestOptimizationClient.TestManagementResponseTestPropertiesAttri
                 if (isRetry)
                 {
                     testTags.TestIsRetry = "true";
-                    testTags.TestRetryReason = "attempt_to_fix";
+                    testTags.TestRetryReason = TestTags.TestRetryReasonAttemptToFix;
                 }
             }
 
@@ -273,4 +274,49 @@ internal static void CheckFaultyThreshold(Test test, long nTestCases, long tTest
             }
         }
     }
+
+    /// <summary>
+    /// Calculates the final status for a test based on execution results and test management tags.
+    /// Priority order (first match wins):
+    /// 1. Quarantined/disabled -> skip (always mask to skip)
+    /// 2. For ATF tests: any execution failed -> fail (flaky test = fix didn't work)
+    /// 3. Any execution passed -> pass
+    /// 4. Skip/inconclusive AND no pass -> skip
+    /// 5. All executions failed -> fail
+    /// </summary>
+    /// <param name="anyExecutionPassed">True if any execution (initial or retry) passed.</param>
+    /// <param name="anyExecutionFailed">True if any execution (initial or retry) failed.</param>
+    /// <param name="isSkippedOrInconclusive">True if the current/last execution was skip or inconclusive.</param>
+    /// <param name="testTags">The test tags to check for quarantine/disabled/ATF status.</param>
+    /// <returns>The final status string: "pass", "fail", or "skip".</returns>
+    internal static string CalculateFinalStatus(bool anyExecutionPassed, bool anyExecutionFailed, bool isSkippedOrInconclusive, TestSpanTags? testTags)
+    {
+        // Priority 1: Quarantined/disabled tests always mask to skip
+        if (testTags?.IsQuarantined == "true" || testTags?.IsDisabled == "true")
+        {
+            return TestTags.StatusSkip;
+        }
+
+        // Priority 2: For ATF tests, any failure means fix didn't work (test is still flaky)
+        // This must be checked BEFORE anyPassed for ATF tests
+        if (testTags?.IsAttemptToFix == "true" && anyExecutionFailed)
+        {
+            return TestTags.StatusFail;
+        }
+
+        // Priority 3: Any execution passed -> pass (pass takes precedence over skip)
+        if (anyExecutionPassed)
+        {
+            return TestTags.StatusPass;
+        }
+
+        // Priority 4: Skip/inconclusive AND no pass -> skip
+        if (isSkippedOrInconclusive)
+        {
+            return TestTags.StatusSkip;
+        }
+
+        // Priority 5: All executions failed -> fail
+        return TestTags.StatusFail;
+    }
 }

tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Testing/MsTestV2/SkipTestMethodExecutor.cs

@@ -8,6 +8,7 @@
 using System.Reflection;
 using System.Threading.Tasks;
 using Datadog.Trace.Ci;
+using Datadog.Trace.Ci.Tags;
 using Datadog.Trace.DuckTyping;
 
 namespace Datadog.Trace.ClrProfiler.AutoInstrumentation.Testing.MsTestV2;
@@ -37,8 +38,13 @@ protected void ProcessTestMethod(object testMethod)
         if (testMethod.TryDuckCast<ITestMethod>(out var testMethodInfo))
         {
             // Create the skip span
-            MsTestIntegration.OnMethodBegin(testMethodInfo, testMethod.GetType(), isRetry: false)?
-               .Close(TestStatus.Skip, TimeSpan.Zero, _skipReason);
+            var test = MsTestIntegration.OnMethodBegin(testMethodInfo, testMethod.GetType(), isRetry: false);
+            if (test is not null)
+            {
+                // Set final_status = skip for pre-execution skipped tests (ITR/attribute-based skips)
+                test.GetTags().FinalStatus = TestTags.StatusSkip;
+                test.Close(TestStatus.Skip, TimeSpan.Zero, _skipReason);
+            }
         }
     }