Skip to content

Commit 32acee9

Browse files
authored
[CI Visibility] Implement ITR code coverage backfill (#8626)
## Summary of changes - Implements ITR code coverage backfill for .NET test runs that skip tests while a line-coverage source is active. - Parses backend `meta.coverage` bitmaps from skippable-tests responses using the .NET `FileBitmap` representation and persists scoped backfill data across child processes, injected sessions, AppDomains, and coverage tool adapters. - Backfills line coverage for Datadog internal coverage, Coverlet in-memory results, Coverlet collector XML attachments, Cobertura/OpenCover XML, and line-capable Microsoft CodeCoverage XML reports. - Rewrites supported customer-visible report files before downstream coverage tools read their final results. - Publishes one session-level `test.code_coverage.lines_pct` result through source arbitration, with `test.code_coverage.backfilled` when the selected result used backend-aware coverage data. - Keeps ITR skip behavior aligned with the Java reference implementation: when line coverage is active, a skippable test is denied only when the backend marks that candidate with `_missing_line_code_coverage=true`. Missing or invalid aggregate `meta.coverage` disables safe backfill publication, but does not globally disable ITR skipping. - Sends custom test configuration tags to the backend and trusts the backend response, matching Java. The tracer does not locally filter skippable tests by `custom` configurations. ## Reason for change ITR can skip tests that would otherwise contribute line coverage. Without backfill, coverage tools and the Datadog session metric can under-report line coverage for a run where skipped tests were intentionally omitted. The backend now returns skipped-test line coverage in the skippable-tests response, so the tracer can merge backend line bitmaps with local coverage before publishing the session result or rewriting external reports. References: - RFC: https://docs.google.com/document/d/1234bLTvL-v9PzA3zIAf_2NN8DdTpfLM8NX4WiniPssw/edit?tab=t.0#heading=h.rnd972k0hiye - Java reference implementation: DataDog/dd-trace-java#7367 ## Java compatibility notes This PR follows the Java reference behavior for the user-visible ITR decision: - `custom` test configurations are sent to the backend, but are not used for local post-filtering. - `test.bundle`/module scope is used to associate backend candidates with the current module scope. - `_missing_line_code_coverage=true` prevents skipping that candidate when line coverage is active. - Missing, absent, or invalid aggregate `meta.coverage` does not prevent ITR from skipping tests. It only means the .NET tracer must avoid publishing a backfilled coverage result that cannot be proven safe. The intentional implementation difference is bitmap representation: Java decodes backend coverage into `BitSet`, while .NET keeps the existing `FileBitmap` representation for both decoding backend coverage and sending/persisting coverage data. ## Review diagrams ### End-to-end backfill flow ```mermaid flowchart TD A["dotnet test / vstest / coverage command starts"] B["CoverageBackfillCapability inspects command, runsettings, response files, and report outputs"] C{"Line-capable coverage source active?"} D["Legacy ITR behavior; no backfill required"] E["Request scoped skippable tests for the current test.bundle/module"] F["Backend returns skippable candidates, _missing_line_code_coverage flags, and optional meta.coverage"] G["Java-compatible skip gate decides whether each candidate may be skipped"] H["Framework integration records the actual ITR skip only after it commits to skipping"] I["CoverageBackfillData decodes meta.coverage with FileBitmap"] J["CoverageBackfillDataStore persists backend coverage, actual-skip state, scoped backfillable state, IPC results, and IPC failures"] K["Coverage producers apply or rewrite line coverage results"] L["Validation checks paths, represented backend lines, producer completeness, and IPC state"] M["CodeCoverageResultAggregator selects one publishable source by priority"] N["Session publishes test.code_coverage.lines_pct and test.code_coverage.backfilled when applicable"] A --> B --> C C -- "no" --> D C -- "yes" --> E --> F F --> G --> H --> J F --> I --> J J --> K --> L --> M --> N ``` ### Skippable scope and Java-compatible skip gate ```mermaid flowchart TD A["Skippable-tests response item"] B{"Top-level test.bundle mismatches scoped request?"} C["Drop only this mismatched candidate"] D["Keep candidate; do not filter by custom configs"] E{"Line coverage active?"} F["Use legacy ITR skip behavior"] G{"Candidate has _missing_line_code_coverage=true?"} H["Do not skip this candidate"] I["Allow ITR skip"] J["meta.coverage is decoded separately and is not part of the skip decision"] A --> B B -- "yes" --> C B -- "no" --> D D --> E E -- "no" --> F --> I E -- "yes" --> G G -- "yes" --> H G -- "no or absent" --> I J -. "matches Java" .-> G ``` ### Coverage producers, persistence, and publication ```mermaid flowchart LR A["Backend meta.coverage FileBitmap data"] B["Actual ITR skip markers"] C["CoverageBackfillDataStore"] subgraph Producers["Coverage producers"] D["Datadog internal coverage"] E["Coverlet in-memory result"] F["Coverlet collector XML fallback"] G["Cobertura/OpenCover XML"] H["Microsoft CodeCoverage XML"] end I["Apply backend line bitmaps or rewrite customer-visible XML"] J["Send live IPC result or persist result/failure marker"] K["Validate source paths, represented lines, partial merges, and IPC completeness"] L["Suppress stale or unsafe results"] M["Select highest-priority publishable source"] N["Publish final session coverage tags"] A --> C B --> C C --> Producers Producers --> I --> J --> K K -- "safe" --> M --> N K -- "unsafe" --> L ``` ## Implementation details - Adds `CoverageBackfillData` parsing for backend `meta.coverage`, including bitmap decoding, validation, merging, and source-path normalization. - Adds `CoverageBackfillDataStore` to persist backend coverage, actual-skip markers, backfillable scoped skip state, coverage IPC results, and coverage IPC failures across process and AppDomain boundaries. - Adds scoped skippable-tests requests when coverage backfill is required, using `test.bundle` plus runtime/configuration fingerprinting so backend coverage matches the current local execution scope. - Separates skippable candidates from tests actually skipped by ITR. Framework integrations record coverage-backfill skip state when they commit to an ITR skip; `Test.Close` records the session-level "ITR skip happened" tag state later. - Uses Java-compatible skip gating: the skip decision checks the backend candidate's `_missing_line_code_coverage` flag, while aggregate backend coverage validity is tracked separately for deciding whether a backfilled coverage result can be published. - Adds `CoverageBackfillCapability` and command-line parsing for activation checks across `dotnet test`, `vstest`, `dotnet-coverage`, Coverlet collector/MSBuild, Microsoft CodeCoverage, response files, runsettings, MSBuild properties, filters, framework selectors, thresholds, and generated report paths. - Adds shared path matching and validation state so partial coverage reports can only publish when their merged result represents all required backend skipped-test lines and did not rely on unsafe local suffix matches. - Adds adapters for Datadog internal global coverage, Coverlet modules, Coverlet collector XML fallback, external Cobertura/OpenCover XML, and Microsoft CodeCoverage XML. - Adds `CodeCoverageResultAggregator` to select the final session result by source priority and suppress ambiguous count-based same-source merges. ## Key behavior cases - If no coverage report source is active, ITR keeps the legacy skip behavior and no coverage backfill is applied. - If line coverage is active, a skippable test is not skipped only when the backend marks that candidate as missing line coverage. - If backend aggregate `meta.coverage` is missing or invalid, ITR can still skip tests, but the session will not publish a backfilled coverage result unless another validated producer result is available. - If coverage is active but no test is actually skipped by ITR, reports can still be read normally and backend backfill is not applied. - If an external XML report is selected, it has higher priority than internal or IPC-based coverage results. - If a child coverage tool cannot deliver a complete result when backfill publication is required, the session fails closed for coverage publication instead of publishing stale lower-priority coverage. - Branch, method, condition, complexity, and threshold metadata are preserved but not reconstructed from backend data; backend coverage is line bitmap data only. ## Source priority `CodeCoverageResultAggregator` selects the highest-priority publishable result: 1. `ExternalXml` 2. `CoverletXmlFallback` 3. `Coverlet` 4. `MicrosoftCodeCoverage` 5. `Unknown` 6. `DatadogInternal` Same-source results with line counts are only publishable when they were explicitly merged by the producer or can be validated without ambiguous count aggregation. ## Type guide | Type | Responsibility | | --- | --- | | `CoverageBackfillData` | Decodes and validates backend `meta.coverage` bitmaps. | | `CoverageBackfillDataStore` | Persists backend coverage, actual-skip state, scoped backfillable skip state, IPC results, and IPC failures. | | `CoverageBackfillCapability` | Determines whether backfill is required and whether the active coverage command/report path is safe. | | `CoverageBackfillCommandLine` | Parses command lines, response files, runsettings, MSBuild properties, and coverage output options for capability checks. | | `CoverageBackfillPathMatcher` / `CoverageBackfillPathMatchTracker` | Matches backend source paths to local report paths and detects unsafe suffix ambiguity. | | `CoverageBackfillApplicator` | Applies backend line bitmaps to Datadog internal global coverage. | | `ExternalCoverageXmlBackfill` | Parses, validates, rewrites, and merges line-capable XML reports. | | `CoverletCoverageBackfill` | Mutates Coverlet's in-memory modules before Coverlet calculates its line summary. | | `CodeCoverageBackfillValidation` | Carries cross-result validation state so partial producer results only publish when the merged set is complete and safe. | | `TestOptimizationSkippableFeature` | Requests scoped skippable tests, gates coverage-active skips, and records actual ITR skip state. | | `SessionCodeCoverageMessage` | Sends source-specific coverage results from child tools/domains to the parent test session. | | `CodeCoverageResultAggregator` | Suppresses unsafe results, merges validated producer output, and selects the final session coverage source by priority. | ## Test coverage Added and expanded coverage across: - backend `meta.coverage` parsing, bitmap merging, invalid payload handling, missing coverage, and missing-line metadata; - Java-compatible skippable parsing behavior for `custom` configurations and `test.bundle` module scope; - scoped skippable requests, actual-skip tracking, bundle/config fingerprinting, and coverage-publication safety state; - Datadog internal coverage backfill and `LineCallCount` semantics; - Coverlet module mutation, rollback, validation, partial result merging, and IPC failure paths; - Cobertura, OpenCover, Microsoft CodeCoverage XML parsing/rewrite/merge paths; - path normalization, suffix ambiguity detection, source-root ambiguity, represented-line validation, and multi-report validation; - `dotnet test`, VSTest, Coverlet, Microsoft CodeCoverage, `dotnet-coverage`, response file, runsettings, and MSBuild command detection; - session result arbitration, persisted IPC recovery, atomic marker/result races, late Coverlet XML fallback, and injected session behavior. Recent validation: - Java-parity focused unit tests for skippable parsing, skip gating, and backend coverage decoding passed on `net10.0`. - `AttributeTests.AttributesInstantiationsOnlyUseBuiltinTypes` passed on `net10.0` and `net9.0` x64. - `IpcTests.IpcClientCanSendCoverageMessageWithBackfillValidation` passed on `net10.0`. - `ManagedVanguardStopIntegrationTests.DeduplicatesEquivalentMicrosoftLineXmlPathsBeforeBackfill` passed on `net10.0`. - `git diff --check`: clean.
1 parent 2d3de8b commit 32acee9

112 files changed

Lines changed: 49953 additions & 586 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

tracer/src/Datadog.Trace.Coverage.collector/AssemblyProcessor.cs

Lines changed: 183 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,11 @@ internal class AssemblyProcessor
5151
"Xunit.SkippableFact.dll",
5252
};
5353

54+
private static readonly string[] IgnoredAssemblyPrefixes =
55+
{
56+
"coverlet.",
57+
};
58+
5459
private readonly CoverageSettings _settings;
5560
private readonly ICollectorLogger _logger;
5661
private readonly string _assemblyFilePath;
@@ -97,9 +102,10 @@ public unsafe void Process()
97102
_logger.Debug($"Processing: {_assemblyFilePath}");
98103

99104
// Check if the assembly is in the ignored assemblies list.
100-
var assemblyFullName = Path.GetFileName(_assemblyFilePath);
101-
if (Array.Exists(IgnoredAssemblies, i => assemblyFullName == i))
105+
var assemblyFileName = Path.GetFileName(_assemblyFilePath);
106+
if (IsIgnoredAssembly(assemblyFileName))
102107
{
108+
_logger.Debug($"Assembly: {FilePath}, ignored by assembly name.");
103109
return;
104110
}
105111

@@ -170,6 +176,18 @@ public unsafe void Process()
170176
_logger.Debug($"{snkFilePath} exists.");
171177
_strongNameKeyBlob = File.ReadAllBytes(snkFilePath);
172178
_logger.Debug($"{snkFilePath} loaded.");
179+
180+
if (!TryGetStrongNamePublicKey(_strongNameKeyBlob, out _))
181+
{
182+
_logger.Warning($"Assembly: {FilePath}, the .snk file configured in {Configuration.ConfigurationKeys.CIVisibility.CodeCoverageSnkFile} could not be read. Skipping coverage instrumentation to avoid changing the assembly identity.");
183+
return;
184+
}
185+
186+
if (!StrongNameKeyMatchesAssemblyPublicKey(assemblyDefinition.Name, _strongNameKeyBlob))
187+
{
188+
_logger.Warning($"Assembly: {FilePath}, is signed with a different strong-name key than the .snk file configured in {Configuration.ConfigurationKeys.CIVisibility.CodeCoverageSnkFile}. Skipping coverage instrumentation to avoid changing the assembly identity.");
189+
return;
190+
}
173191
}
174192
else if (tracerTarget == TracerTarget.Net461)
175193
{
@@ -774,6 +792,169 @@ internal static void WriteTargetAssembly(AssemblyDefinition assemblyDefinition,
774792
});
775793
}
776794

795+
internal static bool IsIgnoredAssembly(string? assemblyFileName)
796+
{
797+
if (assemblyFileName is not { Length: > 0 } fileName)
798+
{
799+
return false;
800+
}
801+
802+
return Array.Exists(IgnoredAssemblies, ignoredAssembly => string.Equals(fileName, ignoredAssembly, StringComparison.OrdinalIgnoreCase)) ||
803+
Array.Exists(IgnoredAssemblyPrefixes, ignoredPrefix => fileName.StartsWith(ignoredPrefix, StringComparison.OrdinalIgnoreCase));
804+
}
805+
806+
internal static bool IsIgnoredAssemblyDependencyManifest(string? depsJsonFileName)
807+
{
808+
const string depsJsonSuffix = ".deps.json";
809+
if (depsJsonFileName is not { Length: > 0 } fileName ||
810+
!fileName.EndsWith(depsJsonSuffix, StringComparison.OrdinalIgnoreCase))
811+
{
812+
return false;
813+
}
814+
815+
var assemblyFileName = fileName.Substring(0, fileName.Length - depsJsonSuffix.Length) + ".dll";
816+
return IsIgnoredAssembly(assemblyFileName);
817+
}
818+
819+
// Cecil derives the assembly identity from the strong-name public key blob before signing.
820+
// Mirror that conversion so mismatched configured SNK files are detected before rewriting.
821+
internal static bool StrongNameKeyMatchesAssemblyPublicKey(AssemblyNameDefinition assemblyName, byte[] strongNameKeyBlob)
822+
{
823+
return TryGetStrongNamePublicKey(strongNameKeyBlob, out var publicKey) &&
824+
assemblyName.PublicKey is { Length: > 0 } assemblyPublicKey &&
825+
assemblyPublicKey.SequenceEqual(publicKey);
826+
}
827+
828+
internal static bool TryGetStrongNamePublicKey(byte[]? strongNameKeyBlob, out byte[] publicKey)
829+
{
830+
publicKey = Array.Empty<byte>();
831+
832+
if (strongNameKeyBlob is not { Length: > 20 })
833+
{
834+
return false;
835+
}
836+
837+
if (IsStrongNamePublicKeyBlob(strongNameKeyBlob))
838+
{
839+
publicKey = strongNameKeyBlob;
840+
return true;
841+
}
842+
843+
byte[] capiPublicKeyBlob;
844+
if (IsCapiPublicKeyBlob(strongNameKeyBlob))
845+
{
846+
capiPublicKeyBlob = strongNameKeyBlob;
847+
}
848+
else if (!TryGetCapiPublicKeyBlobFromPrivateKey(strongNameKeyBlob, out capiPublicKeyBlob))
849+
{
850+
return false;
851+
}
852+
853+
publicKey = CreateStrongNamePublicKey(capiPublicKeyBlob);
854+
return true;
855+
}
856+
857+
private static bool IsStrongNamePublicKeyBlob(byte[] blob)
858+
{
859+
return blob.Length > 12 &&
860+
blob[0] == 0x00 &&
861+
blob[12] == 0x06 &&
862+
IsCapiPublicKeyBlob(blob, 12);
863+
}
864+
865+
private static bool IsCapiPublicKeyBlob(byte[] blob, int offset = 0)
866+
{
867+
if (blob.Length < offset + 20 ||
868+
blob[offset] != 0x06 ||
869+
blob[offset + 1] != 0x02 ||
870+
blob[offset + 2] != 0x00 ||
871+
blob[offset + 3] != 0x00 ||
872+
ReadUInt32LittleEndian(blob, offset + 8) != 0x31415352)
873+
{
874+
return false;
875+
}
876+
877+
var bitLength = ReadInt32LittleEndian(blob, offset + 12);
878+
if (bitLength <= 0 || bitLength % 8 != 0)
879+
{
880+
return false;
881+
}
882+
883+
var keyLength = bitLength / 8;
884+
return blob.Length >= offset + 20 + keyLength;
885+
}
886+
887+
private static bool TryGetCapiPublicKeyBlobFromPrivateKey(byte[] privateKeyBlob, out byte[] publicKeyBlob)
888+
{
889+
publicKeyBlob = Array.Empty<byte>();
890+
891+
if (privateKeyBlob.Length < 20 ||
892+
privateKeyBlob[0] != 0x07 ||
893+
privateKeyBlob[1] != 0x02 ||
894+
privateKeyBlob[2] != 0x00 ||
895+
privateKeyBlob[3] != 0x00 ||
896+
ReadUInt32LittleEndian(privateKeyBlob, 8) != 0x32415352)
897+
{
898+
return false;
899+
}
900+
901+
var bitLength = ReadInt32LittleEndian(privateKeyBlob, 12);
902+
if (bitLength <= 0 || bitLength % 8 != 0)
903+
{
904+
return false;
905+
}
906+
907+
var keyLength = bitLength / 8;
908+
if (privateKeyBlob.Length < 20 + keyLength)
909+
{
910+
return false;
911+
}
912+
913+
publicKeyBlob = new byte[20 + keyLength];
914+
publicKeyBlob[0] = 0x06;
915+
publicKeyBlob[1] = 0x02;
916+
publicKeyBlob[5] = 0x24;
917+
publicKeyBlob[8] = 0x52;
918+
publicKeyBlob[9] = 0x53;
919+
publicKeyBlob[10] = 0x41;
920+
publicKeyBlob[11] = 0x31;
921+
Buffer.BlockCopy(privateKeyBlob, 12, publicKeyBlob, 12, 8);
922+
Buffer.BlockCopy(privateKeyBlob, 20, publicKeyBlob, 20, keyLength);
923+
return true;
924+
}
925+
926+
private static byte[] CreateStrongNamePublicKey(byte[] capiPublicKeyBlob)
927+
{
928+
var publicKey = new byte[12 + capiPublicKeyBlob.Length];
929+
Buffer.BlockCopy(capiPublicKeyBlob, 0, publicKey, 12, capiPublicKeyBlob.Length);
930+
publicKey[1] = 0x24;
931+
publicKey[4] = 0x04;
932+
publicKey[5] = 0x80;
933+
WriteInt32LittleEndian(publicKey, 8, capiPublicKeyBlob.Length);
934+
return publicKey;
935+
}
936+
937+
private static int ReadInt32LittleEndian(byte[] buffer, int offset)
938+
{
939+
return (int)ReadUInt32LittleEndian(buffer, offset);
940+
}
941+
942+
private static uint ReadUInt32LittleEndian(byte[] buffer, int offset)
943+
{
944+
return buffer[offset] |
945+
((uint)buffer[offset + 1] << 8) |
946+
((uint)buffer[offset + 2] << 16) |
947+
((uint)buffer[offset + 3] << 24);
948+
}
949+
950+
private static void WriteInt32LittleEndian(byte[] buffer, int offset, int value)
951+
{
952+
buffer[offset] = (byte)value;
953+
buffer[offset + 1] = (byte)(value >> 8);
954+
buffer[offset + 2] = (byte)(value >> 16);
955+
buffer[offset + 3] = (byte)(value >> 24);
956+
}
957+
777958
private static void RemoveShortOpCodes(Instruction instruction)
778959
{
779960
if (instruction.OpCode == OpCodes.Br_S) { instruction.OpCode = OpCodes.Br; }

tracer/src/Datadog.Trace.Coverage.collector/CoverageCollector.cs

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,12 @@ private void ProcessFolder(string folder, SearchOption searchOption)
252252
var version = typeof(Instrumentation).Assembly.GetName().Version?.ToString();
253253
foreach (var depsJsonPath in Directory.EnumerateFiles(directory, "*.deps.json", SearchOption.TopDirectoryOnly))
254254
{
255+
if (AssemblyProcessor.IsIgnoredAssemblyDependencyManifest(Path.GetFileName(depsJsonPath)))
256+
{
257+
_logger?.Debug($"Skipping dependency manifest for ignored assembly: {depsJsonPath}");
258+
continue;
259+
}
260+
255261
try
256262
{
257263
var json = JObject.Parse(File.ReadAllText(depsJsonPath));

0 commit comments

Comments
 (0)