Commit 32acee9
authored
[CI Visibility] Implement ITR code coverage backfill (#8626)
## Summary of changes
- Implements ITR code coverage backfill for .NET test runs that skip
tests while a line-coverage source is active.
- Parses backend `meta.coverage` bitmaps from skippable-tests responses
using the .NET `FileBitmap` representation and persists scoped backfill
data across child processes, injected sessions, AppDomains, and coverage
tool adapters.
- Backfills line coverage for Datadog internal coverage, Coverlet
in-memory results, Coverlet collector XML attachments,
Cobertura/OpenCover XML, and line-capable Microsoft CodeCoverage XML
reports.
- Rewrites supported customer-visible report files before downstream
coverage tools read their final results.
- Publishes one session-level `test.code_coverage.lines_pct` result
through source arbitration, with `test.code_coverage.backfilled` when
the selected result used backend-aware coverage data.
- Keeps ITR skip behavior aligned with the Java reference
implementation: when line coverage is active, a skippable test is denied
only when the backend marks that candidate with
`_missing_line_code_coverage=true`. Missing or invalid aggregate
`meta.coverage` disables safe backfill publication, but does not
globally disable ITR skipping.
- Sends custom test configuration tags to the backend and trusts the
backend response, matching Java. The tracer does not locally filter
skippable tests by `custom` configurations.
## Reason for change
ITR can skip tests that would otherwise contribute line coverage.
Without backfill, coverage tools and the Datadog session metric can
under-report line coverage for a run where skipped tests were
intentionally omitted. The backend now returns skipped-test line
coverage in the skippable-tests response, so the tracer can merge
backend line bitmaps with local coverage before publishing the session
result or rewriting external reports.
References:
- RFC:
https://docs.google.com/document/d/1234bLTvL-v9PzA3zIAf_2NN8DdTpfLM8NX4WiniPssw/edit?tab=t.0#heading=h.rnd972k0hiye
- Java reference implementation:
DataDog/dd-trace-java#7367
## Java compatibility notes
This PR follows the Java reference behavior for the user-visible ITR
decision:
- `custom` test configurations are sent to the backend, but are not used
for local post-filtering.
- `test.bundle`/module scope is used to associate backend candidates
with the current module scope.
- `_missing_line_code_coverage=true` prevents skipping that candidate
when line coverage is active.
- Missing, absent, or invalid aggregate `meta.coverage` does not prevent
ITR from skipping tests. It only means the .NET tracer must avoid
publishing a backfilled coverage result that cannot be proven safe.
The intentional implementation difference is bitmap representation: Java
decodes backend coverage into `BitSet`, while .NET keeps the existing
`FileBitmap` representation for both decoding backend coverage and
sending/persisting coverage data.
## Review diagrams
### End-to-end backfill flow
```mermaid
flowchart TD
A["dotnet test / vstest / coverage command starts"]
B["CoverageBackfillCapability inspects command, runsettings, response files, and report outputs"]
C{"Line-capable coverage source active?"}
D["Legacy ITR behavior; no backfill required"]
E["Request scoped skippable tests for the current test.bundle/module"]
F["Backend returns skippable candidates, _missing_line_code_coverage flags, and optional meta.coverage"]
G["Java-compatible skip gate decides whether each candidate may be skipped"]
H["Framework integration records the actual ITR skip only after it commits to skipping"]
I["CoverageBackfillData decodes meta.coverage with FileBitmap"]
J["CoverageBackfillDataStore persists backend coverage, actual-skip state, scoped backfillable state, IPC results, and IPC failures"]
K["Coverage producers apply or rewrite line coverage results"]
L["Validation checks paths, represented backend lines, producer completeness, and IPC state"]
M["CodeCoverageResultAggregator selects one publishable source by priority"]
N["Session publishes test.code_coverage.lines_pct and test.code_coverage.backfilled when applicable"]
A --> B --> C
C -- "no" --> D
C -- "yes" --> E --> F
F --> G --> H --> J
F --> I --> J
J --> K --> L --> M --> N
```
### Skippable scope and Java-compatible skip gate
```mermaid
flowchart TD
A["Skippable-tests response item"]
B{"Top-level test.bundle mismatches scoped request?"}
C["Drop only this mismatched candidate"]
D["Keep candidate; do not filter by custom configs"]
E{"Line coverage active?"}
F["Use legacy ITR skip behavior"]
G{"Candidate has _missing_line_code_coverage=true?"}
H["Do not skip this candidate"]
I["Allow ITR skip"]
J["meta.coverage is decoded separately and is not part of the skip decision"]
A --> B
B -- "yes" --> C
B -- "no" --> D
D --> E
E -- "no" --> F --> I
E -- "yes" --> G
G -- "yes" --> H
G -- "no or absent" --> I
J -. "matches Java" .-> G
```
### Coverage producers, persistence, and publication
```mermaid
flowchart LR
A["Backend meta.coverage FileBitmap data"]
B["Actual ITR skip markers"]
C["CoverageBackfillDataStore"]
subgraph Producers["Coverage producers"]
D["Datadog internal coverage"]
E["Coverlet in-memory result"]
F["Coverlet collector XML fallback"]
G["Cobertura/OpenCover XML"]
H["Microsoft CodeCoverage XML"]
end
I["Apply backend line bitmaps or rewrite customer-visible XML"]
J["Send live IPC result or persist result/failure marker"]
K["Validate source paths, represented lines, partial merges, and IPC completeness"]
L["Suppress stale or unsafe results"]
M["Select highest-priority publishable source"]
N["Publish final session coverage tags"]
A --> C
B --> C
C --> Producers
Producers --> I --> J --> K
K -- "safe" --> M --> N
K -- "unsafe" --> L
```
## Implementation details
- Adds `CoverageBackfillData` parsing for backend `meta.coverage`,
including bitmap decoding, validation, merging, and source-path
normalization.
- Adds `CoverageBackfillDataStore` to persist backend coverage,
actual-skip markers, backfillable scoped skip state, coverage IPC
results, and coverage IPC failures across process and AppDomain
boundaries.
- Adds scoped skippable-tests requests when coverage backfill is
required, using `test.bundle` plus runtime/configuration fingerprinting
so backend coverage matches the current local execution scope.
- Separates skippable candidates from tests actually skipped by ITR.
Framework integrations record coverage-backfill skip state when they
commit to an ITR skip; `Test.Close` records the session-level "ITR skip
happened" tag state later.
- Uses Java-compatible skip gating: the skip decision checks the backend
candidate's `_missing_line_code_coverage` flag, while aggregate backend
coverage validity is tracked separately for deciding whether a
backfilled coverage result can be published.
- Adds `CoverageBackfillCapability` and command-line parsing for
activation checks across `dotnet test`, `vstest`, `dotnet-coverage`,
Coverlet collector/MSBuild, Microsoft CodeCoverage, response files,
runsettings, MSBuild properties, filters, framework selectors,
thresholds, and generated report paths.
- Adds shared path matching and validation state so partial coverage
reports can only publish when their merged result represents all
required backend skipped-test lines and did not rely on unsafe local
suffix matches.
- Adds adapters for Datadog internal global coverage, Coverlet modules,
Coverlet collector XML fallback, external Cobertura/OpenCover XML, and
Microsoft CodeCoverage XML.
- Adds `CodeCoverageResultAggregator` to select the final session result
by source priority and suppress ambiguous count-based same-source
merges.
## Key behavior cases
- If no coverage report source is active, ITR keeps the legacy skip
behavior and no coverage backfill is applied.
- If line coverage is active, a skippable test is not skipped only when
the backend marks that candidate as missing line coverage.
- If backend aggregate `meta.coverage` is missing or invalid, ITR can
still skip tests, but the session will not publish a backfilled coverage
result unless another validated producer result is available.
- If coverage is active but no test is actually skipped by ITR, reports
can still be read normally and backend backfill is not applied.
- If an external XML report is selected, it has higher priority than
internal or IPC-based coverage results.
- If a child coverage tool cannot deliver a complete result when
backfill publication is required, the session fails closed for coverage
publication instead of publishing stale lower-priority coverage.
- Branch, method, condition, complexity, and threshold metadata are
preserved but not reconstructed from backend data; backend coverage is
line bitmap data only.
## Source priority
`CodeCoverageResultAggregator` selects the highest-priority publishable
result:
1. `ExternalXml`
2. `CoverletXmlFallback`
3. `Coverlet`
4. `MicrosoftCodeCoverage`
5. `Unknown`
6. `DatadogInternal`
Same-source results with line counts are only publishable when they were
explicitly merged by the producer or can be validated without ambiguous
count aggregation.
## Type guide
| Type | Responsibility |
| --- | --- |
| `CoverageBackfillData` | Decodes and validates backend `meta.coverage`
bitmaps. |
| `CoverageBackfillDataStore` | Persists backend coverage, actual-skip
state, scoped backfillable skip state, IPC results, and IPC failures. |
| `CoverageBackfillCapability` | Determines whether backfill is required
and whether the active coverage command/report path is safe. |
| `CoverageBackfillCommandLine` | Parses command lines, response files,
runsettings, MSBuild properties, and coverage output options for
capability checks. |
| `CoverageBackfillPathMatcher` / `CoverageBackfillPathMatchTracker` |
Matches backend source paths to local report paths and detects unsafe
suffix ambiguity. |
| `CoverageBackfillApplicator` | Applies backend line bitmaps to Datadog
internal global coverage. |
| `ExternalCoverageXmlBackfill` | Parses, validates, rewrites, and
merges line-capable XML reports. |
| `CoverletCoverageBackfill` | Mutates Coverlet's in-memory modules
before Coverlet calculates its line summary. |
| `CodeCoverageBackfillValidation` | Carries cross-result validation
state so partial producer results only publish when the merged set is
complete and safe. |
| `TestOptimizationSkippableFeature` | Requests scoped skippable tests,
gates coverage-active skips, and records actual ITR skip state. |
| `SessionCodeCoverageMessage` | Sends source-specific coverage results
from child tools/domains to the parent test session. |
| `CodeCoverageResultAggregator` | Suppresses unsafe results, merges
validated producer output, and selects the final session coverage source
by priority. |
## Test coverage
Added and expanded coverage across:
- backend `meta.coverage` parsing, bitmap merging, invalid payload
handling, missing coverage, and missing-line metadata;
- Java-compatible skippable parsing behavior for `custom` configurations
and `test.bundle` module scope;
- scoped skippable requests, actual-skip tracking, bundle/config
fingerprinting, and coverage-publication safety state;
- Datadog internal coverage backfill and `LineCallCount` semantics;
- Coverlet module mutation, rollback, validation, partial result
merging, and IPC failure paths;
- Cobertura, OpenCover, Microsoft CodeCoverage XML parsing/rewrite/merge
paths;
- path normalization, suffix ambiguity detection, source-root ambiguity,
represented-line validation, and multi-report validation;
- `dotnet test`, VSTest, Coverlet, Microsoft CodeCoverage,
`dotnet-coverage`, response file, runsettings, and MSBuild command
detection;
- session result arbitration, persisted IPC recovery, atomic
marker/result races, late Coverlet XML fallback, and injected session
behavior.
Recent validation:
- Java-parity focused unit tests for skippable parsing, skip gating, and
backend coverage decoding passed on `net10.0`.
- `AttributeTests.AttributesInstantiationsOnlyUseBuiltinTypes` passed on
`net10.0` and `net9.0` x64.
- `IpcTests.IpcClientCanSendCoverageMessageWithBackfillValidation`
passed on `net10.0`.
-
`ManagedVanguardStopIntegrationTests.DeduplicatesEquivalentMicrosoftLineXmlPathsBeforeBackfill`
passed on `net10.0`.
- `git diff --check`: clean.1 parent 2d3de8b commit 32acee9
112 files changed
Lines changed: 49953 additions & 586 deletions
File tree
- tracer
- src
- Datadog.Trace.Coverage.collector
- Datadog.Trace.Tools.Runner
- Datadog.Trace.Trimming/build
- Datadog.Trace
- Ci
- Agent/MessagePack
- CiEnvironment
- Coverage
- Backfill
- Models/Global
- Ipc
- Messages
- Net
- Tags
- ClrProfiler/AutoInstrumentation/Testing
- DotnetTest
- MsTestV2
- NUnit
- XUnit
- V3
- Configuration
- test
- Datadog.Trace.ClrProfiler.IntegrationTests/CI
- Datadog.Trace.Tests
- Ci
- Ipc
- Telemetry
- Datadog.Trace.Tools.Runner.IntegrationTests
- Datadog.Trace.Tools.Runner.Tests
- snapshots
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 183 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
54 | 59 | | |
55 | 60 | | |
56 | 61 | | |
| |||
97 | 102 | | |
98 | 103 | | |
99 | 104 | | |
100 | | - | |
101 | | - | |
| 105 | + | |
| 106 | + | |
102 | 107 | | |
| 108 | + | |
103 | 109 | | |
104 | 110 | | |
105 | 111 | | |
| |||
170 | 176 | | |
171 | 177 | | |
172 | 178 | | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
173 | 191 | | |
174 | 192 | | |
175 | 193 | | |
| |||
774 | 792 | | |
775 | 793 | | |
776 | 794 | | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
777 | 958 | | |
778 | 959 | | |
779 | 960 | | |
| |||
Lines changed: 6 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
255 | 261 | | |
256 | 262 | | |
257 | 263 | | |
| |||
0 commit comments