[CI Visibility] Bound global coverage memory by tonyredondo · Pull Request #8926 · DataDog/dd-trace-dotnet

tonyredondo · 2026-07-22T10:38:20Z

Summary of changes

Stop retaining every completed test's native coverage counter buffers when global code coverage is enabled.
Capture each completed context as compact executed-line bitmaps, merge those bitmaps into a bounded process-wide accumulator, and release the context's ModuleValue buffers immediately.
Publish one atomic, run-scoped coverage artifact per instrumented process and combine the process artifacts after dotnet test completes.
Preserve per-test coverage, global coverage percentages, executable-line counts, and the existing lock-free coverage-probe ABI.

Reason for change

DefaultWithGlobalCoverageEventHandler previously kept every completed CoverageContextContainer in _coverages until global coverage was calculated at the end of the test process. Each context retained one or more ModuleValue instances backed by Marshal.AllocHGlobal, so native memory grew with the number of completed tests rather than with the amount of unique code being measured.

APMS-20099 reports this failure mode in a Windows VSTest/NUnit run with 5,134 tests: 60,863 live ModuleValue instances retained approximately 16 GB of native buffers and caused testhost.exe to run out of memory before TestSessionEnd could perform the only cleanup.

The fix must release completed contexts during the run without losing their contribution to the final union. It must also support multiple testhost processes, preserve the per-test payload, and avoid adding synchronization to the generated counter-increment hot path.

Implementation details

Release completed contexts without losing coverage

CoverageSessionHandle binds each test to the exact handler, context, and lifecycle admission created at test start. This avoids closing the wrong AsyncLocal context and makes construction, abort, and teardown cleanup explicit.
At test end, ModuleCoverageData scans the existing byte or integer counter buffer once. A line is covered when its counter is non-zero; this preserves both line-execution and line-call-count modes, including counter overflow.
The captured executed-line bitmaps are used for the existing per-test coverage payload and are also merged into GlobalCoverageAccumulator.
The context is disposed in finally after capture and merge, so its native buffers are released even if coverage processing fails.
Completed-test memory now scales with the compact bitmaps for unique instrumented modules and files, not with the number or raw counter size of completed test contexts.

Preserve global coverage correctness

The accumulator keys in-process coverage by loaded Module and retains the original metadata and executable bitmap for that module.
Executed bitmaps are combined with a byte-wise OR. Repeated or overlapping contexts are therefore monotonic and idempotent: each executable line contributes at most once to the covered-line count.
The executable-line denominator continues to come exclusively from the original executable bitmap; merging additional tests cannot inflate it.
Snapshots clone mutable executed bitmaps and use lifecycle admissions so a snapshot cannot be published as complete while starts, active contexts, or another snapshot are still in flight.
Metadata inconsistencies, allocation failures, configured limits, or publication failures suppress the incomplete global result instead of publishing a plausible but partial percentage. Per-test native cleanup still runs.

Keep the probe hot path and IL contract unchanged

Generated instrumentation still calls CoverageReporter<TMeta>.GetFileCounter(int) and receives the same native pointer used by the existing counter increment IL.
The populated module cache remains lock-free on the normal probe path. Synchronization stays in module creation, context lifecycle, merging, snapshots, and publication rather than in generated counter increments.
A closed AsyncLocal flow falls back to the process-global module buffer instead of reopening or writing to a disposed test context.
The production coverage rewriter is not changed by this PR.

Publish complete coverage across testhost processes

Each instrumented process creates a durable pending marker for its output directory and Test Optimization run.
At process shutdown, the in-process collector or the TestOptimization shutdown hook seals the same idempotent finalization path, writes one immutable JSON artifact atomically, and removes the pending marker only after the artifact is durable.
Artifact and marker names contain a SHA-256 token derived from the Test Optimization run ID plus a process identity, so output directories can be reused without mixing different runs or colliding process outputs.
The outer dotnet test instrumentation combines only artifacts belonging to its run, applies the existing ITR backfill, and atomically publishes the final session-coverage-*.json file.
A remaining pending marker or malformed selected artifact makes run-scoped combination fail closed. The standalone legacy coverage command keeps its previous best-effort behavior for unrelated JSON files.
Readers, incremental combiners, and writers enforce bounds for serialized size, JSON depth and scalars, bitmap sizes, retained bitmap bytes, components, modules, files, and identity strings, bounding retained coverage state and artifact processing even for large or malformed inputs.

Test coverage

Unit tests cover both counter modes, exact bitmap and percentage results, overlapping contexts, multiple modules and files, unchanged executable-line denominators, late contexts, owner-bound cleanup, stale AsyncLocal flows, concurrent starts/snapshots/sealing, allocation and metadata failures, configured memory limits, atomic publication, run isolation, malformed artifacts, and ITR consumer behavior.
A 10,000-context regression allocates a 128 KiB native buffer per context, verifies that every buffer is released immediately at context close, and verifies the exact final union: 10,000 of 20,000 executable lines, or 50%.
Rewriter tests verify that coverage instrumentation retains the existing GetFileCounter pointer shape, does not add exception handlers or invalidate tail. methods, and still releases buffers when instrumented code throws.
Windows end-to-end tests execute 6,000 non-parallel NUnit cases in one VSTest testhost through the production dd-trace ci run tool and packaged Datadog collector, using Coverlet 3.2.0 and 6.0.0.
The end-to-end assertions verify bounded private-memory growth, one testhost PID, exactly 6,000 started/closed/disposed/merged contexts, zero active context buffers after sealing, balanced native allocations and frees, and the expected common/first/middle/last executed-line sentinels in the final artifact read by the production coverage reader.
CI runs the end-to-end regression on Windows x64 and x86. All four Coverlet/architecture executions and the focused correctness, lifecycle, consumer, publication, and IL tests passed for the current PR head.

Other details

Jira: APMS-20099

dd-trace-dotnet-ci-bot · 2026-07-22T11:42:42Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8926) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	71.83 ± (71.79 - 72.13) ms	73.71 ± (73.62 - 74.13) ms	+2.6%	✅⬆️
.NET Framework 4.8 - Bailout
duration	80.27 ± (80.46 - 81.58) ms	76.06 ± (75.99 - 76.49) ms	-5.2%	✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1094.89 ± (1093.39 - 1099.55) ms	1087.52 ± (1087.35 - 1093.68) ms	-0.7%	✅
.NET Core 3.1 - Baseline
process.internal_duration_ms	22.40 ± (22.35 - 22.45) ms	22.34 ± (22.28 - 22.39) ms	-0.3%	✅
process.time_to_main_ms	84.76 ± (84.47 - 85.06) ms	84.54 ± (84.22 - 84.85) ms	-0.3%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	11.03 ± (11.03 - 11.04) MB	11.00 ± (10.99 - 11.00) MB	-0.3%	✅
runtime.dotnet.threads.count	12 ± (12 - 12)	12 ± (12 - 12)	+0.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	22.33 ± (22.29 - 22.37) ms	21.98 ± (21.95 - 22.02) ms	-1.5%	✅
process.time_to_main_ms	85.28 ± (85.01 - 85.56) ms	82.66 ± (82.49 - 82.82) ms	-3.1%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	11.06 ± (11.06 - 11.07) MB	11.04 ± (11.03 - 11.04) MB	-0.2%	✅
runtime.dotnet.threads.count	13 ± (13 - 13)	13 ± (13 - 13)	+0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	210.28 ± (209.34 - 211.21) ms	210.96 ± (210.08 - 211.83) ms	+0.3%	✅⬆️
process.time_to_main_ms	538.93 ± (537.63 - 540.23) ms	540.42 ± (539.28 - 541.57) ms	+0.3%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	49.64 ± (49.60 - 49.67) MB	49.57 ± (49.54 - 49.60) MB	-0.1%	✅
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.4%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	21.25 ± (21.20 - 21.30) ms	20.84 ± (20.81 - 20.87) ms	-1.9%	✅
process.time_to_main_ms	74.78 ± (74.49 - 75.07) ms	71.11 ± (70.95 - 71.27) ms	-4.9%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.72 ± (10.72 - 10.73) MB	10.72 ± (10.72 - 10.72) MB	-0.0%	✅
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 6 - Bailout
process.internal_duration_ms	20.82 ± (20.79 - 20.86) ms	21.13 ± (21.09 - 21.17) ms	+1.5%	✅⬆️
process.time_to_main_ms	72.52 ± (72.35 - 72.69) ms	75.05 ± (74.81 - 75.30) ms	+3.5%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.83 ± (10.83 - 10.84) MB	10.85 ± (10.85 - 10.85) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	371.98 ± (369.86 - 374.09) ms	369.61 ± (367.22 - 372.00) ms	-0.6%	✅
process.time_to_main_ms	546.78 ± (545.59 - 547.96) ms	544.33 ± (543.12 - 545.54) ms	-0.4%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	50.65 ± (50.63 - 50.68) MB	50.68 ± (50.65 - 50.70) MB	+0.0%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	-0.4%	✅
.NET 8 - Baseline
process.internal_duration_ms	19.21 ± (19.17 - 19.25) ms	19.06 ± (19.04 - 19.09) ms	-0.8%	✅
process.time_to_main_ms	71.47 ± (71.20 - 71.74) ms	70.87 ± (70.64 - 71.09) ms	-0.8%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.76 ± (7.76 - 7.76) MB	7.77 ± (7.76 - 7.77) MB	+0.1%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 8 - Bailout
process.internal_duration_ms	19.44 ± (19.40 - 19.49) ms	18.96 ± (18.93 - 18.99) ms	-2.5%	✅
process.time_to_main_ms	74.88 ± (74.62 - 75.13) ms	71.09 ± (70.95 - 71.23) ms	-5.1%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.83 ± (7.82 - 7.84) MB	7.85 ± (7.84 - 7.87) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	296.66 ± (294.51 - 298.81) ms	298.90 ± (296.65 - 301.16) ms	+0.8%	✅⬆️
process.time_to_main_ms	492.46 ± (491.49 - 493.44) ms	496.40 ± (495.20 - 497.60) ms	+0.8%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	38.07 ± (38.04 - 38.10) MB	38.18 ± (38.15 - 38.21) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	27 ± (27 - 27)	27 ± (27 - 27)	-0.6%	✅

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	210.98 ± (210.61 - 211.56) ms	202.94 ± (202.91 - 203.89) ms	-3.8%	✅
.NET Framework 4.8 - Bailout
duration	214.99 ± (214.44 - 215.34) ms	207.63 ± (207.30 - 208.32) ms	-3.4%	✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1265.27 ± (1264.17 - 1271.43) ms	1217.76 ± (1217.85 - 1223.55) ms	-3.8%	✅
.NET Core 3.1 - Baseline
process.internal_duration_ms	203.57 ± (203.12 - 204.02) ms	194.85 ± (194.35 - 195.34) ms	-4.3%	✅
process.time_to_main_ms	89.89 ± (89.58 - 90.20) ms	85.06 ± (84.80 - 85.32) ms	-5.4%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.07 ± (16.05 - 16.09) MB	16.22 ± (16.20 - 16.24) MB	+0.9%	✅⬆️
runtime.dotnet.threads.count	20 ± (20 - 20)	20 ± (20 - 20)	-1.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	202.98 ± (202.52 - 203.44) ms	193.65 ± (193.36 - 193.94) ms	-4.6%	✅
process.time_to_main_ms	91.51 ± (91.27 - 91.76) ms	86.37 ± (86.22 - 86.52) ms	-5.6%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.08 ± (16.06 - 16.09) MB	16.29 ± (16.27 - 16.31) MB	+1.3%	✅⬆️
runtime.dotnet.threads.count	21 ± (21 - 21)	21 ± (21 - 21)	-1.3%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	401.33 ± (399.93 - 402.73) ms	389.07 ± (387.74 - 390.39) ms	-3.1%	✅
process.time_to_main_ms	566.20 ± (565.03 - 567.37) ms	550.51 ± (549.34 - 551.68) ms	-2.8%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	59.80 ± (59.70 - 59.91) MB	58.42 ± (58.22 - 58.62) MB	-2.3%	✅
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	-0.9%	✅
.NET 6 - Baseline
process.internal_duration_ms	208.47 ± (208.06 - 208.87) ms	200.96 ± (200.58 - 201.33) ms	-3.6%	✅
process.time_to_main_ms	78.92 ± (78.61 - 79.23) ms	74.97 ± (74.71 - 75.23) ms	-5.0%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.46 ± (16.44 - 16.48) MB	16.41 ± (16.39 - 16.44) MB	-0.3%	✅
runtime.dotnet.threads.count	20 ± (19 - 20)	19 ± (19 - 19)	-1.3%	✅
.NET 6 - Bailout
process.internal_duration_ms	207.40 ± (206.88 - 207.93) ms	199.95 ± (199.50 - 200.40) ms	-3.6%	✅
process.time_to_main_ms	79.58 ± (79.36 - 79.80) ms	75.96 ± (75.79 - 76.14) ms	-4.5%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.48 ± (16.46 - 16.50) MB	16.52 ± (16.49 - 16.54) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	21 ± (20 - 21)	20 ± (20 - 20)	-1.5%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	579.91 ± (577.66 - 582.17) ms	586.22 ± (583.85 - 588.58) ms	+1.1%	✅⬆️
process.time_to_main_ms	579.70 ± (578.49 - 580.91) ms	559.36 ± (558.19 - 560.53) ms	-3.5%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	61.61 ± (61.53 - 61.69) MB	61.79 ± (61.68 - 61.90) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	31 ± (31 - 31)	31 ± (31 - 31)	+0.6%	✅⬆️
.NET 8 - Baseline
process.internal_duration_ms	209.30 ± (208.84 - 209.75) ms	198.64 ± (198.21 - 199.08) ms	-5.1%	✅
process.time_to_main_ms	78.75 ± (78.48 - 79.01) ms	74.01 ± (73.80 - 74.22) ms	-6.0%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.78 ± (11.76 - 11.80) MB	11.89 ± (11.87 - 11.90) MB	+0.9%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	19 ± (18 - 19)	-2.6%	✅
.NET 8 - Bailout
process.internal_duration_ms	209.21 ± (208.71 - 209.70) ms	199.36 ± (198.83 - 199.89) ms	-4.7%	✅
process.time_to_main_ms	80.08 ± (79.86 - 80.29) ms	75.60 ± (75.41 - 75.79) ms	-5.6%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.83 ± (11.81 - 11.85) MB	11.91 ± (11.88 - 11.93) MB	+0.7%	✅⬆️
runtime.dotnet.threads.count	20 ± (20 - 20)	20 ± (20 - 20)	-1.7%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	517.32 ± (511.82 - 522.81) ms	508.84 ± (506.39 - 511.29) ms	-1.6%	✅
process.time_to_main_ms	531.98 ± (531.07 - 532.89) ms	511.71 ± (510.84 - 512.59) ms	-3.8%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	51.36 ± (51.31 - 51.41) MB	51.46 ± (51.43 - 51.50) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	-0.1%	✅

Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts

FakeDbCommand (.NET Framework 4.8)

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (74ms)  : 70, 78
    master - mean (72ms)  : 69, 75

    section Bailout
    This PR (8926) - mean (76ms)  : 73, 80
    master - mean (81ms)  : 73, 89

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (1,091ms)  : 1045, 1136
    master - mean (1,096ms)  : 1052, 1141

FakeDbCommand (.NET Core 3.1)

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (114ms)  : 108, 120
    master - mean (115ms)  : 108, 121

    section Bailout
    This PR (8926) - mean (110ms)  : 108, 113
    master - mean (115ms)  : 108, 122

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (787ms)  : 764, 810
    master - mean (785ms)  : 763, 807

FakeDbCommand (.NET 6)

gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (97ms)  : 95, 100
    master - mean (103ms)  : 97, 109

    section Bailout
    This PR (8926) - mean (102ms)  : 98, 107
    master - mean (99ms)  : 96, 103

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (941ms)  : 907, 974
    master - mean (950ms)  : 901, 999

FakeDbCommand (.NET 8)

gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (97ms)  : 92, 102
    master - mean (98ms)  : 92, 105

    section Bailout
    This PR (8926) - mean (97ms)  : 95, 99
    master - mean (102ms)  : 96, 108

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (826ms)  : 789, 862
    master - mean (822ms)  : 782, 862

HttpMessageHandler (.NET Framework 4.8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (203ms)  : 197, 209
    master - mean (211ms)  : 205, 217

    section Bailout
    This PR (8926) - mean (208ms)  : 202, 213
    master - mean (215ms)  : 211, 219

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (1,221ms)  : 1180, 1261
    master - mean (1,268ms)  : 1214, 1321

HttpMessageHandler (.NET Core 3.1)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (289ms)  : 282, 296
    master - mean (303ms)  : 294, 312

    section Bailout
    This PR (8926) - mean (289ms)  : 286, 293
    master - mean (305ms)  : 295, 315

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (977ms)  : 954, 1000
    master - mean (1,009ms)  : 984, 1035

HttpMessageHandler (.NET 6)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (285ms)  : 278, 291
    master - mean (298ms)  : 291, 305

    section Bailout
    This PR (8926) - mean (285ms)  : 278, 292
    master - mean (297ms)  : 291, 303

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (1,176ms)  : 1140, 1212
    master - mean (1,200ms)  : 1160, 1239

HttpMessageHandler (.NET 8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8926) - mean (283ms)  : 278, 288
    master - mean (300ms)  : 294, 306

    section Bailout
    This PR (8926) - mean (285ms)  : 279, 292
    master - mean (301ms)  : 293, 308

    section CallTarget+Inlining+NGEN
    This PR (8926) - mean (1,053ms)  : 1015, 1092
    master - mean (1,094ms)  : 989, 1199

tonyredondo · 2026-07-22T13:25:42Z

@codex review

chatgpt-codex-connector

💡 Codex Review

dd-trace-dotnet/tracer/src/Datadog.Trace.Tools.Runner/CiUtils.cs

Lines 427 to 429 in 31856d7

    
           else if (isDotnetTestCommand) 
        
           { 
        
               Log.Debug("DatadogCoverage data collector was not added because Microsoft Testing Platform coverage is already selected.");

Seal global coverage when Testing Platform skips the collector

When dotnet test uses Microsoft Testing Platform coverage, this branch deliberately omits the Datadog in-proc collector. However, RequestSeal() is only invoked from that collector's TestSessionEnd, while code coverage remains enabled and TestModule still publishes the new pending/raw global-coverage artifacts. Those artifacts therefore never receive ready markers; reusing an explicit coverage directory leaves an incomplete protocol set that makes subsequent reconciliation reject both the stale and newly produced coverage files. Add an equivalent end-of-run sealing path for the Testing Platform case, or avoid creating protocol artifacts there.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pr-commenter · 2026-07-22T15:25:26Z

Benchmarks

Benchmark execution time: 2026-07-31 22:17:15

Comparing candidate commit f74fb63 in PR branch fix/apms-20099-global-coverage-memory with baseline commit 1dbaad3 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 1 performance regressions! Performance is the same for 71 metrics, 0 unstable metrics, 61 known flaky benchmarks, 65 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.HttpClientBenchmark.SendAsync net472

🟥 throughput [-5359.959op/s; -5107.166op/s] or [-6.119%; -5.830%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472

🟥 throughput [-12686.335op/s; -10927.299op/s] or [-15.042%; -12.957%]

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1

🟥 throughput [-10977.089op/s; -9594.388op/s] or [-11.161%; -9.755%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472

🟥 execution_time [+304.298ms; +307.078ms] or [+151.003%; +152.383%]
🟥 throughput [-53.174op/s; -48.904op/s] or [-9.567%; -8.799%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

🟥 execution_time [+380.080ms; +381.719ms] or [+300.286%; +301.581%]
🟩 throughput [+91.448op/s; +94.248op/s] or [+12.057%; +12.426%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

🟥 execution_time [+398.039ms; +401.589ms] or [+352.249%; +355.391%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472

🟥 allocated_mem [+4.692KB; +4.693KB] or [+98.785%; +98.801%]
🟥 throughput [-61000.309op/s; -60479.622op/s] or [-47.461%; -47.056%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

🟥 allocated_mem [+3.816KB; +3.816KB] or [+80.699%; +80.711%]
🟩 execution_time [-15.533ms; -11.356ms] or [-7.254%; -5.304%]
🟥 throughput [-59522.315op/s; -56764.631op/s] or [-43.448%; -41.435%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1

🟥 allocated_mem [+4.544KB; +4.544KB] or [+98.261%; +98.274%]
🟥 throughput [-48890.691op/s; -46638.044op/s] or [-44.203%; -42.166%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

🟥 allocated_mem [+1.315KB; +1.315KB] or [+106.388%; +106.404%]
🟥 throughput [-268140.762op/s; -262851.578op/s] or [-27.378%; -26.838%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0

🟥 allocated_mem [+479 bytes; +480 bytes] or [+39.212%; +39.221%]
🟩 execution_time [-27.002ms; -22.125ms] or [-12.042%; -9.867%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

🟥 allocated_mem [+1.280KB; +1.280KB] or [+105.947%; +105.963%]
🟥 throughput [-159486.108op/s; -143355.944op/s] or [-22.915%; -20.597%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472

🟥 allocated_mem [+3.378KB; +3.378KB] or [+89.003%; +89.017%]
🟥 throughput [-72612.021op/s; -71846.419op/s] or [-48.867%; -48.352%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

🟥 allocated_mem [+3.336KB; +3.336KB] or [+88.150%; +88.161%]
🟥 throughput [-73253.294op/s; -70375.363op/s] or [-46.610%; -44.779%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

🟥 allocated_mem [+3.264KB; +3.264KB] or [+88.493%; +88.506%]
🟥 throughput [-56789.860op/s; -54171.235op/s] or [-45.241%; -43.154%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

🟩 throughput [+198017.009op/s; +212483.530op/s] or [+6.603%; +7.085%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

🟩 execution_time [-17.443ms; -12.086ms] or [-8.040%; -5.571%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472

🟥 execution_time [+300.415ms; +301.108ms] or [+150.107%; +150.454%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0

🟥 execution_time [+298.885ms; +302.578ms] or [+150.728%; +152.591%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

🟥 execution_time [+300.679ms; +303.114ms] or [+151.459%; +152.685%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472

🟥 execution_time [+296.546ms; +297.719ms] or [+145.652%; +146.228%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

🟥 execution_time [+296.811ms; +299.287ms] or [+145.100%; +146.310%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

🟥 execution_time [+300.927ms; +303.651ms] or [+150.403%; +151.765%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0

🟥 execution_time [+24.392µs; +48.030µs] or [+7.787%; +15.333%]
🟥 throughput [-445.243op/s; -246.367op/s] or [-13.880%; -7.680%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472

🟥 execution_time [+299.253ms; +300.007ms] or [+149.358%; +149.734%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

unstable execution_time [+407.869ms; +419.308ms] or [+443.167%; +455.595%]
🟩 throughput [+808.352op/s; +963.374op/s] or [+6.642%; +7.916%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

🟥 execution_time [+368.192ms; +373.706ms] or [+279.565%; +283.751%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

unstable execution_time [+371.502ms; +436.805ms] or [+170.813%; +200.839%]
🟥 throughput [-567.486op/s; -517.153op/s] or [-51.420%; -46.859%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

unstable execution_time [+204.310ms; +337.619ms] or [+87.068%; +143.879%]
🟥 throughput [-692.891op/s; -609.044op/s] or [-46.216%; -40.623%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

🟥 execution_time [+352.871ms; +363.266ms] or [+211.058%; +217.275%]
🟥 throughput [-453.980op/s; -419.210op/s] or [-31.610%; -29.189%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0

🟩 execution_time [-176.240µs; -122.953µs] or [-8.928%; -6.228%]
🟩 throughput [+35.852op/s; +49.805op/s] or [+7.078%; +9.832%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472

🟥 execution_time [+301.516ms; +303.089ms] or [+151.838%; +152.630%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

🟥 execution_time [+302.158ms; +303.941ms] or [+151.412%; +152.305%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1

🟥 execution_time [+300.980ms; +304.225ms] or [+151.200%; +152.830%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

🟥 execution_time [+301.503ms; +303.260ms] or [+151.405%; +152.287%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

🟥 execution_time [+299.227ms; +301.628ms] or [+147.955%; +149.141%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

🟥 execution_time [+303.918ms; +307.548ms] or [+154.039%; +155.879%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472

🟥 execution_time [+299.371ms; +301.203ms] or [+150.257%; +151.177%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

🟥 execution_time [+301.442ms; +303.975ms] or [+150.241%; +151.504%]
🟩 throughput [+44646.933op/s; +49213.302op/s] or [+8.865%; +9.772%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

🟥 execution_time [+298.852ms; +301.358ms] or [+148.676%; +149.923%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0

🟩 execution_time [-17.186ms; -13.552ms] or [-7.992%; -6.302%]
🟩 throughput [+20765.066op/s; +27839.142op/s] or [+5.696%; +7.637%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472

unstable execution_time [+23.972µs; +69.710µs] or [+5.921%; +17.219%]
🟥 throughput [-357.462op/s; -138.631op/s] or [-14.385%; -5.579%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

🟩 allocated_mem [-18.462KB; -18.440KB] or [-6.735%; -6.727%]
unstable execution_time [-36.535µs; +23.294µs] or [-7.221%; +4.604%]
unstable throughput [-75.280op/s; +135.035op/s] or [-3.757%; +6.738%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

unstable execution_time [-53.955µs; +6.674µs] or [-9.350%; +1.157%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

unstable execution_time [+5.971µs; +10.407µs] or [+14.114%; +24.600%]
🟥 throughput [-4719.329op/s; -2874.629op/s] or [-19.867%; -12.101%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

unstable execution_time [-15.017µs; -7.513µs] or [-23.298%; -11.657%]
🟩 throughput [+1922.155op/s; +3501.978op/s] or [+11.793%; +21.486%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472

🟥 execution_time [+303.140ms; +304.531ms] or [+153.224%; +153.927%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0

🟥 execution_time [+302.496ms; +304.891ms] or [+153.969%; +155.188%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

🟥 execution_time [+298.072ms; +300.247ms] or [+149.222%; +150.310%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net6.0

🟩 throughput [+34568.167op/s; +39150.593op/s] or [+6.543%; +7.410%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472

🟥 execution_time [+297.762ms; +300.048ms] or [+148.408%; +149.547%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0

🟥 execution_time [+301.082ms; +304.041ms] or [+151.189%; +152.675%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1

🟥 execution_time [+302.191ms; +304.769ms] or [+153.252%; +154.559%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472

🟥 execution_time [+299.378ms; +300.238ms] or [+149.331%; +149.760%]
🟩 throughput [+61144636.934op/s; +61476873.626op/s] or [+44.529%; +44.771%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

🟥 execution_time [+422.183ms; +424.780ms] or [+525.060%; +528.289%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

🟥 execution_time [+299.047ms; +300.106ms] or [+149.158%; +149.686%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0

🟩 throughput [+101761.785op/s; +107879.592op/s] or [+9.501%; +10.072%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1

🟩 throughput [+44989.482op/s; +66920.272op/s] or [+5.207%; +7.746%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0

🟩 throughput [+81825.262op/s; +112949.739op/s] or [+6.333%; +8.743%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1

🟩 throughput [+67025.558op/s; +76811.228op/s] or [+6.657%; +7.629%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0

🟩 throughput [+48467.886op/s; +52564.067op/s] or [+8.801%; +9.545%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0

🟩 throughput [+61266.615op/s; +82638.610op/s] or [+6.845%; +9.233%]

Known flaky benchmarks without significant changes:

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled netcoreapp3.1
scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net472
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net472
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net472
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472
scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1
scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net472
scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472
scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net472
scenario:Benchmarks.Trace.RedisBenchmark.SendReceive netcoreapp3.1
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net472
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net472
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net472
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

tonyredondo · 2026-07-23T13:34:12Z

💡 Codex Review

dd-trace-dotnet/tracer/src/Datadog.Trace.Tools.Runner/CiUtils.cs

Lines 427 to 429 in 31856d7

else if (isDotnetTestCommand)

{

Log.Debug("DatadogCoverage data collector was not added because Microsoft Testing Platform coverage is already selected.");

Seal global coverage when Testing Platform skips the collector
When dotnet test uses Microsoft Testing Platform coverage, this branch deliberately omits the Datadog in-proc collector. However, RequestSeal() is only invoked from that collector's TestSessionEnd, while code coverage remains enabled and TestModule still publishes the new pending/raw global-coverage artifacts. Those artifacts therefore never receive ready markers; reusing an explicit coverage directory leaves an incomplete protocol set that makes subsequent reconciliation reject both the stale and newly produced coverage files. Add an equivalent end-of-run sealing path for the Testing Platform case, or avoid creating protocol artifacts there.

ℹ️ About Codex in GitHub

@chatgpt-codex-connector Addressed in a79fa2d. Global coverage is now finalized from TestOptimization shutdown after active modules are closed, so Microsoft Testing Platform runs seal their output even when the Datadog in-proc collector is intentionally omitted. The finalization path is idempotent and shared with the collector. I also added unit coverage plus an MTP end-to-end test that runs twice against the same coverage directory and verifies successful reconciliation.

chatgpt-codex-connector · 2026-07-23T13:34:22Z

To use Codex here, create an environment for this repo.

tonyredondo · 2026-07-23T13:42:31Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c62a892b39

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tonyredondo · 2026-07-23T15:49:53Z

@codex review

chatgpt-codex-connector · 2026-07-23T15:53:13Z

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 82ec03f4b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-07-24T13:41:47Z

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

github-actions Bot added the area:ci-visibility label Jul 22, 2026

chatgpt-codex-connector Bot reviewed Jul 22, 2026

View reviewed changes

tonyredondo force-pushed the fix/apms-20099-global-coverage-memory branch from eee3854 to c62a892 Compare July 23, 2026 13:41

tonyredondo marked this pull request as ready for review July 23, 2026 13:42

tonyredondo requested review from a team as code owners July 23, 2026 13:42

tonyredondo requested review from vandonr and removed request for a team July 23, 2026 13:42

chatgpt-codex-connector Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread tracer/src/Datadog.Trace/Ci/Coverage/GlobalCoverageReconciliationLease.cs Outdated

Comment thread tracer/src/Datadog.Trace/Ci/Coverage/GlobalCoverageInputReader.cs

tonyredondo force-pushed the fix/apms-20099-global-coverage-memory branch from 6cb87db to bfd8734 Compare July 24, 2026 08:43

tonyredondo added 9 commits July 30, 2026 18:38

[CI Visibility] Bound global coverage memory

e98513b

[CI Visibility] Fix global coverage CI failures

0d88b48

[CI Visibility] Seal global coverage during shutdown

dc7e740

[CI Visibility] Simplify global coverage processing

549b170

[CI Visibility] Simplify coverage metadata validation

19d514d

[CI Visibility] Simplify global coverage diagnostics

8cd3c81

[CI Visibility] Harden coverage artifact reconciliation

f1f4eb6

[CI Visibility] Strengthen global coverage regression tests

2120529

[CI Visibility] Harden global coverage lifecycle correctness

94c6545

tonyredondo added 4 commits July 30, 2026 18:38

[CI Visibility] Bound global coverage native memory

0e3a4bf

[CI Visibility] Release global coverage buffers per test

b1af3f6

[CI Visibility] Isolate coverage reconciliation by test run

9c18601

[CI Visibility] Regenerate call target definitions

36bb32a

tonyredondo force-pushed the fix/apms-20099-global-coverage-memory branch from 8a3be12 to 36bb32a Compare July 30, 2026 16:43

tonyredondo added 12 commits July 30, 2026 20:43

[CI Visibility] Run global coverage Windows regression tests

aab6c90

[CI Visibility] Test global coverage with packaged collector

50ab29c

[CI Visibility] Fix global coverage Windows smoke tests

d968455

[CI Visibility] Exercise SDK 10 coverage hook in Windows smoke test

c9679e7

[CI Visibility] Build SDK 10 coverage smoke sample

59f80e3

[CI Visibility] Align SDK 10 smoke restore

fbe978f

[CI Visibility] Match rendered coverage diagnostics

706e76b

[CI Visibility] Avoid persistent compiler in coverage smoke test

5bbb2f4

[CI Visibility] Simplify global coverage publication

0772d7d

[CI Visibility] Make coverage publication atomic

852e4f2

[CI Visibility] Report global coverage publication failures

5e09de1

[CI Visibility] Use run-scoped coverage fixture

f74fb63

	else if (isDotnetTestCommand)
	{
	Log.Debug("DatadogCoverage data collector was not added because Microsoft Testing Platform coverage is already selected.");

Uh oh!

Conversation

tonyredondo commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Release completed contexts without losing coverage

Preserve global coverage correctness

Keep the probe hot path and IL contract unchanged

Publish complete coverage across testhost processes

Test coverage

Other details

Uh oh!

dd-trace-dotnet-ci-bot Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

FakeDbCommand

HttpMessageHandler

Uh oh!

tonyredondo commented Jul 22, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

pr-commenter Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:Benchmarks.Trace.HttpClientBenchmark.SendAsync net472

Known flaky benchmarks

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0

tonyredondo commented Jul 22, 2026 •

edited

Loading

dd-trace-dotnet-ci-bot Bot commented Jul 22, 2026 •

edited

Loading

pr-commenter Bot commented Jul 22, 2026 •

edited

Loading