Skip to content

Support GitHub/GitLab/Azure DevOps/Bitbucket Server SourceLink variants - #8505

Draft
OmerRaviv wants to merge 1 commit into
masterfrom
omer/sourcelink-enterprise-variants
Draft

Support GitHub/GitLab/Azure DevOps/Bitbucket Server SourceLink variants#8505
OmerRaviv wants to merge 1 commit into
masterfrom
omer/sourcelink-enterprise-variants

Conversation

@OmerRaviv

Copy link
Copy Markdown
Collaborator

Summary of changes

Extends the SourceLink URL parsers under tracer/src/Datadog.Trace/PDBs/SourceLink/ to cover on-prem and enterprise URL variants emitted by real-world PDBs that the existing parsers rejected.

Parser What's added
GitHub GitHub Enterprise with subdomain isolation (raw.{host}/...) and main-host path form ({host}/raw/...), in addition to the existing raw.githubusercontent.com
Azure DevOps TFS on-prem, virtual directories, collections, and custom ports — all handled uniformly via a single /_apis/git/repositories/ anchor instead of host-specific branching
GitLab New /-/raw/ path format (GitLab ≥ 12.0) and arbitrary nested groups/subgroups
Bitbucket Server New parser for /projects/{p}/repos/{r}/raw/*?at=SHA (≥ 4.7) and /browse/*?at=SHA&raw (< 4.7), with support for base path deployments

Also fixes a latent bug in SourceLinkInformationExtractor where an InformationalVersion like "1.0.0+" (empty after the plus) was treated as a valid-but-empty commit SHA.

Reason for change

Several users have PDBs that embed SourceLink URLs from GHE / GitLab self-hosted / Azure DevOps Server / Bitbucket Server. The old parsers only handled the canonical cloud variants, so the repository URL and commit SHA were not surfaced from those PDBs.

Implementation details

  • Azure DevOps: replaced per-host segment-index parsing with a single /_apis/git/repositories/ anchor. Everything before _apis becomes the project prefix path (supporting /{vdir}/{collection}/{project}/...); the repo URL is {scheme}://{authority}{prefix}/_git/{repo}.
  • GitHub: three cases (cloud, GHE subdomain isolation, GHE /raw/ path), with shared helpers for the /{owner}/{repo}/{sha}/* and /raw/{owner}/{repo}/{sha}/* layouts.
  • GitLab: anchor-based search using LastIndexOf("/-/raw/") then LastIndexOf("/raw/"), with a trimmed segment count to ensure at least {group}/{repo} before the anchor.
  • Bitbucket Server: new parser that requires /projects/ to precede /repos/, walks the query string once to extract at={sha} and — for the /browse/ variant — a standalone raw flag (not a substring match).

Test coverage

  • 84 tests in Datadog.Trace.Tests.Pdb.SourceLink.* (covering each parser individually and via the composite router).
  • 37 tests in Datadog.Trace.ClrProfiler.Managed.Tests.SourceLinkInformationExtractorTests (assembly-attribute extraction).
  • New positive cases: GHE subdomain isolation & /raw/ path, TFS on-prem (DefaultCollection, virtual dirs, custom ports), GitLab -/raw/ and nested groups, Bitbucket Server ≥ 4.7 and < 4.7 with base paths and varied query-param ordering.
  • New negative cases: raw. with too few segments, missing /projects/ prefix, raws= false-positive check, empty InformationalVersion metadata.

All 121 tests pass locally on net10.0.

Other details

  • Datadog.Trace.ClrProfiler.Managed.Tests/SourceLinkInformationExtractorTests.cs was previously untracked in the working tree; it's now included and the extractor bug it surfaces is fixed.

🤖 Generated with Claude Code

Extends SourceLink URL parsers to cover on-prem and enterprise URL
variants emitted by real-world PDBs that the existing parsers rejected:

- GitHub: GitHub Enterprise with subdomain isolation (raw.{host}) and
  main-host /raw/ path form, in addition to raw.githubusercontent.com
- Azure DevOps: TFS on-prem, virtual directories, collections, custom
  ports - handled uniformly via a single /_apis/git/repositories/
  anchor instead of host-specific branching
- GitLab: new /-/raw/ path format (>= 12.0) and nested groups/subgroups
- Bitbucket Server: new parser for /projects/{p}/repos/{r}/raw/*?at=
  (>= 4.7) and /browse/*?at=&raw (< 4.7), with base path support

Also fixes a latent bug in SourceLinkInformationExtractor where an
InformationalVersion like "1.0.0+" (empty after the plus) was treated
as a valid but empty commit SHA.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@dd-trace-dotnet-ci-bot

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8505) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration74.94 ± (74.81 - 75.38) ms72.52 ± (72.55 - 72.97) ms-3.2%
.NET Framework 4.8 - Bailout
duration77.87 ± (77.79 - 78.20) ms78.98 ± (78.71 - 79.19) ms+1.4%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1073.37 ± (1073.94 - 1081.08) ms1071.39 ± (1071.94 - 1080.00) ms-0.2%
.NET Core 3.1 - Baseline
process.internal_duration_ms22.82 ± (22.76 - 22.88) ms22.79 ± (22.73 - 22.86) ms-0.1%
process.time_to_main_ms85.93 ± (85.62 - 86.24) ms86.92 ± (86.63 - 87.21) ms+1.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.85 ± (10.84 - 10.85) MB10.93 ± (10.92 - 10.93) MB+0.7%✅⬆️
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%
.NET Core 3.1 - Bailout
process.internal_duration_ms22.48 ± (22.44 - 22.52) ms22.55 ± (22.50 - 22.60) ms+0.3%✅⬆️
process.time_to_main_ms85.02 ± (84.81 - 85.23) ms87.86 ± (87.59 - 88.14) ms+3.3%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.94 ± (10.94 - 10.95) MB10.97 ± (10.97 - 10.97) MB+0.2%✅⬆️
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms212.84 ± (211.89 - 213.78) ms211.54 ± (210.69 - 212.39) ms-0.6%
process.time_to_main_ms526.54 ± (525.05 - 528.03) ms527.38 ± (526.03 - 528.72) ms+0.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed47.87 ± (47.84 - 47.91) MB47.95 ± (47.92 - 47.97) MB+0.2%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.0%
.NET 6 - Baseline
process.internal_duration_ms21.20 ± (21.16 - 21.24) ms21.38 ± (21.33 - 21.43) ms+0.9%✅⬆️
process.time_to_main_ms73.07 ± (72.88 - 73.26) ms73.89 ± (73.64 - 74.14) ms+1.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.61 ± (10.61 - 10.61) MB10.64 ± (10.64 - 10.64) MB+0.3%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 6 - Bailout
process.internal_duration_ms21.38 ± (21.32 - 21.43) ms21.46 ± (21.41 - 21.52) ms+0.4%✅⬆️
process.time_to_main_ms75.59 ± (75.28 - 75.90) ms76.78 ± (76.48 - 77.07) ms+1.6%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.72 ± (10.71 - 10.72) MB10.75 ± (10.75 - 10.76) MB+0.3%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms383.41 ± (381.55 - 385.27) ms384.03 ± (381.94 - 386.13) ms+0.2%✅⬆️
process.time_to_main_ms524.24 ± (523.11 - 525.36) ms527.90 ± (526.66 - 529.14) ms+0.7%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed49.24 ± (49.21 - 49.26) MB49.38 ± (49.35 - 49.41) MB+0.3%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.2%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms19.73 ± (19.68 - 19.78) ms19.69 ± (19.62 - 19.77) ms-0.2%
process.time_to_main_ms74.50 ± (74.23 - 74.76) ms73.41 ± (73.13 - 73.69) ms-1.5%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.67 ± (7.66 - 7.67) MB7.66 ± (7.65 - 7.66) MB-0.2%
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 8 - Bailout
process.internal_duration_ms19.42 ± (19.37 - 19.47) ms19.46 ± (19.42 - 19.50) ms+0.2%✅⬆️
process.time_to_main_ms74.21 ± (74.03 - 74.39) ms73.99 ± (73.81 - 74.17) ms-0.3%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.72 ± (7.71 - 7.72) MB7.71 ± (7.70 - 7.72) MB-0.1%
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms302.35 ± (300.35 - 304.34) ms303.83 ± (301.44 - 306.22) ms+0.5%✅⬆️
process.time_to_main_ms489.41 ± (488.23 - 490.59) ms488.82 ± (487.86 - 489.78) ms-0.1%
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed36.54 ± (36.50 - 36.58) MB36.61 ± (36.55 - 36.67) MB+0.2%✅⬆️
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)+0.1%✅⬆️

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration206.29 ± (205.25 - 206.94) ms200.54 ± (200.38 - 201.50) ms-2.8%
.NET Framework 4.8 - Bailout
duration205.76 ± (205.78 - 207.14) ms203.61 ± (203.32 - 204.40) ms-1.0%
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1198.38 ± (1196.57 - 1204.27) ms1185.33 ± (1188.16 - 1196.82) ms-1.1%
.NET Core 3.1 - Baseline
process.internal_duration_ms196.39 ± (195.61 - 197.18) ms193.92 ± (193.33 - 194.50) ms-1.3%
process.time_to_main_ms85.18 ± (84.79 - 85.57) ms84.10 ± (83.82 - 84.38) ms-1.3%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.04 ± (16.02 - 16.06) MB16.05 ± (16.03 - 16.07) MB+0.1%✅⬆️
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (20 - 20)-0.7%
.NET Core 3.1 - Bailout
process.internal_duration_ms197.88 ± (197.09 - 198.67) ms194.18 ± (193.57 - 194.80) ms-1.9%
process.time_to_main_ms86.93 ± (86.52 - 87.34) ms85.91 ± (85.63 - 86.19) ms-1.2%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.01 ± (15.99 - 16.03) MB16.07 ± (16.05 - 16.09) MB+0.4%✅⬆️
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (21 - 21)-0.2%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms390.36 ± (388.39 - 392.32) ms387.38 ± (386.01 - 388.76) ms-0.8%
process.time_to_main_ms531.42 ± (529.60 - 533.24) ms525.80 ± (524.50 - 527.11) ms-1.1%
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed57.28 ± (57.06 - 57.51) MB57.34 ± (57.17 - 57.52) MB+0.1%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.2%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms202.91 ± (202.14 - 203.68) ms200.25 ± (199.58 - 200.92) ms-1.3%
process.time_to_main_ms74.03 ± (73.70 - 74.37) ms73.51 ± (73.20 - 73.82) ms-0.7%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.31 ± (16.28 - 16.33) MB16.30 ± (16.28 - 16.32) MB-0.0%
runtime.dotnet.threads.count19 ± (19 - 19)19 ± (19 - 19)+0.3%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms202.67 ± (201.84 - 203.50) ms198.62 ± (198.10 - 199.14) ms-2.0%
process.time_to_main_ms75.44 ± (75.13 - 75.75) ms73.85 ± (73.62 - 74.09) ms-2.1%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed16.26 ± (16.24 - 16.28) MB16.31 ± (16.29 - 16.33) MB+0.3%✅⬆️
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (20 - 20)-1.0%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms593.08 ± (590.41 - 595.74) ms595.88 ± (593.08 - 598.68) ms+0.5%✅⬆️
process.time_to_main_ms534.16 ± (532.71 - 535.61) ms525.31 ± (524.12 - 526.50) ms-1.7%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed60.89 ± (60.79 - 61.00) MB60.96 ± (60.86 - 61.06) MB+0.1%✅⬆️
runtime.dotnet.threads.count31 ± (31 - 31)31 ± (31 - 31)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms200.26 ± (199.38 - 201.13) ms198.32 ± (197.62 - 199.02) ms-1.0%
process.time_to_main_ms73.44 ± (73.11 - 73.77) ms73.16 ± (72.86 - 73.45) ms-0.4%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.63 ± (11.61 - 11.65) MB11.64 ± (11.62 - 11.65) MB+0.1%✅⬆️
runtime.dotnet.threads.count19 ± (18 - 19)18 ± (18 - 18)-1.0%
.NET 8 - Bailout
process.internal_duration_ms199.44 ± (198.62 - 200.25) ms196.10 ± (195.58 - 196.61) ms-1.7%
process.time_to_main_ms74.80 ± (74.44 - 75.16) ms73.18 ± (72.99 - 73.38) ms-2.2%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.69 ± (11.68 - 11.71) MB11.69 ± (11.68 - 11.71) MB-0.0%
runtime.dotnet.threads.count19 ± (19 - 20)19 ± (19 - 19)-1.0%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms528.05 ± (522.73 - 533.37) ms521.85 ± (519.01 - 524.70) ms-1.2%
process.time_to_main_ms493.28 ± (491.93 - 494.62) ms481.00 ± (479.93 - 482.08) ms-2.5%
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed50.33 ± (50.27 - 50.38) MB50.32 ± (50.28 - 50.36) MB-0.0%
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+0.2%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (73ms)  : 70, 76
    master - mean (75ms)  : 71, 79

    section Bailout
    This PR (8505) - mean (79ms)  : 75, 83
    master - mean (78ms)  : 75, 81

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (1,076ms)  : 1017, 1135
    master - mean (1,078ms)  : 1025, 1130

Loading
FakeDbCommand (.NET Core 3.1)
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (117ms)  : 111, 123
    master - mean (116ms)  : 110, 121

    section Bailout
    This PR (8505) - mean (118ms)  : 112, 124
    master - mean (114ms)  : 111, 117

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (777ms)  : 748, 805
    master - mean (778ms)  : 755, 801

Loading
FakeDbCommand (.NET 6)
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (102ms)  : 97, 106
    master - mean (101ms)  : 97, 104

    section Bailout
    This PR (8505) - mean (105ms)  : 101, 109
    master - mean (104ms)  : 97, 110

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (938ms)  : 903, 974
    master - mean (937ms)  : 901, 974

Loading
FakeDbCommand (.NET 8)
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (101ms)  : 95, 106
    master - mean (102ms)  : 96, 108

    section Bailout
    This PR (8505) - mean (101ms)  : 98, 104
    master - mean (101ms)  : 99, 104

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (823ms)  : 786, 861
    master - mean (823ms)  : 790, 856

Loading
HttpMessageHandler (.NET Framework 4.8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (201ms)  : 193, 209
    master - mean (206ms)  : 193, 219

    section Bailout
    This PR (8505) - mean (204ms)  : 196, 212
    master - mean (206ms)  : 197, 216

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (1,192ms)  : 1130, 1255
    master - mean (1,200ms)  : 1144, 1256

Loading
HttpMessageHandler (.NET Core 3.1)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (287ms)  : 274, 301
    master - mean (292ms)  : 274, 310

    section Bailout
    This PR (8505) - mean (289ms)  : 276, 303
    master - mean (295ms)  : 277, 313

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (948ms)  : 920, 975
    master - mean (956ms)  : 937, 976

Loading
HttpMessageHandler (.NET 6)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (282ms)  : 266, 299
    master - mean (288ms)  : 270, 306

    section Bailout
    This PR (8505) - mean (281ms)  : 269, 292
    master - mean (288ms)  : 270, 305

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (1,149ms)  : 1093, 1205
    master - mean (1,158ms)  : 1100, 1216

Loading
HttpMessageHandler (.NET 8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8505) - mean (282ms)  : 267, 298
    master - mean (285ms)  : 266, 305

    section Bailout
    This PR (8505) - mean (280ms)  : 269, 290
    master - mean (287ms)  : 266, 308

    section CallTarget+Inlining+NGEN
    This PR (8505) - mean (1,037ms)  : 991, 1082
    master - mean (1,051ms)  : 980, 1123

Loading

@pr-commenter

pr-commenter Bot commented Apr 23, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-04-23 05:26:30

Comparing candidate commit 647d1c7 in PR branch omer/sourcelink-enterprise-variants with baseline commit 0f47fd3 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 27 metrics, 0 unstable metrics, 58 known flaky benchmarks, 29 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0

  • 🟩 throughput [+10019.517op/s; +12351.038op/s] or [+8.422%; +10.382%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 execution_time [+309.295ms; +314.250ms] or [+153.483%; +155.942%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+381.269ms; +384.230ms] or [+301.226%; +303.565%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+400.337ms; +402.564ms] or [+354.283%; +356.254%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472

  • 🟥 allocated_mem [+1.308KB; +1.308KB] or [+27.529%; +27.541%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+9.977%; +9.987%]
  • 🟩 execution_time [-15.878ms; -11.706ms] or [-7.416%; -5.467%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+27.502%; +27.510%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

  • 🟥 allocated_mem [+1.307KB; +1.307KB] or [+105.746%; +105.759%]
  • 🟥 throughput [-231452.226op/s; -228007.685op/s] or [-23.632%; -23.281%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+38.558%; +38.566%]
  • 🟩 execution_time [-25.492ms; -20.649ms] or [-11.369%; -9.209%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+105.292%; +105.304%]
  • 🟥 throughput [-137666.784op/s; -121487.935op/s] or [-19.780%; -17.455%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

  • 🟩 throughput [+11443.693op/s; +14374.564op/s] or [+7.281%; +9.146%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

  • 🟩 throughput [+8801.584op/s; +11460.073op/s] or [+7.012%; +9.129%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

  • 🟩 throughput [+438319.768op/s; +464090.981op/s] or [+14.615%; +15.475%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • 🟩 execution_time [-19.062ms; -14.700ms] or [-8.787%; -6.776%]
  • 🟩 throughput [+201792.380op/s; +255463.182op/s] or [+8.010%; +10.140%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472

  • 🟥 execution_time [+300.340ms; +301.087ms] or [+150.070%; +150.443%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0

  • 🟥 execution_time [+300.185ms; +303.474ms] or [+151.384%; +153.043%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

  • 🟥 execution_time [+299.308ms; +301.803ms] or [+150.768%; +152.025%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472

  • 🟥 execution_time [+297.049ms; +297.652ms] or [+145.899%; +146.195%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

  • 🟥 execution_time [+295.228ms; +299.158ms] or [+144.326%; +146.247%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • 🟥 execution_time [+300.293ms; +302.434ms] or [+150.086%; +151.156%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0

  • 🟥 execution_time [+26.499µs; +50.386µs] or [+8.460%; +16.086%]
  • 🟥 throughput [-462.184op/s; -262.069op/s] or [-14.408%; -8.169%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472

  • 🟥 execution_time [+299.820ms; +300.506ms] or [+149.641%; +149.983%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

  • 🟥 execution_time [+410.374ms; +414.900ms] or [+445.888%; +450.806%]
  • 🟩 throughput [+1080.959op/s; +1205.475op/s] or [+8.882%; +9.906%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

  • unstable execution_time [+261.339ms; +304.831ms] or [+198.432%; +231.455%]
  • 🟩 throughput [+654.551op/s; +854.413op/s] or [+6.336%; +8.271%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • unstable execution_time [+316.721ms; +396.301ms] or [+145.625%; +182.215%]
  • 🟥 throughput [-588.743op/s; -522.633op/s] or [-53.346%; -47.356%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • unstable execution_time [+205.167ms; +338.385ms] or [+87.433%; +144.205%]
  • 🟥 throughput [-744.805op/s; -661.356op/s] or [-49.679%; -44.113%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 allocated_mem [+2.304KB; +2.308KB] or [+5.441%; +5.449%]
  • 🟥 execution_time [+337.171ms; +346.673ms] or [+201.668%; +207.351%]
  • 🟥 throughput [-409.309op/s; -373.463op/s] or [-28.500%; -26.004%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472

  • 🟥 execution_time [+303.986ms; +305.290ms] or [+153.082%; +153.738%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

  • 🟥 execution_time [+302.492ms; +304.115ms] or [+151.579%; +152.392%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1

  • 🟥 execution_time [+301.949ms; +305.337ms] or [+151.686%; +153.388%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

  • 🟥 execution_time [+302.756ms; +304.153ms] or [+152.034%; +152.735%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

  • 🟥 execution_time [+298.463ms; +300.350ms] or [+147.577%; +148.510%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

  • 🟥 execution_time [+302.864ms; +306.473ms] or [+153.505%; +155.334%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472

  • 🟥 execution_time [+302.249ms; +303.755ms] or [+151.702%; +152.458%]
  • 🟥 throughput [-22147.868op/s; -20086.737op/s] or [-5.746%; -5.211%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

  • 🟥 execution_time [+300.430ms; +310.092ms] or [+149.737%; +154.552%]
  • 🟩 throughput [+60619.823op/s; +70731.970op/s] or [+12.037%; +14.045%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

  • 🟥 execution_time [+298.369ms; +301.286ms] or [+148.436%; +149.887%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0

  • 🟩 execution_time [-17.140ms; -13.477ms] or [-7.970%; -6.267%]
  • 🟩 throughput [+23794.220op/s; +30387.779op/s] or [+6.527%; +8.336%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472

  • unstable execution_time [+13.181µs; +55.030µs] or [+3.256%; +13.593%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

  • 🟩 allocated_mem [-19.742KB; -19.721KB] or [-7.201%; -7.194%]
  • unstable execution_time [-38.120µs; +14.573µs] or [-7.534%; +2.880%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

  • 🟩 allocated_mem [-15.684KB; -15.668KB] or [-5.717%; -5.712%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

  • 🟥 execution_time [+6.348µs; +10.429µs] or [+15.005%; +24.650%]
  • 🟥 throughput [-4716.700op/s; -2955.384op/s] or [-19.856%; -12.441%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

  • unstable execution_time [-14.211µs; -6.951µs] or [-22.048%; -10.784%]
  • 🟩 throughput [+1827.930op/s; +3352.101op/s] or [+11.215%; +20.566%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472

  • 🟥 execution_time [+302.044ms; +303.313ms] or [+152.670%; +153.311%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+303.713ms; +305.874ms] or [+154.589%; +155.689%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+299.745ms; +301.686ms] or [+150.059%; +151.031%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net6.0

  • 🟩 throughput [+35738.719op/s; +41848.197op/s] or [+6.765%; +7.921%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472

  • 🟥 execution_time [+299.231ms; +301.084ms] or [+149.140%; +150.063%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+301.673ms; +302.902ms] or [+151.486%; +152.103%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+303.363ms; +306.235ms] or [+153.846%; +155.303%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472

  • 🟥 execution_time [+300.137ms; +300.740ms] or [+149.710%; +150.011%]
  • 🟩 throughput [+61012174.829op/s; +61276915.737op/s] or [+44.433%; +44.626%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

  • unstable execution_time [+383.780ms; +412.935ms] or [+477.298%; +513.558%]
  • 🟩 throughput [+971.292op/s; +1146.532op/s] or [+7.509%; +8.863%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • 🟥 execution_time [+299.314ms; +300.235ms] or [+149.291%; +149.750%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net472

  • 🟥 throughput [-51587.358op/s; -48313.049op/s] or [-5.757%; -5.392%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0

  • 🟩 throughput [+94038.990op/s; +103104.643op/s] or [+8.780%; +9.627%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0

  • 🟩 throughput [+86395.601op/s; +116632.936op/s] or [+6.687%; +9.028%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1

  • 🟩 throughput [+86833.483op/s; +95622.482op/s] or [+8.624%; +9.497%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0

  • 🟩 throughput [+48877.489op/s; +53214.676op/s] or [+8.875%; +9.663%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0

  • 🟩 throughput [+47911.136op/s; +66064.824op/s] or [+5.353%; +7.381%]

Known flaky benchmarks without significant changes:

  • scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472
  • scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark netcoreapp3.1
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack netcoreapp3.1
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool netcoreapp3.1
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1
  • scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net472
  • scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1
  • scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472
  • scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net472
  • scenario:Benchmarks.Trace.RedisBenchmark.SendReceive netcoreapp3.1
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1
  • scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472
  • scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant