Fix dsm throughput tests - #8981
Conversation
BenchmarksBenchmark execution time: 2026-08-03 14:51:24 Comparing candidate commit a3b4460 in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 71 metrics, 0 unstable metrics, 65 known flaky benchmarks, 61 flaky benchmarks without significant changes.
|
cba5efa to
7c52704
Compare
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8981) and master.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 190.87 ± (190.92 - 191.61) ms | 204.85 ± (204.67 - 205.53) ms | +7.3% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 195.47 ± (195.18 - 195.61) ms | 207.91 ± (207.48 - 208.21) ms | +6.4% | ❌⬆️ |
Full Metrics Comparison
FakeDbCommand
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 70.59 ± (70.67 - 70.98) ms | 70.29 ± (70.51 - 70.93) ms | -0.4% | ✅ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 74.80 ± (74.71 - 75.04) ms | 76.61 ± (76.49 - 77.12) ms | +2.4% | ✅⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1085.33 ± (1085.33 - 1091.55) ms | 1089.26 ± (1089.35 - 1097.10) ms | +0.4% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 22.19 ± (22.16 - 22.22) ms | 22.11 ± (22.07 - 22.15) ms | -0.4% | ✅ |
| process.time_to_main_ms | 81.50 ± (81.33 - 81.67) ms | 81.22 ± (81.06 - 81.39) ms | -0.3% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.01 ± (11.00 - 11.01) MB | 10.99 ± (10.98 - 10.99) MB | -0.2% | ✅ |
| runtime.dotnet.threads.count | 12 ± (12 - 12) | 12 ± (12 - 12) | +0.0% | ✅ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 21.93 ± (21.91 - 21.96) ms | 22.29 ± (22.24 - 22.34) ms | +1.6% | ✅⬆️ |
| process.time_to_main_ms | 82.46 ± (82.30 - 82.62) ms | 85.92 ± (85.61 - 86.24) ms | +4.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.03 ± (11.03 - 11.04) MB | 11.02 ± (11.02 - 11.02) MB | -0.1% | ✅ |
| runtime.dotnet.threads.count | 13 ± (13 - 13) | 13 ± (13 - 13) | +0.0% | ✅ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 211.05 ± (210.16 - 211.94) ms | 210.29 ± (209.33 - 211.26) ms | -0.4% | ✅ |
| process.time_to_main_ms | 538.95 ± (537.71 - 540.18) ms | 538.44 ± (537.18 - 539.70) ms | -0.1% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 49.60 ± (49.57 - 49.63) MB | 49.46 ± (49.42 - 49.50) MB | -0.3% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.3% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 20.90 ± (20.87 - 20.93) ms | 21.11 ± (21.06 - 21.15) ms | +1.0% | ✅⬆️ |
| process.time_to_main_ms | 71.03 ± (70.90 - 71.15) ms | 74.03 ± (73.76 - 74.30) ms | +4.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.73 ± (10.73 - 10.73) MB | 10.73 ± (10.73 - 10.73) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 20.69 ± (20.66 - 20.72) ms | 20.91 ± (20.87 - 20.94) ms | +1.0% | ✅⬆️ |
| process.time_to_main_ms | 71.87 ± (71.74 - 72.01) ms | 74.15 ± (73.90 - 74.40) ms | +3.2% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 10.85 ± (10.85 - 10.85) MB | 10.85 ± (10.84 - 10.85) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 371.09 ± (368.81 - 373.36) ms | 371.24 ± (368.69 - 373.79) ms | +0.0% | ✅⬆️ |
| process.time_to_main_ms | 543.01 ± (541.90 - 544.11) ms | 546.32 ± (544.99 - 547.64) ms | +0.6% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 50.65 ± (50.63 - 50.67) MB | 50.64 ± (50.62 - 50.66) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 28 ± (28 - 28) | 28 ± (28 - 28) | +0.1% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 19.42 ± (19.38 - 19.47) ms | 19.08 ± (19.05 - 19.10) ms | -1.8% | ✅ |
| process.time_to_main_ms | 73.05 ± (72.79 - 73.30) ms | 69.90 ± (69.78 - 70.03) ms | -4.3% | ✅ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.78 ± (7.77 - 7.78) MB | 7.77 ± (7.77 - 7.78) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 10 ± (10 - 10) | 10 ± (10 - 10) | +0.0% | ✅ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 19.06 ± (19.03 - 19.08) ms | 19.34 ± (19.30 - 19.39) ms | +1.5% | ✅⬆️ |
| process.time_to_main_ms | 71.70 ± (71.55 - 71.85) ms | 74.33 ± (74.01 - 74.65) ms | +3.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 7.81 ± (7.80 - 7.81) MB | 7.82 ± (7.81 - 7.83) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 11 ± (11 - 11) | 11 ± (11 - 11) | +0.0% | ✅ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 297.56 ± (295.57 - 299.54) ms | 298.39 ± (296.08 - 300.69) ms | +0.3% | ✅⬆️ |
| process.time_to_main_ms | 491.00 ± (490.07 - 491.93) ms | 493.32 ± (492.32 - 494.31) ms | +0.5% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 0 ± (0 - 0) | 0 ± (0 - 0) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 38.06 ± (38.03 - 38.08) MB | 38.10 ± (38.07 - 38.13) MB | +0.1% | ✅⬆️ |
| runtime.dotnet.threads.count | 27 ± (27 - 27) | 27 ± (27 - 27) | +0.7% | ✅⬆️ |
HttpMessageHandler
| Metric | Master (Mean ± 95% CI) | Current (Mean ± 95% CI) | Change | Status |
|---|---|---|---|---|
| .NET Framework 4.8 - Baseline | ||||
| duration | 190.87 ± (190.92 - 191.61) ms | 204.85 ± (204.67 - 205.53) ms | +7.3% | ❌⬆️ |
| .NET Framework 4.8 - Bailout | ||||
| duration | 195.47 ± (195.18 - 195.61) ms | 207.91 ± (207.48 - 208.21) ms | +6.4% | ❌⬆️ |
| .NET Framework 4.8 - CallTarget+Inlining+NGEN | ||||
| duration | 1165.40 ± (1164.04 - 1169.25) ms | 1224.24 ± (1225.22 - 1233.05) ms | +5.0% | ✅⬆️ |
| .NET Core 3.1 - Baseline | ||||
| process.internal_duration_ms | 184.56 ± (184.24 - 184.87) ms | 196.14 ± (195.80 - 196.48) ms | +6.3% | ✅⬆️ |
| process.time_to_main_ms | 80.56 ± (80.37 - 80.75) ms | 85.95 ± (85.67 - 86.24) ms | +6.7% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.31 ± (16.24 - 16.37) MB | 16.24 ± (16.21 - 16.27) MB | -0.4% | ✅ |
| runtime.dotnet.threads.count | 20 ± (19 - 20) | 20 ± (20 - 20) | +0.3% | ✅⬆️ |
| .NET Core 3.1 - Bailout | ||||
| process.internal_duration_ms | 183.85 ± (183.60 - 184.10) ms | 197.32 ± (196.75 - 197.89) ms | +7.3% | ✅⬆️ |
| process.time_to_main_ms | 81.57 ± (81.46 - 81.68) ms | 87.49 ± (87.24 - 87.74) ms | +7.3% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 16.03 ± (15.88 - 16.17) MB | 16.22 ± (16.20 - 16.24) MB | +1.2% | ✅⬆️ |
| runtime.dotnet.threads.count | 20 ± (20 - 20) | 21 ± (21 - 21) | +3.1% | ✅⬆️ |
| .NET Core 3.1 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 373.15 ± (371.86 - 374.43) ms | 389.17 ± (387.98 - 390.36) ms | +4.3% | ✅⬆️ |
| process.time_to_main_ms | 523.06 ± (522.19 - 523.92) ms | 548.75 ± (547.56 - 549.94) ms | +4.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 3 ± (3 - 3) | 3 ± (3 - 3) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 58.02 ± (57.96 - 58.08) MB | 58.61 ± (58.37 - 58.84) MB | +1.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 30 ± (30 - 30) | 30 ± (30 - 30) | +0.3% | ✅⬆️ |
| .NET 6 - Baseline | ||||
| process.internal_duration_ms | 188.65 ± (188.38 - 188.92) ms | 200.82 ± (200.44 - 201.20) ms | +6.5% | ✅⬆️ |
| process.time_to_main_ms | 70.41 ± (70.25 - 70.56) ms | 75.22 ± (74.97 - 75.47) ms | +6.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.91 ± (15.73 - 16.09) MB | 16.47 ± (16.44 - 16.50) MB | +3.5% | ✅⬆️ |
| runtime.dotnet.threads.count | 17 ± (17 - 18) | 19 ± (19 - 19) | +11.1% | ✅⬆️ |
| .NET 6 - Bailout | ||||
| process.internal_duration_ms | 187.12 ± (186.93 - 187.32) ms | 200.34 ± (199.94 - 200.75) ms | +7.1% | ✅⬆️ |
| process.time_to_main_ms | 71.38 ± (71.33 - 71.43) ms | 76.24 ± (76.05 - 76.43) ms | +6.8% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 15.75 ± (15.58 - 15.93) MB | 16.54 ± (16.51 - 16.56) MB | +5.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 18 ± (18 - 19) | 20 ± (20 - 20) | +9.3% | ✅⬆️ |
| .NET 6 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 584.74 ± (581.93 - 587.56) ms | 586.81 ± (584.48 - 589.13) ms | +0.4% | ✅⬆️ |
| process.time_to_main_ms | 532.03 ± (531.16 - 532.90) ms | 558.68 ± (557.72 - 559.64) ms | +5.0% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 61.67 ± (61.60 - 61.75) MB | 61.87 ± (61.78 - 61.97) MB | +0.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 31 ± (31 - 31) | 31 ± (31 - 31) | +0.4% | ✅⬆️ |
| .NET 8 - Baseline | ||||
| process.internal_duration_ms | 186.48 ± (186.24 - 186.72) ms | 199.55 ± (199.17 - 199.93) ms | +7.0% | ✅⬆️ |
| process.time_to_main_ms | 69.66 ± (69.52 - 69.81) ms | 74.75 ± (74.52 - 74.97) ms | +7.3% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.73 ± (11.61 - 11.84) MB | 11.87 ± (11.85 - 11.90) MB | +1.3% | ✅⬆️ |
| runtime.dotnet.threads.count | 17 ± (17 - 18) | 18 ± (18 - 19) | +6.3% | ✅⬆️ |
| .NET 8 - Bailout | ||||
| process.internal_duration_ms | 185.53 ± (185.38 - 185.69) ms | 199.12 ± (198.80 - 199.43) ms | +7.3% | ✅⬆️ |
| process.time_to_main_ms | 70.79 ± (70.73 - 70.86) ms | 75.48 ± (75.33 - 75.62) ms | +6.6% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 11.59 ± (11.46 - 11.72) MB | 11.94 ± (11.92 - 11.97) MB | +3.0% | ✅⬆️ |
| runtime.dotnet.threads.count | 18 ± (18 - 19) | 20 ± (20 - 20) | +7.4% | ✅⬆️ |
| .NET 8 - CallTarget+Inlining+NGEN | ||||
| process.internal_duration_ms | 512.16 ± (509.37 - 514.94) ms | 510.83 ± (507.97 - 513.70) ms | -0.3% | ✅ |
| process.time_to_main_ms | 482.45 ± (481.84 - 483.05) ms | 510.76 ± (509.95 - 511.56) ms | +5.9% | ✅⬆️ |
| runtime.dotnet.exceptions.count | 4 ± (4 - 4) | 4 ± (4 - 4) | +0.0% | ✅ |
| runtime.dotnet.mem.committed | 51.52 ± (51.48 - 51.55) MB | 51.49 ± (51.46 - 51.53) MB | -0.0% | ✅ |
| runtime.dotnet.threads.count | 29 ± (29 - 29) | 30 ± (30 - 30) | +1.6% | ✅⬆️ |
Comparison explanation
Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (71ms) : 68, 74
master - mean (71ms) : 69, 73
section Bailout
This PR (8981) - mean (77ms) : 73, 81
master - mean (75ms) : 73, 77
section CallTarget+Inlining+NGEN
This PR (8981) - mean (1,093ms) : 1037, 1149
master - mean (1,088ms) : 1044, 1133
FakeDbCommand (.NET Core 3.1)
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (110ms) : 107, 113
master - mean (110ms) : 106, 114
section Bailout
This PR (8981) - mean (115ms) : 109, 122
master - mean (111ms) : 108, 113
section CallTarget+Inlining+NGEN
This PR (8981) - mean (785ms) : 765, 806
master - mean (788ms) : 767, 809
FakeDbCommand (.NET 6)
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (102ms) : 96, 107
master - mean (98ms) : 95, 101
section Bailout
This PR (8981) - mean (102ms) : 96, 108
master - mean (99ms) : 97, 100
section CallTarget+Inlining+NGEN
This PR (8981) - mean (948ms) : 908, 987
master - mean (943ms) : 898, 988
FakeDbCommand (.NET 8)
gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (96ms) : 94, 99
master - mean (100ms) : 94, 106
section Bailout
This PR (8981) - mean (102ms) : 96, 108
master - mean (98ms) : 94, 101
section CallTarget+Inlining+NGEN
This PR (8981) - mean (824ms) : 784, 864
master - mean (818ms) : 786, 851
HttpMessageHandler (.NET Framework 4.8)
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (205ms) : 201, 210
master - mean (191ms) : 188, 195
section Bailout
This PR (8981) - mean (208ms) : crit, 204, 211
master - mean (195ms) : 193, 197
section CallTarget+Inlining+NGEN
This PR (8981) - mean (1,229ms) : 1173, 1285
master - mean (1,167ms) : 1135, 1199
HttpMessageHandler (.NET Core 3.1)
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (293ms) : 285, 300
master - mean (274ms) : 270, 278
section Bailout
This PR (8981) - mean (295ms) : crit, 287, 303
master - mean (274ms) : 271, 277
section CallTarget+Inlining+NGEN
This PR (8981) - mean (979ms) : 956, 1003
master - mean (937ms) : 918, 956
HttpMessageHandler (.NET 6)
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (285ms) : 278, 293
master - mean (267ms) : 264, 271
section Bailout
This PR (8981) - mean (285ms) : crit, 279, 292
master - mean (267ms) : 264, 270
section CallTarget+Inlining+NGEN
This PR (8981) - mean (1,175ms) : 1127, 1223
master - mean (1,146ms) : 1102, 1190
HttpMessageHandler (.NET 8)
gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8981) - mean (285ms) : 280, 289
master - mean (266ms) : 262, 270
section Bailout
This PR (8981) - mean (285ms) : crit, 281, 289
master - mean (266ms) : 263, 268
section CallTarget+Inlining+NGEN
This PR (8981) - mean (1,053ms) : 1001, 1105
master - mean (1,026ms) : 985, 1068
7c52704 to
1c1d2e7
Compare
1c1d2e7 to
a3b4460
Compare
Summary of changes
Fix failing dsm throughput check
Reason for change
The DSM throughput tests are failing on master because of
signature manifest not found, because they're using an old benchmarking platform image (v7, instead of v8)Implementation details
Update to use the new image
Test coverage
Forced a test, and it succeeded