Skip to content

Fix Pekko HTTP async test exception flakiness#10799

Closed
bm1549 wants to merge 3 commits intomasterfrom
brian.marks/fix-pekko-http-async-flaky-test-v2
Closed

Fix Pekko HTTP async test exception flakiness#10799
bm1549 wants to merge 3 commits intomasterfrom
brian.marks/fix-pekko-http-async-flaky-test-v2

Conversation

@bm1549
Copy link
Copy Markdown
Contributor

@bm1549 bm1549 commented Mar 10, 2026 •
edited
Loading

What Does This Do

Fixes the root cause of flaky PekkoHttpServerInstrumentationAsyncTest and PekkoHttpServerInstrumentationAsyncHttp2Test "test exception" failures. Removes the @Flaky annotation.

Motivation

Root Cause

When the async handler throws an exception, the Future fails. The instrumentation's error transform callback calls span.finish() while Scala Promise.Transformation continuations are still pending (pendingReferenceCount > 0). The root span goes to ROOT_BUFFERED → DiscardingPendingTraceBuffer.enqueue() (a no-op under strictTraceWrites=true). The trace IS eventually written when continuations clean up (removeContinuation → count reaches 0 → write()), but this happens asynchronously. Under extreme CI load, the gap between the HTTP response and continuation cleanup can exceed the test's 20-second waitForTraces timeout.

Why Deterministic Local Reproduction Is Not Feasible

The race window is between Transformation.run() body completion and the endTaskScope call in the bytecode advice exit handler — nanoseconds under normal conditions. A simple Thread.sleep() in the Future body delays execution start, not the gap between span finish and continuation cleanup. Reproducing this requires >20s OS thread preemption, which only occurs under extreme CI load. Confirmed: 10 local runs with various delay strategies produced 0 failures.

Fix

Added .recover { case e: Exception => HttpResponse(500) } to the asyncHandler's Future. This converts the failed Future to a successful one, ensuring the Scala Future chain resolves on the success path throughout. The .recover Transformation runs between the body's failure and the error callback, meaning by the time the span is finished, all continuations have been cleaned up sequentially. This eliminates the race by design.

Why the Fix Is Correct (by analysis, not reproduction)

  • Server span: Still marked as errored via HTTP 500 status check in the decorator
  • Controller span: Unaffected — runUnderTrace("controller", closure) captures the exception (error.type, error.msg, error.stack) and re-throws BEFORE .recover catches it
  • expectedExtraErrorInformation: Uses it == null || ... predicates, passing regardless of error propagation mechanism
  • Relates to PekkoHttpServerInstrumentationAsyncHttp2Test."test exception" is flaky #9396

Additional Notes

  • The @Flaky annotation on "test exception" is removed
  • The syncHandler's exception behavior is unchanged
  • All test assertions pass identically

Jira ticket: N/A

🤖 Generated with Claude Code

@bm1549 @claude
Fix Pekko HTTP async test exception flakiness
c5a177c 
The async handler's exception path caused a failed Future whose span
completion depended on Scala continuation cleanup. With strict trace
writes enabled in tests, if the root span finished while continuations
were still pending, the trace was enqueued to a discarding buffer and
never written, causing a 20-second timeout in waitForTraces.

Fix by recovering from exceptions in the async handler to return a
proper 500 HTTP response instead of a failed Future. This routes span
completion through the success path of the DatadogAsyncHandlerWrapper
transform callback, avoiding the problematic continuation cleanup race.

Also remove the @flaky annotation from the "test exception" test since
the root cause is now fixed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bm1549 bm1549 added type: bug Bug report and fix tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM tag: flaky test Flaky tests labels Mar 10, 2026
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Mar 10, 2026 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master brian.marks/fix-pekko-http-async-flaky-test-v2
git_commit_date 1773169596 1773757680
git_commit_sha cc12228 ede5582
release_version 1.61.0-SNAPSHOT~cc122288e5 1.61.0-SNAPSHOT~ede558242d
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1773962892 1773962892
ci_job_id 1523998674 1523998674
ci_pipeline_id 103634483 103634483
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-kcj535kg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-kcj535kg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 66 metrics, 5 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1059685
Total [baseline] (11.053 s) : 0, 11052836
Agent [candidate] (1.058 s) : 0, 1058340
Total [candidate] (11.031 s) : 0, 11030504
section appsec
Agent [baseline] (1.244 s) : 0, 1244318
Total [baseline] (11.118 s) : 0, 11118058
Agent [candidate] (1.245 s) : 0, 1245215
Total [candidate] (11.183 s) : 0, 11183450
section iast
Agent [baseline] (1.227 s) : 0, 1227349
Total [baseline] (11.349 s) : 0, 11349269
Agent [candidate] (1.239 s) : 0, 1238789
Total [candidate] (11.466 s) : 0, 11466064
section profiling
Agent [baseline] (1.179 s) : 0, 1179289
Total [baseline] (11.03 s) : 0, 11030252
Agent [candidate] (1.179 s) : 0, 1178599
Total [candidate] (11.092 s) : 0, 11092033
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.06 s -
Agent appsec 1.244 s 184.632 ms (17.4%)
Agent iast 1.227 s 167.664 ms (15.8%)
Agent profiling 1.179 s 119.603 ms (11.3%)
Total tracing 11.053 s -
Total appsec 11.118 s 65.221 ms (0.6%)
Total iast 11.349 s 296.433 ms (2.7%)
Total profiling 11.03 s -22.584 ms (-0.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.058 s -
Agent appsec 1.245 s 186.875 ms (17.7%)
Agent iast 1.239 s 180.449 ms (17.1%)
Agent profiling 1.179 s 120.259 ms (11.4%)
Total tracing 11.031 s -
Total appsec 11.183 s 152.946 ms (1.4%)
Total iast 11.466 s 435.56 ms (3.9%)
Total profiling 11.092 s 61.529 ms (0.6%) 
gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.182 ms) : 0, 1182
BytebuddyAgent [baseline] (629.251 ms) : 0, 629251
BytebuddyAgent [candidate] (627.929 ms) : 0, 627929
AgentMeter [baseline] (29.13 ms) : 0, 29130
AgentMeter [candidate] (29.01 ms) : 0, 29010
GlobalTracer [baseline] (257.523 ms) : 0, 257523
GlobalTracer [candidate] (257.201 ms) : 0, 257201
AppSec [baseline] (31.522 ms) : 0, 31522
AppSec [candidate] (31.702 ms) : 0, 31702
Debugger [baseline] (59.448 ms) : 0, 59448
Debugger [candidate] (59.702 ms) : 0, 59702
Remote Config [baseline] (586.336 µs) : 0, 586
Remote Config [candidate] (586.041 µs) : 0, 586
Telemetry [baseline] (8.604 ms) : 0, 8604
Telemetry [candidate] (8.664 ms) : 0, 8664
Flare Poller [baseline] (6.429 ms) : 0, 6429
Flare Poller [candidate] (6.41 ms) : 0, 6410
section appsec
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.193 ms) : 0, 1193
BytebuddyAgent [baseline] (657.353 ms) : 0, 657353
BytebuddyAgent [candidate] (657.687 ms) : 0, 657687
AgentMeter [baseline] (11.978 ms) : 0, 11978
AgentMeter [candidate] (11.989 ms) : 0, 11989
GlobalTracer [baseline] (257.859 ms) : 0, 257859
GlobalTracer [candidate] (257.986 ms) : 0, 257986
AppSec [baseline] (176.95 ms) : 0, 176950
AppSec [candidate] (177.396 ms) : 0, 177396
Debugger [baseline] (65.753 ms) : 0, 65753
Debugger [candidate] (65.73 ms) : 0, 65730
Remote Config [baseline] (577.621 µs) : 0, 578
Remote Config [candidate] (564.972 µs) : 0, 565
Telemetry [baseline] (9.027 ms) : 0, 9027
Telemetry [candidate] (8.973 ms) : 0, 8973
Flare Poller [baseline] (3.626 ms) : 0, 3626
Flare Poller [candidate] (3.596 ms) : 0, 3596
IAST [baseline] (23.823 ms) : 0, 23823
IAST [candidate] (23.916 ms) : 0, 23916
section iast
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (796.097 ms) : 0, 796097
BytebuddyAgent [candidate] (804.742 ms) : 0, 804742
AgentMeter [baseline] (11.322 ms) : 0, 11322
AgentMeter [candidate] (11.617 ms) : 0, 11617
GlobalTracer [baseline] (247.24 ms) : 0, 247240
GlobalTracer [candidate] (249.002 ms) : 0, 249002
AppSec [baseline] (26.313 ms) : 0, 26313
AppSec [candidate] (26.749 ms) : 0, 26749
Debugger [baseline] (65.694 ms) : 0, 65694
Debugger [candidate] (65.628 ms) : 0, 65628
Remote Config [baseline] (540.832 µs) : 0, 541
Remote Config [candidate] (533.479 µs) : 0, 533
Telemetry [baseline] (13.596 ms) : 0, 13596
Telemetry [candidate] (13.441 ms) : 0, 13441
Flare Poller [baseline] (4.316 ms) : 0, 4316
Flare Poller [candidate] (4.333 ms) : 0, 4333
IAST [baseline] (25.042 ms) : 0, 25042
IAST [candidate] (25.411 ms) : 0, 25411
section profiling
ProfilingAgent [baseline] (93.55 ms) : 0, 93550
ProfilingAgent [candidate] (93.65 ms) : 0, 93650
crashtracking [baseline] (1.162 ms) : 0, 1162
crashtracking [candidate] (1.163 ms) : 0, 1163
BytebuddyAgent [baseline] (680.558 ms) : 0, 680558
BytebuddyAgent [candidate] (680.862 ms) : 0, 680862
AgentMeter [baseline] (8.621 ms) : 0, 8621
AgentMeter [candidate] (8.597 ms) : 0, 8597
GlobalTracer [baseline] (215.249 ms) : 0, 215249
GlobalTracer [candidate] (214.979 ms) : 0, 214979
AppSec [baseline] (31.831 ms) : 0, 31831
AppSec [candidate] (31.832 ms) : 0, 31832
Debugger [baseline] (65.298 ms) : 0, 65298
Debugger [candidate] (64.784 ms) : 0, 64784
Remote Config [baseline] (597.085 µs) : 0, 597
Remote Config [candidate] (575.03 µs) : 0, 575
Telemetry [baseline] (8.147 ms) : 0, 8147
Telemetry [candidate] (8.219 ms) : 0, 8219
Flare Poller [baseline] (3.441 ms) : 0, 3441
Flare Poller [candidate] (3.496 ms) : 0, 3496
Profiling [baseline] (94.117 ms) : 0, 94117
Profiling [candidate] (94.215 ms) : 0, 94215
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1064213
Total [baseline] (8.897 s) : 0, 8897343
Agent [candidate] (1.058 s) : 0, 1057828
Total [candidate] (8.834 s) : 0, 8834070
section iast
Agent [baseline] (1.227 s) : 0, 1226706
Total [baseline] (9.536 s) : 0, 9535795
Agent [candidate] (1.225 s) : 0, 1225383
Total [candidate] (9.54 s) : 0, 9540011
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent iast 1.227 s 162.493 ms (15.3%)
Total tracing 8.897 s -
Total iast 9.536 s 638.451 ms (7.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.058 s -
Agent iast 1.225 s 167.555 ms (15.8%)
Total tracing 8.834 s -
Total iast 9.54 s 705.941 ms (8.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.199 ms) : 0, 1199
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (631.803 ms) : 0, 631803
BytebuddyAgent [candidate] (627.248 ms) : 0, 627248
AgentMeter [baseline] (29.233 ms) : 0, 29233
AgentMeter [candidate] (29.146 ms) : 0, 29146
GlobalTracer [baseline] (258.391 ms) : 0, 258391
GlobalTracer [candidate] (256.864 ms) : 0, 256864
AppSec [baseline] (31.788 ms) : 0, 31788
AppSec [candidate] (31.55 ms) : 0, 31550
Debugger [baseline] (59.205 ms) : 0, 59205
Debugger [candidate] (58.741 ms) : 0, 58741
Remote Config [baseline] (600.22 µs) : 0, 600
Remote Config [candidate] (601.331 µs) : 0, 601
Telemetry [baseline] (8.712 ms) : 0, 8712
Telemetry [candidate] (8.604 ms) : 0, 8604
Flare Poller [baseline] (7.239 ms) : 0, 7239
Flare Poller [candidate] (7.917 ms) : 0, 7917
section iast
crashtracking [baseline] (1.205 ms) : 0, 1205
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (796.107 ms) : 0, 796107
BytebuddyAgent [candidate] (795.894 ms) : 0, 795894
AgentMeter [baseline] (11.319 ms) : 0, 11319
AgentMeter [candidate] (11.324 ms) : 0, 11324
GlobalTracer [baseline] (246.95 ms) : 0, 246950
GlobalTracer [candidate] (247.041 ms) : 0, 247041
IAST [baseline] (25.129 ms) : 0, 25129
IAST [candidate] (25.028 ms) : 0, 25028
AppSec [baseline] (26.388 ms) : 0, 26388
AppSec [candidate] (26.331 ms) : 0, 26331
Debugger [baseline] (63.014 ms) : 0, 63014
Debugger [candidate] (62.36 ms) : 0, 62360
Remote Config [baseline] (527.177 µs) : 0, 527
Remote Config [candidate] (523.289 µs) : 0, 523
Telemetry [baseline] (14.886 ms) : 0, 14886
Telemetry [candidate] (14.777 ms) : 0, 14777
Flare Poller [baseline] (4.975 ms) : 0, 4975
Flare Poller [candidate] (4.876 ms) : 0, 4876
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master brian.marks/fix-pekko-http-async-flaky-test-v2
git_commit_date 1773169596 1773757680
git_commit_sha cc12228 ede5582
release_version 1.61.0-SNAPSHOT~cc122288e5 1.61.0-SNAPSHOT~ede558242d
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1773963371 1773963371
ci_job_id 1523998675 1523998675
ci_pipeline_id 103634483 103634483
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-cmeg92ad 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-cmeg92ad 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 19 metrics, 17 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.18 ms) : 1168, 1191
.   : milestone, 1180,
iast (3.217 ms) : 3172, 3262
.   : milestone, 3217,
iast_FULL (6.264 ms) : 6199, 6329
.   : milestone, 6264,
iast_GLOBAL (3.508 ms) : 3459, 3557
.   : milestone, 3508,
profiling (2.217 ms) : 2198, 2237
.   : milestone, 2217,
tracing (1.814 ms) : 1799, 1830
.   : milestone, 1814,
section candidate
no_agent (1.173 ms) : 1162, 1185
.   : milestone, 1173,
iast (3.144 ms) : 3103, 3186
.   : milestone, 3144,
iast_FULL (6.025 ms) : 5964, 6085
.   : milestone, 6025,
iast_GLOBAL (3.612 ms) : 3560, 3663
.   : milestone, 3612,
profiling (2.048 ms) : 2030, 2066
.   : milestone, 2048,
tracing (1.8 ms) : 1785, 1815
.   : milestone, 1800,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.18 ms [1.168 ms, 1.191 ms] -
iast 3.217 ms [3.172 ms, 3.262 ms] 2.037 ms (172.7%)
iast_FULL 6.264 ms [6.199 ms, 6.329 ms] 5.084 ms (431.0%)
iast_GLOBAL 3.508 ms [3.459 ms, 3.557 ms] 2.328 ms (197.4%)
profiling 2.217 ms [2.198 ms, 2.237 ms] 1.038 ms (88.0%)
tracing 1.814 ms [1.799 ms, 1.83 ms] 634.432 µs (53.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.173 ms [1.162 ms, 1.185 ms] -
iast 3.144 ms [3.103 ms, 3.186 ms] 1.971 ms (168.0%)
iast_FULL 6.025 ms [5.964 ms, 6.085 ms] 4.851 ms (413.4%)
iast_GLOBAL 3.612 ms [3.56 ms, 3.663 ms] 2.438 ms (207.8%)
profiling 2.048 ms [2.03 ms, 2.066 ms] 874.379 µs (74.5%)
tracing 1.8 ms [1.785 ms, 1.815 ms] 626.853 µs (53.4%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.337 ms) : 18146, 18528
.   : milestone, 18337,
appsec (18.582 ms) : 18395, 18769
.   : milestone, 18582,
code_origins (17.906 ms) : 17727, 18085
.   : milestone, 17906,
iast (17.974 ms) : 17794, 18154
.   : milestone, 17974,
profiling (19.541 ms) : 19340, 19743
.   : milestone, 19541,
tracing (18.194 ms) : 18009, 18379
.   : milestone, 18194,
section candidate
no_agent (18.61 ms) : 18418, 18802
.   : milestone, 18610,
appsec (18.641 ms) : 18450, 18831
.   : milestone, 18641,
code_origins (17.946 ms) : 17763, 18129
.   : milestone, 17946,
iast (17.728 ms) : 17550, 17906
.   : milestone, 17728,
profiling (18.75 ms) : 18563, 18936
.   : milestone, 18750,
tracing (18.289 ms) : 18104, 18474
.   : milestone, 18289,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.337 ms [18.146 ms, 18.528 ms] -
appsec 18.582 ms [18.395 ms, 18.769 ms] 244.825 µs (1.3%)
code_origins 17.906 ms [17.727 ms, 18.085 ms] -430.851 µs (-2.3%)
iast 17.974 ms [17.794 ms, 18.154 ms] -362.99 µs (-2.0%)
profiling 19.541 ms [19.34 ms, 19.743 ms] 1.204 ms (6.6%)
tracing 18.194 ms [18.009 ms, 18.379 ms] -142.924 µs (-0.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.61 ms [18.418 ms, 18.802 ms] -
appsec 18.641 ms [18.45 ms, 18.831 ms] 30.351 µs (0.2%)
code_origins 17.946 ms [17.763 ms, 18.129 ms] -664.003 µs (-3.6%)
iast 17.728 ms [17.55 ms, 17.906 ms] -882.242 µs (-4.7%)
profiling 18.75 ms [18.563 ms, 18.936 ms] 139.544 µs (0.7%)
tracing 18.289 ms [18.104 ms, 18.474 ms] -321.012 µs (-1.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master brian.marks/fix-pekko-http-async-flaky-test-v2
git_commit_date 1773169596 1773757680
git_commit_sha cc12228 ede5582
release_version 1.61.0-SNAPSHOT~cc122288e5 1.61.0-SNAPSHOT~ede558242d
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1773963055 1773963055
ci_job_id 1523998676 1523998676
ci_pipeline_id 103634483 103634483
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-28fv915a 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-28fv915a 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.473 ms) : 1462, 1485
.   : milestone, 1473,
appsec (3.825 ms) : 3602, 4047
.   : milestone, 3825,
iast (2.258 ms) : 2189, 2327
.   : milestone, 2258,
iast_GLOBAL (2.308 ms) : 2239, 2377
.   : milestone, 2308,
profiling (2.074 ms) : 2020, 2128
.   : milestone, 2074,
tracing (2.086 ms) : 2032, 2139
.   : milestone, 2086,
section candidate
no_agent (1.478 ms) : 1466, 1489
.   : milestone, 1478,
appsec (3.825 ms) : 3603, 4047
.   : milestone, 3825,
iast (2.26 ms) : 2192, 2329
.   : milestone, 2260,
iast_GLOBAL (2.307 ms) : 2237, 2376
.   : milestone, 2307,
profiling (2.098 ms) : 2042, 2154
.   : milestone, 2098,
tracing (2.062 ms) : 2009, 2116
.   : milestone, 2062,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.473 ms [1.462 ms, 1.485 ms] -
appsec 3.825 ms [3.602 ms, 4.047 ms] 2.352 ms (159.6%)
iast 2.258 ms [2.189 ms, 2.327 ms] 784.613 µs (53.3%)
iast_GLOBAL 2.308 ms [2.239 ms, 2.377 ms] 834.816 µs (56.7%)
profiling 2.074 ms [2.02 ms, 2.128 ms] 600.965 µs (40.8%)
tracing 2.086 ms [2.032 ms, 2.139 ms] 612.479 µs (41.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.478 ms [1.466 ms, 1.489 ms] -
appsec 3.825 ms [3.603 ms, 4.047 ms] 2.347 ms (158.8%)
iast 2.26 ms [2.192 ms, 2.329 ms] 782.648 µs (53.0%)
iast_GLOBAL 2.307 ms [2.237 ms, 2.376 ms] 828.928 µs (56.1%)
profiling 2.098 ms [2.042 ms, 2.154 ms] 620.575 µs (42.0%)
tracing 2.062 ms [2.009 ms, 2.116 ms] 584.52 µs (39.6%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~ede558242d, baseline=1.61.0-SNAPSHOT~cc122288e5
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.474 s) : 15474000, 15474000
.   : milestone, 15474000,
appsec (14.875 s) : 14875000, 14875000
.   : milestone, 14875000,
iast (18.139 s) : 18139000, 18139000
.   : milestone, 18139000,
iast_GLOBAL (17.839 s) : 17839000, 17839000
.   : milestone, 17839000,
profiling (14.703 s) : 14703000, 14703000
.   : milestone, 14703000,
tracing (15.111 s) : 15111000, 15111000
.   : milestone, 15111000,
section candidate
no_agent (15.465 s) : 15465000, 15465000
.   : milestone, 15465000,
appsec (15.065 s) : 15065000, 15065000
.   : milestone, 15065000,
iast (18.042 s) : 18042000, 18042000
.   : milestone, 18042000,
iast_GLOBAL (17.789 s) : 17789000, 17789000
.   : milestone, 17789000,
profiling (14.91 s) : 14910000, 14910000
.   : milestone, 14910000,
tracing (15.032 s) : 15032000, 15032000
.   : milestone, 15032000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.474 s [15.474 s, 15.474 s] -
appsec 14.875 s [14.875 s, 14.875 s] -599.0 ms (-3.9%)
iast 18.139 s [18.139 s, 18.139 s] 2.665 s (17.2%)
iast_GLOBAL 17.839 s [17.839 s, 17.839 s] 2.365 s (15.3%)
profiling 14.703 s [14.703 s, 14.703 s] -771.0 ms (-5.0%)
tracing 15.111 s [15.111 s, 15.111 s] -363.0 ms (-2.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.465 s [15.465 s, 15.465 s] -
appsec 15.065 s [15.065 s, 15.065 s] -400.0 ms (-2.6%)
iast 18.042 s [18.042 s, 18.042 s] 2.577 s (16.7%)
iast_GLOBAL 17.789 s [17.789 s, 17.789 s] 2.324 s (15.0%)
profiling 14.91 s [14.91 s, 14.91 s] -555.0 ms (-3.6%)
tracing 15.032 s [15.032 s, 15.032 s] -433.0 ms (-2.8%)

@bm1549 bm1549 marked this pull request as ready for review March 10, 2026 21:07
@bm1549 bm1549 requested review from a team as code owners March 10, 2026 21:07
bm1549 and others added 2 commits March 16, 2026 22:52
@bm1549 @claude
Add artificial delay for deterministic race condition reproduction
d6f5c4e 
Add Thread.sleep(50) inside the async handler's Future to widen the race
window between Future continuation cleanup and span completion. Without
the .recover fix, this delay causes the failed-Future trace-drop to
trigger reliably. With .recover, the delay is harmless because the
Future always succeeds.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bm1549 @claude
Remove non-functional Thread.sleep(50) from async handler
ede5582 
The sleep doesn't reproduce the flake — it delays the Future body execution,
not the gap between span finish and Scala continuation cleanup. The race
window is nanoseconds and only manifests under extreme CI load (>20s OS
preemption). The .recover fix eliminates the race by design: converting the
failed Future to a successful one ensures the span is finished deterministically
before the HTTP response is sent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bm1549 bm1549 closed this Apr 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez manuel-alvarez-alvarez is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

tag: ai generated Largely based on code generated by an AI or LLM tag: flaky test Flaky tests tag: no release notes Changes to exclude from release notes type: bug Bug report and fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bm1549