Update AgentTestRunner to use JUnit5 - Continuation pt 2 by sarahchen6 · Pull Request #9431 · DataDog/dd-trace-java

sarahchen6 · 2025-08-28T02:07:12Z

What Does This Do

This PR builds on #9264 to address the failing instrumentation tests.

Motivation

Update AgentTestRunner to use JUnit 5, necessary to support Java 25 and test JUnit 6.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

datadog-official · 2025-08-28T02:42:03Z

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 59.60% (+0.01%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 9db45c5 | Docs | Was this helpful? Give us feedback!}

pr-commenter · 2025-08-28T02:55:53Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/experiment-2
git_commit_date	1756404933	1756409984
git_commit_sha	`d07fee9`	`9db45c5`
release_version	1.53.0-SNAPSHOT~d07fee98fb	1.53.0-SNAPSHOT~9db45c5209

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1756411947	1756411947
ci_job_id	1104331454	1104331454
ci_pipeline_id	75023878	75023878
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-g5zn9kvj 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-g5zn9kvj 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 16 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.047 s) : 0, 1046972
Total [baseline] (8.634 s) : 0, 8633849
Agent [candidate] (1.044 s) : 0, 1044453
Total [candidate] (8.59 s) : 0, 8590440
section iast
Agent [baseline] (1.181 s) : 0, 1180553
Total [baseline] (9.334 s) : 0, 9334190
Agent [candidate] (1.193 s) : 0, 1193463
Total [candidate] (9.326 s) : 0, 9325702

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	iast	1.181 s	133.58 ms (12.8%)
Total	tracing	8.634 s	-
Total	iast	9.334 s	700.342 ms (8.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.044 s	-
Agent	iast	1.193 s	149.01 ms (14.3%)
Total	tracing	8.59 s	-
Total	iast	9.326 s	735.263 ms (8.6%)

gantt
    title insecure-bank - break down per module: candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.446 ms) : 0, 1446
crashtracking [candidate] (1.448 ms) : 0, 1448
BytebuddyAgent [baseline] (732.173 ms) : 0, 732173
BytebuddyAgent [candidate] (731.03 ms) : 0, 731030
GlobalTracer [baseline] (241.941 ms) : 0, 241941
GlobalTracer [candidate] (242.043 ms) : 0, 242043
AppSec [baseline] (29.992 ms) : 0, 29992
AppSec [candidate] (30.108 ms) : 0, 30108
Debugger [baseline] (6.047 ms) : 0, 6047
Debugger [candidate] (6.038 ms) : 0, 6038
Remote Config [baseline] (659.042 µs) : 0, 659
Remote Config [candidate] (659.703 µs) : 0, 660
Telemetry [baseline] (13.768 ms) : 0, 13768
Telemetry [candidate] (12.23 ms) : 0, 12230
section iast
crashtracking [baseline] (1.468 ms) : 0, 1468
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (851.744 ms) : 0, 851744
BytebuddyAgent [candidate] (863.436 ms) : 0, 863436
GlobalTracer [baseline] (234.331 ms) : 0, 234331
GlobalTracer [candidate] (235.849 ms) : 0, 235849
IAST [baseline] (27.735 ms) : 0, 27735
IAST [candidate] (28.573 ms) : 0, 28573
AppSec [baseline] (27.824 ms) : 0, 27824
AppSec [candidate] (28.371 ms) : 0, 28371
Debugger [baseline] (6.664 ms) : 0, 6664
Debugger [candidate] (5.72 ms) : 0, 5720
Remote Config [baseline] (603.104 µs) : 0, 603
Remote Config [candidate] (611.102 µs) : 0, 611
Telemetry [baseline] (9.138 ms) : 0, 9138
Telemetry [candidate] (8.285 ms) : 0, 8285

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.054 s) : 0, 1053711
Total [baseline] (10.75 s) : 0, 10749586
Agent [candidate] (1.053 s) : 0, 1052538
Total [candidate] (10.774 s) : 0, 10773997
section appsec
Agent [baseline] (1.222 s) : 0, 1222111
Total [baseline] (10.816 s) : 0, 10815938
Agent [candidate] (1.224 s) : 0, 1224283
Total [candidate] (10.854 s) : 0, 10854194
section iast
Agent [baseline] (1.193 s) : 0, 1192547
Total [baseline] (11.06 s) : 0, 11059761
Agent [candidate] (1.198 s) : 0, 1197589
Total [candidate] (10.886 s) : 0, 10886078
section profiling
Agent [baseline] (1.2 s) : 0, 1200136
Total [baseline] (10.942 s) : 0, 10941809
Agent [candidate] (1.193 s) : 0, 1193100
Total [candidate] (10.882 s) : 0, 10882314

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	appsec	1.222 s	168.4 ms (16.0%)
Agent	iast	1.193 s	138.836 ms (13.2%)
Agent	profiling	1.2 s	146.425 ms (13.9%)
Total	tracing	10.75 s	-
Total	appsec	10.816 s	66.352 ms (0.6%)
Total	iast	11.06 s	310.175 ms (2.9%)
Total	profiling	10.942 s	192.223 ms (1.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	appsec	1.224 s	171.745 ms (16.3%)
Agent	iast	1.198 s	145.051 ms (13.8%)
Agent	profiling	1.193 s	140.562 ms (13.4%)
Total	tracing	10.774 s	-
Total	appsec	10.854 s	80.198 ms (0.7%)
Total	iast	10.886 s	112.082 ms (1.0%)
Total	profiling	10.882 s	108.317 ms (1.0%)

gantt
    title petclinic - break down per module: candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.47 ms) : 0, 1470
crashtracking [candidate] (1.461 ms) : 0, 1461
BytebuddyAgent [baseline] (736.63 ms) : 0, 736630
BytebuddyAgent [candidate] (736.436 ms) : 0, 736436
GlobalTracer [baseline] (243.597 ms) : 0, 243597
GlobalTracer [candidate] (243.896 ms) : 0, 243896
AppSec [baseline] (30.268 ms) : 0, 30268
AppSec [candidate] (30.47 ms) : 0, 30470
Debugger [baseline] (6.074 ms) : 0, 6074
Debugger [candidate] (6.064 ms) : 0, 6064
Remote Config [baseline] (675.065 µs) : 0, 675
Remote Config [candidate] (676.237 µs) : 0, 676
Telemetry [baseline] (13.837 ms) : 0, 13837
Telemetry [candidate] (12.407 ms) : 0, 12407
section appsec
crashtracking [baseline] (1.456 ms) : 0, 1456
crashtracking [candidate] (1.453 ms) : 0, 1453
BytebuddyAgent [baseline] (754.715 ms) : 0, 754715
BytebuddyAgent [candidate] (756.209 ms) : 0, 756209
GlobalTracer [baseline] (234.691 ms) : 0, 234691
GlobalTracer [candidate] (235.611 ms) : 0, 235611
IAST [baseline] (23.565 ms) : 0, 23565
IAST [candidate] (23.615 ms) : 0, 23615
AppSec [baseline] (170.017 ms) : 0, 170017
AppSec [candidate] (168.313 ms) : 0, 168313
Debugger [baseline] (7.407 ms) : 0, 7407
Debugger [candidate] (8.801 ms) : 0, 8801
Remote Config [baseline] (655.281 µs) : 0, 655
Remote Config [candidate] (653.024 µs) : 0, 653
Telemetry [baseline] (8.455 ms) : 0, 8455
Telemetry [candidate] (8.432 ms) : 0, 8432
section iast
crashtracking [baseline] (1.482 ms) : 0, 1482
crashtracking [candidate] (1.465 ms) : 0, 1465
BytebuddyAgent [baseline] (860.774 ms) : 0, 860774
BytebuddyAgent [candidate] (865.257 ms) : 0, 865257
GlobalTracer [baseline] (234.515 ms) : 0, 234515
GlobalTracer [candidate] (235.779 ms) : 0, 235779
IAST [baseline] (30.145 ms) : 0, 30145
IAST [candidate] (28.91 ms) : 0, 28910
AppSec [baseline] (26.547 ms) : 0, 26547
AppSec [candidate] (27.495 ms) : 0, 27495
Debugger [baseline] (8.58 ms) : 0, 8580
Debugger [candidate] (8.378 ms) : 0, 8378
Remote Config [baseline] (623.02 µs) : 0, 623
Remote Config [candidate] (611.688 µs) : 0, 612
Telemetry [baseline] (8.527 ms) : 0, 8527
Telemetry [candidate] (8.407 ms) : 0, 8407
section profiling
crashtracking [baseline] (1.445 ms) : 0, 1445
crashtracking [candidate] (1.441 ms) : 0, 1441
BytebuddyAgent [baseline] (763.275 ms) : 0, 763275
BytebuddyAgent [candidate] (759.382 ms) : 0, 759382
GlobalTracer [baseline] (223.0 ms) : 0, 223000
GlobalTracer [candidate] (221.677 ms) : 0, 221677
AppSec [baseline] (30.116 ms) : 0, 30116
AppSec [candidate] (29.947 ms) : 0, 29947
Debugger [baseline] (6.294 ms) : 0, 6294
Debugger [candidate] (7.027 ms) : 0, 7027
Remote Config [baseline] (710.909 µs) : 0, 711
Remote Config [candidate] (708.843 µs) : 0, 709
Telemetry [baseline] (16.454 ms) : 0, 16454
Telemetry [candidate] (15.65 ms) : 0, 15650
ProfilingAgent [baseline] (109.086 ms) : 0, 109086
ProfilingAgent [candidate] (107.68 ms) : 0, 107680
Profiling [baseline] (109.731 ms) : 0, 109731
Profiling [candidate] (108.345 ms) : 0, 108345

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/experiment-2
git_commit_date	1756404933	1756409984
git_commit_sha	`d07fee9`	`9db45c5`
release_version	1.53.0-SNAPSHOT~d07fee98fb	1.53.0-SNAPSHOT~9db45c5209

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1756411710	1756411710
ci_job_id	1104331455	1104331455
ci_pipeline_id	75023878	75023878
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-2h7alhmb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-2h7alhmb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:no_agent:high_load	better [-2.402ms; -1.755ms] or [-6.313%; -4.612%]	unstable [-1.574op/s; +15.749op/s] or [-1.282%; +12.821%]	35.971ms	129.925op/s	38.050ms	122.838op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb
    dateFormat X
    axisFormat %s
section baseline
no_agent (38.05 ms) : 37735, 38364
.   : milestone, 38050,
appsec (46.947 ms) : 46546, 47349
.   : milestone, 46947,
code_origins (45.453 ms) : 45070, 45835
.   : milestone, 45453,
iast (46.238 ms) : 45829, 46647
.   : milestone, 46238,
profiling (48.318 ms) : 47905, 48731
.   : milestone, 48318,
tracing (43.424 ms) : 43047, 43801
.   : milestone, 43424,
section candidate
no_agent (35.971 ms) : 35685, 36258
.   : milestone, 35971,
appsec (47.708 ms) : 47281, 48135
.   : milestone, 47708,
code_origins (45.75 ms) : 45352, 46147
.   : milestone, 45750,
iast (45.131 ms) : 44746, 45515
.   : milestone, 45131,
profiling (49.515 ms) : 49047, 49983
.   : milestone, 49515,
tracing (43.095 ms) : 42734, 43455
.   : milestone, 43095,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	38.05 ms [37.735 ms, 38.364 ms]	-
appsec	46.947 ms [46.546 ms, 47.349 ms]	8.898 ms (23.4%)
code_origins	45.453 ms [45.07 ms, 45.835 ms]	7.403 ms (19.5%)
iast	46.238 ms [45.829 ms, 46.647 ms]	8.188 ms (21.5%)
profiling	48.318 ms [47.905 ms, 48.731 ms]	10.269 ms (27.0%)
tracing	43.424 ms [43.047 ms, 43.801 ms]	5.375 ms (14.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	35.971 ms [35.685 ms, 36.258 ms]	-
appsec	47.708 ms [47.281 ms, 48.135 ms]	11.736 ms (32.6%)
code_origins	45.75 ms [45.352 ms, 46.147 ms]	9.779 ms (27.2%)
iast	45.131 ms [44.746 ms, 45.515 ms]	9.159 ms (25.5%)
profiling	49.515 ms [49.047 ms, 49.983 ms]	13.544 ms (37.7%)
tracing	43.095 ms [42.734 ms, 43.455 ms]	7.123 ms (19.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.431 ms) : 4381, 4480
.   : milestone, 4431,
iast (9.466 ms) : 9300, 9633
.   : milestone, 9466,
iast_FULL (13.633 ms) : 13361, 13904
.   : milestone, 13633,
iast_GLOBAL (10.369 ms) : 10185, 10553
.   : milestone, 10369,
profiling (9.003 ms) : 8863, 9143
.   : milestone, 9003,
tracing (7.454 ms) : 7351, 7558
.   : milestone, 7454,
section candidate
no_agent (4.387 ms) : 4336, 4438
.   : milestone, 4387,
iast (9.507 ms) : 9342, 9672
.   : milestone, 9507,
iast_FULL (14.003 ms) : 13723, 14282
.   : milestone, 14003,
iast_GLOBAL (10.166 ms) : 9988, 10344
.   : milestone, 10166,
profiling (8.789 ms) : 8637, 8941
.   : milestone, 8789,
tracing (7.635 ms) : 7522, 7749
.   : milestone, 7635,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.431 ms [4.381 ms, 4.48 ms]	-
iast	9.466 ms [9.3 ms, 9.633 ms]	5.036 ms (113.7%)
iast_FULL	13.633 ms [13.361 ms, 13.904 ms]	9.202 ms (207.7%)
iast_GLOBAL	10.369 ms [10.185 ms, 10.553 ms]	5.938 ms (134.0%)
profiling	9.003 ms [8.863 ms, 9.143 ms]	4.572 ms (103.2%)
tracing	7.454 ms [7.351 ms, 7.558 ms]	3.024 ms (68.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.387 ms [4.336 ms, 4.438 ms]	-
iast	9.507 ms [9.342 ms, 9.672 ms]	5.12 ms (116.7%)
iast_FULL	14.003 ms [13.723 ms, 14.282 ms]	9.616 ms (219.2%)
iast_GLOBAL	10.166 ms [9.988 ms, 10.344 ms]	5.779 ms (131.7%)
profiling	8.789 ms [8.637 ms, 8.941 ms]	4.402 ms (100.3%)
tracing	7.635 ms [7.522 ms, 7.749 ms]	3.248 ms (74.0%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/experiment-2
git_commit_date	1756404933	1756409984
git_commit_sha	`d07fee9`	`9db45c5`
release_version	1.53.0-SNAPSHOT~d07fee98fb	1.53.0-SNAPSHOT~9db45c5209

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1756412248	1756412248
ci_job_id	1104331456	1104331456
ci_pipeline_id	75023878	75023878
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-i6rj11g1 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-i6rj11g1 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1464, 1487
.   : milestone, 1476,
appsec (3.672 ms) : 3456, 3889
.   : milestone, 3672,
iast (2.209 ms) : 2146, 2271
.   : milestone, 2209,
iast_GLOBAL (2.244 ms) : 2181, 2307
.   : milestone, 2244,
profiling (2.041 ms) : 1990, 2091
.   : milestone, 2041,
tracing (2.018 ms) : 1970, 2067
.   : milestone, 2018,
section candidate
no_agent (1.477 ms) : 1466, 1489
.   : milestone, 1477,
appsec (3.657 ms) : 3440, 3874
.   : milestone, 3657,
iast (2.205 ms) : 2142, 2268
.   : milestone, 2205,
iast_GLOBAL (2.24 ms) : 2177, 2303
.   : milestone, 2240,
profiling (2.027 ms) : 1977, 2077
.   : milestone, 2027,
tracing (2.018 ms) : 1970, 2067
.   : milestone, 2018,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.464 ms, 1.487 ms]	-
appsec	3.672 ms [3.456 ms, 3.889 ms]	2.197 ms (148.9%)
iast	2.209 ms [2.146 ms, 2.271 ms]	732.989 µs (49.7%)
iast_GLOBAL	2.244 ms [2.181 ms, 2.307 ms]	768.155 µs (52.1%)
profiling	2.041 ms [1.99 ms, 2.091 ms]	565.031 µs (38.3%)
tracing	2.018 ms [1.97 ms, 2.067 ms]	542.834 µs (36.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.489 ms]	-
appsec	3.657 ms [3.44 ms, 3.874 ms]	2.18 ms (147.5%)
iast	2.205 ms [2.142 ms, 2.268 ms]	727.542 µs (49.2%)
iast_GLOBAL	2.24 ms [2.177 ms, 2.303 ms]	762.696 µs (51.6%)
profiling	2.027 ms [1.977 ms, 2.077 ms]	549.83 µs (37.2%)
tracing	2.018 ms [1.97 ms, 2.067 ms]	541.17 µs (36.6%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~9db45c5209, baseline=1.53.0-SNAPSHOT~d07fee98fb
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.71 s) : 14710000, 14710000
.   : milestone, 14710000,
appsec (14.935 s) : 14935000, 14935000
.   : milestone, 14935000,
iast (18.147 s) : 18147000, 18147000
.   : milestone, 18147000,
iast_GLOBAL (18.047 s) : 18047000, 18047000
.   : milestone, 18047000,
profiling (15.991 s) : 15991000, 15991000
.   : milestone, 15991000,
tracing (14.705 s) : 14705000, 14705000
.   : milestone, 14705000,
section candidate
no_agent (15.242 s) : 15242000, 15242000
.   : milestone, 15242000,
appsec (14.568 s) : 14568000, 14568000
.   : milestone, 14568000,
iast (18.778 s) : 18778000, 18778000
.   : milestone, 18778000,
iast_GLOBAL (17.828 s) : 17828000, 17828000
.   : milestone, 17828000,
profiling (15.173 s) : 15173000, 15173000
.   : milestone, 15173000,
tracing (14.845 s) : 14845000, 14845000
.   : milestone, 14845000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.71 s [14.71 s, 14.71 s]	-
appsec	14.935 s [14.935 s, 14.935 s]	225.0 ms (1.5%)
iast	18.147 s [18.147 s, 18.147 s]	3.437 s (23.4%)
iast_GLOBAL	18.047 s [18.047 s, 18.047 s]	3.337 s (22.7%)
profiling	15.991 s [15.991 s, 15.991 s]	1.281 s (8.7%)
tracing	14.705 s [14.705 s, 14.705 s]	-5.0 ms (-0.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.242 s [15.242 s, 15.242 s]	-
appsec	14.568 s [14.568 s, 14.568 s]	-674.0 ms (-4.4%)
iast	18.778 s [18.778 s, 18.778 s]	3.536 s (23.2%)
iast_GLOBAL	17.828 s [17.828 s, 17.828 s]	2.586 s (17.0%)
profiling	15.173 s [15.173 s, 15.173 s]	-69.0 ms (-0.5%)
tracing	14.845 s [14.845 s, 14.845 s]	-397.0 ms (-2.6%)

pr-commenter · 2025-08-28T02:57:09Z

Kafka / producer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	nikita-tkachenko/junit-6	sarahchen6/experiment-2
git_commit_date	1754670324	1756409984
git_commit_sha	`7c7286a`	`9db45c5`

See matching parameters

	Baseline	Candidate
ci_job_date	1756411256	1756411256
ci_job_id	1104331459	1104331459
ci_pipeline_id	75023878	75023878
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.25	11.0.25
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.25+9-post-Ubuntu-1ubuntu122.04	11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce	unsure [-7423.632op/s; -857.325op/s] or [-4.540%; -0.524%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce	same

pr-commenter · 2025-08-28T03:09:59Z

Kafka / consumer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	nikita-tkachenko/junit-6	sarahchen6/experiment-2
git_commit_date	1754670324	1756409984
git_commit_sha	`7c7286a`	`9db45c5`

See matching parameters

	Baseline	Candidate
ci_job_date	1756411300	1756411300
ci_job_id	1104331460	1104331460
ci_pipeline_id	75023878	75023878
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.25	11.0.25
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.25+9-post-Ubuntu-1ubuntu122.04	11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same

sarahchen6 · 2025-08-28T17:13:53Z

    final url = "${address}/iast/sources/form"
-    final body = new Request.Builder().add('name', 'value').build()
+    final body = new FormBody.Builder().add('name', 'value').build()
    final request = new Request.Builder().url(url).post(body).build()


add method needs to be FormBody. See FormBody.Builder vs Request.Builder.

sarahchen6 · 2025-08-28T17:16:14Z

    then:
-    1 * endCb.apply(_, _)
+    1 * endCb.apply(_, _) >> Flow.ResultFlow.empty()



Without >> Flow.ResultFlow.empty(), this was resulting in a NPE where flow.getAction() was called on null objects.

sarahchen6 · 2025-08-29T17:10:01Z

        }
        if (exception) {
-          errorTags(exception.class, exception.message)
+          // PlayWS classes throw different exception types for the same connection failures


I resolved these failures by loosening the expected exception types specifically for PlayWS classes. Is there a better way to address this?

sarahchen6 · 2025-09-02T14:30:49Z

Relevant changes included in #9445

sarahchen6 added 3 commits August 27, 2025 21:50

Fix Iast tests

3c5cb5d

Fix servlet/request-2 test

d9ec3cd

Fix instrumentation test

efd8ebc

sarahchen6 changed the base branch from master to nikita-tkachenko/junit-6 August 28, 2025 02:07

sarahchen6 added 3 commits August 27, 2025 22:19

Fix jetty test

6b729bb

Give more time for aws sqs, kafka, and vertx tests

5a6ec68

Run latestDepJava11Test on Java 11 only

2a02021

sarahchen6 force-pushed the sarahchen6/experiment-2 branch from 774f994 to 2a02021 Compare August 28, 2025 02:20

Suppress CIVis warning

aac4305

sarahchen6 added 2 commits August 28, 2025 10:24

Fix vertx-web tests

dccc4ea

Propagate karate test skip for Java 24+

23a0369

sarahchen6 commented Aug 28, 2025

View reviewed changes

sarahchen6 added 3 commits August 28, 2025 14:20

Fix play-ws tests

d015ca3

Address codenarc issue

c82fa6c

Loosen exception expectations for PlayWS only

9db45c5

sarahchen6 commented Aug 29, 2025

View reviewed changes

sarahchen6 mentioned this pull request Aug 29, 2025

Update AgentTestRunner to use JUnit5 - Continuation #9413

Closed

sarahchen6 closed this Sep 2, 2025

sarahchen6 deleted the sarahchen6/experiment-2 branch October 16, 2025 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update AgentTestRunner to use JUnit5 - Continuation pt 2#9431

Update AgentTestRunner to use JUnit5 - Continuation pt 2#9431
sarahchen6 wants to merge 12 commits intonikita-tkachenko/junit-6from
sarahchen6/experiment-2

sarahchen6 commented Aug 28, 2025 •

edited

Loading

Uh oh!

datadog-official Bot commented Aug 28, 2025 •

edited by datadog-datadog-prod-us1 Bot

Loading

Uh oh!

pr-commenter Bot commented Aug 28, 2025 •

edited

Loading

Uh oh!

pr-commenter Bot commented Aug 28, 2025 •

edited

Loading

Uh oh!

pr-commenter Bot commented Aug 28, 2025 •

edited

Loading

Uh oh!

sarahchen6 Aug 28, 2025

Uh oh!

sarahchen6 Aug 28, 2025 •

edited

Loading

Uh oh!

sarahchen6 Aug 29, 2025

Uh oh!

sarahchen6 commented Sep 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

sarahchen6 commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-official Bot commented Aug 28, 2025 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

pr-commenter Bot commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kafka / producer-benchmark

Parameters

Summary

Uh oh!

pr-commenter Bot commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Kafka / consumer-benchmark

Parameters

Summary

Uh oh!

sarahchen6 Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

sarahchen6 commented Sep 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

sarahchen6 commented Aug 28, 2025 •

edited

Loading

datadog-official Bot commented Aug 28, 2025 •

edited by datadog-datadog-prod-us1 Bot

Loading

pr-commenter Bot commented Aug 28, 2025 •

edited

Loading

pr-commenter Bot commented Aug 28, 2025 •

edited

Loading

pr-commenter Bot commented Aug 28, 2025 •

edited

Loading

sarahchen6 Aug 28, 2025 •

edited

Loading