Add throughput benchmark for petclinic load testing by bm1549 · Pull Request #11098 · DataDog/dd-trace-java

bm1549 · 2026-04-13T14:44:14Z

What Does This Do

Adds a new throughput benchmark under benchmark/throughput/petclinic/ that measures tracer throughput overhead under load at varying heap sizes using spring-petclinic 3.3.0-SNAPSHOT and JMeter.

Motivation

Automates the manual process of running petclinic at various heap sizes (256m → 64m) with and without the agent, measuring stabilized throughput, and comparing the delta.

The script:

Auto-downloads JMeter 5.6.3 and builds spring-petclinic 3.3.0-SNAPSHOT (pinned to commit 7034d17)
Iterates over configurable heap sizes (default: 256/192/128/96/80/64 MiB)
For each heap size: starts petclinic, runs JMeter (8 threads, GET /owners/3), captures throughput from the measurement window (discarding warmup)
Detects OOM (server crash during load)
Outputs a comparison table with throughput deltas and saves CSV + per-run JTL files

Example output from a local run (Zulu 17, Apple Silicon):

Heap      Baseline        Candidate       Delta         
------    -----------     -----------     -----------   
256m      17712 req/s     12549 req/s     -29.1%
192m      19649 req/s     13154 req/s     -33.1%
128m      19111 req/s     10769 req/s     -43.6%
96m       13575 req/s     9122 req/s      -32.8%
80m       12875 req/s     8451 req/s      -34.4%
64m       10258 req/s     6162 req/s      -39.9%

Differences from the original manual benchmark

Fixed duration vs run-until-stable: The original benchmark runs JMeter indefinitely until throughput stabilizes, then manually records the number. This script uses a fixed warmup + measurement window (default 30s + 60s). This is simpler to automate but may not fully capture stabilization, especially at lower heap sizes where GC behavior is more variable.
Same-machine load generation: The original runs JMeter separately from petclinic. This script runs both on the same machine, which means JMeter and petclinic compete for CPU/memory. This inflates overhead percentages compared to the original results.
No CPU pinning: The original uses taskset to pin server processes to specific cores. This script does not pin, so results are noisier due to scheduling jitter.
Two variants only: The original tests no-agent vs multiple tracer versions side-by-side. This script compares no-agent vs a single candidate agent jar. To compare multiple versions, run the script multiple times with --skip-baseline and different --agent paths.
H2 database: Both use the embedded H2 database (not an external Postgres), though an external DB would be more realistic.

Additional Notes

Supports --jfr flag for JFR profiling during measurement
Supports --skip-baseline for comparing multiple candidate builds without re-running baseline
Supports --agent-opts for testing with specific features disabled
tools/ and results/ directories are gitignored

tag: ai generated

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels
Avoid using close, fix, or linking keywords
Update the CODEOWNERS file on source file addition
Update public documentation — N/A (internal benchmark tooling)

🤖 Generated with Claude Code

Adds a new benchmark category under benchmark/memory/petclinic/ that measures tracer throughput overhead across decreasing heap sizes using spring-petclinic 3.3.0 and JMeter. Based on Doug's methodology for evaluating memory overhead of the Java tracer agent. The script auto-downloads JMeter and builds petclinic, iterates over configurable heap sizes (default: 256/192/128/96/80/64 MiB), runs each with and without the agent, and outputs a comparison table with throughput delta. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

dougqh · 2026-04-13T15:34:34Z

I think it is more important to see the difference of dd-java-agent with and without the change. Is that something we can add?

pr-commenter · 2026-04-13T15:36:56Z

Benchmarks

⚠️ Warning: Baseline build not found for merge-base commit. Comparing against the latest commit on master instead.

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	worktree-brian.marks+memory-benchmark
git_commit_date	1776101218	1776108463
git_commit_sha	`140eee9`	`f8fc1df`
release_version	1.62.0-SNAPSHOT~140eee9add	1.62.0-SNAPSHOT~f8fc1df1dd

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1776110184	1776110184
ci_job_id	1591250291	1591250291
ci_pipeline_id	107457101	107457101
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-zbu1ksmy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-zbu1ksmy 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1058565
Total [baseline] (8.841 s) : 0, 8841421
Agent [candidate] (1.063 s) : 0, 1062830
Total [candidate] (8.861 s) : 0, 8861391
section iast
Agent [baseline] (1.222 s) : 0, 1221718
Total [baseline] (9.587 s) : 0, 9587037
Agent [candidate] (1.222 s) : 0, 1221562
Total [candidate] (9.573 s) : 0, 9573381

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.059 s	-
Agent	iast	1.222 s	163.153 ms (15.4%)
Total	tracing	8.841 s	-
Total	iast	9.587 s	745.617 ms (8.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.063 s	-
Agent	iast	1.222 s	158.732 ms (14.9%)
Total	tracing	8.861 s	-
Total	iast	9.573 s	711.99 ms (8.0%)

gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.236 ms) : 0, 1236
crashtracking [candidate] (1.24 ms) : 0, 1240
BytebuddyAgent [baseline] (632.428 ms) : 0, 632428
BytebuddyAgent [candidate] (637.12 ms) : 0, 637120
AgentMeter [baseline] (29.45 ms) : 0, 29450
AgentMeter [candidate] (29.717 ms) : 0, 29717
GlobalTracer [baseline] (249.268 ms) : 0, 249268
GlobalTracer [candidate] (250.218 ms) : 0, 250218
AppSec [baseline] (32.187 ms) : 0, 32187
AppSec [candidate] (32.32 ms) : 0, 32320
Debugger [baseline] (59.334 ms) : 0, 59334
Debugger [candidate] (59.364 ms) : 0, 59364
Remote Config [baseline] (605.919 µs) : 0, 606
Remote Config [candidate] (956.552 µs) : 0, 957
Telemetry [baseline] (8.092 ms) : 0, 8092
Telemetry [candidate] (8.09 ms) : 0, 8090
Flare Poller [baseline] (9.811 ms) : 0, 9811
Flare Poller [candidate] (7.48 ms) : 0, 7480
section iast
crashtracking [baseline] (1.238 ms) : 0, 1238
crashtracking [candidate] (1.227 ms) : 0, 1227
BytebuddyAgent [baseline] (799.44 ms) : 0, 799440
BytebuddyAgent [candidate] (799.159 ms) : 0, 799159
AgentMeter [baseline] (11.365 ms) : 0, 11365
AgentMeter [candidate] (11.355 ms) : 0, 11355
GlobalTracer [baseline] (238.61 ms) : 0, 238610
GlobalTracer [candidate] (238.92 ms) : 0, 238920
AppSec [baseline] (30.187 ms) : 0, 30187
AppSec [candidate] (28.69 ms) : 0, 28690
Debugger [baseline] (61.955 ms) : 0, 61955
Debugger [candidate] (63.231 ms) : 0, 63231
Remote Config [baseline] (536.609 µs) : 0, 537
Remote Config [candidate] (1.136 ms) : 0, 1136
Telemetry [baseline] (12.602 ms) : 0, 12602
Telemetry [candidate] (12.226 ms) : 0, 12226
Flare Poller [baseline] (3.592 ms) : 0, 3592
Flare Poller [candidate] (3.391 ms) : 0, 3391
IAST [baseline] (25.788 ms) : 0, 25788
IAST [candidate] (25.904 ms) : 0, 25904

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.065 s) : 0, 1064885
Total [baseline] (11.048 s) : 0, 11048054
Agent [candidate] (1.054 s) : 0, 1053681
Total [candidate] (11.068 s) : 0, 11067791
section appsec
Agent [baseline] (1.257 s) : 0, 1257289
Total [baseline] (11.132 s) : 0, 11131814
Agent [candidate] (1.253 s) : 0, 1253073
Total [candidate] (11.105 s) : 0, 11104869
section iast
Agent [baseline] (1.224 s) : 0, 1224324
Total [baseline] (11.307 s) : 0, 11307190
Agent [candidate] (1.226 s) : 0, 1225748
Total [candidate] (11.338 s) : 0, 11338057
section profiling
Agent [baseline] (1.186 s) : 0, 1185927
Total [baseline] (11.006 s) : 0, 11005533
Agent [candidate] (1.186 s) : 0, 1185906
Total [candidate] (11.073 s) : 0, 11073065

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.065 s	-
Agent	appsec	1.257 s	192.404 ms (18.1%)
Agent	iast	1.224 s	159.439 ms (15.0%)
Agent	profiling	1.186 s	121.041 ms (11.4%)
Total	tracing	11.048 s	-
Total	appsec	11.132 s	83.761 ms (0.8%)
Total	iast	11.307 s	259.137 ms (2.3%)
Total	profiling	11.006 s	-42.521 ms (-0.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	appsec	1.253 s	199.392 ms (18.9%)
Agent	iast	1.226 s	172.067 ms (16.3%)
Agent	profiling	1.186 s	132.224 ms (12.5%)
Total	tracing	11.068 s	-
Total	appsec	11.105 s	37.078 ms (0.3%)
Total	iast	11.338 s	270.266 ms (2.4%)
Total	profiling	11.073 s	5.274 ms (0.0%)

gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.255 ms) : 0, 1255
crashtracking [candidate] (1.226 ms) : 0, 1226
BytebuddyAgent [baseline] (637.272 ms) : 0, 637272
BytebuddyAgent [candidate] (631.021 ms) : 0, 631021
AgentMeter [baseline] (29.659 ms) : 0, 29659
AgentMeter [candidate] (29.481 ms) : 0, 29481
GlobalTracer [baseline] (250.117 ms) : 0, 250117
GlobalTracer [candidate] (248.703 ms) : 0, 248703
AppSec [baseline] (32.199 ms) : 0, 32199
AppSec [candidate] (31.997 ms) : 0, 31997
Debugger [baseline] (60.371 ms) : 0, 60371
Debugger [candidate] (59.906 ms) : 0, 59906
Remote Config [baseline] (600.001 µs) : 0, 600
Remote Config [candidate] (591.612 µs) : 0, 592
Telemetry [baseline] (8.126 ms) : 0, 8126
Telemetry [candidate] (8.026 ms) : 0, 8026
Flare Poller [baseline] (8.994 ms) : 0, 8994
Flare Poller [candidate] (6.64 ms) : 0, 6640
section appsec
crashtracking [baseline] (1.226 ms) : 0, 1226
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (666.632 ms) : 0, 666632
BytebuddyAgent [candidate] (664.449 ms) : 0, 664449
AgentMeter [baseline] (12.193 ms) : 0, 12193
AgentMeter [candidate] (12.153 ms) : 0, 12153
GlobalTracer [baseline] (250.981 ms) : 0, 250981
GlobalTracer [candidate] (250.272 ms) : 0, 250272
AppSec [baseline] (185.751 ms) : 0, 185751
AppSec [candidate] (184.887 ms) : 0, 184887
Debugger [baseline] (66.249 ms) : 0, 66249
Debugger [candidate] (66.32 ms) : 0, 66320
Remote Config [baseline] (604.934 µs) : 0, 605
Remote Config [candidate] (592.358 µs) : 0, 592
Telemetry [baseline] (8.774 ms) : 0, 8774
Telemetry [candidate] (8.618 ms) : 0, 8618
Flare Poller [baseline] (3.641 ms) : 0, 3641
Flare Poller [candidate] (3.562 ms) : 0, 3562
IAST [baseline] (24.774 ms) : 0, 24774
IAST [candidate] (24.548 ms) : 0, 24548
section iast
crashtracking [baseline] (1.223 ms) : 0, 1223
crashtracking [candidate] (1.234 ms) : 0, 1234
BytebuddyAgent [baseline] (800.598 ms) : 0, 800598
BytebuddyAgent [candidate] (802.862 ms) : 0, 802862
AgentMeter [baseline] (11.424 ms) : 0, 11424
AgentMeter [candidate] (11.428 ms) : 0, 11428
GlobalTracer [baseline] (239.382 ms) : 0, 239382
GlobalTracer [candidate] (239.22 ms) : 0, 239220
AppSec [baseline] (32.64 ms) : 0, 32640
AppSec [candidate] (31.648 ms) : 0, 31648
Debugger [baseline] (59.153 ms) : 0, 59153
Debugger [candidate] (59.824 ms) : 0, 59824
Remote Config [baseline] (532.634 µs) : 0, 533
Remote Config [candidate] (1.096 ms) : 0, 1096
Telemetry [baseline] (13.624 ms) : 0, 13624
Telemetry [candidate] (12.846 ms) : 0, 12846
Flare Poller [baseline] (3.487 ms) : 0, 3487
Flare Poller [candidate] (3.469 ms) : 0, 3469
IAST [baseline] (25.871 ms) : 0, 25871
IAST [candidate] (25.758 ms) : 0, 25758
section profiling
crashtracking [baseline] (1.184 ms) : 0, 1184
crashtracking [candidate] (1.195 ms) : 0, 1195
BytebuddyAgent [baseline] (692.08 ms) : 0, 692080
BytebuddyAgent [candidate] (693.371 ms) : 0, 693371
AgentMeter [baseline] (9.161 ms) : 0, 9161
AgentMeter [candidate] (9.084 ms) : 0, 9084
GlobalTracer [baseline] (207.527 ms) : 0, 207527
GlobalTracer [candidate] (207.129 ms) : 0, 207129
AppSec [baseline] (32.598 ms) : 0, 32598
AppSec [candidate] (32.472 ms) : 0, 32472
Debugger [baseline] (65.648 ms) : 0, 65648
Debugger [candidate] (65.3 ms) : 0, 65300
Remote Config [baseline] (582.152 µs) : 0, 582
Remote Config [candidate] (564.828 µs) : 0, 565
Telemetry [baseline] (7.838 ms) : 0, 7838
Telemetry [candidate] (7.8 ms) : 0, 7800
Flare Poller [baseline] (3.537 ms) : 0, 3537
Flare Poller [candidate] (3.587 ms) : 0, 3587
ProfilingAgent [baseline] (94.124 ms) : 0, 94124
ProfilingAgent [candidate] (93.732 ms) : 0, 93732
Profiling [baseline] (94.684 ms) : 0, 94684
Profiling [candidate] (94.287 ms) : 0, 94287

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	worktree-brian.marks+memory-benchmark
git_commit_date	1776101218	1776108463
git_commit_sha	`140eee9`	`f8fc1df`
release_version	1.62.0-SNAPSHOT~140eee9add	1.62.0-SNAPSHOT~f8fc1df1dd

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1776110750	1776110750
ci_job_id	1591250294	1591250294
ci_pipeline_id	107457101	107457101
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-2u7jtu81 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-2u7jtu81 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 0 performance regressions! Performance is the same for 18 metrics, 16 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-255.038µs; -105.814µs] or [-13.643%; -5.661%]	unstable [-1463.279µs; -444.837µs] or [-25.111%; -7.634%]	unstable [+48.667op/s; +529.583op/s] or [+2.593%; +28.222%]	1.689ms	4.873ms	2165.625op/s	1.869ms	5.827ms	1876.500op/s
scenario:load:petclinic:appsec:high_load	better [-1.790ms; -0.570ms] or [-8.935%; -2.845%]	unsure [-2.299ms; -0.495ms] or [-7.193%; -1.550%]	unstable [-11.612op/s; +35.924op/s] or [-5.033%; +15.573%]	18.855ms	30.558ms	242.844op/s	20.035ms	31.955ms	230.688op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.247 ms) : 1234, 1260
.   : milestone, 1247,
iast (3.45 ms) : 3407, 3494
.   : milestone, 3450,
iast_FULL (6.035 ms) : 5974, 6097
.   : milestone, 6035,
iast_GLOBAL (3.736 ms) : 3671, 3800
.   : milestone, 3736,
profiling (2.419 ms) : 2393, 2445
.   : milestone, 2419,
tracing (1.906 ms) : 1890, 1923
.   : milestone, 1906,
section candidate
no_agent (1.223 ms) : 1211, 1234
.   : milestone, 1223,
iast (3.34 ms) : 3298, 3382
.   : milestone, 3340,
iast_FULL (5.997 ms) : 5935, 6059
.   : milestone, 5997,
iast_GLOBAL (3.625 ms) : 3571, 3679
.   : milestone, 3625,
profiling (2.087 ms) : 2068, 2106
.   : milestone, 2087,
tracing (1.919 ms) : 1901, 1938
.   : milestone, 1919,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.247 ms [1.234 ms, 1.26 ms]	-
iast	3.45 ms [3.407 ms, 3.494 ms]	2.203 ms (176.7%)
iast_FULL	6.035 ms [5.974 ms, 6.097 ms]	4.788 ms (384.0%)
iast_GLOBAL	3.736 ms [3.671 ms, 3.8 ms]	2.489 ms (199.6%)
profiling	2.419 ms [2.393 ms, 2.445 ms]	1.172 ms (94.0%)
tracing	1.906 ms [1.89 ms, 1.923 ms]	659.504 µs (52.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.223 ms [1.211 ms, 1.234 ms]	-
iast	3.34 ms [3.298 ms, 3.382 ms]	2.118 ms (173.2%)
iast_FULL	5.997 ms [5.935 ms, 6.059 ms]	4.774 ms (390.5%)
iast_GLOBAL	3.625 ms [3.571 ms, 3.679 ms]	2.402 ms (196.5%)
profiling	2.087 ms [2.068 ms, 2.106 ms]	864.309 µs (70.7%)
tracing	1.919 ms [1.901 ms, 1.938 ms]	696.95 µs (57.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.359 ms) : 18170, 18549
.   : milestone, 18359,
appsec (20.238 ms) : 20030, 20447
.   : milestone, 20238,
code_origins (17.965 ms) : 17788, 18142
.   : milestone, 17965,
iast (19.056 ms) : 18868, 19244
.   : milestone, 19056,
profiling (18.157 ms) : 17981, 18332
.   : milestone, 18157,
tracing (17.614 ms) : 17442, 17785
.   : milestone, 17614,
section candidate
no_agent (18.379 ms) : 18194, 18564
.   : milestone, 18379,
appsec (19.217 ms) : 19024, 19411
.   : milestone, 19217,
code_origins (17.973 ms) : 17795, 18151
.   : milestone, 17973,
iast (19.53 ms) : 19334, 19727
.   : milestone, 19530,
profiling (18.542 ms) : 18359, 18724
.   : milestone, 18542,
tracing (17.807 ms) : 17634, 17981
.   : milestone, 17807,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.359 ms [18.17 ms, 18.549 ms]	-
appsec	20.238 ms [20.03 ms, 20.447 ms]	1.879 ms (10.2%)
code_origins	17.965 ms [17.788 ms, 18.142 ms]	-394.897 µs (-2.2%)
iast	19.056 ms [18.868 ms, 19.244 ms]	696.73 µs (3.8%)
profiling	18.157 ms [17.981 ms, 18.332 ms]	-202.766 µs (-1.1%)
tracing	17.614 ms [17.442 ms, 17.785 ms]	-745.918 µs (-4.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.379 ms [18.194 ms, 18.564 ms]	-
appsec	19.217 ms [19.024 ms, 19.411 ms]	838.657 µs (4.6%)
code_origins	17.973 ms [17.795 ms, 18.151 ms]	-405.345 µs (-2.2%)
iast	19.53 ms [19.334 ms, 19.727 ms]	1.151 ms (6.3%)
profiling	18.542 ms [18.359 ms, 18.724 ms]	162.815 µs (0.9%)
tracing	17.807 ms [17.634 ms, 17.981 ms]	-571.399 µs (-3.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	worktree-brian.marks+memory-benchmark
git_commit_date	1776101218	1776108463
git_commit_sha	`140eee9`	`f8fc1df`
release_version	1.62.0-SNAPSHOT~140eee9add	1.62.0-SNAPSHOT~f8fc1df1dd

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1776110519	1776110519
ci_job_id	1591250298	1591250298
ci_pipeline_id	107457101	107457101
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-jllxio6c 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-jllxio6c 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.488 ms) : 1477, 1500
.   : milestone, 1488,
appsec (2.539 ms) : 2484, 2593
.   : milestone, 2539,
iast (2.268 ms) : 2199, 2337
.   : milestone, 2268,
iast_GLOBAL (2.317 ms) : 2247, 2386
.   : milestone, 2317,
profiling (2.099 ms) : 2044, 2154
.   : milestone, 2099,
tracing (2.077 ms) : 2023, 2130
.   : milestone, 2077,
section candidate
no_agent (1.487 ms) : 1476, 1499
.   : milestone, 1487,
appsec (3.846 ms) : 3622, 4069
.   : milestone, 3846,
iast (2.272 ms) : 2202, 2341
.   : milestone, 2272,
iast_GLOBAL (2.308 ms) : 2239, 2377
.   : milestone, 2308,
profiling (2.098 ms) : 2043, 2153
.   : milestone, 2098,
tracing (2.084 ms) : 2030, 2137
.   : milestone, 2084,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.488 ms [1.477 ms, 1.5 ms]	-
appsec	2.539 ms [2.484 ms, 2.593 ms]	1.05 ms (70.6%)
iast	2.268 ms [2.199 ms, 2.337 ms]	779.47 µs (52.4%)
iast_GLOBAL	2.317 ms [2.247 ms, 2.386 ms]	828.409 µs (55.7%)
profiling	2.099 ms [2.044 ms, 2.154 ms]	610.171 µs (41.0%)
tracing	2.077 ms [2.023 ms, 2.13 ms]	588.13 µs (39.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.487 ms [1.476 ms, 1.499 ms]	-
appsec	3.846 ms [3.622 ms, 4.069 ms]	2.358 ms (158.5%)
iast	2.272 ms [2.202 ms, 2.341 ms]	784.052 µs (52.7%)
iast_GLOBAL	2.308 ms [2.239 ms, 2.377 ms]	820.398 µs (55.2%)
profiling	2.098 ms [2.043 ms, 2.153 ms]	610.556 µs (41.0%)
tracing	2.084 ms [2.03 ms, 2.137 ms]	596.457 µs (40.1%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~f8fc1df1dd, baseline=1.62.0-SNAPSHOT~140eee9add
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.179 s) : 15179000, 15179000
.   : milestone, 15179000,
appsec (14.794 s) : 14794000, 14794000
.   : milestone, 14794000,
iast (18.214 s) : 18214000, 18214000
.   : milestone, 18214000,
iast_GLOBAL (18.157 s) : 18157000, 18157000
.   : milestone, 18157000,
profiling (14.948 s) : 14948000, 14948000
.   : milestone, 14948000,
tracing (14.742 s) : 14742000, 14742000
.   : milestone, 14742000,
section candidate
no_agent (15.485 s) : 15485000, 15485000
.   : milestone, 15485000,
appsec (15.083 s) : 15083000, 15083000
.   : milestone, 15083000,
iast (18.449 s) : 18449000, 18449000
.   : milestone, 18449000,
iast_GLOBAL (17.892 s) : 17892000, 17892000
.   : milestone, 17892000,
profiling (15.085 s) : 15085000, 15085000
.   : milestone, 15085000,
tracing (14.919 s) : 14919000, 14919000
.   : milestone, 14919000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.179 s [15.179 s, 15.179 s]	-
appsec	14.794 s [14.794 s, 14.794 s]	-385.0 ms (-2.5%)
iast	18.214 s [18.214 s, 18.214 s]	3.035 s (20.0%)
iast_GLOBAL	18.157 s [18.157 s, 18.157 s]	2.978 s (19.6%)
profiling	14.948 s [14.948 s, 14.948 s]	-231.0 ms (-1.5%)
tracing	14.742 s [14.742 s, 14.742 s]	-437.0 ms (-2.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.485 s [15.485 s, 15.485 s]	-
appsec	15.083 s [15.083 s, 15.083 s]	-402.0 ms (-2.6%)
iast	18.449 s [18.449 s, 18.449 s]	2.964 s (19.1%)
iast_GLOBAL	17.892 s [17.892 s, 17.892 s]	2.407 s (15.5%)
profiling	15.085 s [15.085 s, 15.085 s]	-400.0 ms (-2.6%)
tracing	14.919 s [14.919 s, 14.919 s]	-566.0 ms (-3.7%)

dougqh · 2026-04-13T17:00:52Z

@@ -0,0 +1,2 @@
+tools/


Placing these in a "memory" folder seems off to me.

The intent of running at multiple memory settings is to simulate how the application behaves under load relative to available resources, so I don't think of this as a memory benchmark.

Moved it to benchmark/throughput/petclinic/.

Address review feedback: the benchmark measures throughput under load at varying resource constraints, not memory specifically. Renamed directory and updated all references. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

bm1549 · 2026-04-13T19:35:17Z

You can do that today by running twice with different --agent paths and --skip-baseline on the second run:

./run.sh --agent /path/to/dd-java-agent-1.61.0.jar --output ./results/before
./run.sh --agent /path/to/dd-java-agent-SNAPSHOT.jar --output ./results/after --skip-baseline

I could also add a --agent-before/--agent-after mode that does both in one invocation if that'd be less annoying to use.

bm1549 added the tag: no release notes Changes to exclude from release notes label Apr 13, 2026

Remove name references from benchmark files

e61cb36

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

dougqh reviewed Apr 13, 2026

View reviewed changes

bm1549 changed the title ~~Add memory-pressure benchmark for petclinic throughput testing~~ Add throughput benchmark for petclinic load testing Apr 13, 2026

bm1549 closed this Apr 28, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add throughput benchmark for petclinic load testing#11098

Add throughput benchmark for petclinic load testing#11098
bm1549 wants to merge 3 commits into
masterfrom
worktree-brian.marks+memory-benchmark

bm1549 commented Apr 13, 2026 •

edited

Loading

Uh oh!

dougqh commented Apr 13, 2026

Uh oh!

pr-commenter Bot commented Apr 13, 2026 •

edited

Loading

Uh oh!

dougqh Apr 13, 2026

Uh oh!

bm1549 Apr 13, 2026

Uh oh!

bm1549 commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bm1549 commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Differences from the original manual benchmark

Additional Notes

Contributor Checklist

Uh oh!

dougqh commented Apr 13, 2026

Uh oh!

pr-commenter Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

dougqh Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

bm1549 Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

bm1549 commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bm1549 commented Apr 13, 2026 •

edited

Loading

pr-commenter Bot commented Apr 13, 2026 •

edited

Loading