Skip to content

Pin Spring Petclinic in benchmarks#8969

Merged
smola merged 1 commit intomasterfrom
smola/pin-benchmark-apps
Jun 12, 2025
Merged

Pin Spring Petclinic in benchmarks#8969
smola merged 1 commit intomasterfrom
smola/pin-benchmark-apps

Conversation

@smola
Copy link
Copy Markdown
Member

@smola smola commented Jun 12, 2025 •
edited
Loading

What Does This Do

Spring Petclinic in benchmark for reproducibility.
Pinned to latest commit in main brach. Spring Petclinci repo has no tags.

@smola smola requested a review from a team as a code owner June 12, 2025 06:55
@smola smola added the tag: no release notes Changes to exclude from release notes label Jun 12, 2025
@smola smola removed the request for review from a team June 12, 2025 06:55
@smola smola added the comp: tooling Build & Tooling label Jun 12, 2025
PerfectSlayer
PerfectSlayer approved these changes Jun 12, 2025
View reviewed changes
Comment thread benchmark/Dockerfile
@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Jun 12, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/pin-benchmark-apps
git_commit_date 1749673130 1749711966
git_commit_sha 2442b1f 9b97427184
release_version 1.50.0-SNAPSHOT~2442b1fa6d 1.50.0-SNAPSHOT~b9b97427184
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749713635 1749713635
ci_job_id 977794536 977794536
ci_pipeline_id 67477454 67477454
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-63wmvl1-project-304-concurrent-0-8n2osn3u 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-63wmvl1-project-304-concurrent-0-8n2osn3u 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 55 metrics, 16 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~b9b97427184, baseline=1.50.0-SNAPSHOT~2442b1fa6d

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1021467
Total [baseline] (8.515 s) : 0, 8515062
Agent [candidate] (1.023 s) : 0, 1023154
Total [candidate] (8.542 s) : 0, 8541890
section iast
Agent [baseline] (1.148 s) : 0, 1148302
Total [baseline] (9.164 s) : 0, 9163649
Agent [candidate] (1.164 s) : 0, 1164432
Total [candidate] (9.195 s) : 0, 9194665
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.152 s) : 0, 1152100
Total [baseline] (9.148 s) : 0, 9148062
Agent [candidate] (1.153 s) : 0, 1152957
Total [candidate] (9.121 s) : 0, 9121032
section iast_TELEMETRY_OFF
Agent [baseline] (1.146 s) : 0, 1146295
Total [baseline] (9.217 s) : 0, 9217424
Agent [candidate] (1.155 s) : 0, 1154581
Total [candidate] (9.223 s) : 0, 9222521
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.021 s -
Agent iast 1.148 s 126.834 ms (12.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.152 s 130.633 ms (12.8%)
Agent iast_TELEMETRY_OFF 1.146 s 124.827 ms (12.2%)
Total tracing 8.515 s -
Total iast 9.164 s 648.587 ms (7.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.148 s 633.0 ms (7.4%)
Total iast_TELEMETRY_OFF 9.217 s 702.363 ms (8.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.023 s -
Agent iast 1.164 s 141.278 ms (13.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.153 s 129.803 ms (12.7%)
Agent iast_TELEMETRY_OFF 1.155 s 131.427 ms (12.8%)
Total tracing 8.542 s -
Total iast 9.195 s 652.774 ms (7.6%)
Total iast_HARDCODED_SECRET_DISABLED 9.121 s 579.141 ms (6.8%)
Total iast_TELEMETRY_OFF 9.223 s 680.631 ms (8.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~b9b97427184, baseline=1.50.0-SNAPSHOT~2442b1fa6d

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.484 ms) : 0, 682484
BytebuddyAgent [candidate] (683.017 ms) : 0, 683017
GlobalTracer [baseline] (239.473 ms) : 0, 239473
GlobalTracer [candidate] (240.838 ms) : 0, 240838
AppSec [baseline] (56.235 ms) : 0, 56235
AppSec [candidate] (55.621 ms) : 0, 55621
Debugger [baseline] (6.146 ms) : 0, 6146
Debugger [candidate] (6.207 ms) : 0, 6207
Remote Config [baseline] (736.173 µs) : 0, 736
Remote Config [candidate] (750.096 µs) : 0, 750
Telemetry [baseline] (12.859 ms) : 0, 12859
Telemetry [candidate] (13.201 ms) : 0, 13201
section iast
BytebuddyAgent [baseline] (801.17 ms) : 0, 801170
BytebuddyAgent [candidate] (814.065 ms) : 0, 814065
GlobalTracer [baseline] (229.889 ms) : 0, 229889
GlobalTracer [candidate] (232.857 ms) : 0, 232857
IAST [baseline] (26.967 ms) : 0, 26967
IAST [candidate] (26.123 ms) : 0, 26123
AppSec [baseline] (52.343 ms) : 0, 52343
AppSec [candidate] (53.049 ms) : 0, 53049
Debugger [baseline] (5.918 ms) : 0, 5918
Debugger [candidate] (5.983 ms) : 0, 5983
Remote Config [baseline] (606.438 µs) : 0, 606
Remote Config [candidate] (614.323 µs) : 0, 614
Telemetry [baseline] (7.914 ms) : 0, 7914
Telemetry [candidate] (7.923 ms) : 0, 7923
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.979 ms) : 0, 802979
BytebuddyAgent [candidate] (803.584 ms) : 0, 803584
GlobalTracer [baseline] (231.03 ms) : 0, 231030
GlobalTracer [candidate] (231.253 ms) : 0, 231253
IAST [baseline] (28.027 ms) : 0, 28027
IAST [candidate] (26.34 ms) : 0, 26340
AppSec [baseline] (51.807 ms) : 0, 51807
AppSec [candidate] (53.695 ms) : 0, 53695
Debugger [baseline] (6.016 ms) : 0, 6016
Debugger [candidate] (5.986 ms) : 0, 5986
Remote Config [baseline] (605.159 µs) : 0, 605
Remote Config [candidate] (594.719 µs) : 0, 595
Telemetry [baseline] (8.006 ms) : 0, 8006
Telemetry [candidate] (7.933 ms) : 0, 7933
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (798.738 ms) : 0, 798738
BytebuddyAgent [candidate] (805.989 ms) : 0, 805989
GlobalTracer [baseline] (230.618 ms) : 0, 230618
GlobalTracer [candidate] (231.491 ms) : 0, 231491
IAST [baseline] (28.669 ms) : 0, 28669
IAST [candidate] (29.459 ms) : 0, 29459
AppSec [baseline] (50.299 ms) : 0, 50299
AppSec [candidate] (49.467 ms) : 0, 49467
Debugger [baseline] (5.958 ms) : 0, 5958
Debugger [candidate] (6.019 ms) : 0, 6019
Remote Config [baseline] (637.23 µs) : 0, 637
Remote Config [candidate] (604.169 µs) : 0, 604
Telemetry [baseline] (7.859 ms) : 0, 7859
Telemetry [candidate] (7.89 ms) : 0, 7890
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~b9b97427184, baseline=1.50.0-SNAPSHOT~2442b1fa6d

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.023 s) : 0, 1022588
Total [baseline] (11.116 s) : 0, 11115897
Agent [candidate] (1.038 s) : 0, 1037613
Total [candidate] (11.182 s) : 0, 11182260
section appsec
Agent [baseline] (1.172 s) : 0, 1172229
Total [baseline] (11.202 s) : 0, 11201906
Agent [candidate] (1.186 s) : 0, 1185580
Total [candidate] (11.293 s) : 0, 11292570
section iast
Agent [baseline] (1.165 s) : 0, 1165076
Total [baseline] (11.369 s) : 0, 11369254
Agent [candidate] (1.154 s) : 0, 1153723
Total [candidate] (11.425 s) : 0, 11425281
section profiling
Agent [baseline] (1.266 s) : 0, 1265752
Total [baseline] (11.474 s) : 0, 11474398
Agent [candidate] (1.268 s) : 0, 1267979
Total [candidate] (11.446 s) : 0, 11445981
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.023 s -
Agent appsec 1.172 s 149.641 ms (14.6%)
Agent iast 1.165 s 142.488 ms (13.9%)
Agent profiling 1.266 s 243.164 ms (23.8%)
Total tracing 11.116 s -
Total appsec 11.202 s 86.009 ms (0.8%)
Total iast 11.369 s 253.358 ms (2.3%)
Total profiling 11.474 s 358.501 ms (3.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.038 s -
Agent appsec 1.186 s 147.966 ms (14.3%)
Agent iast 1.154 s 116.11 ms (11.2%)
Agent profiling 1.268 s 230.366 ms (22.2%)
Total tracing 11.182 s -
Total appsec 11.293 s 110.31 ms (1.0%)
Total iast 11.425 s 243.021 ms (2.2%)
Total profiling 11.446 s 263.72 ms (2.4%) 
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~b9b97427184, baseline=1.50.0-SNAPSHOT~2442b1fa6d

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.165 ms) : 0, 682165
BytebuddyAgent [candidate] (692.645 ms) : 0, 692645
GlobalTracer [baseline] (240.338 ms) : 0, 240338
GlobalTracer [candidate] (244.024 ms) : 0, 244024
AppSec [baseline] (57.648 ms) : 0, 57648
AppSec [candidate] (59.235 ms) : 0, 59235
Debugger [baseline] (6.874 ms) : 0, 6874
Debugger [candidate] (6.255 ms) : 0, 6255
Remote Config [baseline] (740.143 µs) : 0, 740
Remote Config [candidate] (729.989 µs) : 0, 730
Telemetry [baseline] (11.306 ms) : 0, 11306
Telemetry [candidate] (10.757 ms) : 0, 10757
section appsec
BytebuddyAgent [baseline] (703.683 ms) : 0, 703683
BytebuddyAgent [candidate] (711.703 ms) : 0, 711703
GlobalTracer [baseline] (233.741 ms) : 0, 233741
GlobalTracer [candidate] (237.569 ms) : 0, 237569
IAST [baseline] (21.76 ms) : 0, 21760
IAST [candidate] (22.312 ms) : 0, 22312
AppSec [baseline] (175.656 ms) : 0, 175656
AppSec [candidate] (176.274 ms) : 0, 176274
Debugger [baseline] (5.95 ms) : 0, 5950
Debugger [candidate] (5.982 ms) : 0, 5982
Remote Config [baseline] (627.211 µs) : 0, 627
Remote Config [candidate] (622.861 µs) : 0, 623
Telemetry [baseline] (7.312 ms) : 0, 7312
Telemetry [candidate] (7.381 ms) : 0, 7381
section iast
BytebuddyAgent [baseline] (813.431 ms) : 0, 813431
BytebuddyAgent [candidate] (804.309 ms) : 0, 804309
GlobalTracer [baseline] (232.974 ms) : 0, 232974
GlobalTracer [candidate] (231.24 ms) : 0, 231240
IAST [baseline] (28.834 ms) : 0, 28834
IAST [candidate] (26.358 ms) : 0, 26358
AppSec [baseline] (51.447 ms) : 0, 51447
AppSec [candidate] (53.674 ms) : 0, 53674
Debugger [baseline] (5.976 ms) : 0, 5976
Debugger [candidate] (5.975 ms) : 0, 5975
Remote Config [baseline] (599.825 µs) : 0, 600
Remote Config [candidate] (591.556 µs) : 0, 592
Telemetry [baseline] (8.045 ms) : 0, 8045
Telemetry [candidate] (7.911 ms) : 0, 7911
section profiling
BytebuddyAgent [baseline] (675.037 ms) : 0, 675037
BytebuddyAgent [candidate] (675.998 ms) : 0, 675998
GlobalTracer [baseline] (359.425 ms) : 0, 359425
GlobalTracer [candidate] (360.618 ms) : 0, 360618
AppSec [baseline] (61.439 ms) : 0, 61439
AppSec [candidate] (61.772 ms) : 0, 61772
Debugger [baseline] (6.072 ms) : 0, 6072
Debugger [candidate] (6.077 ms) : 0, 6077
Remote Config [baseline] (654.025 µs) : 0, 654
Remote Config [candidate] (649.267 µs) : 0, 649
Telemetry [baseline] (8.079 ms) : 0, 8079
Telemetry [candidate] (8.147 ms) : 0, 8147
ProfilingAgent [baseline] (104.248 ms) : 0, 104248
ProfilingAgent [candidate] (103.616 ms) : 0, 103616
Profiling [baseline] (104.272 ms) : 0, 104272
Profiling [candidate] (103.64 ms) : 0, 103640
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-06-12T07:17:18 2025-06-12T07:19:28
git_branch master smola/pin-benchmark-apps
git_commit_date 1749673130 1749711967
git_commit_sha 2442b1f 9b97427184
release_version 1.50.0-SNAPSHOT~2442b1fa6d 1.50.0-SNAPSHOT~b9b97427184
start_time 2025-06-12T07:14:35 2025-06-12T07:18:53
See matching parameters
Baseline Candidate
application petclinic petclinic
ci_job_date 1749712768 1749712768
ci_job_id 977794537 977794537
ci_pipeline_id 67477454 67477454
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-63wmvl1-project-304-concurrent-1-sfgbmta9 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-63wmvl1-project-304-concurrent-1-sfgbmta9 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results results results
variant appsec appsec

Summary

Found 0 performance improvements and 3 performance regressions! Performance is the same for 0 metrics, 9 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:no_agent unstable
[+57.311ms; +59.297ms] or [+667.395%; +690.523%]		 worse
[-521.981op/s; -482.721op/s] or [-90.694%; -83.873%]		 66.892ms 73.188op/s 8.587ms 575.540op/s
scenario:load:petclinic:profiling worse
[+82.068ms; +84.791ms] or [+inf%; +inf%]		 unstable
[-24515.882op/s; -15368.689op/s] or [-122.579%; -76.843%]		 83429377.567ns 57.714op/s 0.000ns 20000.000op/s
scenario:load:petclinic:tracing unstable
[+64.115ms; +66.438ms] or [+747.171%; +774.241%]		 worse
[-521.495op/s; -488.229op/s] or [-91.262%; -85.440%]		 73.857ms 66.567op/s 8.581ms 571.429op/s

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/pin-benchmark-apps
git_commit_date 1749673130 1749711960
git_commit_sha 2442b1f 9b97427184
release_version 1.50.0-SNAPSHOT~2442b1fa6d 1.50.0-SNAPSHOT~b9b97427184
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1749713410 1749713410
ci_job_id 977794538 977794538
ci_pipeline_id 67477454 67477454
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-pz8nwqvr-project-304-concurrent-2-n4soeluj 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-pz8nwqvr-project-304-concurrent-2-n4soeluj 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~b9b97427184, baseline=1.50.0-SNAPSHOT~2442b1fa6d
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (2.407 ms) : 2358, 2455
.   : milestone, 2407,
iast (2.189 ms) : 2127, 2250
.   : milestone, 2189,
iast_GLOBAL (2.236 ms) : 2175, 2298
.   : milestone, 2236,
profiling (2.485 ms) : 2314, 2656
.   : milestone, 2485,
tracing (2.006 ms) : 1959, 2054
.   : milestone, 2006,
section candidate
no_agent (1.482 ms) : 1471, 1494
.   : milestone, 1482,
appsec (2.42 ms) : 2371, 2469
.   : milestone, 2420,
iast (2.199 ms) : 2138, 2260
.   : milestone, 2199,
iast_GLOBAL (2.242 ms) : 2180, 2304
.   : milestone, 2242,
profiling (2.061 ms) : 2011, 2112
.   : milestone, 2061,
tracing (2.024 ms) : 1976, 2072
.   : milestone, 2024,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.479 ms [1.468 ms, 1.491 ms] -
appsec 2.407 ms [2.358 ms, 2.455 ms] 927.776 µs (62.7%)
iast 2.189 ms [2.127 ms, 2.25 ms] 709.494 µs (48.0%)
iast_GLOBAL 2.236 ms [2.175 ms, 2.298 ms] 757.004 µs (51.2%)
profiling 2.485 ms [2.314 ms, 2.656 ms] 1.006 ms (68.0%)
tracing 2.006 ms [1.959 ms, 2.054 ms] 526.964 µs (35.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.482 ms [1.471 ms, 1.494 ms] -
appsec 2.42 ms [2.371 ms, 2.469 ms] 938.026 µs (63.3%)
iast 2.199 ms [2.138 ms, 2.26 ms] 716.777 µs (48.4%)
iast_GLOBAL 2.242 ms [2.18 ms, 2.304 ms] 759.913 µs (51.3%)
profiling 2.061 ms [2.011 ms, 2.112 ms] 578.955 µs (39.1%)
tracing 2.024 ms [1.976 ms, 2.072 ms] 541.919 µs (36.6%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~b9b97427184, baseline=1.50.0-SNAPSHOT~2442b1fa6d
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.036 s) : 15036000, 15036000
.   : milestone, 15036000,
appsec (14.811 s) : 14811000, 14811000
.   : milestone, 14811000,
iast (18.34 s) : 18340000, 18340000
.   : milestone, 18340000,
iast_GLOBAL (18.238 s) : 18238000, 18238000
.   : milestone, 18238000,
profiling (15.614 s) : 15614000, 15614000
.   : milestone, 15614000,
tracing (14.826 s) : 14826000, 14826000
.   : milestone, 14826000,
section candidate
no_agent (15.279 s) : 15279000, 15279000
.   : milestone, 15279000,
appsec (14.843 s) : 14843000, 14843000
.   : milestone, 14843000,
iast (18.313 s) : 18313000, 18313000
.   : milestone, 18313000,
iast_GLOBAL (18.412 s) : 18412000, 18412000
.   : milestone, 18412000,
profiling (15.329 s) : 15329000, 15329000
.   : milestone, 15329000,
tracing (15.24 s) : 15240000, 15240000
.   : milestone, 15240000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.036 s [15.036 s, 15.036 s] -
appsec 14.811 s [14.811 s, 14.811 s] -225.0 ms (-1.5%)
iast 18.34 s [18.34 s, 18.34 s] 3.304 s (22.0%)
iast_GLOBAL 18.238 s [18.238 s, 18.238 s] 3.202 s (21.3%)
profiling 15.614 s [15.614 s, 15.614 s] 578.0 ms (3.8%)
tracing 14.826 s [14.826 s, 14.826 s] -210.0 ms (-1.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.279 s [15.279 s, 15.279 s] -
appsec 14.843 s [14.843 s, 14.843 s] -436.0 ms (-2.9%)
iast 18.313 s [18.313 s, 18.313 s] 3.034 s (19.9%)
iast_GLOBAL 18.412 s [18.412 s, 18.412 s] 3.133 s (20.5%)
profiling 15.329 s [15.329 s, 15.329 s] 50.0 ms (0.3%)
tracing 15.24 s [15.24 s, 15.24 s] -39.0 ms (-0.3%)

@smola smola merged commit 58aca3d into master Jun 12, 2025
522 of 524 checks passed
@smola smola deleted the smola/pin-benchmark-apps branch June 12, 2025 07:45
@github-actions github-actions bot added this to the 1.50.0 milestone Jun 12, 2025
sarahchen6 pushed a commit that referenced this pull request Jun 12, 2025
@smola @sarahchen6
Pin Spring Petclinic in benchmarks (#8969)
d5205a3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bric3 bric3 bric3 approved these changes

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez

Assignees

No one assigned

Labels

comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.50.0

Development

Successfully merging this pull request may close these issues.

3 participants

@smola @bric3 @PerfectSlayer