WIP: instrumenter index experiment by mcculls · Pull Request #9031 · DataDog/dd-trace-java

mcculls · 2025-06-24T23:04:56Z

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-24T23:40:49Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mcculls/instrumenter-index-experiment
git_commit_date	1750839907	1750806252
git_commit_sha	`c5581ea`	`22f46ba`
release_version	1.51.0-SNAPSHOT~c5581eae59	1.51.0-SNAPSHOT~22f46ba16f

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750844293	1750844293
ci_job_id	997462979	997462979
ci_pipeline_id	68681310	68681310
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-bziamzy-project-304-concurrent-0-agd5m9xz 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-bziamzy-project-304-concurrent-0-agd5m9xz 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 3 performance improvements and 3 performance regressions! Performance is the same for 37 metrics, 10 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:iast:BytebuddyAgent	better [-24.683ms; -19.034ms] or [-3.065%; -2.364%]	783.337ms	805.195ms
scenario:startup:insecure-bank:iast:GlobalTracer	worse [+6.014ms; +8.812ms] or [+2.595%; +3.803%]	239.139ms	231.727ms
scenario:startup:insecure-bank:tracing:BytebuddyAgent	better [-19.179ms; -13.943ms] or [-2.793%; -2.031%]	670.069ms	686.630ms
scenario:startup:petclinic:appsec:BytebuddyAgent	better [-26.633ms; -14.932ms] or [-3.756%; -2.106%]	688.280ms	709.063ms
scenario:startup:petclinic:appsec:GlobalTracer	worse [+6.093ms; +12.601ms] or [+2.590%; +5.356%]	244.608ms	235.261ms
scenario:startup:petclinic:iast:GlobalTracer	worse [+5.764ms; +13.519ms] or [+2.484%; +5.826%]	241.692ms	232.050ms

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (993.939 ms) : 0, 993939
Total [baseline] (8.535 s) : 0, 8534971
Agent [candidate] (977.466 ms) : 0, 977466
Total [candidate] (8.523 s) : 0, 8523050
section iast
Agent [baseline] (1.127 s) : 0, 1126634
Total [baseline] (9.23 s) : 0, 9230270
Agent [candidate] (1.113 s) : 0, 1112623
Total [candidate] (9.255 s) : 0, 9254605

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	993.939 ms	-
Agent	iast	1.127 s	132.695 ms (13.4%)
Total	tracing	8.535 s	-
Total	iast	9.23 s	695.299 ms (8.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	977.466 ms	-
Agent	iast	1.113 s	135.157 ms (13.8%)
Total	tracing	8.523 s	-
Total	iast	9.255 s	731.555 ms (8.6%)

gantt
    title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.63 ms) : 0, 686630
BytebuddyAgent [candidate] (670.069 ms) : 0, 670069
GlobalTracer [baseline] (241.645 ms) : 0, 241645
GlobalTracer [candidate] (241.58 ms) : 0, 241580
AppSec [baseline] (30.029 ms) : 0, 30029
AppSec [candidate] (30.01 ms) : 0, 30010
Debugger [baseline] (6.054 ms) : 0, 6054
Debugger [candidate] (6.015 ms) : 0, 6015
Remote Config [baseline] (658.682 µs) : 0, 659
Remote Config [candidate] (655.444 µs) : 0, 655
Telemetry [baseline] (8.106 ms) : 0, 8106
Telemetry [candidate] (8.267 ms) : 0, 8267
section iast
BytebuddyAgent [baseline] (805.195 ms) : 0, 805195
BytebuddyAgent [candidate] (783.337 ms) : 0, 783337
GlobalTracer [baseline] (231.727 ms) : 0, 231727
GlobalTracer [candidate] (239.139 ms) : 0, 239139
AppSec [baseline] (24.858 ms) : 0, 24858
AppSec [candidate] (31.176 ms) : 0, 31176
Debugger [baseline] (5.781 ms) : 0, 5781
Debugger [candidate] (5.706 ms) : 0, 5706
Remote Config [baseline] (585.039 µs) : 0, 585
Remote Config [candidate] (557.426 µs) : 0, 557
Telemetry [baseline] (7.903 ms) : 0, 7903
Telemetry [candidate] (8.712 ms) : 0, 8712
IAST [baseline] (29.97 ms) : 0, 29970
IAST [candidate] (23.087 ms) : 0, 23087

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (998.751 ms) : 0, 998751
Total [baseline] (10.705 s) : 0, 10705191
Agent [candidate] (981.507 ms) : 0, 981507
Total [candidate] (10.654 s) : 0, 10653802
section appsec
Agent [baseline] (1.171 s) : 0, 1170512
Total [baseline] (10.699 s) : 0, 10698515
Agent [candidate] (1.161 s) : 0, 1160925
Total [candidate] (10.784 s) : 0, 10783520
section iast
Agent [baseline] (1.129 s) : 0, 1129178
Total [baseline] (10.814 s) : 0, 10813742
Agent [candidate] (1.128 s) : 0, 1127566
Total [candidate] (11.029 s) : 0, 11029151
section profiling
Agent [baseline] (1.24 s) : 0, 1240424
Total [baseline] (10.957 s) : 0, 10956680
Agent [candidate] (1.231 s) : 0, 1230830
Total [candidate] (10.968 s) : 0, 10967575

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	998.751 ms	-
Agent	appsec	1.171 s	171.761 ms (17.2%)
Agent	iast	1.129 s	130.427 ms (13.1%)
Agent	profiling	1.24 s	241.672 ms (24.2%)
Total	tracing	10.705 s	-
Total	appsec	10.699 s	-6.676 ms (-0.1%)
Total	iast	10.814 s	108.551 ms (1.0%)
Total	profiling	10.957 s	251.488 ms (2.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	981.507 ms	-
Agent	appsec	1.161 s	179.418 ms (18.3%)
Agent	iast	1.128 s	146.058 ms (14.9%)
Agent	profiling	1.231 s	249.322 ms (25.4%)
Total	tracing	10.654 s	-
Total	appsec	10.784 s	129.717 ms (1.2%)
Total	iast	11.029 s	375.349 ms (3.5%)
Total	profiling	10.968 s	313.773 ms (2.9%)

gantt
    title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~c5581eae59

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (689.608 ms) : 0, 689608
BytebuddyAgent [candidate] (674.549 ms) : 0, 674549
GlobalTracer [baseline] (243.037 ms) : 0, 243037
GlobalTracer [candidate] (241.145 ms) : 0, 241145
AppSec [baseline] (30.236 ms) : 0, 30236
AppSec [candidate] (29.952 ms) : 0, 29952
Debugger [baseline] (6.085 ms) : 0, 6085
Debugger [candidate] (6.001 ms) : 0, 6001
Remote Config [baseline] (661.75 µs) : 0, 662
Remote Config [candidate] (640.932 µs) : 0, 641
Telemetry [baseline] (8.182 ms) : 0, 8182
Telemetry [candidate] (8.198 ms) : 0, 8198
section appsec
BytebuddyAgent [baseline] (709.063 ms) : 0, 709063
BytebuddyAgent [candidate] (688.28 ms) : 0, 688280
GlobalTracer [baseline] (235.261 ms) : 0, 235261
GlobalTracer [candidate] (244.608 ms) : 0, 244608
AppSec [baseline] (168.974 ms) : 0, 168974
AppSec [candidate] (170.092 ms) : 0, 170092
Debugger [baseline] (5.804 ms) : 0, 5804
Debugger [candidate] (5.804 ms) : 0, 5804
Remote Config [baseline] (610.149 µs) : 0, 610
Remote Config [candidate] (610.003 µs) : 0, 610
Telemetry [baseline] (8.188 ms) : 0, 8188
Telemetry [candidate] (8.366 ms) : 0, 8366
IAST [baseline] (21.855 ms) : 0, 21855
IAST [candidate] (22.329 ms) : 0, 22329
section iast
BytebuddyAgent [baseline] (806.59 ms) : 0, 806590
BytebuddyAgent [candidate] (793.986 ms) : 0, 793986
GlobalTracer [baseline] (232.05 ms) : 0, 232050
GlobalTracer [candidate] (241.692 ms) : 0, 241692
AppSec [baseline] (26.096 ms) : 0, 26096
AppSec [candidate] (32.771 ms) : 0, 32771
Debugger [baseline] (5.789 ms) : 0, 5789
Debugger [candidate] (5.865 ms) : 0, 5865
Remote Config [baseline] (580.906 µs) : 0, 581
Remote Config [candidate] (580.16 µs) : 0, 580
Telemetry [baseline] (7.924 ms) : 0, 7924
Telemetry [candidate] (7.993 ms) : 0, 7993
IAST [baseline] (29.368 ms) : 0, 29368
IAST [candidate] (23.522 ms) : 0, 23522
section profiling
BytebuddyAgent [baseline] (676.487 ms) : 0, 676487
BytebuddyAgent [candidate] (665.179 ms) : 0, 665179
GlobalTracer [baseline] (360.677 ms) : 0, 360677
GlobalTracer [candidate] (361.767 ms) : 0, 361767
AppSec [baseline] (32.081 ms) : 0, 32081
AppSec [candidate] (30.586 ms) : 0, 30586
Debugger [baseline] (7.514 ms) : 0, 7514
Debugger [candidate] (11.163 ms) : 0, 11163
Remote Config [baseline] (678.241 µs) : 0, 678
Remote Config [candidate] (677.715 µs) : 0, 678
Telemetry [baseline] (10.953 ms) : 0, 10953
Telemetry [candidate] (8.827 ms) : 0, 8827
ProfilingAgent [baseline] (102.732 ms) : 0, 102732
ProfilingAgent [candidate] (104.054 ms) : 0, 104054
Profiling [baseline] (102.758 ms) : 0, 102758
Profiling [candidate] (104.078 ms) : 0, 104078

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mcculls/instrumenter-index-experiment
git_commit_date	1750788935	1750806252
git_commit_sha	`faeb62c`	`22f46ba`
release_version	1.51.0-SNAPSHOT~faeb62cfb1	1.51.0-SNAPSHOT~22f46ba16f

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750807926	1750807926
ci_job_id	997130998	997130998
ci_pipeline_id	68681310	68681310
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-vaym2f8s-project-304-concurrent-0-73h7z8i5 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-vaym2f8s-project-304-concurrent-0-73h7z8i5 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-923.042µs; -617.688µs] or [-9.865%; -6.602%]	unstable [-20.180op/s; +108.305op/s] or [-4.068%; +21.834%]	8.586ms	540.094op/s	9.357ms	496.031op/s
scenario:load:petclinic:code_origins:high_load	worse [+2.003ms; +2.816ms] or [+4.592%; +6.457%]	unstable [-13.280op/s; +2.055op/s] or [-12.383%; +1.916%]	46.025ms	101.638op/s	43.616ms	107.250op/s
scenario:load:petclinic:iast:high_load	worse [+2.206ms; +3.102ms] or [+4.954%; +6.967%]	unstable [-13.815op/s; +1.965op/s] or [-13.143%; +1.870%]	47.182ms	99.188op/s	44.528ms	105.112op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (35.892 ms) : 35604, 36181
.   : milestone, 35892,
appsec (48.025 ms) : 47601, 48450
.   : milestone, 48025,
code_origins (43.616 ms) : 43249, 43982
.   : milestone, 43616,
iast (44.528 ms) : 44131, 44925
.   : milestone, 44528,
profiling (50.022 ms) : 49528, 50517
.   : milestone, 50022,
tracing (44.908 ms) : 44548, 45269
.   : milestone, 44908,
section candidate
no_agent (36.639 ms) : 36342, 36936
.   : milestone, 36639,
appsec (47.064 ms) : 46646, 47481
.   : milestone, 47064,
code_origins (46.025 ms) : 45636, 46414
.   : milestone, 46025,
iast (47.182 ms) : 46747, 47617
.   : milestone, 47182,
profiling (51.075 ms) : 50554, 51596
.   : milestone, 51075,
tracing (44.247 ms) : 43868, 44626
.   : milestone, 44247,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	35.892 ms [35.604 ms, 36.181 ms]	-
appsec	48.025 ms [47.601 ms, 48.45 ms]	12.133 ms (33.8%)
code_origins	43.616 ms [43.249 ms, 43.982 ms]	7.723 ms (21.5%)
iast	44.528 ms [44.131 ms, 44.925 ms]	8.636 ms (24.1%)
profiling	50.022 ms [49.528 ms, 50.517 ms]	14.13 ms (39.4%)
tracing	44.908 ms [44.548 ms, 45.269 ms]	9.016 ms (25.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.639 ms [36.342 ms, 36.936 ms]	-
appsec	47.064 ms [46.646 ms, 47.481 ms]	10.425 ms (28.5%)
code_origins	46.025 ms [45.636 ms, 46.414 ms]	9.386 ms (25.6%)
iast	47.182 ms [46.747 ms, 47.617 ms]	10.543 ms (28.8%)
profiling	51.075 ms [50.554 ms, 51.596 ms]	14.437 ms (39.4%)
tracing	44.247 ms [43.868 ms, 44.626 ms]	7.608 ms (20.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.468 ms) : 4410, 4526
.   : milestone, 4468,
iast (9.094 ms) : 8948, 9240
.   : milestone, 9094,
iast_FULL (13.839 ms) : 13570, 14109
.   : milestone, 13839,
iast_GLOBAL (9.967 ms) : 9777, 10157
.   : milestone, 9967,
profiling (9.357 ms) : 9204, 9509
.   : milestone, 9357,
tracing (7.749 ms) : 7635, 7863
.   : milestone, 7749,
section candidate
no_agent (4.492 ms) : 4439, 4544
.   : milestone, 4492,
iast (9.377 ms) : 9225, 9528
.   : milestone, 9377,
iast_FULL (13.575 ms) : 13302, 13847
.   : milestone, 13575,
iast_GLOBAL (10.084 ms) : 9910, 10259
.   : milestone, 10084,
profiling (8.586 ms) : 8456, 8716
.   : milestone, 8586,
tracing (7.756 ms) : 7645, 7868
.   : milestone, 7756,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.468 ms [4.41 ms, 4.526 ms]	-
iast	9.094 ms [8.948 ms, 9.24 ms]	4.626 ms (103.5%)
iast_FULL	13.839 ms [13.57 ms, 14.109 ms]	9.371 ms (209.7%)
iast_GLOBAL	9.967 ms [9.777 ms, 10.157 ms]	5.499 ms (123.1%)
profiling	9.357 ms [9.204 ms, 9.509 ms]	4.888 ms (109.4%)
tracing	7.749 ms [7.635 ms, 7.863 ms]	3.281 ms (73.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.492 ms [4.439 ms, 4.544 ms]	-
iast	9.377 ms [9.225 ms, 9.528 ms]	4.885 ms (108.8%)
iast_FULL	13.575 ms [13.302 ms, 13.847 ms]	9.083 ms (202.2%)
iast_GLOBAL	10.084 ms [9.91 ms, 10.259 ms]	5.593 ms (124.5%)
profiling	8.586 ms [8.456 ms, 8.716 ms]	4.095 ms (91.2%)
tracing	7.756 ms [7.645 ms, 7.868 ms]	3.265 ms (72.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mcculls/instrumenter-index-experiment
git_commit_date	1750788935	1750806252
git_commit_sha	`faeb62c`	`22f46ba`
release_version	1.51.0-SNAPSHOT~faeb62cfb1	1.51.0-SNAPSHOT~22f46ba16f

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1750808414	1750808414
ci_job_id	997131000	997131000
ci_pipeline_id	68681310	68681310
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-6z7wq9aq-project-304-concurrent-0-09ta9ift 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-6z7wq9aq-project-304-concurrent-0-09ta9ift 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.606 s) : 15606000, 15606000
.   : milestone, 15606000,
appsec (14.84 s) : 14840000, 14840000
.   : milestone, 14840000,
iast (18.382 s) : 18382000, 18382000
.   : milestone, 18382000,
iast_GLOBAL (18.036 s) : 18036000, 18036000
.   : milestone, 18036000,
profiling (15.2 s) : 15200000, 15200000
.   : milestone, 15200000,
tracing (14.56 s) : 14560000, 14560000
.   : milestone, 14560000,
section candidate
no_agent (15.552 s) : 15552000, 15552000
.   : milestone, 15552000,
appsec (14.965 s) : 14965000, 14965000
.   : milestone, 14965000,
iast (18.307 s) : 18307000, 18307000
.   : milestone, 18307000,
iast_GLOBAL (18.052 s) : 18052000, 18052000
.   : milestone, 18052000,
profiling (15.278 s) : 15278000, 15278000
.   : milestone, 15278000,
tracing (14.812 s) : 14812000, 14812000
.   : milestone, 14812000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.606 s [15.606 s, 15.606 s]	-
appsec	14.84 s [14.84 s, 14.84 s]	-766.0 ms (-4.9%)
iast	18.382 s [18.382 s, 18.382 s]	2.776 s (17.8%)
iast_GLOBAL	18.036 s [18.036 s, 18.036 s]	2.43 s (15.6%)
profiling	15.2 s [15.2 s, 15.2 s]	-406.0 ms (-2.6%)
tracing	14.56 s [14.56 s, 14.56 s]	-1.046 s (-6.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.552 s [15.552 s, 15.552 s]	-
appsec	14.965 s [14.965 s, 14.965 s]	-587.0 ms (-3.8%)
iast	18.307 s [18.307 s, 18.307 s]	2.755 s (17.7%)
iast_GLOBAL	18.052 s [18.052 s, 18.052 s]	2.5 s (16.1%)
profiling	15.278 s [15.278 s, 15.278 s]	-274.0 ms (-1.8%)
tracing	14.812 s [14.812 s, 14.812 s]	-740.0 ms (-4.8%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~22f46ba16f, baseline=1.51.0-SNAPSHOT~faeb62cfb1
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.48 ms) : 1468, 1492
.   : milestone, 1480,
appsec (2.412 ms) : 2363, 2461
.   : milestone, 2412,
iast (2.192 ms) : 2131, 2253
.   : milestone, 2192,
iast_GLOBAL (2.242 ms) : 2180, 2304
.   : milestone, 2242,
profiling (2.043 ms) : 1992, 2093
.   : milestone, 2043,
tracing (2.017 ms) : 1970, 2065
.   : milestone, 2017,
section candidate
no_agent (1.488 ms) : 1476, 1499
.   : milestone, 1488,
appsec (2.408 ms) : 2359, 2456
.   : milestone, 2408,
iast (2.194 ms) : 2133, 2255
.   : milestone, 2194,
iast_GLOBAL (2.225 ms) : 2164, 2286
.   : milestone, 2225,
profiling (2.03 ms) : 1981, 2078
.   : milestone, 2030,
tracing (2.011 ms) : 1964, 2059
.   : milestone, 2011,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.48 ms [1.468 ms, 1.492 ms]	-
appsec	2.412 ms [2.363 ms, 2.461 ms]	931.777 µs (63.0%)
iast	2.192 ms [2.131 ms, 2.253 ms]	711.899 µs (48.1%)
iast_GLOBAL	2.242 ms [2.18 ms, 2.304 ms]	762.004 µs (51.5%)
profiling	2.043 ms [1.992 ms, 2.093 ms]	562.716 µs (38.0%)
tracing	2.017 ms [1.97 ms, 2.065 ms]	537.308 µs (36.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.488 ms [1.476 ms, 1.499 ms]	-
appsec	2.408 ms [2.359 ms, 2.456 ms]	919.884 µs (61.8%)
iast	2.194 ms [2.133 ms, 2.255 ms]	706.183 µs (47.5%)
iast_GLOBAL	2.225 ms [2.164 ms, 2.286 ms]	737.346 µs (49.6%)
profiling	2.03 ms [1.981 ms, 2.078 ms]	541.796 µs (36.4%)
tracing	2.011 ms [1.964 ms, 2.059 ms]	523.487 µs (35.2%)

pr-commenter · 2025-06-24T23:43:07Z

Kafka / producer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	mcculls/cleanup-instrumenter-enablement	mcculls/instrumenter-index-experiment
git_commit_date	1750792614	1750806252
git_commit_sha	`0e4a616`	`22f46ba`

See matching parameters

	Baseline	Candidate
ci_job_date	1750807497	1750807497
ci_job_id	997131004	997131004
ci_pipeline_id	68681310	68681310
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.25	11.0.25
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.25+9-post-Ubuntu-1ubuntu122.04	11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce	same

pr-commenter · 2025-06-24T23:56:16Z

Kafka / consumer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	mcculls/cleanup-instrumenter-enablement	mcculls/instrumenter-index-experiment
git_commit_date	1750792614	1750806252
git_commit_sha	`0e4a616`	`22f46ba`

See matching parameters

	Baseline	Candidate
ci_job_date	1750807541	1750807541
ci_job_id	997131006	997131006
ci_pipeline_id	68681310	68681310
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.25	11.0.25
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant	-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.25+9-post-Ubuntu-1ubuntu122.04	11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume	unsure [+1539.435op/s; +15558.679op/s] or [+0.499%; +5.040%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same

github-actions · 2025-10-01T03:19:01Z

This pull request has been marked as stale because it has not had activity over the past quarter. It will be closed in 7 days if no further activity occurs. Feel free to reopen the PR if you are still working on it.

github-actions · 2025-11-01T03:19:20Z

This pull request has been closed because it has not had activity over the past quarter. Feel free to reopen the PR if you are still working on it.

Experiment with recording targetSystem in the instrumenter index

22f46ba

mcculls added the tag: do not merge Do not merge changes label Jun 24, 2025

mcculls changed the base branch from master to mcculls/cleanup-instrumenter-enablement June 24, 2025 23:05

github-actions Bot added the tag: stale Stale pull requests label Oct 1, 2025

github-actions Bot closed this Nov 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: instrumenter index experiment#9031

WIP: instrumenter index experiment#9031
mcculls wants to merge 1 commit intomcculls/cleanup-instrumenter-enablementfrom
mcculls/instrumenter-index-experiment

mcculls commented Jun 24, 2025

Uh oh!

pr-commenter Bot commented Jun 24, 2025 •

edited

Loading

Uh oh!

pr-commenter Bot commented Jun 24, 2025

Uh oh!

pr-commenter Bot commented Jun 24, 2025

Uh oh!

github-actions Bot commented Oct 1, 2025

Uh oh!

github-actions Bot commented Nov 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mcculls commented Jun 24, 2025

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter Bot commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

pr-commenter Bot commented Jun 24, 2025

Kafka / producer-benchmark

Parameters

Summary

Uh oh!

pr-commenter Bot commented Jun 24, 2025

Kafka / consumer-benchmark

Parameters

Summary

Uh oh!

github-actions Bot commented Oct 1, 2025

Uh oh!

github-actions Bot commented Nov 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pr-commenter Bot commented Jun 24, 2025 •

edited

Loading