Refactors environment variables handling to make it testable. by AlexeyKuznetsov-DD · Pull Request #9586 · DataDog/dd-trace-java

AlexeyKuznetsov-DD · 2025-09-22T19:46:57Z

What Does This Do

Refactors environment variables handling to make it testable.

Motivation

Relying directly on environment variables in tests is considered a bad practice.
@PerfectSlayer introduced a shared component, EnvironmentVariables, which is already used across the codebase.

This PR makes EnvironmentVariables testable by introducing a provider that allows configuring initial values for environment variables in tests.

Additional Notes

In one case, I had to initialize environment variables in a static block, because the test expected instrumentation to be auto-enabled by a special variable before instrumentation was loaded.

If class loading order cannot be avoided, I added support for a special environment variable:
TEST_ENV_PROPAGATE_VARS — a comma-separated list of variables that should be propagated from the real environment into the test environment.

This change also removed the need for the JUnit4 @Rule API and the legacy system-rules library.

pr-commenter · 2025-09-22T19:55:56Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	alexeyk/refactored-env-vars-to-be-testable
git_commit_date	1760109324	1760113974
git_commit_sha	`30e2ee8`	`06b8d5d`
release_version	1.55.0-SNAPSHOT~30e2ee8311	1.54.0-SNAPSHOT~06b8d5d7f7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1760115905	1760115905
ci_job_id	1174464441	1174464441
ci_pipeline_id	78994051	78994051
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-4wfgesxa 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-4wfgesxa 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 7 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1020606
Total [baseline] (10.728 s) : 0, 10728466
Agent [candidate] (1.019 s) : 0, 1018639
Total [candidate] (10.72 s) : 0, 10720487
section appsec
Agent [baseline] (1.2 s) : 0, 1200459
Total [baseline] (11.124 s) : 0, 11124409
Agent [candidate] (1.199 s) : 0, 1199012
Total [candidate] (11.029 s) : 0, 11029324
section iast
Agent [baseline] (1.158 s) : 0, 1158433
Total [baseline] (10.994 s) : 0, 10994376
Agent [candidate] (1.155 s) : 0, 1155275
Total [candidate] (11.003 s) : 0, 11002932
section profiling
Agent [baseline] (1.16 s) : 0, 1159735
Total [baseline] (11.063 s) : 0, 11062672
Agent [candidate] (1.181 s) : 0, 1181262
Total [candidate] (11.182 s) : 0, 11181872

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.021 s	-
Agent	appsec	1.2 s	179.853 ms (17.6%)
Agent	iast	1.158 s	137.827 ms (13.5%)
Agent	profiling	1.16 s	139.129 ms (13.6%)
Total	tracing	10.728 s	-
Total	appsec	11.124 s	395.943 ms (3.7%)
Total	iast	10.994 s	265.91 ms (2.5%)
Total	profiling	11.063 s	334.206 ms (3.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.019 s	-
Agent	appsec	1.199 s	180.373 ms (17.7%)
Agent	iast	1.155 s	136.636 ms (13.4%)
Agent	profiling	1.181 s	162.623 ms (16.0%)
Total	tracing	10.72 s	-
Total	appsec	11.029 s	308.837 ms (2.9%)
Total	iast	11.003 s	282.444 ms (2.6%)
Total	profiling	11.182 s	461.384 ms (4.3%)

gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.464 ms) : 0, 1464
crashtracking [candidate] (1.474 ms) : 0, 1474
BytebuddyAgent [baseline] (694.723 ms) : 0, 694723
BytebuddyAgent [candidate] (695.052 ms) : 0, 695052
GlobalTracer [baseline] (242.296 ms) : 0, 242296
GlobalTracer [candidate] (242.63 ms) : 0, 242630
AppSec [baseline] (32.516 ms) : 0, 32516
AppSec [candidate] (32.378 ms) : 0, 32378
Debugger [baseline] (6.406 ms) : 0, 6406
Debugger [candidate] (6.423 ms) : 0, 6423
Remote Config [baseline] (771.164 µs) : 0, 771
Remote Config [candidate] (713.206 µs) : 0, 713
Telemetry [baseline] (9.464 ms) : 0, 9464
Telemetry [candidate] (9.398 ms) : 0, 9398
Flare Poller [baseline] (11.7 ms) : 0, 11700
Flare Poller [candidate] (9.388 ms) : 0, 9388
section appsec
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.464 ms) : 0, 1464
BytebuddyAgent [baseline] (721.928 ms) : 0, 721928
BytebuddyAgent [candidate] (721.413 ms) : 0, 721413
GlobalTracer [baseline] (235.917 ms) : 0, 235917
GlobalTracer [candidate] (235.947 ms) : 0, 235947
IAST [baseline] (24.748 ms) : 0, 24748
IAST [candidate] (24.645 ms) : 0, 24645
AppSec [baseline] (175.999 ms) : 0, 175999
AppSec [candidate] (175.355 ms) : 0, 175355
Debugger [baseline] (6.127 ms) : 0, 6127
Debugger [candidate] (6.096 ms) : 0, 6096
Remote Config [baseline] (637.888 µs) : 0, 638
Remote Config [candidate] (626.732 µs) : 0, 627
Telemetry [baseline] (8.467 ms) : 0, 8467
Telemetry [candidate] (8.438 ms) : 0, 8438
Flare Poller [baseline] (3.976 ms) : 0, 3976
Flare Poller [candidate] (3.897 ms) : 0, 3897
section iast
crashtracking [baseline] (1.466 ms) : 0, 1466
crashtracking [candidate] (1.454 ms) : 0, 1454
BytebuddyAgent [baseline] (821.04 ms) : 0, 821040
BytebuddyAgent [candidate] (818.93 ms) : 0, 818930
GlobalTracer [baseline] (233.02 ms) : 0, 233020
GlobalTracer [candidate] (232.398 ms) : 0, 232398
IAST [baseline] (26.708 ms) : 0, 26708
IAST [candidate] (26.291 ms) : 0, 26291
AppSec [baseline] (35.134 ms) : 0, 35134
AppSec [candidate] (34.982 ms) : 0, 34982
Debugger [baseline] (6.179 ms) : 0, 6179
Debugger [candidate] (6.155 ms) : 0, 6155
Remote Config [baseline] (611.853 µs) : 0, 612
Remote Config [candidate] (621.797 µs) : 0, 622
Telemetry [baseline] (8.596 ms) : 0, 8596
Telemetry [candidate] (8.629 ms) : 0, 8629
Flare Poller [baseline] (4.219 ms) : 0, 4219
Flare Poller [candidate] (4.18 ms) : 0, 4180
section profiling
crashtracking [baseline] (1.437 ms) : 0, 1437
crashtracking [candidate] (1.473 ms) : 0, 1473
BytebuddyAgent [baseline] (720.528 ms) : 0, 720528
BytebuddyAgent [candidate] (732.558 ms) : 0, 732558
GlobalTracer [baseline] (217.125 ms) : 0, 217125
GlobalTracer [candidate] (221.051 ms) : 0, 221051
AppSec [baseline] (32.429 ms) : 0, 32429
AppSec [candidate] (32.792 ms) : 0, 32792
Debugger [baseline] (6.451 ms) : 0, 6451
Debugger [candidate] (8.237 ms) : 0, 8237
Remote Config [baseline] (874.507 µs) : 0, 875
Remote Config [candidate] (796.313 µs) : 0, 796
Telemetry [baseline] (16.116 ms) : 0, 16116
Telemetry [candidate] (14.714 ms) : 0, 14714
Flare Poller [baseline] (4.108 ms) : 0, 4108
Flare Poller [candidate] (4.282 ms) : 0, 4282
ProfilingAgent [baseline] (107.422 ms) : 0, 107422
ProfilingAgent [candidate] (111.339 ms) : 0, 111339
Profiling [baseline] (108.581 ms) : 0, 108581
Profiling [candidate] (112.49 ms) : 0, 112490

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.014 s) : 0, 1013786
Total [baseline] (8.701 s) : 0, 8700789
Agent [candidate] (1.022 s) : 0, 1022413
Total [candidate] (8.718 s) : 0, 8718341
section iast
Agent [baseline] (1.151 s) : 0, 1151068
Total [baseline] (9.348 s) : 0, 9347646
Agent [candidate] (1.151 s) : 0, 1150627
Total [candidate] (9.292 s) : 0, 9292093

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.014 s	-
Agent	iast	1.151 s	137.282 ms (13.5%)
Total	tracing	8.701 s	-
Total	iast	9.348 s	646.857 ms (7.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.022 s	-
Agent	iast	1.151 s	128.215 ms (12.5%)
Total	tracing	8.718 s	-
Total	iast	9.292 s	573.752 ms (6.6%)

gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.472 ms) : 0, 1472
BytebuddyAgent [baseline] (691.994 ms) : 0, 691994
BytebuddyAgent [candidate] (697.296 ms) : 0, 697296
GlobalTracer [baseline] (240.986 ms) : 0, 240986
GlobalTracer [candidate] (243.05 ms) : 0, 243050
AppSec [baseline] (32.231 ms) : 0, 32231
AppSec [candidate] (32.503 ms) : 0, 32503
Debugger [baseline] (6.379 ms) : 0, 6379
Debugger [candidate] (6.476 ms) : 0, 6476
Remote Config [baseline] (710.849 µs) : 0, 711
Remote Config [candidate] (715.085 µs) : 0, 715
Telemetry [baseline] (9.318 ms) : 0, 9318
Telemetry [candidate] (9.401 ms) : 0, 9401
Flare Poller [baseline] (9.552 ms) : 0, 9552
Flare Poller [candidate] (10.191 ms) : 0, 10191
section iast
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.487 ms) : 0, 1487
BytebuddyAgent [baseline] (815.646 ms) : 0, 815646
BytebuddyAgent [candidate] (815.198 ms) : 0, 815198
GlobalTracer [baseline] (231.707 ms) : 0, 231707
GlobalTracer [candidate] (231.252 ms) : 0, 231252
IAST [baseline] (26.3 ms) : 0, 26300
IAST [candidate] (26.352 ms) : 0, 26352
AppSec [baseline] (35.141 ms) : 0, 35141
AppSec [candidate] (35.294 ms) : 0, 35294
Debugger [baseline] (6.137 ms) : 0, 6137
Debugger [candidate] (6.14 ms) : 0, 6140
Remote Config [baseline] (612.473 µs) : 0, 612
Remote Config [candidate] (605.64 µs) : 0, 606
Telemetry [baseline] (8.624 ms) : 0, 8624
Telemetry [candidate] (8.606 ms) : 0, 8606
Flare Poller [baseline] (4.169 ms) : 0, 4169
Flare Poller [candidate] (4.238 ms) : 0, 4238

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	alexeyk/refactored-env-vars-to-be-testable
git_commit_date	1760109324	1760113974
git_commit_sha	`30e2ee8`	`06b8d5d`
release_version	1.55.0-SNAPSHOT~30e2ee8311	1.54.0-SNAPSHOT~06b8d5d7f7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1760115484	1760115484
ci_job_id	1174464442	1174464442
ci_pipeline_id	78994051	78994051
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-i1k4chy7 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-i1k4chy7 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 4 performance regressions! Performance is the same for 6 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:iast_FULL:high_load	worse [+581.826µs; +1189.507µs] or [+4.210%; +8.607%]	unstable [-59.200op/s; +18.700op/s] or [-17.560%; +5.547%]	14.706ms	316.875op/s	13.820ms	337.125op/s
scenario:load:insecure-bank:profiling:high_load	worse [+299.470µs; +610.288µs] or [+3.535%; +7.204%]	unstable [-100.778op/s; +45.653op/s] or [-18.410%; +8.340%]	8.927ms	519.844op/s	8.472ms	547.406op/s
scenario:load:petclinic:code_origins:high_load	worse [+2.239ms; +3.069ms] or [+5.109%; +7.004%]	unstable [-13.648op/s; +1.573op/s] or [-12.785%; +1.474%]	46.478ms	100.713op/s	43.823ms	106.750op/s
scenario:load:petclinic:no_agent:high_load	better [-3.309ms; -2.680ms] or [-8.648%; -7.003%]	unstable [+1.405op/s; +19.220op/s] or [+1.150%; +15.722%]	35.270ms	132.562op/s	38.265ms	122.250op/s
scenario:load:petclinic:appsec:high_load	worse [+1.644ms; +2.597ms] or [+3.365%; +5.316%]	unstable [-10.835op/s; +3.160op/s] or [-11.312%; +3.299%]	50.972ms	91.950op/s	48.852ms	95.787op/s
scenario:load:petclinic:profiling:high_load	better [-2.315ms; -1.301ms] or [-4.616%; -2.595%]	unstable [-3.753op/s; +10.778op/s] or [-4.020%; +11.545%]	48.335ms	96.862op/s	50.143ms	93.350op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
    dateFormat X
    axisFormat %s
section baseline
no_agent (38.265 ms) : 37964, 38565
.   : milestone, 38265,
appsec (48.852 ms) : 48414, 49290
.   : milestone, 48852,
code_origins (43.823 ms) : 43444, 44202
.   : milestone, 43823,
iast (46.497 ms) : 46093, 46901
.   : milestone, 46497,
profiling (50.143 ms) : 49641, 50645
.   : milestone, 50143,
tracing (45.258 ms) : 44869, 45646
.   : milestone, 45258,
section candidate
no_agent (35.27 ms) : 34986, 35554
.   : milestone, 35270,
appsec (50.972 ms) : 50525, 51420
.   : milestone, 50972,
code_origins (46.478 ms) : 46085, 46870
.   : milestone, 46478,
iast (46.368 ms) : 45975, 46761
.   : milestone, 46368,
profiling (48.335 ms) : 47897, 48772
.   : milestone, 48335,
tracing (45.576 ms) : 45182, 45970
.   : milestone, 45576,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	38.265 ms [37.964 ms, 38.565 ms]	-
appsec	48.852 ms [48.414 ms, 49.29 ms]	10.587 ms (27.7%)
code_origins	43.823 ms [43.444 ms, 44.202 ms]	5.559 ms (14.5%)
iast	46.497 ms [46.093 ms, 46.901 ms]	8.232 ms (21.5%)
profiling	50.143 ms [49.641 ms, 50.645 ms]	11.878 ms (31.0%)
tracing	45.258 ms [44.869 ms, 45.646 ms]	6.993 ms (18.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	35.27 ms [34.986 ms, 35.554 ms]	-
appsec	50.972 ms [50.525 ms, 51.42 ms]	15.702 ms (44.5%)
code_origins	46.478 ms [46.085 ms, 46.87 ms]	11.207 ms (31.8%)
iast	46.368 ms [45.975 ms, 46.761 ms]	11.098 ms (31.5%)
profiling	48.335 ms [47.897 ms, 48.772 ms]	13.064 ms (37.0%)
tracing	45.576 ms [45.182 ms, 45.97 ms]	10.306 ms (29.2%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.385 ms) : 4335, 4434
.   : milestone, 4385,
iast (9.307 ms) : 9144, 9470
.   : milestone, 9307,
iast_FULL (13.82 ms) : 13548, 14093
.   : milestone, 13820,
iast_GLOBAL (10.688 ms) : 10494, 10881
.   : milestone, 10688,
profiling (8.472 ms) : 8340, 8604
.   : milestone, 8472,
tracing (7.877 ms) : 7763, 7991
.   : milestone, 7877,
section candidate
no_agent (4.327 ms) : 4273, 4382
.   : milestone, 4327,
iast (9.59 ms) : 9413, 9767
.   : milestone, 9590,
iast_FULL (14.706 ms) : 14415, 14998
.   : milestone, 14706,
iast_GLOBAL (11.112 ms) : 10913, 11312
.   : milestone, 11112,
profiling (8.927 ms) : 8771, 9083
.   : milestone, 8927,
tracing (7.81 ms) : 7696, 7924
.   : milestone, 7810,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.385 ms [4.335 ms, 4.434 ms]	-
iast	9.307 ms [9.144 ms, 9.47 ms]	4.922 ms (112.3%)
iast_FULL	13.82 ms [13.548 ms, 14.093 ms]	9.436 ms (215.2%)
iast_GLOBAL	10.688 ms [10.494 ms, 10.881 ms]	6.303 ms (143.7%)
profiling	8.472 ms [8.34 ms, 8.604 ms]	4.087 ms (93.2%)
tracing	7.877 ms [7.763 ms, 7.991 ms]	3.492 ms (79.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.327 ms [4.273 ms, 4.382 ms]	-
iast	9.59 ms [9.413 ms, 9.767 ms]	5.263 ms (121.6%)
iast_FULL	14.706 ms [14.415 ms, 14.998 ms]	10.379 ms (239.8%)
iast_GLOBAL	11.112 ms [10.913 ms, 11.312 ms]	6.785 ms (156.8%)
profiling	8.927 ms [8.771 ms, 9.083 ms]	4.599 ms (106.3%)
tracing	7.81 ms [7.696 ms, 7.924 ms]	3.483 ms (80.5%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	alexeyk/refactored-env-vars-to-be-testable
git_commit_date	1760109324	1760113974
git_commit_sha	`30e2ee8`	`06b8d5d`
release_version	1.55.0-SNAPSHOT~30e2ee8311	1.54.0-SNAPSHOT~06b8d5d7f7

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1760116062	1760116062
ci_job_id	1174464443	1174464443
ci_pipeline_id	78994051	78994051
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-ogxwka8s 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-ogxwka8s 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:dacapo:tomcat:appsec	better [-1.380ms; -1.045ms] or [-37.461%; -28.364%]	2.471ms	3.683ms

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.901 s) : 14901000, 14901000
.   : milestone, 14901000,
appsec (14.985 s) : 14985000, 14985000
.   : milestone, 14985000,
iast (18.848 s) : 18848000, 18848000
.   : milestone, 18848000,
iast_GLOBAL (17.822 s) : 17822000, 17822000
.   : milestone, 17822000,
profiling (15.314 s) : 15314000, 15314000
.   : milestone, 15314000,
tracing (15.108 s) : 15108000, 15108000
.   : milestone, 15108000,
section candidate
no_agent (15.722 s) : 15722000, 15722000
.   : milestone, 15722000,
appsec (15.057 s) : 15057000, 15057000
.   : milestone, 15057000,
iast (18.64 s) : 18640000, 18640000
.   : milestone, 18640000,
iast_GLOBAL (17.983 s) : 17983000, 17983000
.   : milestone, 17983000,
profiling (15.261 s) : 15261000, 15261000
.   : milestone, 15261000,
tracing (15.35 s) : 15350000, 15350000
.   : milestone, 15350000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.901 s [14.901 s, 14.901 s]	-
appsec	14.985 s [14.985 s, 14.985 s]	84.0 ms (0.6%)
iast	18.848 s [18.848 s, 18.848 s]	3.947 s (26.5%)
iast_GLOBAL	17.822 s [17.822 s, 17.822 s]	2.921 s (19.6%)
profiling	15.314 s [15.314 s, 15.314 s]	413.0 ms (2.8%)
tracing	15.108 s [15.108 s, 15.108 s]	207.0 ms (1.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.722 s [15.722 s, 15.722 s]	-
appsec	15.057 s [15.057 s, 15.057 s]	-665.0 ms (-4.2%)
iast	18.64 s [18.64 s, 18.64 s]	2.918 s (18.6%)
iast_GLOBAL	17.983 s [17.983 s, 17.983 s]	2.261 s (14.4%)
profiling	15.261 s [15.261 s, 15.261 s]	-461.0 ms (-2.9%)
tracing	15.35 s [15.35 s, 15.35 s]	-372.0 ms (-2.4%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~06b8d5d7f7, baseline=1.55.0-SNAPSHOT~30e2ee8311
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.482 ms) : 1471, 1494
.   : milestone, 1482,
appsec (3.683 ms) : 3469, 3897
.   : milestone, 3683,
iast (2.227 ms) : 2164, 2291
.   : milestone, 2227,
iast_GLOBAL (2.263 ms) : 2199, 2326
.   : milestone, 2263,
profiling (2.051 ms) : 2000, 2102
.   : milestone, 2051,
tracing (2.034 ms) : 1985, 2083
.   : milestone, 2034,
section candidate
no_agent (1.48 ms) : 1469, 1492
.   : milestone, 1480,
appsec (2.471 ms) : 2420, 2522
.   : milestone, 2471,
iast (2.221 ms) : 2157, 2284
.   : milestone, 2221,
iast_GLOBAL (2.265 ms) : 2201, 2329
.   : milestone, 2265,
profiling (2.083 ms) : 2031, 2136
.   : milestone, 2083,
tracing (2.022 ms) : 1973, 2072
.   : milestone, 2022,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.482 ms [1.471 ms, 1.494 ms]	-
appsec	3.683 ms [3.469 ms, 3.897 ms]	2.201 ms (148.5%)
iast	2.227 ms [2.164 ms, 2.291 ms]	745.1 µs (50.3%)
iast_GLOBAL	2.263 ms [2.199 ms, 2.326 ms]	780.365 µs (52.6%)
profiling	2.051 ms [2.0 ms, 2.102 ms]	568.623 µs (38.4%)
tracing	2.034 ms [1.985 ms, 2.083 ms]	551.268 µs (37.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.48 ms [1.469 ms, 1.492 ms]	-
appsec	2.471 ms [2.42 ms, 2.522 ms]	990.524 µs (66.9%)
iast	2.221 ms [2.157 ms, 2.284 ms]	740.455 µs (50.0%)
iast_GLOBAL	2.265 ms [2.201 ms, 2.329 ms]	784.367 µs (53.0%)
profiling	2.083 ms [2.031 ms, 2.136 ms]	602.946 µs (40.7%)
tracing	2.022 ms [1.973 ms, 2.072 ms]	541.827 µs (36.6%)

daniel-mohedano

LGTM, some small feedback regarding changes to CIVis tests

jpbempel

LGTM

AlexeyKuznetsov-DD · 2025-09-24T12:03:54Z

After some thoughts I think that better not to mix testing code and library... I will revert some changes.

The main question is what are you trying to make "testable"?

All environment variables that are used as a way to control the "Java Client Library" behavior should never be accessing directly the System.getenv() nor the EnvironmentVariable.get(). All access will be guarded by @mhlidd ConfigHelper, which will have a way to setup test values -- you should discuss with him about his plans for config. There will be rules in ForbiddenApi gradle checks to prevent it.

The remaining access to environment variables should be to control tests, usually related to environment or CI setup. I'm not sure about the need a programmatic way to set them as we are controlling their value through our test setup.

Is there any other environment variable access I miss that must be testable?

@PerfectSlayer That make sense, I will chat with @mhlidd and rework my PR.
My main idea was to replace system-rules library which used to override access to System.getenv with reflection hack (that actually generates a lot of warnings under lates JDKs) and make sure that test can setup env vars without hacking System.getenv().

mhlidd · 2025-09-24T15:38:44Z

All environment variables that are used as a way to control the "Java Client Library" behavior should never be accessing directly the System.getenv() nor the EnvironmentVariable.get(). All access will be guarded by @mhlidd ConfigHelper, which will have a way to setup test values -- you should discuss with him about his plans for config. There will be rules in ForbiddenApi gradle checks to prevent it.

Adding on, but Config Inversion will only restrict the usages of EnvironmentVariables and System.getenv for src/main. Everything is allowed for testing. I don't think @AlexeyKuznetsov-DD and I will have any conflicting work (if anything I want to use the ControllableEnvironmentVariables util for my own Config Inversion tests. As discussed offline, Alexey will take a second look at my PRs to make sure that there is no conflict between our work. :)

pr-commenter · 2025-09-24T22:45:21Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1760114617	1760114963
end_time	2025-10-10T16:44:58	2025-10-10T16:50:46
git_branch	master	alexeyk/refactored-env-vars-to-be-testable
git_commit_sha	`30e2ee8`	`06b8d5d`
start_time	2025-10-10T16:43:38	2025-10-10T16:49:24

See matching parameters

	Baseline	Candidate
ci_job_id	1174464448	1174464448
ci_pipeline_id	78994051	78994051
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1760113974	1760113974

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-23902.409ns; +25863.274ns] or [-8.421%; +9.112%]	unstable [-35643.665ns; +36793.953ns] or [-10.930%; +11.283%]	unstable [-47.415µs; +45.105µs] or [-13.862%; +13.187%]	unstable [+80.708µs; +310.061µs] or [+9.000%; +34.576%]	same
scenario:basic	same	same	same	unstable [+66.101µs; +247.068µs] or [+9.811%; +36.671%]	unstable [-178.891op/s; +178.891op/s] or [-6.798%; +6.798%]
scenario:loop	unsure [+0.771µs; +5.448µs] or [+0.009%; +0.061%]	same	same	same	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (326.107 µs) : 286, 366
.   : milestone, 326,
basic (281.841 µs) : 275, 289
.   : milestone, 282,
loop (8.968 ms) : 8964, 8972
.   : milestone, 8968,
section candidate
noprobe (326.682 µs) : 299, 354
.   : milestone, 327,
basic (279.222 µs) : 273, 285
.   : milestone, 279,
loop (8.981 ms) : 8953, 9009
.   : milestone, 8981,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	326.107 µs [285.839 µs, 366.375 µs]
basic	281.841 µs [274.744 µs, 288.937 µs]
loop	8.968 ms [8.964 ms, 8.972 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	326.682 µs [299.2 µs, 354.164 µs]
basic	279.222 µs [273.457 µs, 284.988 µs]
loop	8.981 ms [8.953 ms, 9.009 ms]

datadog-official · 2025-09-24T22:47:58Z

🎯 Code Coverage
• Patch Coverage: 85.71%
• Total Coverage: 59.46% (-0.42%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 06b8d5d | Docs | Was this helpful? Give us feedback!}

PerfectSlayer · 2025-09-25T06:11:31Z

+    env.set("DD_GIT_COMMIT_SHA", "sha1");
+    env.set("DD_GIT_REPOSITORY_URL", "http://github.com");


🎯 suggestion: ‏This is typically the kind of test that should be migrated to be tested using config mocked value, not environment variables.

AlexeyKuznetsov-DD · 2025-10-06T17:51:32Z

@nikita-tkachenko-datadog, @PerfectSlayer, @bric3 - I've updated the code with your recommendations, please take a look

bric3 · 2025-10-07T15:15:42Z

  static Object checkStaticAssertions() {
    assert System.getProperty("dd.trace.enabled") == null
-    assert System.getenv("DD_TRACE_ENABLED") == null
+    assert EnvironmentVariables.get("DD_TRACE_ENABLED") == null


question: shouldn't we use ControllableEnvironmentVariables for all test instead ?

# Conflicts: # dd-java-agent/instrumentation/aws-java/aws-java-lambda-handler-1.2/build.gradle

AlexeyKuznetsov-DD added 4 commits September 19, 2025 21:58

Refactored EnvironmentVariables to be testable.

401b3cb

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

4003775

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

10d4bdf

WIP.

348f975

AlexeyKuznetsov-DD self-assigned this Sep 22, 2025

AlexeyKuznetsov-DD added type: enhancement Enhancements and improvements comp: testing Testing tag: no release notes Changes to exclude from release notes labels Sep 22, 2025

AlexeyKuznetsov-DD marked this pull request as ready for review September 22, 2025 19:47

AlexeyKuznetsov-DD requested review from a team as code owners September 22, 2025 19:47

AlexeyKuznetsov-DD requested review from a team, PerfectSlayer, bric3, daniel-mohedano, nikita-tkachenko-datadog, sarahchen6 and shatzi and removed request for a team and PerfectSlayer September 22, 2025 19:47

AlexeyKuznetsov-DD changed the title ~~Refactored ENV variables to be testable.~~ Refactors environment variables handling to make it testable. Sep 22, 2025

daniel-mohedano approved these changes Sep 23, 2025

View reviewed changes

jpbempel approved these changes Sep 23, 2025

View reviewed changes

mhlidd requested review from a team as code owners September 24, 2025 22:15

mhlidd requested review from manuel-alvarez-alvarez, mtoffl01 and smola and removed request for a team September 24, 2025 22:15

mhlidd force-pushed the alexeyk/refactored-env-vars-to-be-testable branch from 54f2e28 to 348f975 Compare September 24, 2025 22:24

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

a0be629

PerfectSlayer reviewed Sep 25, 2025

View reviewed changes

AlexeyKuznetsov-DD added 4 commits September 25, 2025 11:19

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

335f9f2

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

cc64b43

Applied review comments.

ca04c39

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

d230400

AlexeyKuznetsov-DD requested review from PerfectSlayer and daniel-mohedano October 6, 2025 17:57

bric3 approved these changes Oct 7, 2025

View reviewed changes

AlexeyKuznetsov-DD added 3 commits October 9, 2025 11:22

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

3e00962

# Conflicts: # dd-java-agent/instrumentation/aws-java/aws-java-lambda-handler-1.2/build.gradle

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

dd97aa0

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

d8b5aec

bric3 approved these changes Oct 10, 2025

View reviewed changes

Merge branch 'master' into alexeyk/refactored-env-vars-to-be-testable

06b8d5d

AlexeyKuznetsov-DD merged commit a9fd9a4 into master Oct 10, 2025
537 checks passed

AlexeyKuznetsov-DD deleted the alexeyk/refactored-env-vars-to-be-testable branch October 10, 2025 17:28

github-actions Bot added this to the 1.55.0 milestone Oct 10, 2025

		env.set("DD_GIT_COMMIT_SHA", "sha1");
		env.set("DD_GIT_REPOSITORY_URL", "http://github.com");

Conversation

AlexeyKuznetsov-DD commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenter Bot commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

daniel-mohedano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jpbempel left a comment

Choose a reason for hiding this comment

Uh oh!

AlexeyKuznetsov-DD commented Sep 24, 2025

Uh oh!

mhlidd commented Sep 24, 2025

Uh oh!

pr-commenter Bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

datadog-official Bot commented Sep 24, 2025 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

AlexeyKuznetsov-DD commented Oct 6, 2025

Uh oh!

bric3 Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AlexeyKuznetsov-DD commented Sep 22, 2025 •

edited

Loading

pr-commenter Bot commented Sep 22, 2025 •

edited

Loading

pr-commenter Bot commented Sep 24, 2025 •

edited

Loading

datadog-official Bot commented Sep 24, 2025 •

edited by datadog-datadog-prod-us1 Bot

Loading