Skip to content

[Repro] Standalone LLMObs sampling drop on top of #10989#11249

Closed
FouadWahabi wants to merge 10 commits intomasterfrom
wahabi/llmobs-standalone-sampling-repro
Closed

[Repro] Standalone LLMObs sampling drop on top of #10989#11249
FouadWahabi wants to merge 10 commits intomasterfrom
wahabi/llmobs-standalone-sampling-repro

Conversation

@FouadWahabi
Copy link
Copy Markdown

@FouadWahabi FouadWahabi commented Apr 30, 2026

What

Adds a failing smoke test on top of #10989 demonstrating that standalone LLMObs spans (no surrounding APM scope) get silently dropped by the new StandaloneSampler.

Why

In DDLLMObsSpan (dd-java-agent/agent-llmobs/.../DDLLMObsSpan.java):

TraceSegment segment = AgentTracer.get().getTraceSegment();
if (segment != null) {
  segment.setTagTop(Tags.PROPAGATED_TRACE_SOURCE, ProductTraceSource.LLMOBS);
}

getTraceSegment() reads activeSpan() (CoreTracer.java:1505-1515). But LLMObsContext.attach(span.context()) does not activate an AgentScope — it writes to its own ContextKey. So when LLMObs is used without a surrounding APM scope (CLI, batch, worker, fresh thread, before any HTTP handler), activeSpan() is null, segment is null, and the LLMOBS bit is never propagated to the local root. StandaloneSampler only checks the root's traceSource bitfield → trace is dropped.

The existing LlmObsApmDisabledSmokeTest does not catch this because every LLMObs call runs inside a Spring servlet handler, so the request span is the active scope and is the local root for the LLMObs trace.

How this repro works

  • New endpoint /rest-api/llmobs/standalone that runs LLMObs.startLLMSpan(...) on a fresh thread (no inherited APM scope).
  • New test asserts the resulting trace:
    • exists,
    • has the LLMOBS bit (0x20 → \"20\") in _dd.p.ts on the root,
    • has SAMPLER_KEEP.

On top of #10989 the test fails because the trace receives SAMPLER_DROP and _dd.p.ts is missing.

Suggested fix

Use the LLMObs span's own context (already a TraceSegment via DDSpanContext) instead of the active-scope segment:

if (span.context() instanceof TraceSegment) {
  ((TraceSegment) span.context()).setTagTop(Tags.PROPAGATED_TRACE_SOURCE, ProductTraceSource.LLMOBS);
}

DDSpanContext.setTagTop already routes through getRootSpanContextOrThis() (DDSpanContext.java:1363), so it correctly marks the local root regardless of which scope is active.

Test plan

  • Run ./gradlew :dd-smoke-tests:apm-tracing-disabled:test --tests \"*standalone LLMObs*\" against this branch — expected to fail.
  • Apply the fix above on top — expected to pass.

matsumo-and and others added 10 commits April 29, 2026 00:43
@matsumo-and
Fix NullPointerException in LLMObsSystem when tracing is disabled
dd7a96c 
Signed-off-by: matsumo-and <yh134.toisanda@gmail.com>
@matsumo-and
Support DD_APM_TRACING_ENABLED=false with DD_LLMOBS_ENABLED=true
c27b597 
Fixes #10051

When DD_APM_TRACING_ENABLED=false is set, APM tracing should be disabled
while allowing other products like LLM Observability to function. Previously,
setting DD_APM_TRACING_ENABLED=false would inadvertently disable LLMObs or
allow APM traces to leak through when no other products were enabled.

Signed-off-by: matsumo-and <yh134.toisanda@gmail.com>
@matsumo-and
Improve LLMObs trace-disabled smoke test
762717e 
Signed-off-by: matsumo-and <yh134.toisanda@gmail.com>
@matsumo-and
format: spotless apply
2cdbdd7 
Signed-off-by: matsumo-and <yh134.toisanda@gmail.com>
@matsumo-and
Handle both LLMObs and ASM enabled in standalone sampler
3df930e 
When APM tracing is disabled but both LLMObs and ASM are enabled,
the previous code returned LlmObsStandaloneSampler which never sent
the 1 APM trace/minute required by ASM for billing and service catalog.

Introduce LlmObsAndAsmStandaloneSampler that keeps all LLMObs and
ASM traces while rate-limiting plain APM traces to 1 per minute.

Also clarify the log message for the ASM-only standalone case.

Signed-off-by: matsumo-and <yh134.toisanda@gmail.com>
@matsumo-and
Unify standalone samplers into a single StandaloneSampler
1a0d979 
Replace the three separate sampler classes (AsmStandaloneSampler,
LlmObsStandaloneSampler, LlmObsAndAsmStandaloneSampler) with a single
StandaloneSampler that iterates over a list of active StandaloneProduct
entries, making it trivially extensible to future products.

Also add ProductTraceSource.isAnyStandaloneProductMarked() to simplify
the TraceCollector force-keep bypass check.

Signed-off-by: matsumo-and <yh134.toisanda@gmail.com>
@matsumo-and
Improve standalone sampler test coverage
e9010e7 
- Replace individual SamplerTest cases with @unroll matrix that covers all
product-flag combinations and asserts activeProducts contents directly;
add package-private getActiveProducts() getter to StandaloneSampler to
enable this
- Add StandaloneSamplerTest case for spans with both LLMOBS and ASM bits set
simultaneously, verifying LLMOBS wins via list ordering
- Add TraceCollectorTest exercising the full span-finish → CoreTracer write
path to verify that setSamplingPriorityIfNecessary skips the sampler when
APM is disabled, a standalone product flag is set, and priority is already
non-UNSET

Signed-off-by: matsumo-and <yh134.toisanda@gmail.com>
@FouadWahabi
Add smoke test reproducing standalone LLMObs sampling drop
aabdb06 
When an LLMObs span is created without an active surrounding APM
AgentScope, DDLLMObsSpan calls AgentTracer.get().getTraceSegment()
which reads activeSpan() — but LLMObsContext.attach does not activate
an AgentScope, so the segment is null and the LLMOBS trace-source bit
is never set on the local root. StandaloneSampler then drops the trace
because it only checks the root's traceSource bitfield.

Add a controller endpoint that creates the LLMObs span on a fresh
thread (no inherited APM scope) and a smoke test that asserts the
standalone LLMObs trace is kept and carries the LLMOBS bit (0x20) in
_dd.p.ts. The test fails on top of the current PR and will pass once
the bit is propagated regardless of active scope.
@FouadWahabi
Implement abstract traceAgentProtocolVersion() in LLMObs smoke tests
0e45599 
Required by AbstractApmTracingDisabledSmokeTest. Without this override,
the smoke test sources fail to compile.
@FouadWahabi
Add braces to single-statement if blocks in SamplerTest
f4dbbf9 
CodeNarc IfStatementBraces rule requires braces even for single
statements.
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Apr 30, 2026

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master wahabi/llmobs-standalone-sampling-repro
git_commit_date 1777386837 1777574060
git_commit_sha e6cac64 f4dbbf9
release_version 1.62.0-SNAPSHOT~e6cac64dfd 1.62.0-SNAPSHOT~f4dbbf9945
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1777575798 1777575798
ci_job_id 1647548536 1647548536
ci_pipeline_id 110780158 110780158
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-wzv6fx18 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-wzv6fx18 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 11 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:petclinic:profiling:Remote Config better
[-42.972µs; -14.052µs] or [-7.123%; -2.329%]		 574.793µs 603.305µs
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1064377
Total [baseline] (8.841 s) : 0, 8841160
Agent [candidate] (1.065 s) : 0, 1064691
Total [candidate] (8.842 s) : 0, 8842147
section iast
Agent [baseline] (1.244 s) : 0, 1244463
Total [baseline] (9.497 s) : 0, 9496974
Agent [candidate] (1.239 s) : 0, 1239490
Total [candidate] (9.489 s) : 0, 9489254
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent iast 1.244 s 180.085 ms (16.9%)
Total tracing 8.841 s -
Total iast 9.497 s 655.814 ms (7.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.065 s -
Agent iast 1.239 s 174.798 ms (16.4%)
Total tracing 8.842 s -
Total iast 9.489 s 647.107 ms (7.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.241 ms) : 0, 1241
crashtracking [candidate] (1.222 ms) : 0, 1222
BytebuddyAgent [baseline] (635.868 ms) : 0, 635868
BytebuddyAgent [candidate] (635.128 ms) : 0, 635128
AgentMeter [baseline] (29.48 ms) : 0, 29480
AgentMeter [candidate] (29.393 ms) : 0, 29393
GlobalTracer [baseline] (249.399 ms) : 0, 249399
GlobalTracer [candidate] (249.563 ms) : 0, 249563
AppSec [baseline] (32.76 ms) : 0, 32760
AppSec [candidate] (33.007 ms) : 0, 33007
Debugger [baseline] (59.959 ms) : 0, 59959
Debugger [candidate] (60.244 ms) : 0, 60244
Remote Config [baseline] (602.938 µs) : 0, 603
Remote Config [candidate] (618.222 µs) : 0, 618
Telemetry [baseline] (8.389 ms) : 0, 8389
Telemetry [candidate] (9.87 ms) : 0, 9870
Flare Poller [baseline] (10.636 ms) : 0, 10636
Flare Poller [candidate] (9.723 ms) : 0, 9723
section iast
crashtracking [baseline] (1.232 ms) : 0, 1232
crashtracking [candidate] (1.215 ms) : 0, 1215
BytebuddyAgent [baseline] (823.708 ms) : 0, 823708
BytebuddyAgent [candidate] (820.62 ms) : 0, 820620
AgentMeter [baseline] (11.284 ms) : 0, 11284
AgentMeter [candidate] (11.276 ms) : 0, 11276
GlobalTracer [baseline] (237.865 ms) : 0, 237865
GlobalTracer [candidate] (237.198 ms) : 0, 237198
AppSec [baseline] (31.376 ms) : 0, 31376
AppSec [candidate] (30.428 ms) : 0, 30428
Debugger [baseline] (62.467 ms) : 0, 62467
Debugger [candidate] (61.989 ms) : 0, 61989
Remote Config [baseline] (536.068 µs) : 0, 536
Remote Config [candidate] (519.007 µs) : 0, 519
Telemetry [baseline] (7.97 ms) : 0, 7970
Telemetry [candidate] (7.867 ms) : 0, 7867
Flare Poller [baseline] (3.351 ms) : 0, 3351
Flare Poller [candidate] (3.343 ms) : 0, 3343
IAST [baseline] (28.612 ms) : 0, 28612
IAST [candidate] (29.05 ms) : 0, 29050
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1062541
Total [baseline] (11.002 s) : 0, 11001881
Agent [candidate] (1.07 s) : 0, 1070103
Total [candidate] (11.029 s) : 0, 11029254
section appsec
Agent [baseline] (1.266 s) : 0, 1265796
Total [baseline] (11.022 s) : 0, 11021523
Agent [candidate] (1.266 s) : 0, 1265670
Total [candidate] (11.072 s) : 0, 11071694
section iast
Agent [baseline] (1.247 s) : 0, 1247458
Total [baseline] (11.319 s) : 0, 11319043
Agent [candidate] (1.254 s) : 0, 1253852
Total [candidate] (11.313 s) : 0, 11313446
section profiling
Agent [baseline] (1.198 s) : 0, 1198065
Total [baseline] (10.958 s) : 0, 10957704
Agent [candidate] (1.192 s) : 0, 1192250
Total [candidate] (11.033 s) : 0, 11033130
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.063 s -
Agent appsec 1.266 s 203.255 ms (19.1%)
Agent iast 1.247 s 184.917 ms (17.4%)
Agent profiling 1.198 s 135.524 ms (12.8%)
Total tracing 11.002 s -
Total appsec 11.022 s 19.642 ms (0.2%)
Total iast 11.319 s 317.162 ms (2.9%)
Total profiling 10.958 s -44.177 ms (-0.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.07 s -
Agent appsec 1.266 s 195.567 ms (18.3%)
Agent iast 1.254 s 183.749 ms (17.2%)
Agent profiling 1.192 s 122.147 ms (11.4%)
Total tracing 11.029 s -
Total appsec 11.072 s 42.44 ms (0.4%)
Total iast 11.313 s 284.192 ms (2.6%)
Total profiling 11.033 s 3.876 ms (0.0%) 
gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.221 ms) : 0, 1221
crashtracking [candidate] (1.225 ms) : 0, 1225
BytebuddyAgent [baseline] (634.362 ms) : 0, 634362
BytebuddyAgent [candidate] (639.534 ms) : 0, 639534
AgentMeter [baseline] (29.378 ms) : 0, 29378
AgentMeter [candidate] (29.651 ms) : 0, 29651
GlobalTracer [baseline] (248.631 ms) : 0, 248631
GlobalTracer [candidate] (250.516 ms) : 0, 250516
AppSec [baseline] (32.662 ms) : 0, 32662
AppSec [candidate] (33.071 ms) : 0, 33071
Debugger [baseline] (60.595 ms) : 0, 60595
Debugger [candidate] (61.085 ms) : 0, 61085
Remote Config [baseline] (605.598 µs) : 0, 606
Remote Config [candidate] (598.971 µs) : 0, 599
Telemetry [baseline] (9.974 ms) : 0, 9974
Telemetry [candidate] (8.43 ms) : 0, 8430
Flare Poller [baseline] (9.125 ms) : 0, 9125
Flare Poller [candidate] (9.844 ms) : 0, 9844
section appsec
crashtracking [baseline] (1.232 ms) : 0, 1232
crashtracking [candidate] (1.217 ms) : 0, 1217
BytebuddyAgent [baseline] (675.632 ms) : 0, 675632
BytebuddyAgent [candidate] (675.394 ms) : 0, 675394
AgentMeter [baseline] (12.195 ms) : 0, 12195
AgentMeter [candidate] (12.24 ms) : 0, 12240
GlobalTracer [baseline] (249.566 ms) : 0, 249566
GlobalTracer [candidate] (249.053 ms) : 0, 249053
IAST [baseline] (24.637 ms) : 0, 24637
IAST [candidate] (24.636 ms) : 0, 24636
AppSec [baseline] (185.056 ms) : 0, 185056
AppSec [candidate] (185.061 ms) : 0, 185061
Debugger [baseline] (64.6 ms) : 0, 64600
Debugger [candidate] (64.647 ms) : 0, 64647
Remote Config [baseline] (559.433 µs) : 0, 559
Remote Config [candidate] (602.596 µs) : 0, 603
Telemetry [baseline] (7.758 ms) : 0, 7758
Telemetry [candidate] (7.835 ms) : 0, 7835
Flare Poller [baseline] (8.566 ms) : 0, 8566
Flare Poller [candidate] (7.821 ms) : 0, 7821
section iast
crashtracking [baseline] (1.226 ms) : 0, 1226
crashtracking [candidate] (1.237 ms) : 0, 1237
BytebuddyAgent [baseline] (825.029 ms) : 0, 825029
BytebuddyAgent [candidate] (829.294 ms) : 0, 829294
AgentMeter [baseline] (11.324 ms) : 0, 11324
AgentMeter [candidate] (11.424 ms) : 0, 11424
GlobalTracer [baseline] (238.355 ms) : 0, 238355
GlobalTracer [candidate] (239.415 ms) : 0, 239415
IAST [baseline] (28.267 ms) : 0, 28267
IAST [candidate] (28.322 ms) : 0, 28322
AppSec [baseline] (31.655 ms) : 0, 31655
AppSec [candidate] (31.808 ms) : 0, 31808
Debugger [baseline] (63.471 ms) : 0, 63471
Debugger [candidate] (64.021 ms) : 0, 64021
Remote Config [baseline] (531.124 µs) : 0, 531
Remote Config [candidate] (541.071 µs) : 0, 541
Telemetry [baseline] (7.987 ms) : 0, 7987
Telemetry [candidate] (8.099 ms) : 0, 8099
Flare Poller [baseline] (3.439 ms) : 0, 3439
Flare Poller [candidate] (3.534 ms) : 0, 3534
section profiling
crashtracking [baseline] (1.188 ms) : 0, 1188
crashtracking [candidate] (1.184 ms) : 0, 1184
BytebuddyAgent [baseline] (699.053 ms) : 0, 699053
BytebuddyAgent [candidate] (696.85 ms) : 0, 696850
AgentMeter [baseline] (8.979 ms) : 0, 8979
AgentMeter [candidate] (8.976 ms) : 0, 8976
GlobalTracer [baseline] (210.799 ms) : 0, 210799
GlobalTracer [candidate] (209.364 ms) : 0, 209364
AppSec [baseline] (33.52 ms) : 0, 33520
AppSec [candidate] (32.693 ms) : 0, 32693
Debugger [baseline] (66.677 ms) : 0, 66677
Debugger [candidate] (65.486 ms) : 0, 65486
Remote Config [baseline] (603.305 µs) : 0, 603
Remote Config [candidate] (574.793 µs) : 0, 575
Telemetry [baseline] (8.181 ms) : 0, 8181
Telemetry [candidate] (8.116 ms) : 0, 8116
Flare Poller [baseline] (3.564 ms) : 0, 3564
Flare Poller [candidate] (3.549 ms) : 0, 3549
ProfilingAgent [baseline] (93.907 ms) : 0, 93907
ProfilingAgent [candidate] (93.674 ms) : 0, 93674
Profiling [baseline] (94.484 ms) : 0, 94484
Profiling [candidate] (94.224 ms) : 0, 94224
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master wahabi/llmobs-standalone-sampling-repro
git_commit_date 1777386837 1777574060
git_commit_sha e6cac64 f4dbbf9
release_version 1.62.0-SNAPSHOT~e6cac64dfd 1.62.0-SNAPSHOT~f4dbbf9945
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1777576284 1777576284
ci_job_id 1647548537 1647548537
ci_pipeline_id 110780158 110780158
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-ellqqdak 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-ellqqdak 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 1 performance regressions! Performance is the same for 18 metrics, 15 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast:high_load better
[-163.904µs; -63.185µs] or [-6.111%; -2.356%]		 same
[-481.347µs; +57.072µs] or [-6.206%; +0.736%]		 unstable
[-100.848op/s; +195.910op/s] or [-7.538%; +14.644%]		 2.569ms 7.543ms 1385.344op/s 2.682ms 7.756ms 1337.812op/s
scenario:load:petclinic:no_agent:high_load better
[-1.763ms; -0.528ms] or [-9.512%; -2.846%]		 unsure
[-2.516ms; -0.165ms] or [-8.256%; -0.541%]		 unstable
[-13.854op/s; +40.354op/s] or [-5.643%; +16.437%]		 17.393ms 29.134ms 258.750op/s 18.538ms 30.475ms 245.500op/s
scenario:load:petclinic:iast:high_load worse
[+0.585ms; +1.736ms] or [+3.327%; +9.861%]		 unsure
[+0.407ms; +2.271ms] or [+1.426%; +7.949%]		 unstable
[-40.418op/s; +13.668op/s] or [-15.530%; +5.252%]		 18.760ms 29.914ms 246.875op/s 17.599ms 28.574ms 260.250op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.321 ms) : 1309, 1333
.   : milestone, 1321,
iast (3.424 ms) : 3378, 3470
.   : milestone, 3424,
iast_FULL (6.008 ms) : 5947, 6068
.   : milestone, 6008,
iast_GLOBAL (3.645 ms) : 3584, 3707
.   : milestone, 3645,
profiling (2.152 ms) : 2131, 2174
.   : milestone, 2152,
tracing (1.862 ms) : 1847, 1877
.   : milestone, 1862,
section candidate
no_agent (1.25 ms) : 1239, 1261
.   : milestone, 1250,
iast (3.304 ms) : 3259, 3349
.   : milestone, 3304,
iast_FULL (6.01 ms) : 5950, 6070
.   : milestone, 6010,
iast_GLOBAL (3.753 ms) : 3687, 3818
.   : milestone, 3753,
profiling (2.164 ms) : 2145, 2183
.   : milestone, 2164,
tracing (1.953 ms) : 1934, 1972
.   : milestone, 1953,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.321 ms [1.309 ms, 1.333 ms] -
iast 3.424 ms [3.378 ms, 3.47 ms] 2.103 ms (159.2%)
iast_FULL 6.008 ms [5.947 ms, 6.068 ms] 4.687 ms (354.8%)
iast_GLOBAL 3.645 ms [3.584 ms, 3.707 ms] 2.324 ms (176.0%)
profiling 2.152 ms [2.131 ms, 2.174 ms] 831.487 µs (62.9%)
tracing 1.862 ms [1.847 ms, 1.877 ms] 540.795 µs (40.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.25 ms [1.239 ms, 1.261 ms] -
iast 3.304 ms [3.259 ms, 3.349 ms] 2.054 ms (164.3%)
iast_FULL 6.01 ms [5.95 ms, 6.07 ms] 4.76 ms (380.8%)
iast_GLOBAL 3.753 ms [3.687 ms, 3.818 ms] 2.503 ms (200.2%)
profiling 2.164 ms [2.145 ms, 2.183 ms] 913.807 µs (73.1%)
tracing 1.953 ms [1.934 ms, 1.972 ms] 702.902 µs (56.2%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.012 ms) : 18822, 19202
.   : milestone, 19012,
appsec (18.773 ms) : 18585, 18960
.   : milestone, 18773,
code_origins (18.01 ms) : 17833, 18186
.   : milestone, 18010,
iast (17.928 ms) : 17753, 18103
.   : milestone, 17928,
profiling (18.503 ms) : 18322, 18684
.   : milestone, 18503,
tracing (18.157 ms) : 17980, 18335
.   : milestone, 18157,
section candidate
no_agent (18.037 ms) : 17855, 18219
.   : milestone, 18037,
appsec (18.792 ms) : 18604, 18981
.   : milestone, 18792,
code_origins (17.779 ms) : 17603, 17955
.   : milestone, 17779,
iast (18.904 ms) : 18711, 19097
.   : milestone, 18904,
profiling (18.355 ms) : 18172, 18537
.   : milestone, 18355,
tracing (17.725 ms) : 17552, 17898
.   : milestone, 17725,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.012 ms [18.822 ms, 19.202 ms] -
appsec 18.773 ms [18.585 ms, 18.96 ms] -239.36 µs (-1.3%)
code_origins 18.01 ms [17.833 ms, 18.186 ms] -1.003 ms (-5.3%)
iast 17.928 ms [17.753 ms, 18.103 ms] -1.084 ms (-5.7%)
profiling 18.503 ms [18.322 ms, 18.684 ms] -508.786 µs (-2.7%)
tracing 18.157 ms [17.98 ms, 18.335 ms] -854.888 µs (-4.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.037 ms [17.855 ms, 18.219 ms] -
appsec 18.792 ms [18.604 ms, 18.981 ms] 755.386 µs (4.2%)
code_origins 17.779 ms [17.603 ms, 17.955 ms] -258.11 µs (-1.4%)
iast 18.904 ms [18.711 ms, 19.097 ms] 866.799 µs (4.8%)
profiling 18.355 ms [18.172 ms, 18.537 ms] 317.596 µs (1.8%)
tracing 17.725 ms [17.552 ms, 17.898 ms] -311.666 µs (-1.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master wahabi/llmobs-standalone-sampling-repro
git_commit_date 1777386837 1777574060
git_commit_sha e6cac64 f4dbbf9
release_version 1.62.0-SNAPSHOT~e6cac64dfd 1.62.0-SNAPSHOT~f4dbbf9945
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1777576102 1777576102
ci_job_id 1647548543 1647548543
ci_pipeline_id 110780158 110780158
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-idgqukp4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-idgqukp4 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.83 s) : 14830000, 14830000
.   : milestone, 14830000,
appsec (15.151 s) : 15151000, 15151000
.   : milestone, 15151000,
iast (18.167 s) : 18167000, 18167000
.   : milestone, 18167000,
iast_GLOBAL (17.859 s) : 17859000, 17859000
.   : milestone, 17859000,
profiling (14.756 s) : 14756000, 14756000
.   : milestone, 14756000,
tracing (15.128 s) : 15128000, 15128000
.   : milestone, 15128000,
section candidate
no_agent (15.549 s) : 15549000, 15549000
.   : milestone, 15549000,
appsec (15.095 s) : 15095000, 15095000
.   : milestone, 15095000,
iast (18.526 s) : 18526000, 18526000
.   : milestone, 18526000,
iast_GLOBAL (18.081 s) : 18081000, 18081000
.   : milestone, 18081000,
profiling (15.514 s) : 15514000, 15514000
.   : milestone, 15514000,
tracing (14.751 s) : 14751000, 14751000
.   : milestone, 14751000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.83 s [14.83 s, 14.83 s] -
appsec 15.151 s [15.151 s, 15.151 s] 321.0 ms (2.2%)
iast 18.167 s [18.167 s, 18.167 s] 3.337 s (22.5%)
iast_GLOBAL 17.859 s [17.859 s, 17.859 s] 3.029 s (20.4%)
profiling 14.756 s [14.756 s, 14.756 s] -74.0 ms (-0.5%)
tracing 15.128 s [15.128 s, 15.128 s] 298.0 ms (2.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.549 s [15.549 s, 15.549 s] -
appsec 15.095 s [15.095 s, 15.095 s] -454.0 ms (-2.9%)
iast 18.526 s [18.526 s, 18.526 s] 2.977 s (19.1%)
iast_GLOBAL 18.081 s [18.081 s, 18.081 s] 2.532 s (16.3%)
profiling 15.514 s [15.514 s, 15.514 s] -35.0 ms (-0.2%)
tracing 14.751 s [14.751 s, 14.751 s] -798.0 ms (-5.1%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~f4dbbf9945, baseline=1.62.0-SNAPSHOT~e6cac64dfd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.487 ms) : 1476, 1499
.   : milestone, 1487,
appsec (3.805 ms) : 3583, 4026
.   : milestone, 3805,
iast (2.274 ms) : 2204, 2343
.   : milestone, 2274,
iast_GLOBAL (2.315 ms) : 2246, 2385
.   : milestone, 2315,
profiling (2.089 ms) : 2035, 2144
.   : milestone, 2089,
tracing (2.072 ms) : 2019, 2125
.   : milestone, 2072,
section candidate
no_agent (1.488 ms) : 1476, 1499
.   : milestone, 1488,
appsec (3.84 ms) : 3616, 4064
.   : milestone, 3840,
iast (2.272 ms) : 2202, 2341
.   : milestone, 2272,
iast_GLOBAL (2.321 ms) : 2251, 2391
.   : milestone, 2321,
profiling (2.091 ms) : 2037, 2145
.   : milestone, 2091,
tracing (2.074 ms) : 2021, 2127
.   : milestone, 2074,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.487 ms [1.476 ms, 1.499 ms] -
appsec 3.805 ms [3.583 ms, 4.026 ms] 2.318 ms (155.9%)
iast 2.274 ms [2.204 ms, 2.343 ms] 786.632 µs (52.9%)
iast_GLOBAL 2.315 ms [2.246 ms, 2.385 ms] 828.373 µs (55.7%)
profiling 2.089 ms [2.035 ms, 2.144 ms] 601.96 µs (40.5%)
tracing 2.072 ms [2.019 ms, 2.125 ms] 584.896 µs (39.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.488 ms [1.476 ms, 1.499 ms] -
appsec 3.84 ms [3.616 ms, 4.064 ms] 2.352 ms (158.1%)
iast 2.272 ms [2.202 ms, 2.341 ms] 783.998 µs (52.7%)
iast_GLOBAL 2.321 ms [2.251 ms, 2.391 ms] 833.726 µs (56.0%)
profiling 2.091 ms [2.037 ms, 2.145 ms] 603.394 µs (40.6%)
tracing 2.074 ms [2.021 ms, 2.127 ms] 586.436 µs (39.4%)

@FouadWahabi FouadWahabi mentioned this pull request May 1, 2026
Open
5 tasks
@FouadWahabi FouadWahabi closed this May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@FouadWahabi @matsumo-and