Fix Synapse 3.0 instrumentation bugs and harden span propagation#10892
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits intomasterfrom Apr 15, 2026
Merged
Conversation
Three deterministic bugs fixed: 1. Key mismatch in PassthruInstrumentation: PR #9422 renamed the context key constant but forgot to update the hardcoded string literal in SynapsePassthruInstrumentation, breaking parent span propagation across the passthru mechanism. 2. http.url test assertion: QueryObfuscator recombines http.url with the query string after obfuscation. The test expected path-only but the actual value correctly includes the query string. 3. Missing peer.service in V1 client spans: SynapseClientDecorator returns relative URIs (no host), so peer.hostname was never set. Added peer hostname/IP/port extraction from NHttpClientConnection. Additionally, hardened the passthru span propagation to read the server span directly from the source connection context rather than relying solely on activeSpan(). SourceHandler.requestReceived() dispatches to a worker thread pool, and while java-concurrent instrumentation normally handles context propagation across ThreadPoolExecutor, the connection- based lookup is more robust. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bm1549
added
type: bug
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1066491
Total [baseline] (11.087 s) : 0, 11087183
Agent [candidate] (1.059 s) : 0, 1059055
Total [candidate] (11.065 s) : 0, 11064785
section appsec
Agent [baseline] (1.247 s) : 0, 1247263
Total [baseline] (11.178 s) : 0, 11177955
Agent [candidate] (1.252 s) : 0, 1252001
Total [candidate] (11.109 s) : 0, 11109499
section iast
Agent [baseline] (1.225 s) : 0, 1225454
Total [baseline] (11.193 s) : 0, 11193190
Agent [candidate] (1.222 s) : 0, 1221723
Total [candidate] (11.239 s) : 0, 11239250
section profiling
Agent [baseline] (1.191 s) : 0, 1191063
Total [baseline] (10.94 s) : 0, 10940156
Agent [candidate] (1.192 s) : 0, 1191715
Total [candidate] (11.054 s) : 0, 11054478
gantt
title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.241 ms) : 0, 1241
crashtracking [candidate] (1.209 ms) : 0, 1209
BytebuddyAgent [baseline] (638.413 ms) : 0, 638413
BytebuddyAgent [candidate] (633.908 ms) : 0, 633908
AgentMeter [baseline] (29.602 ms) : 0, 29602
AgentMeter [candidate] (29.404 ms) : 0, 29404
GlobalTracer [baseline] (250.403 ms) : 0, 250403
GlobalTracer [candidate] (248.798 ms) : 0, 248798
AppSec [baseline] (32.568 ms) : 0, 32568
AppSec [candidate] (32.412 ms) : 0, 32412
Debugger [baseline] (60.205 ms) : 0, 60205
Debugger [candidate] (60.138 ms) : 0, 60138
Remote Config [baseline] (595.3 µs) : 0, 595
Remote Config [candidate] (590.868 µs) : 0, 591
Telemetry [baseline] (8.074 ms) : 0, 8074
Telemetry [candidate] (8.109 ms) : 0, 8109
Flare Poller [baseline] (9.059 ms) : 0, 9059
Flare Poller [candidate] (8.379 ms) : 0, 8379
section appsec
crashtracking [baseline] (1.234 ms) : 0, 1234
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (661.3 ms) : 0, 661300
BytebuddyAgent [candidate] (663.323 ms) : 0, 663323
AgentMeter [baseline] (12.043 ms) : 0, 12043
AgentMeter [candidate] (12.136 ms) : 0, 12136
GlobalTracer [baseline] (248.716 ms) : 0, 248716
GlobalTracer [candidate] (249.957 ms) : 0, 249957
AppSec [baseline] (184.96 ms) : 0, 184960
AppSec [candidate] (185.469 ms) : 0, 185469
Debugger [baseline] (65.571 ms) : 0, 65571
Debugger [candidate] (66.238 ms) : 0, 66238
Remote Config [baseline] (615.537 µs) : 0, 616
Remote Config [candidate] (604.367 µs) : 0, 604
Telemetry [baseline] (8.306 ms) : 0, 8306
Telemetry [candidate] (8.367 ms) : 0, 8367
Flare Poller [baseline] (3.465 ms) : 0, 3465
Flare Poller [candidate] (3.567 ms) : 0, 3567
IAST [baseline] (24.633 ms) : 0, 24633
IAST [candidate] (24.652 ms) : 0, 24652
section iast
crashtracking [baseline] (1.249 ms) : 0, 1249
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (801.158 ms) : 0, 801158
BytebuddyAgent [candidate] (799.26 ms) : 0, 799260
AgentMeter [baseline] (11.333 ms) : 0, 11333
AgentMeter [candidate] (11.322 ms) : 0, 11322
GlobalTracer [baseline] (240.082 ms) : 0, 240082
GlobalTracer [candidate] (239.008 ms) : 0, 239008
AppSec [baseline] (33.861 ms) : 0, 33861
AppSec [candidate] (32.546 ms) : 0, 32546
Debugger [baseline] (58.712 ms) : 0, 58712
Debugger [candidate] (60.186 ms) : 0, 60186
Remote Config [baseline] (526.808 µs) : 0, 527
Remote Config [candidate] (528.437 µs) : 0, 528
Telemetry [baseline] (12.783 ms) : 0, 12783
Telemetry [candidate] (12.196 ms) : 0, 12196
Flare Poller [baseline] (3.457 ms) : 0, 3457
Flare Poller [candidate] (3.605 ms) : 0, 3605
IAST [baseline] (25.933 ms) : 0, 25933
IAST [candidate] (25.746 ms) : 0, 25746
section profiling
ProfilingAgent [baseline] (94.731 ms) : 0, 94731
ProfilingAgent [candidate] (94.232 ms) : 0, 94232
crashtracking [baseline] (1.182 ms) : 0, 1182
crashtracking [candidate] (1.184 ms) : 0, 1184
BytebuddyAgent [baseline] (694.794 ms) : 0, 694794
BytebuddyAgent [candidate] (696.789 ms) : 0, 696789
AgentMeter [baseline] (9.167 ms) : 0, 9167
AgentMeter [candidate] (9.093 ms) : 0, 9093
GlobalTracer [baseline] (208.588 ms) : 0, 208588
GlobalTracer [candidate] (208.255 ms) : 0, 208255
AppSec [baseline] (32.996 ms) : 0, 32996
AppSec [candidate] (32.732 ms) : 0, 32732
Debugger [baseline] (65.945 ms) : 0, 65945
Debugger [candidate] (65.819 ms) : 0, 65819
Remote Config [baseline] (587.797 µs) : 0, 588
Remote Config [candidate] (578.594 µs) : 0, 579
Telemetry [baseline] (7.909 ms) : 0, 7909
Telemetry [candidate] (7.869 ms) : 0, 7869
Flare Poller [baseline] (3.622 ms) : 0, 3622
Flare Poller [candidate] (3.627 ms) : 0, 3627
Profiling [baseline] (95.324 ms) : 0, 95324
Profiling [candidate] (94.809 ms) : 0, 94809
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055878
Total [baseline] (8.859 s) : 0, 8858599
Agent [candidate] (1.053 s) : 0, 1052888
Total [candidate] (8.813 s) : 0, 8812556
section iast
Agent [baseline] (1.225 s) : 0, 1224829
Total [baseline] (9.535 s) : 0, 9534820
Agent [candidate] (1.228 s) : 0, 1228307
Total [candidate] (9.526 s) : 0, 9526320
gantt
title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.252 ms) : 0, 1252
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (634.938 ms) : 0, 634938
BytebuddyAgent [candidate] (631.715 ms) : 0, 631715
AgentMeter [baseline] (29.294 ms) : 0, 29294
AgentMeter [candidate] (29.268 ms) : 0, 29268
GlobalTracer [baseline] (247.431 ms) : 0, 247431
GlobalTracer [candidate] (248.096 ms) : 0, 248096
AppSec [baseline] (32.425 ms) : 0, 32425
AppSec [candidate] (32.303 ms) : 0, 32303
Debugger [baseline] (59.091 ms) : 0, 59091
Debugger [candidate] (58.939 ms) : 0, 58939
Remote Config [baseline] (590.198 µs) : 0, 590
Remote Config [candidate] (587.344 µs) : 0, 587
Telemetry [baseline] (8.028 ms) : 0, 8028
Telemetry [candidate] (8.047 ms) : 0, 8047
Flare Poller [baseline] (6.695 ms) : 0, 6695
Flare Poller [candidate] (6.624 ms) : 0, 6624
section iast
crashtracking [baseline] (1.23 ms) : 0, 1230
crashtracking [candidate] (1.223 ms) : 0, 1223
BytebuddyAgent [baseline] (803.096 ms) : 0, 803096
BytebuddyAgent [candidate] (804.815 ms) : 0, 804815
AgentMeter [baseline] (11.365 ms) : 0, 11365
AgentMeter [candidate] (11.596 ms) : 0, 11596
GlobalTracer [baseline] (238.63 ms) : 0, 238630
GlobalTracer [candidate] (239.602 ms) : 0, 239602
IAST [baseline] (25.75 ms) : 0, 25750
IAST [candidate] (25.935 ms) : 0, 25935
AppSec [baseline] (29.641 ms) : 0, 29641
AppSec [candidate] (31.512 ms) : 0, 31512
Debugger [baseline] (60.814 ms) : 0, 60814
Debugger [candidate] (60.556 ms) : 0, 60556
Remote Config [baseline] (1.147 ms) : 0, 1147
Remote Config [candidate] (1.143 ms) : 0, 1143
Telemetry [baseline] (13.238 ms) : 0, 13238
Telemetry [candidate] (11.879 ms) : 0, 11879
Flare Poller [baseline] (3.556 ms) : 0, 3556
Flare Poller [candidate] (3.443 ms) : 0, 3443
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 5 performance regressions! Performance is the same for 12 metrics, 17 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section baseline
no_agent (17.413 ms) : 17242, 17584
. : milestone, 17413,
appsec (20.056 ms) : 19848, 20263
. : milestone, 20056,
code_origins (17.657 ms) : 17483, 17830
. : milestone, 17657,
iast (18.113 ms) : 17933, 18293
. : milestone, 18113,
profiling (18.285 ms) : 18106, 18464
. : milestone, 18285,
tracing (17.674 ms) : 17498, 17850
. : milestone, 17674,
section candidate
no_agent (18.447 ms) : 18258, 18635
. : milestone, 18447,
appsec (18.481 ms) : 18297, 18665
. : milestone, 18481,
code_origins (18.225 ms) : 18041, 18409
. : milestone, 18225,
iast (17.84 ms) : 17665, 18014
. : milestone, 17840,
profiling (19.149 ms) : 18961, 19337
. : milestone, 19149,
tracing (18.666 ms) : 18479, 18852
. : milestone, 18666,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section baseline
no_agent (1.287 ms) : 1274, 1300
. : milestone, 1287,
iast (3.263 ms) : 3216, 3310
. : milestone, 3263,
iast_FULL (6.119 ms) : 6057, 6182
. : milestone, 6119,
iast_GLOBAL (3.724 ms) : 3660, 3788
. : milestone, 3724,
profiling (2.661 ms) : 2629, 2693
. : milestone, 2661,
tracing (1.91 ms) : 1895, 1926
. : milestone, 1910,
section candidate
no_agent (1.243 ms) : 1231, 1255
. : milestone, 1243,
iast (3.378 ms) : 3328, 3428
. : milestone, 3378,
iast_FULL (6.431 ms) : 6364, 6498
. : milestone, 6431,
iast_GLOBAL (3.782 ms) : 3716, 3848
. : milestone, 3782,
profiling (2.378 ms) : 2353, 2403
. : milestone, 2378,
tracing (1.934 ms) : 1916, 1951
. : milestone, 1934,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section baseline
no_agent (14.79 s) : 14790000, 14790000
. : milestone, 14790000,
appsec (14.413 s) : 14413000, 14413000
. : milestone, 14413000,
iast (18.38 s) : 18380000, 18380000
. : milestone, 18380000,
iast_GLOBAL (18.291 s) : 18291000, 18291000
. : milestone, 18291000,
profiling (15.281 s) : 15281000, 15281000
. : milestone, 15281000,
tracing (15.026 s) : 15026000, 15026000
. : milestone, 15026000,
section candidate
no_agent (14.897 s) : 14897000, 14897000
. : milestone, 14897000,
appsec (14.791 s) : 14791000, 14791000
. : milestone, 14791000,
iast (18.493 s) : 18493000, 18493000
. : milestone, 18493000,
iast_GLOBAL (17.921 s) : 17921000, 17921000
. : milestone, 17921000,
profiling (15.308 s) : 15308000, 15308000
. : milestone, 15308000,
tracing (15.033 s) : 15033000, 15033000
. : milestone, 15033000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~75bcdaadb3, baseline=1.62.0-SNAPSHOT~8d3cec3b17
dateFormat X
axisFormat %s
section baseline
no_agent (1.493 ms) : 1482, 1505
. : milestone, 1493,
appsec (3.858 ms) : 3634, 4082
. : milestone, 3858,
iast (2.291 ms) : 2221, 2362
. : milestone, 2291,
iast_GLOBAL (2.336 ms) : 2265, 2406
. : milestone, 2336,
profiling (2.12 ms) : 2064, 2176
. : milestone, 2120,
tracing (2.093 ms) : 2039, 2147
. : milestone, 2093,
section candidate
no_agent (1.494 ms) : 1482, 1505
. : milestone, 1494,
appsec (3.81 ms) : 3585, 4036
. : milestone, 3810,
iast (2.291 ms) : 2221, 2360
. : milestone, 2291,
iast_GLOBAL (2.338 ms) : 2267, 2408
. : milestone, 2338,
profiling (2.113 ms) : 2057, 2169
. : milestone, 2113,
tracing (2.084 ms) : 2030, 2139
. : milestone, 2084,
|
Contributor
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: acc4558be4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Use DECORATE.onPeerConnection() and setPeerPort() instead of manually setting peer tags. This correctly classifies IPv6 addresses as peer.ipv6 (instead of forcing peer.ipv4) and honors the trace.peer.hostname.enabled config toggle for peer.hostname. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
jordan-wong
approved these changes
Apr 14, 2026
Contributor
jordan-wong
left a comment
jordan-wong
left a comment
There was a problem hiding this comment.
changes deal with failing Synapse tests appropriately. LGTM!
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
gh-worker-dd-mergequeue-cf854d
bot
deleted the
brian.marks/fix-synapse-instrumentation-bugs
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Fixes three deterministic bugs in the Synapse 3.0 instrumentation and hardens span propagation across the passthru mechanism.
Motivation
The Synapse tests have been
@Flakysince 2023. Investigation revealed three deterministic bugs introduced or exposed over time:Bug 1: Key mismatch in PassthruInstrumentation
PR #9422 renamed the context key constant in
SynapseClientDecoratorfrom"dd.trace.synapse.span"to"dd.trace.synapse.context"but forgot to update the hardcoded string literal inSynapsePassthruInstrumentation. The client instrumentation could never find the parent span, breaking trace linkage in proxy scenarios.Fix: Use the
SYNAPSE_CONTEXT_KEYconstant instead of the hardcoded string.Bug 2: http.url test assertion wrong
QueryObfuscator(a tag post-processor) intentionally recombineshttp.url+http.query.stringafter obfuscation. The test expected path-onlyhttp.urlbut the actual value correctly includes the query string -- matching the behavior of all other HTTP server instrumentation tests.Fix: Updated
serverSpanassertion to include the query string inhttp.urlwhen present.Bug 3: Missing peer.service in V1 client spans
SynapseClientDecorator.url()returns relative URIs (no host), soUriBasedClientDecorator.onURI()never setpeer.hostname. Without it, V1 naming (PeerServiceNamingV1) can't computepeer.service.Fix: Added peer hostname/IP/port extraction from
NHttpClientConnectioninClientRequestAdvice, delegating toBaseDecorator.onPeerConnection()which correctly handles IPv4/IPv6 classification and respects thetrace.peer.hostname.enabledconfig toggle.Robustness: Connection-based span propagation
SourceHandler.requestReceived()dispatches request processing to a worker thread pool (NativeWorkerPool->ThreadPoolExecutor). The passthru advice previously relied onactiveSpan()(thread-local) which depends onjava-concurrentinstrumentation propagating context across the thread pool. The fix reads the server span directly from the source connection's context (whereSynapseServerInstrumentationstored it), withactiveSpan()as fallback.Additional Notes
@Flakyannotation removed from the forked test classes since the underlying bugs are fixedContributor Checklist
type:and (comp:orinst:) labelsclose,fix, or any linking keywords when referencing an issueJira ticket: N/A
tag: ai generated
tag: no release note