fix: SNS v2 batch injection bug#11190
fix: SNS v2 batch injection bug#11190gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits intomasterfrom
Conversation
…dropped when number of attributes exceeded limits
| .region(Region.of("us-east-1")) | ||
| .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("test", "test"))) | ||
| .build() | ||
| .endpointOverride(URI.create(endPoint)) |
There was a problem hiding this comment.
These parts and below were auto-formatted on commit
71e79cc to
c1725ee
Compare
| public class SnsInterceptorTest { | ||
|
|
||
| @Test | ||
| void publishBatchPreservesEntriesAndOnlyInjectsBelowTheMessageAttributeLimit() { |
There was a problem hiding this comment.
New test that reproduces the issue
| } | ||
|
|
||
| @Test | ||
| void publishPreservesReadonlyAttributesWhileAddingDatadogContext() { |
There was a problem hiding this comment.
Migrated from Spock to JUnit
…to SnsInterceptorTest
a63cb8c to
f5bcaf7
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 61 metrics, 10 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1056429
Total [baseline] (8.856 s) : 0, 8855732
Agent [candidate] (1.055 s) : 0, 1055168
Total [candidate] (8.825 s) : 0, 8825416
section iast
Agent [baseline] (1.239 s) : 0, 1239404
Total [baseline] (9.583 s) : 0, 9582524
Agent [candidate] (1.237 s) : 0, 1236866
Total [candidate] (9.558 s) : 0, 9557993
gantt
title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.241 ms) : 0, 1241
crashtracking [candidate] (1.222 ms) : 0, 1222
BytebuddyAgent [baseline] (632.965 ms) : 0, 632965
BytebuddyAgent [candidate] (632.147 ms) : 0, 632147
AgentMeter [baseline] (29.716 ms) : 0, 29716
AgentMeter [candidate] (29.486 ms) : 0, 29486
GlobalTracer [baseline] (249.236 ms) : 0, 249236
GlobalTracer [candidate] (248.367 ms) : 0, 248367
AppSec [baseline] (32.536 ms) : 0, 32536
AppSec [candidate] (32.286 ms) : 0, 32286
Debugger [baseline] (59.126 ms) : 0, 59126
Debugger [candidate] (58.73 ms) : 0, 58730
Remote Config [baseline] (615.217 µs) : 0, 615
Remote Config [candidate] (578.341 µs) : 0, 578
Telemetry [baseline] (8.031 ms) : 0, 8031
Telemetry [candidate] (8.723 ms) : 0, 8723
Flare Poller [baseline] (6.681 ms) : 0, 6681
Flare Poller [candidate] (7.514 ms) : 0, 7514
section iast
crashtracking [baseline] (1.244 ms) : 0, 1244
crashtracking [candidate] (1.224 ms) : 0, 1224
BytebuddyAgent [baseline] (815.369 ms) : 0, 815369
BytebuddyAgent [candidate] (814.216 ms) : 0, 814216
AgentMeter [baseline] (11.503 ms) : 0, 11503
AgentMeter [candidate] (11.518 ms) : 0, 11518
GlobalTracer [baseline] (240.171 ms) : 0, 240171
GlobalTracer [candidate] (239.459 ms) : 0, 239459
AppSec [baseline] (28.574 ms) : 0, 28574
AppSec [candidate] (26.688 ms) : 0, 26688
Debugger [baseline] (64.436 ms) : 0, 64436
Debugger [candidate] (64.236 ms) : 0, 64236
Remote Config [baseline] (533.815 µs) : 0, 534
Remote Config [candidate] (531.26 µs) : 0, 531
Telemetry [baseline] (7.786 ms) : 0, 7786
Telemetry [candidate] (7.742 ms) : 0, 7742
Flare Poller [baseline] (3.471 ms) : 0, 3471
Flare Poller [candidate] (3.429 ms) : 0, 3429
IAST [baseline] (30.129 ms) : 0, 30129
IAST [candidate] (30.84 ms) : 0, 30840
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1063768
Total [baseline] (11.07 s) : 0, 11070495
Agent [candidate] (1.057 s) : 0, 1057179
Total [candidate] (11.039 s) : 0, 11039129
section appsec
Agent [baseline] (1.264 s) : 0, 1263554
Total [baseline] (11.086 s) : 0, 11085828
Agent [candidate] (1.264 s) : 0, 1263527
Total [candidate] (11.061 s) : 0, 11061137
section iast
Agent [baseline] (1.236 s) : 0, 1235694
Total [baseline] (11.355 s) : 0, 11355496
Agent [candidate] (1.239 s) : 0, 1239406
Total [candidate] (11.318 s) : 0, 11317687
section profiling
Agent [baseline] (1.187 s) : 0, 1187096
Total [baseline] (10.942 s) : 0, 10941982
Agent [candidate] (1.189 s) : 0, 1189220
Total [candidate] (10.93 s) : 0, 10929985
gantt
title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.231 ms) : 0, 1231
crashtracking [candidate] (1.228 ms) : 0, 1228
BytebuddyAgent [baseline] (634.08 ms) : 0, 634080
BytebuddyAgent [candidate] (631.476 ms) : 0, 631476
AgentMeter [baseline] (29.89 ms) : 0, 29890
AgentMeter [candidate] (29.587 ms) : 0, 29587
GlobalTracer [baseline] (251.021 ms) : 0, 251021
GlobalTracer [candidate] (248.771 ms) : 0, 248771
AppSec [baseline] (32.703 ms) : 0, 32703
AppSec [candidate] (32.314 ms) : 0, 32314
Debugger [baseline] (60.467 ms) : 0, 60467
Debugger [candidate] (59.983 ms) : 0, 59983
Remote Config [baseline] (595.65 µs) : 0, 596
Remote Config [candidate] (590.776 µs) : 0, 591
Telemetry [baseline] (8.92 ms) : 0, 8920
Telemetry [candidate] (8.037 ms) : 0, 8037
Flare Poller [baseline] (8.593 ms) : 0, 8593
Flare Poller [candidate] (9.093 ms) : 0, 9093
section appsec
crashtracking [baseline] (1.25 ms) : 0, 1250
crashtracking [candidate] (1.211 ms) : 0, 1211
BytebuddyAgent [baseline] (673.919 ms) : 0, 673919
BytebuddyAgent [candidate] (675.189 ms) : 0, 675189
AgentMeter [baseline] (12.187 ms) : 0, 12187
AgentMeter [candidate] (12.172 ms) : 0, 12172
GlobalTracer [baseline] (249.861 ms) : 0, 249861
GlobalTracer [candidate] (249.566 ms) : 0, 249566
IAST [baseline] (24.41 ms) : 0, 24410
IAST [candidate] (24.34 ms) : 0, 24340
AppSec [baseline] (186.328 ms) : 0, 186328
AppSec [candidate] (186.933 ms) : 0, 186933
Debugger [baseline] (67.187 ms) : 0, 67187
Debugger [candidate] (65.9 ms) : 0, 65900
Remote Config [baseline] (577.236 µs) : 0, 577
Remote Config [candidate] (579.75 µs) : 0, 580
Telemetry [baseline] (7.945 ms) : 0, 7945
Telemetry [candidate] (7.868 ms) : 0, 7868
Flare Poller [baseline] (3.489 ms) : 0, 3489
Flare Poller [candidate] (3.497 ms) : 0, 3497
section iast
crashtracking [baseline] (1.24 ms) : 0, 1240
crashtracking [candidate] (1.233 ms) : 0, 1233
BytebuddyAgent [baseline] (811.367 ms) : 0, 811367
BytebuddyAgent [candidate] (814.811 ms) : 0, 814811
AgentMeter [baseline] (11.439 ms) : 0, 11439
AgentMeter [candidate] (11.484 ms) : 0, 11484
GlobalTracer [baseline] (240.118 ms) : 0, 240118
GlobalTracer [candidate] (239.707 ms) : 0, 239707
IAST [baseline] (30.1 ms) : 0, 30100
IAST [candidate] (28.298 ms) : 0, 28298
AppSec [baseline] (28.576 ms) : 0, 28576
AppSec [candidate] (28.578 ms) : 0, 28578
Debugger [baseline] (64.915 ms) : 0, 64915
Debugger [candidate] (67.173 ms) : 0, 67173
Remote Config [baseline] (540.522 µs) : 0, 541
Remote Config [candidate] (536.182 µs) : 0, 536
Telemetry [baseline] (7.818 ms) : 0, 7818
Telemetry [candidate] (7.844 ms) : 0, 7844
Flare Poller [baseline] (3.414 ms) : 0, 3414
Flare Poller [candidate] (3.43 ms) : 0, 3430
section profiling
crashtracking [baseline] (1.187 ms) : 0, 1187
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (692.847 ms) : 0, 692847
BytebuddyAgent [candidate] (695.761 ms) : 0, 695761
AgentMeter [baseline] (9.215 ms) : 0, 9215
AgentMeter [candidate] (9.167 ms) : 0, 9167
GlobalTracer [baseline] (207.452 ms) : 0, 207452
GlobalTracer [candidate] (206.713 ms) : 0, 206713
AppSec [baseline] (32.914 ms) : 0, 32914
AppSec [candidate] (32.622 ms) : 0, 32622
Debugger [baseline] (66.072 ms) : 0, 66072
Debugger [candidate] (65.793 ms) : 0, 65793
Remote Config [baseline] (593.137 µs) : 0, 593
Remote Config [candidate] (588.192 µs) : 0, 588
Telemetry [baseline] (7.751 ms) : 0, 7751
Telemetry [candidate] (7.864 ms) : 0, 7864
Flare Poller [baseline] (3.62 ms) : 0, 3620
Flare Poller [candidate] (3.539 ms) : 0, 3539
ProfilingAgent [baseline] (94.132 ms) : 0, 94132
ProfilingAgent [candidate] (94.286 ms) : 0, 94286
Profiling [baseline] (94.704 ms) : 0, 94704
Profiling [candidate] (94.838 ms) : 0, 94838
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 21 metrics, 15 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section baseline
no_agent (18.293 ms) : 18109, 18478
. : milestone, 18293,
appsec (18.801 ms) : 18610, 18992
. : milestone, 18801,
code_origins (17.774 ms) : 17602, 17947
. : milestone, 17774,
iast (18.207 ms) : 18027, 18387
. : milestone, 18207,
profiling (18.993 ms) : 18798, 19188
. : milestone, 18993,
tracing (18.227 ms) : 18047, 18407
. : milestone, 18227,
section candidate
no_agent (18.231 ms) : 18048, 18414
. : milestone, 18231,
appsec (19.012 ms) : 18822, 19203
. : milestone, 19012,
code_origins (17.871 ms) : 17692, 18049
. : milestone, 17871,
iast (18.189 ms) : 18008, 18371
. : milestone, 18189,
profiling (18.532 ms) : 18343, 18720
. : milestone, 18532,
tracing (18.075 ms) : 17895, 18255
. : milestone, 18075,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section baseline
no_agent (1.255 ms) : 1243, 1267
. : milestone, 1255,
iast (3.313 ms) : 3265, 3362
. : milestone, 3313,
iast_FULL (6.022 ms) : 5961, 6084
. : milestone, 6022,
iast_GLOBAL (3.709 ms) : 3648, 3770
. : milestone, 3709,
profiling (2.127 ms) : 2106, 2148
. : milestone, 2127,
tracing (1.883 ms) : 1867, 1899
. : milestone, 1883,
section candidate
no_agent (1.251 ms) : 1239, 1264
. : milestone, 1251,
iast (3.307 ms) : 3266, 3348
. : milestone, 3307,
iast_FULL (5.984 ms) : 5922, 6047
. : milestone, 5984,
iast_GLOBAL (3.744 ms) : 3684, 3805
. : milestone, 3744,
profiling (2.046 ms) : 2028, 2064
. : milestone, 2046,
tracing (1.902 ms) : 1886, 1917
. : milestone, 1902,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section baseline
no_agent (1.491 ms) : 1479, 1502
. : milestone, 1491,
appsec (3.819 ms) : 3597, 4040
. : milestone, 3819,
iast (2.286 ms) : 2216, 2356
. : milestone, 2286,
iast_GLOBAL (2.323 ms) : 2253, 2393
. : milestone, 2323,
profiling (2.092 ms) : 2037, 2147
. : milestone, 2092,
tracing (2.091 ms) : 2037, 2145
. : milestone, 2091,
section candidate
no_agent (1.488 ms) : 1476, 1499
. : milestone, 1488,
appsec (3.747 ms) : 3529, 3964
. : milestone, 3747,
iast (2.286 ms) : 2216, 2356
. : milestone, 2286,
iast_GLOBAL (2.32 ms) : 2249, 2390
. : milestone, 2320,
profiling (2.101 ms) : 2046, 2157
. : milestone, 2101,
tracing (2.099 ms) : 2045, 2153
. : milestone, 2099,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~a03d54e3e8, baseline=1.62.0-SNAPSHOT~63343baba3
dateFormat X
axisFormat %s
section baseline
no_agent (15.576 s) : 15576000, 15576000
. : milestone, 15576000,
appsec (14.691 s) : 14691000, 14691000
. : milestone, 14691000,
iast (18.588 s) : 18588000, 18588000
. : milestone, 18588000,
iast_GLOBAL (18.139 s) : 18139000, 18139000
. : milestone, 18139000,
profiling (14.927 s) : 14927000, 14927000
. : milestone, 14927000,
tracing (14.919 s) : 14919000, 14919000
. : milestone, 14919000,
section candidate
no_agent (15.167 s) : 15167000, 15167000
. : milestone, 15167000,
appsec (15.001 s) : 15001000, 15001000
. : milestone, 15001000,
iast (18.278 s) : 18278000, 18278000
. : milestone, 18278000,
iast_GLOBAL (18.144 s) : 18144000, 18144000
. : milestone, 18144000,
profiling (14.915 s) : 14915000, 14915000
. : milestone, 14915000,
tracing (14.965 s) : 14965000, 14965000
. : milestone, 14965000,
|
| withDatadogAttribute(entry.messageAttributes(), sdkBytes); | ||
| modified = entry.toBuilder().messageAttributes(messageAttributes).build(); | ||
| } | ||
| entries.add(modified); |
There was a problem hiding this comment.
Bug fix. Now add entry unconditionally
There was a problem hiding this comment.
ah yes thanks for pointing it out, though this comment was not showing up in the latest diff :p
| Map<String, MessageAttributeValue> modified = new HashMap<>(attributes); | ||
| // Use Binary since SNS subscription filter policies fail silently with JSON strings | ||
| // https://github.com/DataDog/datadog-lambda-js/pull/269 | ||
| modified.put( |
There was a problem hiding this comment.
Extract duplicated parts to this helper method
| return context.request(); | ||
| } | ||
|
|
||
| private DataStreamsTags getTags(String snsTopicName) { |
There was a problem hiding this comment.
Inlined getTags as it's used only once
| public class SnsInterceptor implements ExecutionInterceptor { | ||
|
|
||
| // SQS subscriber limit; SNS inherits it when SQS is used as a subscriber | ||
| private static final int MAX_MESSAGE_ATTRIBUTES = 10; |
There was a problem hiding this comment.
Introduce a constant for clarity and deduplication
| abstract String expectedOperation(String awsService, String awsOperation) | ||
| abstract String expectedService(String awsService, String awsOperation) | ||
|
|
||
| def "trace details propagated when message attributes are readonly"() { |
There was a problem hiding this comment.
This test has been moved to the JUnit test suit
4463b14 to
46a68e0
Compare
| withDatadogAttribute(entry.messageAttributes(), sdkBytes); | ||
| modified = entry.toBuilder().messageAttributes(messageAttributes).build(); | ||
| } | ||
| entries.add(modified); |
There was a problem hiding this comment.
ah yes thanks for pointing it out, though this comment was not showing up in the latest diff :p
|
@codex review |
46a68e0 to
a03d54e
Compare
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
What Does This Do
Fixes SNS v2 batch injection bug that causes a batch entry to be dropped when the number of attributes exceeded limits.
Motivation
Fix the bug affecting the application's logic due to an instrumentation flaw.
Additional Notes
Ref: #7150
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: APMS-19177
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.