Commit 35ed9fb
authored
Add server.request.body.files_content AppSec address for Tomcat and Netty multipart (#11198)
refactor(appsec): extract MultipartContentDecoder to internal-api for reuse across integrations
test(appsec): add missing corner cases to MultipartContentDecoderTest
fix(appsec): use machine default charset as fallback in MultipartContentDecoder
Replaces hardcoded UTF-8 (no-charset default) and ISO-8859-1 (fallback)
with Charset.defaultCharset() in both cases, per reviewer feedback.
test(appsec): migrate MultipartContentDecoderTest from Groovy/Spock to Java/JUnit 5
feat(appsec): extend files_content WAF address to Tomcat and Netty multipart
Read up to 4096 bytes per file (ISO-8859-1, max 25 files) and fire the
server.request.body.files_content callback after the filenames callback, only
when the request has not already been blocked.
- ParameterCollector: add readContent() via reflection on getInputStream(),
getContents() getter, and MAX_FILES_TO_INSPECT guard
- ParsePartsInstrumentation: dispatch requestFilesContent() after
requestFilesFilenames() when t == null
- HttpPostRequestDecoderInstrumentation: collect file content inline in the
getBodyHttpDatas() loop (in-memory via ByteBuf, on-disk via FileInputStream)
and dispatch requestFilesContent() when thr == null after filenames
- Add ParameterCollectorImplTest covering truncation, 25-file cap, form-field
skip, IOException fallback, and ISO-8859-1 round-trip
fix(appsec): address techdebt in Tomcat/Netty files_content instrumentation
Issue 1 — Netty: extract MAX_CONTENT_BYTES=4096 and MAX_FILES_TO_INSPECT=25 as
named constants in ParseBodyAdvice, replacing raw magic literals.
Issue 3 — Tomcat + Netty: call effectivelyBlocked() after tryCommitBlockingResponse
in both the filenames and files_content blocking paths, matching the behaviour
already present in CommonsFileUploadAppSecInstrumentation.
Issue 4 — Netty: collect file content for ALL FileUpload data items regardless
of whether filename is present or empty; HttpDataType.FileUpload is the
file-vs-form-field discriminator in Netty's model, so the filename-empty check
was incorrectly skipping content for unnamed file parts (inconsistent with
FileItemContentReader which only skips isFormField() items).
fix(appsec): further cleanup of Tomcat/Netty files_content instrumentation
Issue 1 — Tomcat: inspect file content even when submitted filename is empty.
getSubmittedFileName() returns null for form fields (no filename parameter) and ""
for file uploads whose client explicitly sent filename="". Null is now the only
skip condition, so file parts with an empty filename are inspected but not added
to the filenames list, matching the commons-fileupload and Netty behaviour.
Issue 2 — Netty: remove unnecessary (long) casts in Math.min; both operands are
already int so Math.min(int,int) resolves directly.
Issue 3 — Netty: replace manual try/finally on FileInputStream with
try-with-resources, consistent with FileItemContentReader style.
fix(appsec): align blocking paths with canonical pattern in Tomcat/Netty files_content
Move BlockingException assignment inside the brf != null guard so it is
only thrown when the blocking response was actually committed. Also add
effectivelyBlocked() to the Tomcat body-processed path, and remove the
redundant if (t == null) guard there.
For the Netty body-processed path (urlencoded), effectivelyBlocked() is
intentionally omitted: tryCommitBlockingResponse() finishes the Netty
span synchronously, so calling effectivelyBlocked() afterwards would
interact with an already-closed TraceSegment.
fix(appsec): skip file content I/O in Netty when requestFilesContent callback is unregistered
Pre-fetch the requestFilesContent callback before the body-data loop and
guard ByteBuf/FileInputStream reads behind it, consistent with how
CommonsFileUploadAppSecInstrumentation already does. Also flattens the
post-loop dispatch block by one nesting level.
fix(appsec): align Netty filenames blocking and merge CommonsFileUpload loop
Move thr = new BlockingException(...) for the filenames path in Netty
inside the brf != null guard, consistent with the body-processed and
content paths in the same advice and with the Tomcat/Liberty pattern.
Merge the two FileItem iterations in CommonsFileUploadAppSecInstrumentation
into a single loop that builds both filenames and filesContent in one pass,
calling FileItemContentReader.readContent() per item instead of a separate
FileItemContentReader.readContents() sweep.
refactor(appsec): remove dead readContents() from FileItemContentReader
The bulk readContents(List<FileItem>) method is no longer called — the
advice iterates file items inline calling readContent(FileItem) per item.
Remove the method, its now-unused ArrayList/List imports, and the
corresponding test cases. Also fix redundant type witness on ArrayList
construction in CommonsFileUploadAppSecInstrumentation.
fix(appsec): lazy-init filesContent in Netty and align CommonsFileUpload guard style
Skip filesContent ArrayList allocation in Netty ParseBodyAdvice when the
requestFilesContent callback is not registered, matching the null-sentinel
pattern already used in CommonsFileUploadAppSecInstrumentation.
Replace the early-return in CommonsFileUploadAppSecInstrumentation's
filenames blocking branch with a t == null guard on the content dispatch
block, aligning control flow with ParsePartsInstrumentation and
HttpPostRequestDecoderInstrumentation.
fix(appsec): skip file content I/O in Tomcat when requestFilesContent callback is unregistered
Add an inspectContent boolean to ParameterCollectorImpl, set from the
presence of the requestFilesContent callback in ParsePartsInstrumentation
before(). When false, addPart() still collects filenames but skips the
getInputStream() read, matching the lazy-init approach already used by
the Netty and commons-fileupload integrations.
fix(appsec): move BlockingException inside brf guard in all remaining paths
The content path in Netty and both the filenames and content paths in
Tomcat were setting the BlockingException outside the if (brf != null)
block, meaning the exception would be thrown even when no blocking
response had actually been committed. Align all three with the canonical
pattern: tryCommitBlockingResponse + effectivelyBlocked + exception
assignment all inside if (brf != null).
The Netty body-processed path intentionally still omits effectivelyBlocked()
because tryCommitBlockingResponse() closes the Netty span synchronously in
the test environment, making a subsequent effectivelyBlocked() call fail.
refactor(appsec): cache reflection and use try-with-resources in ParameterCollector
- Replace per-call getMethod() with volatile (Class, Method) entry cache so
reflection resolves once per concrete Part class rather than per file per request.
- Use try-with-resources for InputStream in readContent() instead of try/finally.
- Add unit test covering the Tomcat 7 getFilename() fallback path.
test(appsec): add Netty integration tests for multipart filenames and file content
Also fix HttpPostRequestDecoderInstrumentation to fire on PREEPILOGUE+isLastChunk
so that multipart requests delivered as a FullHttpRequest (single shot) trigger the
requestBodyProcessed/requestFilesFilenames/requestFilesContent callbacks. The
PREEPILOGUE→EPILOGUE transition requires a second parseBody() call which never
comes when the full request arrives in one shot; PREEPILOGUE+isLastChunk is
equivalent to EPILOGUE in that scenario.
refactor(appsec): extract Netty file content reading to NettyFileUploadContentReader helper
Move the in-memory/disk-backed FileUpload content reading logic out of ParseBodyAdvice
into a testable static helper class. Also fixes the partial-read risk on disk-backed
uploads (single fis.read() replaced by a loop) and broadens the catch to Exception to
handle IllegalReferenceCountException from released ByteBufs.
test(appsec): add integration tests for file content size and count limits
test(appsec): increase Netty test server aggregator limit to support large multipart payloads
fix(appsec): do not gate Netty file content callback on requestBodyProcessed subscriber
style: apply spotless formatting to HttpServerTest
fix(appsec): remove effectivelyBlocked() calls in Netty filenames and content blocking paths
fix(appsec): set t before effectivelyBlocked() in Tomcat filenames and content blocking paths
perf(appsec): skip attribute I/O in Netty multipart when body callback is absent
When only requestFilesContent is registered, avoid calling getValue()
on form attributes (which may trigger disk reads for large fields).
refactor(appsec): unify file content limits via Config for DD_APPSEC_MAX_FILE_CONTENT_BYTES/COUNT
Replace hardcoded constants MAX_CONTENT_BYTES=4096 and MAX_FILES_TO_INSPECT=25
(duplicated across commons-fileupload, Netty, and Tomcat) with two Config-backed
variables: appsec.max.file-content.bytes and appsec.max.file-content.count.
chore: register DD_APPSEC_MAX_FILE_CONTENT_BYTES/COUNT in supported-configurations.json
test(appsec): use Config values instead of hardcoded literals in ParameterCollectorImplTest
fix(appsec): move MAX_FILES_TO_INSPECT out of advice inner class to fix muzzle validation
Static fields in @RequiresRequestContext advice inner classes trigger muzzle to
treat the advice class itself as a user-classpath class; moving the constant to
NettyFileUploadContentReader (a helper) avoids the self-referential muzzle failure.
style: apply spotless formatting to HttpPostRequestDecoderInstrumentation
fix(appsec): include filenames callback in Netty multipart early-return gate
requestFilesFilenames() was consulted after data collection, so when it was
the only registered AppSec callback (no requestBodyProcessed, no
requestFilesContent) the early-return gate fired and filenames were never
published to the WAF. Fetch all three callbacks upfront and gate on all three.
fix(appsec): use per-part charset for files_content in Tomcat and Netty
Introduces MultipartContentDecoder (internal-api) to decode multipart file bytes
using the charset declared in each part's Content-Type header, with JVM-default
fallback and REPLACE on malformed input. Mirrors the approach in PR #11212 for
commons-fileupload so all three integrations share the same decoding logic.
refactor(appsec): extract shared InputStream read loop into MultipartContentDecoder
Add readInputStream(InputStream, int, String) to eliminate the identical bounded
read loop duplicated across FileItemContentReader, NettyFileUploadContentReader and
ParameterCollector. Each caller still owns its own MAX_CONTENT_BYTES constant.
Also consolidate the two volatile reflection caches in ParameterCollector.readContent
(getInputStream + getContentType) into a single Method[] entry to halve volatile reads
per invocation.
test(appsec): add charset-aware decoding tests for readInputStream, Netty and Tomcat helpers
test(appsec): add getContentType() to Tomcat test part doubles to fix reflective method resolution
perf(appsec): skip filenames list allocation in CommonsFileUpload when filenames callback is absent
refactor(appsec): extract Netty multipart data collection to NettyMultipartHelper and add unit tests
test(appsec): declare ISO-8859-1 content type on RawBytesPart to match decoder charset expectation
refactor(appsec): extract tryBlock helper to NettyMultipartHelper
Deduplicates the identical RequestBlockingAction dispatch pattern
that appeared three times in the Netty multipart advice.
test(appsec): add unit tests for NettyMultipartHelper.tryBlock
refactor(appsec): move MAX_FILES_TO_INSPECT to FileItemContentReader helper with addToContents()
The constant now lives in the helper class (listed in helperClassNames()) where it is actually
used, instead of the outer instrumentation class which is not visible from inlined advice
bytecode in the app classloader. addToContents() encapsulates the limit check alongside
the constant.
refactor(appsec): replace Map.Entry method cache with four flat volatile fields in ParameterCollector
Use separate volatile fields (cachedPartClass, cachedGetInputStream, cachedGetContentType,
cachedGetFilename) instead of Map.Entry<Class<?>, Method[]> + Map.Entry<Class<?>, Method>.
cachedPartClass is written last to safely publish the three methods per JMM volatile ordering.
test(appsec): add unit tests for FileItemContentReader.addToContents() limit behaviour
fix(appsec): preserve decoder exception when body callback does not block in Netty multipart
Unconditionally assigning thr = tryBlock(...) would null out an original
decoder exception (e.g. malformed multipart on an over-limit body) when
tryBlock returns null for a non-blocking flow. Only overwrite thr when
tryBlock returns a blocking exception.
fix(appsec): make Tomcat Part method cache atomic with an immutable holder
Four flat volatile fields have a TOCTOU race: thread A passes the
cachedPartClass check, thread B overwrites the three method fields for a
different class, and thread A invokes the wrong Method on its instance
(IllegalArgumentException is swallowed, the part is silently dropped).
Replace the four fields with a single volatile CachedMethods reference
that holds an immutable snapshot of all three methods. Readers load the
reference once into a local variable, so their snapshot stays consistent
even if another thread publishes a new entry concurrently.
Add a concurrency regression test that runs two threads simultaneously
with different concrete Part classes; it detects dropped parts caused by
the race.
fix(appsec): inject CachedMethods helper class in Tomcat ParsePartsInstrumentation
ByteBuddy only injects classes explicitly listed in helperClassNames(); nested
inner classes are not auto-discovered. Without this entry, the first addPart()
call triggers NoClassDefFoundError on CachedMethods, which is swallowed by the
broad catch in addPart(), silently dropping all filenames and file contents.
Co-authored-by: alejandro.gonzalez <alejandro.gonzalez@datadoghq.com>1 parent 9cb85ec commit 35ed9fb
17 files changed
Lines changed: 1366 additions & 158 deletions
File tree
- dd-java-agent
- instrumentation-testing/src/main/groovy/datadog/trace/agent/test/base
- instrumentation
- commons-fileupload-1.5/src
- main/java/datadog/trace/instrumentation/commons/fileupload
- test/groovy
- netty/netty-4.1/src
- main/java/datadog/trace/instrumentation/netty41
- test/groovy
- tomcat/tomcat-appsec/tomcat-appsec-7.0/src
- main/java/datadog/trace/instrumentation/tomcat7
- test/groovy/datadog/trace/instrumentation/tomcat7
- dd-trace-api/src/main/java/datadog/trace/api
- config
- internal-api/src
- main/java/datadog/trace/api
- http
- test/java/datadog/trace/api/http
- metadata
Lines changed: 91 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
| 139 | + | |
139 | 140 | | |
140 | 141 | | |
141 | 142 | | |
| |||
372 | 373 | | |
373 | 374 | | |
374 | 375 | | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
375 | 380 | | |
376 | 381 | | |
377 | 382 | | |
| |||
1652 | 1657 | | |
1653 | 1658 | | |
1654 | 1659 | | |
| 1660 | + | |
| 1661 | + | |
| 1662 | + | |
| 1663 | + | |
| 1664 | + | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
| 1679 | + | |
| 1680 | + | |
| 1681 | + | |
| 1682 | + | |
| 1683 | + | |
| 1684 | + | |
| 1685 | + | |
| 1686 | + | |
| 1687 | + | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
| 1691 | + | |
| 1692 | + | |
| 1693 | + | |
| 1694 | + | |
| 1695 | + | |
| 1696 | + | |
| 1697 | + | |
| 1698 | + | |
| 1699 | + | |
| 1700 | + | |
| 1701 | + | |
| 1702 | + | |
| 1703 | + | |
| 1704 | + | |
| 1705 | + | |
| 1706 | + | |
| 1707 | + | |
| 1708 | + | |
| 1709 | + | |
| 1710 | + | |
| 1711 | + | |
| 1712 | + | |
| 1713 | + | |
| 1714 | + | |
| 1715 | + | |
| 1716 | + | |
| 1717 | + | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
| 1721 | + | |
| 1722 | + | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
| 1726 | + | |
| 1727 | + | |
| 1728 | + | |
| 1729 | + | |
| 1730 | + | |
| 1731 | + | |
| 1732 | + | |
| 1733 | + | |
| 1734 | + | |
| 1735 | + | |
1655 | 1736 | | |
1656 | 1737 | | |
1657 | 1738 | | |
| |||
2589 | 2670 | | |
2590 | 2671 | | |
2591 | 2672 | | |
| 2673 | + | |
2592 | 2674 | | |
2593 | 2675 | | |
2594 | 2676 | | |
| |||
2765 | 2847 | | |
2766 | 2848 | | |
2767 | 2849 | | |
| 2850 | + | |
| 2851 | + | |
| 2852 | + | |
| 2853 | + | |
| 2854 | + | |
| 2855 | + | |
| 2856 | + | |
| 2857 | + | |
| 2858 | + | |
2768 | 2859 | | |
2769 | 2860 | | |
2770 | 2861 | | |
| |||
Lines changed: 18 additions & 27 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
30 | 29 | | |
31 | 30 | | |
32 | 31 | | |
| |||
54 | 53 | | |
55 | 54 | | |
56 | 55 | | |
57 | | - | |
58 | 56 | | |
59 | 57 | | |
60 | 58 | | |
| |||
73 | 71 | | |
74 | 72 | | |
75 | 73 | | |
76 | | - | |
| 74 | + | |
| 75 | + | |
77 | 76 | | |
78 | 77 | | |
79 | 78 | | |
80 | 79 | | |
81 | 80 | | |
82 | | - | |
| 81 | + | |
83 | 82 | | |
84 | 83 | | |
85 | | - | |
86 | | - | |
87 | | - | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
88 | 87 | | |
89 | 88 | | |
90 | | - | |
91 | | - | |
| 89 | + | |
92 | 90 | | |
93 | 91 | | |
94 | 92 | | |
| |||
98 | 96 | | |
99 | 97 | | |
100 | 98 | | |
101 | | - | |
102 | 99 | | |
103 | 100 | | |
104 | 101 | | |
105 | 102 | | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
123 | 114 | | |
124 | 115 | | |
125 | 116 | | |
| |||
Lines changed: 11 additions & 25 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
6 | | - | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
| 12 | + | |
| 13 | + | |
28 | 14 | | |
29 | 15 | | |
30 | 16 | | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
| 17 | + | |
| 18 | + | |
39 | 19 | | |
40 | 20 | | |
41 | 21 | | |
42 | 22 | | |
43 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
44 | 30 | | |
45 | 31 | | |
Lines changed: 16 additions & 36 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
| 51 | + | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
| 54 | + | |
54 | 55 | | |
55 | 56 | | |
56 | | - | |
| 57 | + | |
57 | 58 | | |
58 | 59 | | |
59 | | - | |
| 60 | + | |
60 | 61 | | |
61 | 62 | | |
62 | | - | |
| 63 | + | |
63 | 64 | | |
64 | | - | |
65 | | - | |
66 | | - | |
| 65 | + | |
| 66 | + | |
67 | 67 | | |
68 | 68 | | |
69 | | - | |
| 69 | + | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
| 72 | + | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
| 75 | + | |
76 | 76 | | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
| 77 | + | |
| 78 | + | |
82 | 79 | | |
83 | 80 | | |
84 | | - | |
| 81 | + | |
85 | 82 | | |
86 | 83 | | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
| 84 | + | |
| 85 | + | |
106 | 86 | | |
107 | 87 | | |
108 | 88 | | |
| |||
0 commit comments