AVRO-4299: [perl] Bound allocation when decoding length-prefixed values and collections#3864
AVRO-4299: [perl] Bound allocation when decoding length-prefixed values and collections#3864iemejia wants to merge 24 commits into
Conversation
…th-prefixed values and collections A bytes or string value is a length prefix followed by that many bytes, and an array or map block is an element count followed by that many items. A malicious or truncated input can declare a huge length or count with little or no data. - _bytes_remaining reports the bytes still readable, found by seeking to the end and restoring the position; readers that cannot seek to the end (e.g. a streaming decompressor) yield undef and are simply skipped. - decode_bytes rejects an over-large declared length above a threshold before allocating. - decode_array/decode_map reject a block whose element count could not be backed by the bytes remaining, using _min_bytes_per_element computed from the element schema so a zero-byte element type (e.g. null) is not falsely rejected. Mirrors the Java SDK's checks (AVRO-4241). Assisted-by: GitHub Copilot:claude-opus-4.8
The Lint CI step (perlcritic, default severity) flagged a Subroutines::ProhibitExplicitReturnUndef violation: 'return undef' in _bytes_remaining. Use a bare 'return', which yields undef in the scalar context the callers use, so a reader that cannot report its size still skips the check. Assisted-by: GitHub Copilot:claude-opus-4.8
There was a problem hiding this comment.
Pull request overview
Hardens the Perl binary decoder against malicious/truncated inputs that advertise extremely large length/count prefixes by validating claimed sizes against the bytes actually remaining on seekable readers, preventing large allocations before short reads are detected.
Changes:
- Add remaining-bytes checks for large
bytes/stringlength-prefixed reads (seekable readers only). - Add remaining-bytes checks for
array/mapblocks using a computed minimum on-wire element size (with special handling to avoid rejecting zero-byte element types likenull). - Extend Perl decoder tests to cover oversized/truncated
bytes/array/mapcases and a largearray<null>that must still decode.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| lang/perl/lib/Avro/BinaryDecoder.pm | Adds seek-based “bytes remaining” logic and pre-allocation/pre-iteration validation for bytes/strings and array/map blocks. |
| lang/perl/t/03_bin_decode.t | Adds regression tests covering oversized length/count prefixes and ensuring array-of-nulls is not falsely rejected. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| my $class = shift; | ||
| my $reader = pop; | ||
| my $size = decode_long($class, undef, undef, $reader); | ||
| _ensure_available($reader, $size); | ||
| $reader->read(my $buf, $size); |
There was a problem hiding this comment.
Fixed in 92df248: decode_bytes now rejects a negative decoded length before reading.
| sub _bytes_remaining { | ||
| my ($reader) = @_; | ||
| my $current = eval { $reader->tell }; | ||
| return if !defined $current || $current < 0; | ||
| return eval { | ||
| $reader->seek(0, 2) # SEEK_END | ||
| or die "seek to end failed\n"; | ||
| my $end = $reader->tell; | ||
| $reader->seek($current, 0); # SEEK_SET, restore position | ||
| die "unknown end position\n" if !defined $end || $end < 0; | ||
| $end - $current; | ||
| }; | ||
| } |
There was a problem hiding this comment.
Fixed in 92df248: _bytes_remaining now always restores the original position after seeking to the end (even if reading the end offset fails) and treats a restore failure as fatal, so the reader is never silently left at EOF.
| # Map keys are strings (>= 1 byte length prefix) plus the value. | ||
| my $min_bytes = 1 + _min_bytes_per_element($writer_values); |
There was a problem hiding this comment.
Fixed in 92df248: decode_map now uses 2 bytes as the minimum key size (1-byte length prefix + at least 1 byte of data), since empty keys are rejected, making the pre-check slightly tighter.
… tighter map key min Review feedback: - decode_bytes rejects a negative decoded length before reading, since Avro bytes/string lengths must be non-negative. - _bytes_remaining now always restores the original position after seeking to the end (even if reading the end offset fails), and treats a restore failure as fatal, so the reader is never left at EOF and silently used for subsequent decoding. - decode_map keys are non-empty strings (empty keys are rejected), so the minimum per-entry size uses 2 bytes for the key (1-byte length prefix plus at least 1 byte of data) instead of 1, making the pre-check slightly tighter. Assisted-by: GitHub Copilot:claude-opus-4.8
| return if $count <= 0 || $min_bytes <= 0; | ||
| my $remaining = _bytes_remaining($reader); | ||
| return unless defined $remaining; | ||
| if ($count * $min_bytes > $remaining) { |
There was a problem hiding this comment.
Fixed in 98aaeb6: _ensure_collection_available compares count against int(remaining / min_bytes) instead of multiplying, avoiding overflow/wrap on 32-bit builds.
Review feedback: $count * $min_bytes could overflow/wrap on some Perl builds (notably 32-bit), defeating the pre-check for very large declared counts. Compare $count against int($remaining / $min_bytes) instead. Assisted-by: GitHub Copilot:claude-opus-4.8
| throw Avro::Schema::Error::Parse( | ||
| "Invalid negative bytes/string length: $size"); | ||
| } | ||
| _ensure_available($reader, $size); | ||
| $reader->read(my $buf, $size); |
There was a problem hiding this comment.
Fixed in fa8332b: decode_bytes now verifies read() returned exactly the declared number of bytes (rejecting truncated input), and skip_bytes (used by skip_string during resolution) rejects a negative declared length.
Review feedback: - decode_bytes now verifies that read() returned exactly the declared number of bytes; a short read (truncated input) is rejected instead of silently returning an under-sized buffer that would misalign the decoder. - skip_bytes (used by skip_string during schema resolution) rejects a negative declared length, consistent with decode_bytes. Assisted-by: GitHub Copilot:claude-opus-4.8
| throw Avro::Schema::Error::Parse( | ||
| "Invalid negative bytes/string length: $size"); | ||
| } | ||
| $reader->seek($size, 0); |
There was a problem hiding this comment.
Fixed in 9a4cf0a: skip_bytes() now uses whence 1 (SEEK_CUR) to skip forward relative to the current position, instead of whence 0 (SEEK_SET) which seeks to the absolute offset.
Review feedback: skip_bytes() used seek($size, 0), i.e. SEEK_SET, which seeks to the absolute offset $size instead of skipping $size bytes forward from the current position. Use whence 1 (SEEK_CUR) for a relative skip. Assisted-by: GitHub Copilot:claude-opus-4.8
| # Skip forward by $size bytes relative to the current position | ||
| # (SEEK_CUR); whence 0 (SEEK_SET) would incorrectly seek to the | ||
| # absolute offset $size. | ||
| $reader->seek($size, 1); | ||
| return; |
There was a problem hiding this comment.
Fixed in 25ab8ab: skip_bytes now calls _ensure_available before skipping (mirroring decode_bytes), so a declared length that exceeds the remaining bytes is rejected on a size-reporting reader instead of silently seeking past EOF, and a failed seek is treated as fatal.
While here I fixed the same absolute-vs-relative seek bug in two related paths that shared it: skip_fixed used seek(size, 0) (SEEK_SET) — now SEEK_CUR — and skip_block's negative-count branch ignored the block-size long, did an absolute seek of a negative offset, and re-looped on the same count (an infinite loop); it now reads the block size and skips it relatively. Added tests for skipping fixed/bytes writer fields absent from the reader schema and for rejecting an oversized skipped bytes length. Full suite (386 tests) and perlcritic pass.
Review feedback: skip_bytes could silently succeed when the declared length exceeded the bytes actually available (a seek past EOF is allowed on many handles), causing subsequent decoding to read EOF as zeros instead of failing on truncated input. - skip_bytes now calls _ensure_available before skipping (mirroring decode_bytes) and treats a failed seek as fatal. - skip_fixed used seek(size, 0) (SEEK_SET, absolute); use SEEK_CUR for a relative skip and throw on a failed seek. - skip_block's negative-count branch ignored the block-size long, did an absolute seek of a negative offset, and re-looped on the same count (an infinite loop). Read the block size and skip it relatively (SEEK_CUR), throwing on a failed seek. Added tests covering skipping fixed/bytes writer fields absent from the reader schema and rejecting an oversized skipped bytes length. Assisted-by: GitHub Copilot:claude-opus-4.8
| my $block_size = decode_long($class, undef, undef, $reader); | ||
| unless ($reader->seek($block_size, 1)) { | ||
| throw Avro::Schema::Error::Parse( | ||
| "Failed to skip block of $block_size bytes"); | ||
| } |
There was a problem hiding this comment.
Fixed in 8af98be: the negative-count path now rejects a negative block size (throwing Invalid negative block size) before seeking, so a corrupt input cannot seek the reader backwards into an infinite loop or mis-decode. It also validates the block size against the bytes remaining via _ensure_available before the relative seek, consistent with skip_bytes.
Review feedback: in skip_block's negative-count path the decoded block-size long was not validated. A corrupted input with a negative block size would make seek($block_size, SEEK_CUR) move the reader backwards, risking an infinite loop or mis-decoding. Reject a negative block size, and validate it against the bytes remaining (via _ensure_available) before seeking. Assisted-by: GitHub Copilot:claude-opus-4.8
| my $block_count = decode_long($class, undef, undef, $reader); | ||
| while ($block_count) { | ||
| if ($block_count < 0) { | ||
| $reader->seek($block_count, 0); | ||
| next; | ||
| # A negative count is followed by a long block size in bytes, which | ||
| # lets the whole block be skipped without decoding each item. Skip |
There was a problem hiding this comment.
Great catch — this was a genuine pre-existing bug. Fixed in fce9c6a: skip_block no longer shifts an implicit $class; it takes ($reader, $block_content) to match the plain-function call style used by skip_array/skip_map, and passes PACKAGE to decode_long (which ignores its first argument). Previously $class captured the reader handle and $reader captured the coderef, so decode_long would call $coderef->read and die whenever an array/map field was skipped during projection. Added a test that skips writer array and map fields absent from the reader schema (it would have crashed on the old code).
| ## union and enum index bounds | ||
| { | ||
| my $union = Avro::Schema->parse(q(["string","null"])); | ||
| # Index 2 (zig-zag long 0x04) is out of range for a 2-branch union. |
There was a problem hiding this comment.
Fixed — comment now says 'zig-zag int'.
| ); | ||
| } 'Avro::Schema::Error::Parse', "union branch index out of range is rejected"; | ||
|
|
||
| # Index -1 (zig-zag long 0x01) must not wrap to a valid branch. |
There was a problem hiding this comment.
Fixed — comment now says 'zig-zag int'.
The union branch index is encoded as an Avro int per the spec; update the two test comments that said "zig-zag long". Assisted-by: GitHub Copilot:claude-opus-4.8
| my $symbols = $writer_schema->symbols; | ||
| ## A negative or out-of-range index is malformed; reject it before indexing | ||
| ## (Perl's negative indexing would otherwise select the wrong symbol). | ||
| if ($index < 0 || $index >= scalar @$symbols) { | ||
| throw Avro::Schema::Error::Parse( | ||
| "Enum symbol index $index out of range for " . scalar(@$symbols) . " symbols"); | ||
| } |
There was a problem hiding this comment.
Unlike skip_union (where the branch index selects which schema to skip, so a wrong index desyncs the stream), the enum index doesn't affect skipping: an enum is encoded as just the index varint with nothing following it, so skip_enum reads the varint and moves on regardless of the value. An out-of-range index is therefore skipped correctly without desync or allocation, so validating it here would only be for parity with decode_enum, not correctness/safety. I've left skip_enum as skip_int to keep the skip path minimal; happy to add the schema-aware check if you'd prefer strict parity.
| sub skip_map { | ||
| my $class = shift; | ||
| my ($schema, $reader) = @_; | ||
| skip_block($reader, sub { | ||
| # Map entries always carry a >= 1 byte key, so pass a positive minimum. | ||
| skip_block($reader, 1 + _min_bytes_per_element($schema->values), sub { |
There was a problem hiding this comment.
Similarly, an empty map key is skipped correctly (a 0-length string = a 1-byte length prefix and no data), so it doesn't desync or over-read during projection — decode_map rejects it for data validity, but skip only needs to advance past it. So there's no correctness/security impact here; it's a decode-vs-skip strictness parity point. Left as-is to keep skip minimal.
| { | ||
| ## The reported exploit: a ~6 byte payload declaring 200,000,000 nulls is | ||
| ## rejected by the default limit, before allocating. | ||
| throws_ok { | ||
| decode_zero_byte_array('"null"', encode_long(200_000_000) . encode_long(0)); | ||
| } qr/more than \d+ zero-byte/, "array of 200M nulls rejected by default limit"; | ||
| } |
There was a problem hiding this comment.
Fixed — the test now does local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = $Avro::BinaryDecoder::DEFAULT_MAX_COLLECTION_ITEMS;, so it's deterministic regardless of AVRO_MAX_COLLECTION_ITEMS in the environment (verified passing with it set to 999999999). Unlike some SDKs the Perl limit is a settable package var, so localizing it works cleanly.
| my $block_size = decode_long(__PACKAGE__, undef, undef, $reader); | ||
| if ($block_size < 0) { | ||
| # A negative block size would seek the reader backwards, | ||
| # risking an infinite loop or mis-decoding of a corrupt input. | ||
| throw Avro::Schema::Error::Parse( | ||
| "Invalid negative block size: $block_size"); | ||
| } | ||
| # Reject a block size that exceeds the bytes actually remaining | ||
| # before skipping, so a truncated input fails instead of seeking | ||
| # past EOF. | ||
| _ensure_available($reader, $block_size, 1); |
There was a problem hiding this comment.
Fixed — skip_block's negative-count branch now rejects a block_size too small to hold the declared count at its minimum on-wire size ($count > int($block_size / $min_bytes) when min_bytes > 0), before seeking.
| # A 64-bit value uses at most 10 bytes; an 11th continuation byte is a | ||
| # malformed overlong varint and must be rejected. |
There was a problem hiding this comment.
Fixed — reworded to 'an 11th byte (10 continuation bytes 0x80 then a terminator)'.
The "200M nulls rejected by default limit" test relied on the default MAX_COLLECTION_ITEMS, which BinaryDecoder.pm initializes from AVRO_MAX_COLLECTION_ITEMS at load. Localize $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS to DEFAULT_MAX_COLLECTION_ITEMS in the test so it is deterministic even if that env var is set in the runner. Assisted-by: GitHub Copilot:claude-opus-4.8
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.
Comments suppressed due to low confidence (1)
lang/perl/lib/Avro/BinaryDecoder.pm:493
decode_arrayiterates withfor (1..$block_count), which allocates the full range list before the loop body runs. For large counts this can blow memory/CPU independent of the collection caps (notably on non-seekable readers where the bytes-remaining check is skipped). Use an index-based loop instead.
_ensure_collection_available($reader, scalar(@array), $block_count, $min_bytes);
for (1..$block_count) {
push @array, $class->decode(
writer_schema => $writer_items,
reader_schema => $reader_items,
| for (1..$block_count) { | ||
| $block_content->(); | ||
| } |
There was a problem hiding this comment.
This is a Perl foreach-range optimization: for (1..$n) in a foreach loop is special-cased and iterated lazily — it does NOT build the list. Verified: for (1..200_000_000) {} uses 0 extra RSS (4636 kB before and after). So there's no materialization here; and the block count is already bounded by the cumulative caps before this loop runs.
| _ensure_collection_available($reader, $decoded, $block_count, $min_bytes); | ||
| $decoded += $block_count; | ||
| for (1..$block_count) { | ||
| my $key = decode_string($class, @_); | ||
| unless (defined $key && length $key) { |
There was a problem hiding this comment.
Same — for (1..$block_count) in decode_map is the foreach-range form, which Perl iterates lazily without materializing the list (confirmed: no RSS growth for 200M). The count is also bounded by ensure_collection_available before the loop.
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = | ||
| $Avro::BinaryDecoder::DEFAULT_MAX_COLLECTION_ITEMS; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
|
|
||
| { | ||
| ## Within a lowered limit, a legitimate small null array still decodes. | ||
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = 1000; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
| } | ||
|
|
||
| { | ||
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = 1000; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
|
|
||
| { | ||
| ## Cumulative across blocks: two blocks of 600 (1200 > 1000). | ||
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = 1000; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
| { | ||
| ## A negative block count (abs value + block byte-size) is normalized and | ||
| ## must still be bounded. | ||
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = 1000; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
|
|
||
| { | ||
| ## A record whose only field is null encodes to zero bytes. | ||
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = 1000; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
| { | ||
| ## Skipping a huge array<null> writer field absent from the reader schema is | ||
| ## bounded (a CPU exhaustion otherwise), via skip_block. | ||
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = 1000; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
|
|
||
| { | ||
| ## The zero-byte cap raises the dedicated exception class. | ||
| local $Avro::BinaryDecoder::MAX_COLLECTION_ITEMS = 1000; |
There was a problem hiding this comment.
Addressed globally: rather than localize MAX_COLLECTION_STRUCTURAL per test, the top of t/03_bin_decode.t now resets BOTH MAX_COLLECTION_ITEMS and MAX_COLLECTION_STRUCTURAL to their defaults (they're both initialized from AVRO_MAX_COLLECTION_ITEMS at load). Verified the whole suite passes with AVRO_MAX_COLLECTION_ITEMS=5 set.
…pendent - skip_block's negative-count branch now also rejects a block_size too small to hold the declared element count at its minimum on-wire size, so a malformed size can't leave the reader mid-element. - BinaryDecoder initializes both collection caps from AVRO_MAX_COLLECTION_ITEMS at load, which affected the whole test suite (not just the limit tests). Reset both caps to their defaults at the top of t/03_bin_decode.t so every test is deterministic regardless of the environment (verified with the env var set to 5); individual tests still local-override where needed. - Fix the overlong-varint test comment wording. Assisted-by: GitHub Copilot:claude-opus-4.8
| ## Zero-byte elements (null, zero-length fixed, all-null records) consume no | ||
| ## input, so the bytes-remaining check cannot bound them. A huge declared block | ||
| ## count of such elements is capped instead. |
There was a problem hiding this comment.
Fixed — removed 'zero-length fixed' from the comment and noted that this SDK's parser requires a positive Fixed size, so Fixed is never a zero-byte element here.
Fixed is not a zero-byte element type in Perl: this SDK's schema parser requires a positive Fixed size. Drop 'zero-length fixed' from the comment. Assisted-by: GitHub Copilot:claude-opus-4.8
What is the purpose of the change
A
bytesorstringvalue is encoded as a length prefix followed by that many bytes of data, and anarrayormapblock is encoded as an element count followed by that many items. A malicious or truncated input can declare a very large length or count while carrying little or no actual data, which causes a correspondingly large allocation before the shortfall is noticed.This applies the equivalent of the Java SDK fix AVRO-4241 to the Perl SDK and extends it to collections. It has two complementary parts.
1. Validate available bytes before allocating
When the source can report how many bytes remain, a declared length (or a collection block count) that exceeds the bytes actually available is rejected before allocating for it. The collection check uses the minimum on-wire size of the element schema, so a zero-byte element type (such as
null) is never falsely rejected. Sources that cannot report their remaining size are unaffected._bytes_remainingreports the bytes still readable, found by seeking to the end and restoring the position; readers that cannot seek to the end (e.g. a streaming decompressor) yield undef and are simply skipped.decode_bytesrejects an over-large declared length above a threshold, anddecode_array/decode_mapreject a block whose element count could not be backed by the bytes remaining, using_min_bytes_per_elementfrom the element schema.2. Cap collection allocation for zero-byte elements
Zero-byte elements (
null, a zero-lengthfixed, or a record with only zero-byte fields) consume no input, so the available-bytes check cannot bound their count: a tiny payload such as{{"type":"array","items":"null"}}declaring a block count of 200,000,000 would otherwise drive an unbounded allocation. In addition to the available-bytes check,_ensure_collection_availablecaps the cumulative count of zero-byte elements ($MAX_COLLECTION_ITEMS, default 10,000,000) and applies a structural cap to every collection ($MAX_COLLECTION_STRUCTURAL, defaultInteger.MAX_VALUE - 8) that also covers non-seekable readers.skip_blockis now element-aware (bounds zero-byte skips tightly, other skips by the structural cap). Rejections raise the dedicatedAvro::BinaryDecoder::Error::CollectionSize. When set, theAVRO_MAX_COLLECTION_ITEMSenvironment variable caps both limits.This folds in and supersedes the standalone collection-limit change (AVRO-4280, #3847), so this PR is the single complete fix for collection/length-prefixed allocation DoS in the Perl SDK.
This is a sub-task of AVRO-4292 and resolves AVRO-4299.
Verifying this change
This change added tests and can be verified as follows:
lang/perl/t/03_bin_decode.twith over-limitbytes/array/maprejection, anarray<null>with a huge block count that must be rejected, and skip bounding.cd lang/perl && prove -Ilib t/Documentation