fix: bound snappy uncompressed length to avoid OOM on untrusted block#5
Open
moshap-firebolt wants to merge 1 commit into
Open
fix: bound snappy uncompressed length to avoid OOM on untrusted block#5moshap-firebolt wants to merge 1 commit into
moshap-firebolt wants to merge 1 commit into
Conversation
## Problem DataFileReaderBase::readDataBlock() decompresses a snappy-coded OCF block with snappy::Uncompress(), which reads the declared uncompressed length from the varint prefix of the block and resize()s the destination string to that full length before decompressing a single byte. The length is attacker-controlled when the Avro data is untrusted (e.g. Iceberg manifest / data files or `... FORMAT Avro` in PackDB). A tiny crafted block can declare a multi-GiB uncompressed length, triggering a huge allocation -- an OOM -- before the decoder discovers the compressed payload is far shorter. Surfaced by PackDB's `avro_input` libFuzzer harness: a 220-byte snappy OCF drove a single malloc(2415919104) (~2.4 GiB), reported as a libFuzzer out-of-memory. This is the same class as the decodeString/decodeBytes OOM fixed in #4, but a distinct code path (the snappy block decompress, not the binary decoder). ## Fix Read the declared uncompressed length without allocating via snappy::GetUncompressedLength() and reject a block whose declared length implausibly dwarfs the compressed block. A valid snappy stream expands by at most ~64x, so a generous 256x cap blocks the bomb while leaving every well-formed block untouched -- allocating proportional to the bytes that are really there, mirroring #4. ## Test - Reproduced the OOM against the PackDB `avro_input` fuzzer, then confirmed the crash input runs clean (exit 0, 0 ms) with this change. - Well-formed snappy Avro fixtures (weather-snappy, nulls, datapage_v2, alltypes_plain) still decode identically with no regression. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DataFileReaderBase::readDataBlock()decompresses a snappy-coded OCF block withsnappy::Uncompress(), which reads the declared uncompressed length from the varint prefix of the block andresize()s the destination string to that full length before decompressing a single byte. The length is attacker-controlled when the Avro data is untrusted (e.g. Iceberg manifest/data files, or... FORMAT Avroin PackDB). A tiny crafted block can declare a multi-GiB uncompressed length, triggering a huge allocation — an OOM — before the decoder discovers the compressed payload is far shorter.Surfaced by PackDB's
avro_inputlibFuzzer harness: a 220-byte snappy OCF drove a singlemalloc(2415919104)(~2.4 GiB), reported as a libFuzzer out-of-memory.This is the same class as the
decodeString/decodeBytesOOM fixed in #4, but a distinct code path (the snappy block decompress, not the binary decoder). Present verbatim in Apache Avro C++main— not fixed upstream.Fix
Read the declared uncompressed length without allocating via
snappy::GetUncompressedLength()and reject a block whose declared length implausibly dwarfs the compressed block. A valid snappy stream expands by at most ~64x, so a generous 256x cap blocks the bomb while leaving every well-formed block untouched — allocating proportional to the bytes that are really there, mirroring #4.Test
avro_inputfuzzer, then confirmed the crash input runs clean (exit 0, 0 ms) with this change.weather-snappy,nulls.snappy,datapage_v2.snappy,alltypes_plain.snappy) still decode identically with no regression.🤖 Generated with Claude Code