feat(zstd): validate Content_Checksum frames (XXH64)#108
Merged
Conversation
The zstd decoder rejected any frame whose Frame_Header set Content_Checksum_Flag with Error::Unsupported, because no XXH64 was implemented. Since the zstd CLI writes a content checksum by default, default `zstd` output only decoded with `--no-check`. Add a streaming XXH64 (canonical, seed 0; verified against reference vectors and against checksums produced by the zstd CLI). The decoder now feeds every decompressed byte through it at each emit site (raw, RLE, and compressed blocks) and validates the 4-byte little-endian trailer — the low 32 bits of XXH64 over the decompressed content — at end of frame, reporting Error::ChecksumMismatch on a mismatch. Replaces the obsolete decode_rejects_checksum_flag test with decode_validates_correct_checksum / decode_rejects_bad_checksum, and adds XXH64 unit tests. Verified end-to-end against `zstd` CLI output across levels 1-19 (checksummed and --no-check). Our encoder still does not emit a content checksum. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The zstd decoder refused any frame whose
Frame_HeadersetContent_Checksum_Flag, returningError::Unsupported, because the crate shipped no XXH64. ThezstdCLI writes a content checksum by default, so defaultzstdoutput only decoded with--no-check.Change
src/zstd/xxhash.rs) — canonical algorithm, seed 0,no_std. Verified against reference vectors ("","a","abc", 64-byte) and a streaming-vs-one-shot equivalence test.Error::ChecksumMismatch. The frame-checksum state machine (ContentChecksumphase) already existed; this wires it up.Tests
decode_rejects_checksum_flagwithdecode_validates_correct_checksumanddecode_rejects_bad_checksum(hand-built frames with a real / corrupted XXH64 trailer — no CLI dependency).zstdCLI output across levels 1–19, checksummed and--no-check, over a range of sizes including block boundaries; corrupting the trailer is correctly rejected.Our encoder still does not emit a content checksum (unchanged).
🤖 Generated with Claude Code