Skip to content

Commit 25737ae

Browse files
dfa1claude
andauthored
feat: bind findDecompressedSize + frameHeaderSize; dedup content-size checks (#30)
Adds two frame-inspection readers on ZstdFrame: - decompressedSize(byte[]/MemorySegment) — exact combined decompressed size of all concatenated frames, summed from each header (throws if any frame omits its size). Complements decompressedBound (upper bound). Binds ZSTD_findDecompressedSize. - headerSize(byte[]/MemorySegment) — size of a frame header computed from just its leading bytes (as few as 5), without a full parse. Binds ZSTD_frameHeaderSize. Extracts Zstd.requireStoredContentSize(long) for the CONTENTSIZE_UNKNOWN / CONTENTSIZE_ERROR sentinel handling that was copy-pasted across three call sites, and unifies the drifted error message ("not valid zstd data" vs "not a valid zstd frame") to one. ZSTD_getDecompressedSize intentionally not bound: deprecated, and blends empty/unknown/error into 0 — the ambiguity ZSTD_getFrameContentSize (already bound as Zstd.decompressedSize) was added to fix. docs/supported.md: frame inspection 10 -> 12/13. CHANGELOG updated. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 4cf8559 commit 25737ae

6 files changed

Lines changed: 220 additions & 15 deletions

File tree

CHANGELOG.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,13 @@ git tags, which trigger publication to Maven Central.
2020
Binds `ZSTD_getDictID_fromDict`.
2121
- `ZstdDictionaryId` value type — a `record` wrapping the 32-bit dictionary id
2222
with an unsigned `value()`, `isPresent()`, and the `NONE` sentinel for "no id".
23+
- `ZstdFrame.decompressedSize(byte[])` / `ZstdFrame.decompressedSize(MemorySegment)`
24+
— the exact combined decompressed size of all concatenated frames, summed from
25+
each frame header (throws if any frame does not record its size). Complements
26+
`decompressedBound` (upper bound). Binds `ZSTD_findDecompressedSize`.
27+
- `ZstdFrame.headerSize(byte[])` / `ZstdFrame.headerSize(MemorySegment)` — the size
28+
of a frame's header computed from just its leading bytes (as few as 5), without a
29+
full parse. Binds `ZSTD_frameHeaderSize`.
2330

2431
### Changed
2532
- Every dictionary-id accessor now returns `ZstdDictionaryId` instead of `int`:

docs/supported.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ rather than the deprecated `ZSTD_getDecompressedSize`.
3434
| Streaming — compress | 3 / 22 | `ZstdOutputStream` (compressStream2 + buffer sizes) |
3535
| Streaming — decompress | 3 / 15 | `ZstdInputStream` (decompressStream + buffer sizes) |
3636
| Advanced parameters | 14 / 38 | all `ZSTD_cParameter` + `ZSTD_dParameter` via `ZstdCompressParameter`/`ZstdDecompressParameter`; `compress2`, `C/DCtx_setParameter`, `C/DCtx_reset`, `C/DCtx_loadDictionary`, `CCtx_refCDict`/`DCtx_refDDict`, `C/DCtx_refPrefix`, `c/dParam_getBounds`; MT inert on single-thread build |
37-
| Frame inspection | 10 / 13 | `ZstdFrame` + getFrameProgression; `_advanced` not bound |
37+
| Frame inspection | 12 / 13 | `ZstdFrame` + getFrameProgression; `_advanced` not bound |
3838
| Memory sizing | 8 / 14 | sizeof_C/DCtx, sizeof_C/DDict, estimate C/DCtx + C/DDict size |
3939
| Low-level block | 0 / 12 | expert block/continue API not bound |
4040
| Sequences | 0 / 5 | sequence producer API not bound |
@@ -81,7 +81,7 @@ rather than the deprecated `ZSTD_getDecompressedSize`.
8181

8282
1. ~~**Streaming**~~ — done: `ZstdOutputStream` / `ZstdInputStream` (`compressStream2`, `decompressStream`, bounded buffers, dictionary constructors, `pledgedSrcSize` via `withPledgedSize`). Remaining: `MemorySegment`-buffer driver.
8383
2. ~~**Advanced parameters**~~ — done: every `ZSTD_cParameter`/`ZSTD_dParameter` via `ZstdCompressParameter`/`ZstdDecompressParameter` (+ `bounds()`), on both contexts; `pledgedSrcSize`. `nbWorkers` is settable but inert until the native build enables multithreading.
84-
3. ~~**Frame inspection**~~ — done: `ZstdFrame` (`isFrame`, `header`, `compressedSize`, `decompressedBound`, `dictId`, skippable, `getFrameProgression`); dict-id from raw/CDict/DDict.
84+
3. ~~**Frame inspection**~~ — done: `ZstdFrame` (`isFrame`, `header`, `compressedSize`, `decompressedBound`, `decompressedSize`, `dictId`, skippable, `getFrameProgression`); dict-id from raw/CDict/DDict.
8585
4. ~~**Better dictionaries**~~ — done: COVER / fast-COVER optimisers, `finalizeDictionary`, `getDictHeaderSize`.
8686
5. ~~**Typed errors**~~ — done: `ZstdException.code()` returns `ZstdErrorCode` (via `getErrorCode`).
8787

@@ -276,15 +276,15 @@ sequence-producer hooks vs this library's frame-inspection and typed-error surfa
276276
| `ZSTD_estimateCCtxSize_usingCCtxParams` |||
277277
| `ZSTD_freeCCtxParams` |||
278278

279-
### Frame inspection (10/13)
279+
### Frame inspection (12/13)
280280

281281
| Symbol | Bound | zstd-jni |
282282
|---|:---:|:---:|
283283
| `ZSTD_getFrameContentSize` |||
284284
| `ZSTD_decompressBound` |||
285-
| `ZSTD_findDecompressedSize` | ||
285+
| `ZSTD_findDecompressedSize` | ||
286286
| `ZSTD_findFrameCompressedSize` |||
287-
| `ZSTD_frameHeaderSize` | ||
287+
| `ZSTD_frameHeaderSize` | ||
288288
| `ZSTD_getDecompressedSize` | — ᵈ ||
289289
| `ZSTD_getFrameHeader` |||
290290
| `ZSTD_getFrameHeader_advanced` |||

zstd/src/main/java/io/github/dfa1/zstd/Bindings.java

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,11 @@ final class Bindings {
5454
NativeLibrary.lookup("ZSTD_decompressBound",
5555
FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG));
5656

57+
// unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize)
58+
static final MethodHandle FIND_DECOMPRESSED_SIZE =
59+
NativeLibrary.lookup("ZSTD_findDecompressedSize",
60+
FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG));
61+
5762
// unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize)
5863
static final MethodHandle GET_DICT_ID_FROM_FRAME =
5964
NativeLibrary.lookup("ZSTD_getDictID_fromFrame",
@@ -64,6 +69,11 @@ final class Bindings {
6469
NativeLibrary.lookup("ZSTD_getFrameHeader",
6570
FunctionDescriptor.of(JAVA_LONG, ADDRESS, ADDRESS, JAVA_LONG));
6671

72+
// size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize)
73+
static final MethodHandle FRAME_HEADER_SIZE =
74+
NativeLibrary.lookup("ZSTD_frameHeaderSize",
75+
FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG));
76+
6777
// unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size)
6878
static final MethodHandle IS_SKIPPABLE_FRAME =
6979
NativeLibrary.lookup("ZSTD_isSkippableFrame",

zstd/src/main/java/io/github/dfa1/zstd/Zstd.java

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -59,13 +59,7 @@ public static byte[] compress(byte[] src, int level) {
5959
/// use {@link #decompress(byte[], int)} for the latter
6060
public static byte[] decompress(byte[] compressed) {
6161
Objects.requireNonNull(compressed, "compressed");
62-
long size = frameContentSize(compressed);
63-
if (size == CONTENTSIZE_UNKNOWN) {
64-
throw new ZstdException("decompressed size not stored in frame; call decompress(src, maxSize)");
65-
}
66-
if (size == CONTENTSIZE_ERROR) {
67-
throw new ZstdException("not a valid zstd frame");
68-
}
62+
long size = requireStoredContentSize(frameContentSize(compressed));
6963
return decompress(compressed, Math.toIntExact(size));
7064
}
7165

@@ -102,6 +96,16 @@ public static long decompressedSize(MemorySegment frame) {
10296
} catch (Throwable t) {
10397
throw NativeCall.rethrow(t);
10498
}
99+
return requireStoredContentSize(size);
100+
}
101+
102+
/// Maps a raw `ZSTD_getFrameContentSize` / `ZSTD_findDecompressedSize` result to
103+
/// a usable length, turning zstd's negative sentinels into exceptions.
104+
///
105+
/// @param size a content size, or a `CONTENTSIZE_UNKNOWN` / `CONTENTSIZE_ERROR` sentinel
106+
/// @return `size` when it is a real length
107+
/// @throws ZstdException if the size is not stored, or the input is not valid zstd data
108+
static long requireStoredContentSize(long size) {
105109
if (size == CONTENTSIZE_UNKNOWN) {
106110
throw new ZstdException("decompressed size not stored in frame");
107111
}

zstd/src/main/java/io/github/dfa1/zstd/ZstdFrame.java

Lines changed: 72 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,35 @@ public static long decompressedBound(MemorySegment data) {
7373
return decompressedBound(data, data.byteSize());
7474
}
7575

76+
/// Exact total decompressed size of all frames in `data`, summed from each
77+
/// frame header.
78+
///
79+
/// Unlike [#decompressedBound(byte[])], which over-estimates so a buffer never
80+
/// overflows, this is the precise combined size — but only when every frame
81+
/// records it. Use it to size an exact destination; fall back to the bound when
82+
/// a frame was produced by a stream that did not pledge its size.
83+
///
84+
/// @param data one or more concatenated zstd frames
85+
/// @return the exact combined decompressed size
86+
/// @throws ZstdException if the input is not valid zstd data, or any frame does
87+
/// not record its decompressed size
88+
public static long decompressedSize(byte[] data) {
89+
try (Arena arena = Arena.ofConfined()) {
90+
return decompressedSize(Zstd.copyIn(arena, data), data.length);
91+
}
92+
}
93+
94+
/// Exact total decompressed size of all frames in native `data`.
95+
/// Otherwise identical to [#decompressedSize(byte[])].
96+
///
97+
/// @param data one or more concatenated zstd frames
98+
/// @return the exact combined decompressed size
99+
/// @throws ZstdException if the input is not valid zstd data, or any frame does
100+
/// not record its decompressed size
101+
public static long decompressedSize(MemorySegment data) {
102+
return decompressedSize(data, data.byteSize());
103+
}
104+
76105
/// Dictionary id recorded in the first frame's header.
77106
///
78107
/// @param data a zstd frame
@@ -92,6 +121,34 @@ public static ZstdDictionaryId dictId(MemorySegment data) {
92121
return dictId(data, data.byteSize());
93122
}
94123

124+
/// Size in bytes of the first frame's header, without parsing it.
125+
///
126+
/// A lighter probe than [#header(byte[])]: it reads only the few leading bytes
127+
/// needed to compute the header length (as few as 5 for a standard frame), so
128+
/// you can learn how many bytes to buffer before a full [#header(byte[])] parse.
129+
/// For an already-parsed header the same value is [ZstdFrameHeader#headerSize()].
130+
///
131+
/// @param data the start of a zstd frame, at least its first few bytes
132+
/// @return the header size in bytes
133+
/// @throws ZstdException if `data` is too short to determine the header size, or
134+
/// is not a valid zstd frame
135+
public static long headerSize(byte[] data) {
136+
try (Arena arena = Arena.ofConfined()) {
137+
return headerSize(Zstd.copyIn(arena, data), data.length);
138+
}
139+
}
140+
141+
/// Size in bytes of the first frame's header in native `data`.
142+
/// Otherwise identical to [#headerSize(byte[])].
143+
///
144+
/// @param data the start of a zstd frame, at least its first few bytes
145+
/// @return the header size in bytes
146+
/// @throws ZstdException if `data` is too short to determine the header size, or
147+
/// is not a valid zstd frame
148+
public static long headerSize(MemorySegment data) {
149+
return headerSize(data, data.byteSize());
150+
}
151+
95152
/// Parses the header of the first frame in `data` without decompressing it.
96153
///
97154
/// @param data a complete zstd frame (or at least its header)
@@ -210,10 +267,22 @@ private static long decompressedBound(MemorySegment data, long size) {
210267
} catch (Throwable t) {
211268
throw NativeCall.rethrow(t);
212269
}
213-
if (bound == Zstd.CONTENTSIZE_ERROR) {
214-
throw new ZstdException("not valid zstd data");
270+
// ZSTD_decompressBound never returns CONTENTSIZE_UNKNOWN, only a bound or the error sentinel
271+
return Zstd.requireStoredContentSize(bound);
272+
}
273+
274+
private static long decompressedSize(MemorySegment data, long size) {
275+
long total;
276+
try {
277+
total = (long) Bindings.FIND_DECOMPRESSED_SIZE.invokeExact(data, size);
278+
} catch (Throwable t) {
279+
throw NativeCall.rethrow(t);
215280
}
216-
return bound;
281+
return Zstd.requireStoredContentSize(total);
282+
}
283+
284+
private static long headerSize(MemorySegment data, long size) {
285+
return NativeCall.checkReturnValue(() -> (long) Bindings.FRAME_HEADER_SIZE.invokeExact(data, size));
217286
}
218287

219288
private static ZstdDictionaryId dictId(MemorySegment data, long size) {

zstd/src/test/java/io/github/dfa1/zstd/ZstdFrameTest.java

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,121 @@ void rejectsGarbage() {
9494
}
9595
}
9696

97+
@Nested
98+
class DecompressedSize {
99+
100+
@Test
101+
void equalsContentSizeForSingleFrame() {
102+
// Given one frame that records its content size
103+
byte[] frame = Zstd.compress(PAYLOAD);
104+
105+
// Then the exact decompressed size is the payload length
106+
assertThat(ZstdFrame.decompressedSize(frame)).isEqualTo(PAYLOAD.length);
107+
}
108+
109+
@Test
110+
void sumsAcrossConcatenatedFrames() throws Exception {
111+
// Given two frames back to back
112+
byte[] second = "second payload".getBytes(StandardCharsets.UTF_8);
113+
ByteArrayOutputStream both = new ByteArrayOutputStream();
114+
both.write(Zstd.compress(PAYLOAD));
115+
both.write(Zstd.compress(second));
116+
117+
// Then the total is the sum of both decompressed sizes
118+
assertThat(ZstdFrame.decompressedSize(both.toByteArray()))
119+
.isEqualTo((long) PAYLOAD.length + second.length);
120+
}
121+
122+
@Test
123+
void matchesThroughTheSegmentOverload() {
124+
// Given a frame in a native segment
125+
byte[] frame = Zstd.compress(PAYLOAD);
126+
try (Arena arena = Arena.ofConfined()) {
127+
MemorySegment seg = Zstd.copyIn(arena, frame);
128+
129+
// Then the segment overload agrees with the byte[] overload
130+
assertThat(ZstdFrame.decompressedSize(seg)).isEqualTo(ZstdFrame.decompressedSize(frame));
131+
}
132+
}
133+
134+
@Test
135+
void failsWhenAFrameDoesNotRecordItsSize() throws Exception {
136+
// Given a streamed frame with no pledged size
137+
ByteArrayOutputStream sink = new ByteArrayOutputStream();
138+
try (ZstdOutputStream zout = new ZstdOutputStream(sink)) {
139+
zout.write(PAYLOAD);
140+
}
141+
byte[] frame = sink.toByteArray();
142+
143+
// When the exact total is requested
144+
ThrowingCallable result = () -> ZstdFrame.decompressedSize(frame);
145+
146+
// Then it fails because the size was never recorded
147+
assertThatThrownBy(result)
148+
.isInstanceOf(ZstdException.class)
149+
.hasMessageContaining("not stored");
150+
}
151+
152+
@Test
153+
void rejectsGarbage() {
154+
// Given bytes that are not valid zstd data
155+
byte[] garbage = "xx".getBytes(StandardCharsets.UTF_8);
156+
157+
// When the exact total is requested
158+
ThrowingCallable result = () -> ZstdFrame.decompressedSize(garbage);
159+
160+
// Then it fails
161+
assertThatThrownBy(result).isInstanceOf(ZstdException.class);
162+
}
163+
}
164+
165+
@Nested
166+
class HeaderSize {
167+
168+
@Test
169+
void matchesTheParsedHeaderSize() {
170+
// Given a frame
171+
byte[] frame = Zstd.compress(PAYLOAD);
172+
173+
// Then the light probe agrees with the fully parsed header
174+
assertThat(ZstdFrame.headerSize(frame)).isEqualTo(ZstdFrame.header(frame).headerSize());
175+
}
176+
177+
@Test
178+
void computedFromJustTheHeaderPrefix() {
179+
// Given only the first 5 bytes of a frame (the minimum prefix)
180+
byte[] frame = Zstd.compress(PAYLOAD);
181+
byte[] prefix = Arrays.copyOf(frame, 5);
182+
183+
// Then the header size can still be determined, matching the full frame
184+
assertThat(ZstdFrame.headerSize(prefix)).isEqualTo(ZstdFrame.headerSize(frame));
185+
}
186+
187+
@Test
188+
void matchesThroughTheSegmentOverload() {
189+
// Given a frame in a native segment
190+
byte[] frame = Zstd.compress(PAYLOAD);
191+
try (Arena arena = Arena.ofConfined()) {
192+
MemorySegment seg = Zstd.copyIn(arena, frame);
193+
194+
// Then the segment overload agrees with the byte[] overload
195+
assertThat(ZstdFrame.headerSize(seg)).isEqualTo(ZstdFrame.headerSize(frame));
196+
}
197+
}
198+
199+
@Test
200+
void rejectsInputShorterThanThePrefix() {
201+
// Given fewer bytes than the header-size prefix
202+
byte[] tooShort = Arrays.copyOf(Zstd.compress(PAYLOAD), 2);
203+
204+
// When the header size is requested
205+
ThrowingCallable result = () -> ZstdFrame.headerSize(tooShort);
206+
207+
// Then it fails instead of guessing
208+
assertThatThrownBy(result).isInstanceOf(ZstdException.class);
209+
}
210+
}
211+
97212
@Nested
98213
class Header {
99214

0 commit comments

Comments
 (0)