Skip to content

Commit 880ab08

Browse files
dfa1claude
andcommitted
feat: frame header parsing + skippable frames
- ZstdFrame.header(): parse a frame header without decompressing, into a ZstdFrameHeader record (content size, window size, frame type, dict id, checksum flag) via ZSTD_getFrameHeader - skippable frames: writeSkippableFrame / readSkippableFrame (ZstdSkippableContent) / isSkippableFrame — embed and read arbitrary metadata a decoder skips - ZstdFrameType enum (STANDARD / SKIPPABLE) - bind ZSTD_getFrameHeader, ZSTD_isSkippableFrame, ZSTD_writeSkippableFrame, ZSTD_readSkippableFrame; compatibility.md now 48/186 Tests: header reports content size / checksum flag / type; skippable frame round-trips content + magic variant; standard frame is not skippable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 1b8edbf commit 880ab08

7 files changed

Lines changed: 236 additions & 9 deletions

File tree

docs/compatibility.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ and semantics follow the [official manual](https://facebook.github.io/zstd/doc/a
55

66
- zstd version: **1.6.0** (vendored `third_party/zstd`)
77
- Public symbols exported by `libzstd`: **186**
8-
- Bound so far: **44** (~24%)
8+
- Bound so far: **48** (~26%)
99

1010
"Bound" means the symbol has a `MethodHandle` in `Bindings` and is reachable
1111
through the public Java API. The rest are reachable from native code but not yet
@@ -28,7 +28,7 @@ low-level variants that an idiomatic Java API does not need.
2828
| Streaming — compress | 3 / 22 | `ZstdOutputStream` (compressStream2 + buffer sizes) |
2929
| Streaming — decompress | 3 / 15 | `ZstdInputStream` (decompressStream + buffer sizes) |
3030
| Advanced parameters | 4 / 38 | `CCtx_setParameter` + `compress2` (level, checksum, LDM, windowLog) and `C/DCtx_loadDictionary` (dictionary streams); MT/getBounds not bound |
31-
| Frame inspection | 5 / 13 | `ZstdFrame`: isFrame, compressedSize, decompressedBound, dictId, + getFrameContentSize; getFrameHeader/skippable not bound |
31+
| Frame inspection | 9 / 13 | `ZstdFrame`: isFrame, header, compressedSize, decompressedBound, dictId, skippable read/write; getFrameProgression/_advanced not bound |
3232
| Memory sizing | 0 / 14 | `sizeof_*` / `estimate*` accounting not bound |
3333
| Low-level block | 0 / 12 | expert block/continue API not bound |
3434
| Sequences | 0 / 5 | sequence producer API not bound |
@@ -56,14 +56,14 @@ low-level variants that an idiomatic Java API does not need.
5656
| `ZSTD_decompressStream`, `ZSTD_DStreamInSize`, `ZSTD_DStreamOutSize` | `ZstdInputStream` |
5757
| `ZSTD_compress2`, `ZSTD_CCtx_setParameter` | `ZstdCompressCtx.parameter` / `checksum` / `longDistanceMatching` / `windowLog` (+ `ZstdCompressParameter`) |
5858
| `ZSTD_CCtx_loadDictionary`, `ZSTD_DCtx_loadDictionary` | `ZstdOutputStream` / `ZstdInputStream` dictionary constructors |
59-
| `ZSTD_isFrame`, `ZSTD_findFrameCompressedSize`, `ZSTD_decompressBound`, `ZSTD_getDictID_fromFrame` | `ZstdFrame` |
59+
| `ZSTD_isFrame`, `ZSTD_findFrameCompressedSize`, `ZSTD_decompressBound`, `ZSTD_getDictID_fromFrame`, `ZSTD_getFrameHeader`, `ZSTD_isSkippableFrame`, `ZSTD_writeSkippableFrame`, `ZSTD_readSkippableFrame` | `ZstdFrame` (+ `ZstdFrameHeader`, `ZstdFrameType`, `ZstdSkippableContent`) |
6060
| `ZSTD_getErrorCode` | `ZstdException.code()` (+ `ZstdErrorCode`) |
6161

6262
## Roadmap (priority order)
6363

6464
1. ~~**Streaming**~~ — done: `ZstdOutputStream` / `ZstdInputStream` (`compressStream2`, `decompressStream`, bounded buffers, dictionary constructors). Remaining: `MemorySegment`-buffer driver, `pledgedSrcSize`.
6565
2. **Advanced parameters** — done for compression: `CCtx_setParameter` + `compress2` via `ZstdCompressCtx` (`checksum`, `longDistanceMatching`, `windowLog`, generic `parameter`). Remaining: `cParam_getBounds`, `pledgedSrcSize`, and `nbWorkers` (needs a multithreaded native build).
66-
3. **Frame inspection** — done: `ZstdFrame` (`isFrame`, `compressedSize`, `decompressedBound`, `dictId`). Remaining: `getFrameHeader` (struct out-param), skippable frames, `getDictID_fromDict/CDict/DDict`.
66+
3. ~~**Frame inspection**~~ — done: `ZstdFrame` (`isFrame`, `header`, `compressedSize`, `decompressedBound`, `dictId`, skippable read/write). Remaining: `getDictID_fromDict/CDict/DDict`, `getFrameProgression`.
6767
4. **Better dictionaries**`ZDICT_optimizeTrainFromBuffer_cover` / `_fastCover`, `finalizeDictionary`.
6868
5. ~~**Typed errors**~~ — done: `ZstdException.code()` returns `ZstdErrorCode` (via `getErrorCode`).
6969

@@ -247,7 +247,7 @@ low-level variants that an idiomatic Java API does not need.
247247
| `ZSTD_estimateCCtxSize_usingCCtxParams` ||
248248
| `ZSTD_freeCCtxParams` ||
249249

250-
### Frame inspection (5/13)
250+
### Frame inspection (9/13)
251251

252252
| Symbol | Bound |
253253
|---|:---:|
@@ -257,13 +257,13 @@ low-level variants that an idiomatic Java API does not need.
257257
| `ZSTD_findFrameCompressedSize` ||
258258
| `ZSTD_frameHeaderSize` ||
259259
| `ZSTD_getDecompressedSize` ||
260-
| `ZSTD_getFrameHeader` | |
260+
| `ZSTD_getFrameHeader` | |
261261
| `ZSTD_getFrameHeader_advanced` ||
262262
| `ZSTD_getFrameProgression` ||
263263
| `ZSTD_isFrame` ||
264-
| `ZSTD_isSkippableFrame` | |
265-
| `ZSTD_readSkippableFrame` | |
266-
| `ZSTD_writeSkippableFrame` | |
264+
| `ZSTD_isSkippableFrame` | |
265+
| `ZSTD_readSkippableFrame` | |
266+
| `ZSTD_writeSkippableFrame` | |
267267

268268
### Memory sizing (0/14)
269269

zstd/src/main/java/io/github/dfa1/zstdffm/Bindings.java

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,22 @@ final class Bindings {
5151
static final MethodHandle GET_DICT_ID_FROM_FRAME =
5252
NativeLibrary.lookup("ZSTD_getDictID_fromFrame",
5353
FunctionDescriptor.of(JAVA_INT, ADDRESS, JAVA_LONG));
54+
// size_t ZSTD_getFrameHeader(ZSTD_FrameHeader* zfh, const void* src, size_t srcSize)
55+
static final MethodHandle GET_FRAME_HEADER =
56+
NativeLibrary.lookup("ZSTD_getFrameHeader",
57+
FunctionDescriptor.of(JAVA_LONG, ADDRESS, ADDRESS, JAVA_LONG));
58+
// unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size)
59+
static final MethodHandle IS_SKIPPABLE_FRAME =
60+
NativeLibrary.lookup("ZSTD_isSkippableFrame",
61+
FunctionDescriptor.of(JAVA_INT, ADDRESS, JAVA_LONG));
62+
// size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCap, const void* src, size_t srcSize, unsigned magicVariant)
63+
static final MethodHandle WRITE_SKIPPABLE_FRAME =
64+
NativeLibrary.lookup("ZSTD_writeSkippableFrame",
65+
FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG, ADDRESS, JAVA_LONG, JAVA_INT));
66+
// size_t ZSTD_readSkippableFrame(void* dst, size_t dstCap, unsigned* magicVariant, const void* src, size_t srcSize)
67+
static final MethodHandle READ_SKIPPABLE_FRAME =
68+
NativeLibrary.lookup("ZSTD_readSkippableFrame",
69+
FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG, ADDRESS, ADDRESS, JAVA_LONG));
5470

5571
// unsigned ZSTD_isError(size_t code)
5672
static final MethodHandle IS_ERROR =

zstd/src/main/java/io/github/dfa1/zstdffm/ZstdFrame.java

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@
33
import java.lang.foreign.Arena;
44
import java.lang.foreign.MemorySegment;
55

6+
import static java.lang.foreign.ValueLayout.JAVA_INT;
7+
import static java.lang.foreign.ValueLayout.JAVA_LONG;
8+
69
/// Inspection of zstd frames without decompressing them — useful for parsing a
710
/// stream of concatenated frames, sizing buffers, or routing by dictionary id.
811
///
@@ -92,6 +95,105 @@ public static int dictId(MemorySegment data) {
9295
return dictId(data, data.byteSize());
9396
}
9497

98+
/// Parses the header of the first frame in `data` without decompressing it.
99+
///
100+
/// @param data a complete zstd frame (or at least its header)
101+
/// @return the parsed frame header
102+
/// @throws ZstdException if the input is not a valid frame or the header is incomplete
103+
public static ZstdFrameHeader header(byte[] data) {
104+
try (Arena arena = Arena.ofConfined()) {
105+
return header(Zstd.copyIn(arena, data), data.length);
106+
}
107+
}
108+
109+
/// Parses the header of the first frame in native `data`.
110+
///
111+
/// @param data a complete zstd frame (or at least its header)
112+
/// @return the parsed frame header
113+
/// @throws ZstdException if the input is not a valid frame or the header is incomplete
114+
public static ZstdFrameHeader header(MemorySegment data) {
115+
return header(data, data.byteSize());
116+
}
117+
118+
/// Tests whether `data` begins with a skippable frame (user data a decoder ignores).
119+
///
120+
/// @param data the bytes to inspect
121+
/// @return `true` if `data` starts with a skippable frame
122+
public static boolean isSkippableFrame(byte[] data) {
123+
try (Arena arena = Arena.ofConfined()) {
124+
return isSkippableFrame(Zstd.copyIn(arena, data), data.length);
125+
}
126+
}
127+
128+
/// Tests whether native `data` begins with a skippable frame.
129+
///
130+
/// @param data the native bytes to inspect
131+
/// @return `true` if `data` starts with a skippable frame
132+
public static boolean isSkippableFrame(MemorySegment data) {
133+
return isSkippableFrame(data, data.byteSize());
134+
}
135+
136+
/// Wraps `content` in a skippable frame — arbitrary metadata that a zstd
137+
/// decoder skips over, letting you interleave it with compressed frames.
138+
///
139+
/// @param content the user bytes to embed
140+
/// @param magicVariant the variant 0..15 selecting one of the skippable magic numbers
141+
/// @return the skippable frame bytes
142+
/// @throws ZstdException if `magicVariant` is out of range
143+
public static byte[] writeSkippableFrame(byte[] content, int magicVariant) {
144+
try (Arena arena = Arena.ofConfined()) {
145+
MemorySegment src = Zstd.copyIn(arena, content);
146+
long cap = content.length + 8L;
147+
MemorySegment dst = arena.allocate(cap);
148+
long written = Zstd.call(() -> (long) Bindings.WRITE_SKIPPABLE_FRAME.invokeExact(
149+
dst, cap, src, (long) content.length, magicVariant));
150+
return Zstd.copyOut(dst, written);
151+
}
152+
}
153+
154+
/// Reads the content of a skippable frame produced by
155+
/// [#writeSkippableFrame(byte[], int)].
156+
///
157+
/// @param frame a skippable frame
158+
/// @return the embedded content and its magic variant
159+
/// @throws ZstdException if `frame` is not a skippable frame
160+
public static ZstdSkippableContent readSkippableFrame(byte[] frame) {
161+
try (Arena arena = Arena.ofConfined()) {
162+
MemorySegment src = Zstd.copyIn(arena, frame);
163+
MemorySegment magic = arena.allocate(JAVA_INT);
164+
MemorySegment dst = arena.allocate(Math.max(frame.length, 1));
165+
long written = Zstd.call(() -> (long) Bindings.READ_SKIPPABLE_FRAME.invokeExact(
166+
dst, (long) frame.length, magic, src, (long) frame.length));
167+
return new ZstdSkippableContent(Zstd.copyOut(dst, written), magic.get(JAVA_INT, 0));
168+
}
169+
}
170+
171+
private static ZstdFrameHeader header(MemorySegment data, long size) {
172+
try (Arena arena = Arena.ofConfined()) {
173+
MemorySegment zfh = arena.allocate(48);
174+
long remaining = Zstd.call(() -> (long) Bindings.GET_FRAME_HEADER.invokeExact(zfh, data, size));
175+
if (remaining != 0) {
176+
throw new ZstdException("incomplete frame header: need " + remaining + " more bytes");
177+
}
178+
return new ZstdFrameHeader(
179+
zfh.get(JAVA_LONG, 0), // frameContentSize
180+
zfh.get(JAVA_LONG, 8), // windowSize
181+
zfh.get(JAVA_INT, 16) & 0xFFFFFFFFL, // blockSizeMax
182+
ZstdFrameType.of(zfh.get(JAVA_INT, 20)), // frameType
183+
zfh.get(JAVA_INT, 24), // headerSize
184+
zfh.get(JAVA_INT, 28), // dictID
185+
zfh.get(JAVA_INT, 32) != 0); // checksumFlag
186+
}
187+
}
188+
189+
private static boolean isSkippableFrame(MemorySegment data, long size) {
190+
try {
191+
return ((int) Bindings.IS_SKIPPABLE_FRAME.invokeExact(data, size)) != 0;
192+
} catch (Throwable t) {
193+
throw rethrow(t);
194+
}
195+
}
196+
95197
private static boolean isZstdFrame(MemorySegment data, long size) {
96198
try {
97199
return ((int) Bindings.IS_FRAME.invokeExact(data, size)) != 0;
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
package io.github.dfa1.zstdffm;
2+
3+
import java.util.OptionalLong;
4+
5+
/// Parsed contents of a zstd frame header, from `ZSTD_getFrameHeader`.
6+
///
7+
/// @param frameContentSize the stored decompressed size, or a sentinel if unknown;
8+
/// prefer [#contentSize()]
9+
/// @param windowSize the back-reference window size needed to decode the frame
10+
/// @param blockSizeMax the maximum block size used in the frame
11+
/// @param frameType whether this is a standard or skippable frame
12+
/// @param headerSize the size of the frame header in bytes
13+
/// @param dictId the dictionary id, or `0` if none (for a skippable frame,
14+
/// the magic variant 0..15)
15+
/// @param hasChecksum whether a 4-byte content checksum follows the frame
16+
public record ZstdFrameHeader(
17+
long frameContentSize,
18+
long windowSize,
19+
long blockSizeMax,
20+
ZstdFrameType frameType,
21+
int headerSize,
22+
int dictId,
23+
boolean hasChecksum) {
24+
25+
/// Sentinel meaning the decompressed size is not recorded in the frame.
26+
private static final long CONTENTSIZE_UNKNOWN = -1L;
27+
28+
/// The decompressed size, if the frame records it.
29+
///
30+
/// @return the content size, or empty if the frame does not store it
31+
public OptionalLong contentSize() {
32+
return frameContentSize == CONTENTSIZE_UNKNOWN
33+
? OptionalLong.empty()
34+
: OptionalLong.of(frameContentSize);
35+
}
36+
}
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
package io.github.dfa1.zstdffm;
2+
3+
/// The kind of a zstd frame, from `ZSTD_FrameType_e`.
4+
public enum ZstdFrameType {
5+
6+
/// A normal compressed zstd frame.
7+
STANDARD,
8+
/// A skippable frame: arbitrary user data a zstd decoder ignores.
9+
SKIPPABLE;
10+
11+
static ZstdFrameType of(int value) {
12+
return value == 1 ? SKIPPABLE : STANDARD;
13+
}
14+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
package io.github.dfa1.zstdffm;
2+
3+
/// The decoded payload of a skippable frame, returned by
4+
/// [ZstdFrame#readSkippableFrame(byte[])].
5+
///
6+
/// @param content the user bytes carried by the skippable frame
7+
/// @param magicVariant the variant 0..15 the frame was written with
8+
public record ZstdSkippableContent(byte[] content, int magicVariant) {
9+
}

zstd/src/test/java/io/github/dfa1/zstdffm/ZstdFrameTest.java

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,56 @@ void rejectsGarbage() {
7878
}
7979
}
8080

81+
@Nested
82+
class Header {
83+
84+
@Test
85+
void reportsContentSizeAndNoChecksumByDefault() {
86+
// Given a default frame
87+
ZstdFrameHeader header = ZstdFrame.header(Zstd.compress(PAYLOAD));
88+
89+
// Then it is a standard frame storing the content size, no checksum, no dict
90+
assertThat(header.frameType()).isEqualTo(ZstdFrameType.STANDARD);
91+
assertThat(header.contentSize()).hasValue(PAYLOAD.length);
92+
assertThat(header.hasChecksum()).isFalse();
93+
assertThat(header.dictId()).isZero();
94+
assertThat(header.windowSize()).isPositive();
95+
}
96+
97+
@Test
98+
void reflectsChecksumFlag() {
99+
byte[] frame;
100+
try (ZstdCompressCtx ctx = new ZstdCompressCtx().checksum(true)) {
101+
frame = ctx.compress(PAYLOAD);
102+
}
103+
assertThat(ZstdFrame.header(frame).hasChecksum()).isTrue();
104+
}
105+
}
106+
107+
@Nested
108+
class Skippable {
109+
110+
@Test
111+
void roundTripsContentAndMagicVariant() {
112+
// Given user metadata wrapped in a skippable frame
113+
byte[] meta = "sidecar metadata".getBytes(StandardCharsets.UTF_8);
114+
byte[] frame = ZstdFrame.writeSkippableFrame(meta, 7);
115+
116+
// Then it is recognised as skippable and decodes back with its variant
117+
assertThat(ZstdFrame.isSkippableFrame(frame)).isTrue();
118+
assertThat(ZstdFrame.header(frame).frameType()).isEqualTo(ZstdFrameType.SKIPPABLE);
119+
120+
ZstdSkippableContent read = ZstdFrame.readSkippableFrame(frame);
121+
assertThat(read.content()).isEqualTo(meta);
122+
assertThat(read.magicVariant()).isEqualTo(7);
123+
}
124+
125+
@Test
126+
void standardFrameIsNotSkippable() {
127+
assertThat(ZstdFrame.isSkippableFrame(Zstd.compress(PAYLOAD))).isFalse();
128+
}
129+
}
130+
81131
@Nested
82132
class DictId {
83133

0 commit comments

Comments
 (0)