feat: frame inspection — ZstdFrame

dfa1 · claude · dfa1 · commit 53aced52bb22 · 2026-06-25T22:44:23.000+02:00
Inspect zstd frames without decompressing: parse concatenated frames, size
buffers, route by dictionary id. byte[] and zero-copy MemorySegment forms.

- ZstdFrame.isZstdFrame / compressedSize (first frame length) / decompressedBound
  (multi-frame upper bound) / dictId (0 if none)
- bind ZSTD_isFrame, ZSTD_findFrameCompressedSize, ZSTD_decompressBound,
  ZSTD_getDictID_fromFrame; compatibility.md now 43/186

Tests: recognises frames vs garbage; compressedSize locates the first-frame
boundary in concatenated frames; decompressedBound &gt;= original; dictId is 0
without a dictionary and matches the dictionary used.

Co-Authored-By: Claude Opus 4.8 &lt;noreply@anthropic.com&gt;
diff --git a/docs/compatibility.md b/docs/compatibility.md
@@ -5,7 +5,7 @@ and semantics follow the [official manual](https://facebook.github.io/zstd/doc/a
 
 - zstd version: **1.6.0** (vendored `third_party/zstd`)
 - Public symbols exported by `libzstd`: **186**
-- Bound so far: **39** (~21%)
+- Bound so far: **43** (~23%)
 
 "Bound" means the symbol has a `MethodHandle` in `Bindings` and is reachable
 through the public Java API. The rest are reachable from native code but not yet
@@ -28,7 +28,7 @@ low-level variants that an idiomatic Java API does not need.
 | Streaming — compress | 3 / 22 | `ZstdOutputStream` (compressStream2 + buffer sizes) |
 | Streaming — decompress | 3 / 15 | `ZstdInputStream` (decompressStream + buffer sizes) |
 | Advanced parameters | 4 / 38 | `CCtx_setParameter` + `compress2` (level, checksum, LDM, windowLog) and `C/DCtx_loadDictionary` (dictionary streams); MT/getBounds not bound |
-| Frame inspection | 1 / 13 | only `getFrameContentSize`; header/skippable/dictID-from-frame not bound |
+| Frame inspection | 5 / 13 | `ZstdFrame`: isFrame, compressedSize, decompressedBound, dictId, + getFrameContentSize; getFrameHeader/skippable not bound |
 | Memory sizing | 0 / 14 | `sizeof_*` / `estimate*` accounting not bound |
 | Low-level block | 0 / 12 | expert block/continue API not bound |
 | Sequences | 0 / 5 | sequence producer API not bound |
@@ -56,12 +56,13 @@ low-level variants that an idiomatic Java API does not need.
 | `ZSTD_decompressStream`, `ZSTD_DStreamInSize`, `ZSTD_DStreamOutSize` | `ZstdInputStream` |
 | `ZSTD_compress2`, `ZSTD_CCtx_setParameter` | `ZstdCompressCtx.parameter` / `checksum` / `longDistanceMatching` / `windowLog` (+ `ZstdCompressParameter`) |
 | `ZSTD_CCtx_loadDictionary`, `ZSTD_DCtx_loadDictionary` | `ZstdOutputStream` / `ZstdInputStream` dictionary constructors |
+| `ZSTD_isFrame`, `ZSTD_findFrameCompressedSize`, `ZSTD_decompressBound`, `ZSTD_getDictID_fromFrame` | `ZstdFrame` |
 
 ## Roadmap (priority order)
 
 1. ~~**Streaming**~~ — done: `ZstdOutputStream` / `ZstdInputStream` (`compressStream2`, `decompressStream`, bounded buffers, dictionary constructors). Remaining: `MemorySegment`-buffer driver, `pledgedSrcSize`.
 2. **Advanced parameters** — done for compression: `CCtx_setParameter` + `compress2` via `ZstdCompressCtx` (`checksum`, `longDistanceMatching`, `windowLog`, generic `parameter`). Remaining: `cParam_getBounds`, `pledgedSrcSize`, and `nbWorkers` (needs a multithreaded native build).
-3. **Frame inspection** — `getFrameHeader`, `isFrame`, `findFrameCompressedSize`, `decompressBound`, `getDictID_from*`, skippable frames.
+3. **Frame inspection** — done: `ZstdFrame` (`isFrame`, `compressedSize`, `decompressedBound`, `dictId`). Remaining: `getFrameHeader` (struct out-param), skippable frames, `getDictID_fromDict/CDict/DDict`.
 4. **Better dictionaries** — `ZDICT_optimizeTrainFromBuffer_cover` / `_fastCover`, `finalizeDictionary`.
 5. **Typed errors** — `ZSTD_getErrorCode` / `ZSTD_getErrorString`.
 
@@ -136,7 +137,7 @@ low-level variants that an idiomatic Java API does not need.
 | `ZSTD_getDictID_fromCDict` | — |
 | `ZSTD_getDictID_fromDDict` | — |
 | `ZSTD_getDictID_fromDict` | — |
-| `ZSTD_getDictID_fromFrame` | — |
+| `ZSTD_getDictID_fromFrame` | ✅ |
 
 ### Dictionary training, ZDICT (4/12)
 
@@ -245,20 +246,20 @@ low-level variants that an idiomatic Java API does not need.
 | `ZSTD_estimateCCtxSize_usingCCtxParams` | — |
 | `ZSTD_freeCCtxParams` | — |
 
-### Frame inspection (1/13)
+### Frame inspection (5/13)
 
 | Symbol | Bound |
 |---|:---:|
 | `ZSTD_getFrameContentSize` | ✅ |
-| `ZSTD_decompressBound` | — |
+| `ZSTD_decompressBound` | ✅ |
 | `ZSTD_findDecompressedSize` | — |
-| `ZSTD_findFrameCompressedSize` | — |
+| `ZSTD_findFrameCompressedSize` | ✅ |
 | `ZSTD_frameHeaderSize` | — |
 | `ZSTD_getDecompressedSize` | — |
 | `ZSTD_getFrameHeader` | — |
 | `ZSTD_getFrameHeader_advanced` | — |
 | `ZSTD_getFrameProgression` | — |
-| `ZSTD_isFrame` | — |
+| `ZSTD_isFrame` | ✅ |
 | `ZSTD_isSkippableFrame` | — |
 | `ZSTD_readSkippableFrame` | — |
 | `ZSTD_writeSkippableFrame` | — |
diff --git a/zstd/src/main/java/io/github/dfa1/zstdffm/Bindings.java b/zstd/src/main/java/io/github/dfa1/zstdffm/Bindings.java
@@ -36,6 +36,22 @@ final class Bindings {
             NativeLibrary.lookup("ZSTD_getFrameContentSize",
                     FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG));
 
+    // unsigned ZSTD_isFrame(const void* buffer, size_t size)
+    static final MethodHandle IS_FRAME =
+            NativeLibrary.lookup("ZSTD_isFrame", FunctionDescriptor.of(JAVA_INT, ADDRESS, JAVA_LONG));
+    // size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize)
+    static final MethodHandle FIND_FRAME_COMPRESSED_SIZE =
+            NativeLibrary.lookup("ZSTD_findFrameCompressedSize",
+                    FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG));
+    // unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize)
+    static final MethodHandle DECOMPRESS_BOUND =
+            NativeLibrary.lookup("ZSTD_decompressBound",
+                    FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_LONG));
+    // unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize)
+    static final MethodHandle GET_DICT_ID_FROM_FRAME =
+            NativeLibrary.lookup("ZSTD_getDictID_fromFrame",
+                    FunctionDescriptor.of(JAVA_INT, ADDRESS, JAVA_LONG));
+
     // unsigned ZSTD_isError(size_t code)
     static final MethodHandle IS_ERROR =
             NativeLibrary.lookup("ZSTD_isError", FunctionDescriptor.of(JAVA_INT, JAVA_LONG));
diff --git a/zstd/src/main/java/io/github/dfa1/zstdffm/ZstdFrame.java b/zstd/src/main/java/io/github/dfa1/zstdffm/ZstdFrame.java
@@ -0,0 +1,136 @@
+package io.github.dfa1.zstdffm;
+
+import java.lang.foreign.Arena;
+import java.lang.foreign.MemorySegment;
+
+/// Inspection of zstd frames without decompressing them — useful for parsing a
+/// stream of concatenated frames, sizing buffers, or routing by dictionary id.
+///
+/// Each method has a `byte[]` form and a zero-copy {@link MemorySegment} form for
+/// data already off-heap (e.g. an mmap slice); see `docs/zero-copy.md`.
+public final class ZstdFrame {
+
+    /// Sentinel returned by `ZSTD_decompressBound` when the input is not valid.
+    private static final long CONTENTSIZE_ERROR = -2L;
+
+    /// Tests whether `data` begins with a valid zstd frame (standard or skippable).
+    ///
+    /// @param data the bytes to inspect
+    /// @return `true` if `data` starts with a zstd frame
+    public static boolean isZstdFrame(byte[] data) {
+        try (Arena arena = Arena.ofConfined()) {
+            return isZstdFrame(Zstd.copyIn(arena, data), data.length);
+        }
+    }
+
+    /// Tests whether the native `data` begins with a valid zstd frame.
+    ///
+    /// @param data the native bytes to inspect
+    /// @return `true` if `data` starts with a zstd frame
+    public static boolean isZstdFrame(MemorySegment data) {
+        return isZstdFrame(data, data.byteSize());
+    }
+
+    /// Compressed size of the first frame in `data`, including its header and
+    /// epilogue — i.e. where the next frame would begin.
+    ///
+    /// @param data one or more concatenated zstd frames
+    /// @return the byte length of the first frame
+    /// @throws ZstdException if the input is not a valid frame
+    public static long compressedSize(byte[] data) {
+        try (Arena arena = Arena.ofConfined()) {
+            return compressedSize(Zstd.copyIn(arena, data), data.length);
+        }
+    }
+
+    /// Compressed size of the first frame in native `data`.
+    ///
+    /// @param data one or more concatenated zstd frames
+    /// @return the byte length of the first frame
+    /// @throws ZstdException if the input is not a valid frame
+    public static long compressedSize(MemorySegment data) {
+        return compressedSize(data, data.byteSize());
+    }
+
+    /// Upper bound on the total decompressed size of all frames in `data`,
+    /// large enough to never overflow during decompression.
+    ///
+    /// @param data one or more concatenated zstd frames
+    /// @return an upper bound on the combined decompressed size
+    /// @throws ZstdException if the input is not valid zstd data
+    public static long decompressedBound(byte[] data) {
+        try (Arena arena = Arena.ofConfined()) {
+            return decompressedBound(Zstd.copyIn(arena, data), data.length);
+        }
+    }
+
+    /// Upper bound on the total decompressed size of all frames in native `data`.
+    ///
+    /// @param data one or more concatenated zstd frames
+    /// @return an upper bound on the combined decompressed size
+    /// @throws ZstdException if the input is not valid zstd data
+    public static long decompressedBound(MemorySegment data) {
+        return decompressedBound(data, data.byteSize());
+    }
+
+    /// Dictionary id recorded in the first frame's header.
+    ///
+    /// @param data a zstd frame
+    /// @return the dictionary id, or `0` if the frame uses no dictionary or does
+    ///         not record one
+    public static int dictId(byte[] data) {
+        try (Arena arena = Arena.ofConfined()) {
+            return dictId(Zstd.copyIn(arena, data), data.length);
+        }
+    }
+
+    /// Dictionary id recorded in native frame `data`.
+    ///
+    /// @param data a zstd frame
+    /// @return the dictionary id, or `0` if none
+    public static int dictId(MemorySegment data) {
+        return dictId(data, data.byteSize());
+    }
+
+    private static boolean isZstdFrame(MemorySegment data, long size) {
+        try {
+            return ((int) Bindings.IS_FRAME.invokeExact(data, size)) != 0;
+        } catch (Throwable t) {
+            throw rethrow(t);
+        }
+    }
+
+    private static long compressedSize(MemorySegment data, long size) {
+        return Zstd.call(() -> (long) Bindings.FIND_FRAME_COMPRESSED_SIZE.invokeExact(data, size));
+    }
+
+    private static long decompressedBound(MemorySegment data, long size) {
+        long bound;
+        try {
+            bound = (long) Bindings.DECOMPRESS_BOUND.invokeExact(data, size);
+        } catch (Throwable t) {
+            throw rethrow(t);
+        }
+        if (bound == CONTENTSIZE_ERROR) {
+            throw new ZstdException("not valid zstd data");
+        }
+        return bound;
+    }
+
+    private static int dictId(MemorySegment data, long size) {
+        try {
+            return (int) Bindings.GET_DICT_ID_FROM_FRAME.invokeExact(data, size);
+        } catch (Throwable t) {
+            throw rethrow(t);
+        }
+    }
+
+    @SuppressWarnings("unchecked")
+    private static <E extends Throwable> RuntimeException rethrow(Throwable t) throws E {
+        throw (E) t;
+    }
+
+    private ZstdFrame() {
+        // no instances
+    }
+}
diff --git a/zstd/src/test/java/io/github/dfa1/zstdffm/ZstdFrameTest.java b/zstd/src/test/java/io/github/dfa1/zstdffm/ZstdFrameTest.java
@@ -0,0 +1,110 @@
+package io.github.dfa1.zstdffm;
+
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+class ZstdFrameTest {
+
+    private static final byte[] PAYLOAD =
+            "frame inspection payload ".repeat(500).getBytes(StandardCharsets.UTF_8);
+
+    @Nested
+    class IsFrame {
+
+        @Test
+        void recognisesAZstdFrame() {
+            assertThat(ZstdFrame.isZstdFrame(Zstd.compress(PAYLOAD))).isTrue();
+        }
+
+        @Test
+        void rejectsGarbage() {
+            assertThat(ZstdFrame.isZstdFrame("not a frame".getBytes(StandardCharsets.UTF_8))).isFalse();
+        }
+    }
+
+    @Nested
+    class CompressedSize {
+
+        @Test
+        void equalsLengthForSingleFrame() {
+            // Given one frame
+            byte[] frame = Zstd.compress(PAYLOAD);
+
+            // Then its compressed size is the whole buffer
+            assertThat(ZstdFrame.compressedSize(frame)).isEqualTo(frame.length);
+        }
+
+        @Test
+        void locatesFirstFrameBoundaryWhenConcatenated() throws Exception {
+            // Given two frames back to back
+            byte[] first = Zstd.compress(PAYLOAD);
+            byte[] second = Zstd.compress("second".getBytes(StandardCharsets.UTF_8));
+            ByteArrayOutputStream both = new ByteArrayOutputStream();
+            both.write(first);
+            both.write(second);
+
+            // Then compressedSize reports where the second frame begins
+            assertThat(ZstdFrame.compressedSize(both.toByteArray())).isEqualTo(first.length);
+        }
+
+        @Test
+        void rejectsGarbage() {
+            assertThatThrownBy(() -> ZstdFrame.compressedSize("xxxx".getBytes(StandardCharsets.UTF_8)))
+                    .isInstanceOf(ZstdException.class);
+        }
+    }
+
+    @Nested
+    class DecompressedBound {
+
+        @Test
+        void boundsTheDecompressedSize() {
+            byte[] frame = Zstd.compress(PAYLOAD);
+            assertThat(ZstdFrame.decompressedBound(frame)).isGreaterThanOrEqualTo(PAYLOAD.length);
+        }
+
+        @Test
+        void rejectsGarbage() {
+            assertThatThrownBy(() -> ZstdFrame.decompressedBound("xx".getBytes(StandardCharsets.UTF_8)))
+                    .isInstanceOf(ZstdException.class);
+        }
+    }
+
+    @Nested
+    class DictId {
+
+        @Test
+        void isZeroForDictionarylessFrame() {
+            assertThat(ZstdFrame.dictId(Zstd.compress(PAYLOAD))).isZero();
+        }
+
+        @Test
+        void matchesTheDictionaryUsed() {
+            // Given a frame compressed with a dictionary
+            ZstdDictionary dict = trainDict();
+            byte[] frame;
+            try (ZstdCompressCtx ctx = new ZstdCompressCtx()) {
+                frame = ctx.compress(PAYLOAD, dict);
+            }
+
+            // Then the frame's dictionary id matches the dictionary
+            assertThat(ZstdFrame.dictId(frame)).isEqualTo(dict.id());
+        }
+
+        private ZstdDictionary trainDict() {
+            List<byte[]> samples = new ArrayList<>();
+            for (int i = 0; i < 3000; i++) {
+                samples.add(("{\"id\":" + i + ",\"k\":\"v" + (i % 30) + "\"}").getBytes(StandardCharsets.UTF_8));
+            }
+            return ZstdDictionary.train(samples, 8 * 1024);
+        }
+    }
+}
