|
10 | 10 |
|
11 | 11 | **zstd-java** is a Java wrapper for [Zstandard](https://github.com/facebook/zstd) |
12 | 12 | built on the **Foreign Function & Memory (FFM) API** — no JNI, no `sun.misc.Unsafe`. |
13 | | -It targets **JDK 25+** (for stable `java.lang.foreign`) and leads with the |
14 | | -feature missing from most JVM zstd bindings: **dictionary compression**, trained |
15 | | -straight from your own data. |
| 13 | +It targets **JDK 25+** (the first LTS with stable `java.lang.foreign`) and leads |
| 14 | +with the two features most JVM zstd bindings lack: |
| 15 | + |
| 16 | +- **Dictionary compression**, trained straight from your own data — the big win on |
| 17 | + small, repetitive records (logs, market-data ticks, JSON/Avro rows, FIX messages). |
| 18 | +- A **zero-copy `MemorySegment` API** — compress/decompress off-heap buffers (an |
| 19 | + mmap'd slice in, an arena buffer out) with no heap copy and no per-call allocation. |
16 | 20 |
|
17 | 21 | > **AI-assisted development:** This project uses Claude Code for implementation — |
18 | 22 | > C header mapping, test generation, docs. Architecture, API design, and all |
19 | 23 | > decisions are human-driven. |
20 | 24 |
|
21 | | -## Documentation |
| 25 | +## Quickstart |
22 | 26 |
|
23 | | -The docs follow the [Diátaxis](https://diataxis.fr) framework: |
| 27 | +One-shot round-trip with `byte[]` — the convenient path: |
24 | 28 |
|
25 | | -| | Purpose | Start here | |
26 | | -|---|---|---| |
27 | | -| **[Tutorial](docs/tutorial.md)** | Learning by doing | Clean checkout → first round-trip | |
28 | | -| **[How-to guides](docs/how-to.md)** | Solving a specific task | Hot paths, dictionaries, zero-copy, self-built lib | |
29 | | -| **[Reference](docs/reference.md)** | Looking up facts | Platforms, API surface, symbol coverage, build | |
30 | | -| **[Explanation](docs/explanation.md)** | Understanding the why | Why FFM + Zig, when zero-copy pays, benchmarks | |
| 29 | +```java |
| 30 | +import io.github.dfa1.zstd.Zstd; |
31 | 31 |
|
32 | | -Architecture decisions are recorded as [ADRs](adr/ADR.md) (MADR 3.0) — the |
33 | | -foundational choices and their trade-offs, one file per decision. |
| 32 | +byte[] data = ...; |
| 33 | +byte[] frame = Zstd.compress(data); // or Zstd.compress(data, level) |
| 34 | +byte[] back = Zstd.decompress(frame); // size read from the frame header |
| 35 | +``` |
| 36 | + |
| 37 | +**Dictionary** — train on a sample of your records, then compress each one against |
| 38 | +the dictionary (huge ratio gains on small, similar messages): |
| 39 | + |
| 40 | +```java |
| 41 | +import io.github.dfa1.zstd.*; |
| 42 | +import java.util.List; |
| 43 | + |
| 44 | +List<byte[]> samples = ...; // representative records |
| 45 | +ZstdDictionary dict = ZstdDictionary.train(samples, 8 * 1024); |
| 46 | + |
| 47 | +byte[] message = ...; |
| 48 | +try (ZstdCompressCtx cctx = new ZstdCompressCtx(); |
| 49 | + ZstdDecompressCtx dctx = new ZstdDecompressCtx()) { |
| 50 | + byte[] frame = cctx.compress(message, dict); |
| 51 | + byte[] back = dctx.decompress(frame, message.length, dict); |
| 52 | +} |
| 53 | +``` |
| 54 | + |
| 55 | +**Zero-copy** — off-heap in, off-heap out, no `byte[]`, no per-call allocation: |
| 56 | + |
| 57 | +```java |
| 58 | +import io.github.dfa1.zstd.*; |
| 59 | +import java.lang.foreign.*; |
| 60 | + |
| 61 | +try (Arena arena = Arena.ofConfined(); |
| 62 | + ZstdCompressCtx cctx = new ZstdCompressCtx(); |
| 63 | + ZstdDecompressCtx dctx = new ZstdDecompressCtx()) { |
| 64 | + |
| 65 | + MemorySegment src = ...; // e.g. an mmap'd file slice |
| 66 | + MemorySegment frame = cctx.compress(arena, src); // off-heap → off-heap |
| 67 | + MemorySegment restored = dctx.decompress(arena, frame); |
| 68 | +} |
| 69 | +``` |
| 70 | + |
| 71 | +Run with `--enable-native-access=ALL-UNNAMED`. Full walkthrough in the |
| 72 | +[tutorial](docs/tutorial.md); hot-path and dictionary recipes in the |
| 73 | +[how-to guides](docs/how-to.md). |
| 74 | + |
| 75 | +## Performance |
| 76 | + |
| 77 | +Microbenchmarks against the common JVM zstd options (JMH; Apple M5, JDK 25, all |
| 78 | +linking the same zstd 1.5.7). Full methodology and tables in |
| 79 | +[docs/benchmarks.md](docs/benchmarks.md) — including the honest ties. |
| 80 | + |
| 81 | +**Best vs best** — our zero-copy `MemorySegment` path vs **zstd-jni's own** |
| 82 | +zero-copy direct-`ByteBuffer` path (golden-corpus fixtures, publication-grade run): |
| 83 | + |
| 84 | +| operation (payload) | zstd-java `MemorySegment` | zstd-jni `ByteBuffer` | edge | |
| 85 | +|---|---:|---:|---:| |
| 86 | +| compress `http` (1.2 KiB) | **353.6** | 322.1 | +9.8% | |
| 87 | +| decompress `http` | **922.7** | 750.8 | +22.9% | |
| 88 | +| decompress `large-literal` (200 KiB) | 56.1 | 55.6 | tie | |
| 89 | + |
| 90 | +*(throughput, ops/ms, higher is better; allocation is **~0 B/op on both** — both genuinely zero-copy)* |
| 91 | + |
| 92 | +The edge is FFM's lower per-call overhead — **largest on small payloads**, |
| 93 | +converging to a tie when codec/bandwidth dominates. Against the *convenient* |
| 94 | +`byte[]` / JNI APIs (which allocate the output every call), the segment path is |
| 95 | +additionally **allocation-free**: flat ~0 B/op at any size vs MB/op that scales |
| 96 | +with the payload — no GC pressure on the hot path. |
34 | 97 |
|
35 | 98 | ## Install |
36 | 99 |
|
@@ -79,11 +142,26 @@ plus only the `zstd-native-<classifier>` you target. |
79 | 142 | ``` |
80 | 143 |
|
81 | 144 | Classifiers: `osx-aarch64`, `osx-x86_64`, `linux-x86_64`, `linux-aarch64`, |
82 | | -`windows-x86_64`, `windows-aarch64`. Gradle and more detail in the |
83 | | -[tutorial](docs/tutorial.md). Requires JDK 25+ and |
| 145 | +`windows-x86_64`, `windows-aarch64` — each verified on real hardware by the |
| 146 | +[release smoke matrix](.github/workflows/release-smoke.yml). Gradle and more |
| 147 | +detail in the [tutorial](docs/tutorial.md). Requires JDK 25+ and |
84 | 148 | `--enable-native-access=ALL-UNNAMED` at runtime. Building from source is for |
85 | 149 | contributors — see the [reference](docs/reference.md). |
86 | 150 |
|
| 151 | +## Documentation |
| 152 | + |
| 153 | +The docs follow the [Diátaxis](https://diataxis.fr) framework: |
| 154 | + |
| 155 | +| | Purpose | Start here | |
| 156 | +|---|---|---| |
| 157 | +| **[Tutorial](docs/tutorial.md)** | Learning by doing | Clean checkout → first round-trip | |
| 158 | +| **[How-to guides](docs/how-to.md)** | Solving a specific task | Hot paths, dictionaries, zero-copy, self-built lib | |
| 159 | +| **[Reference](docs/reference.md)** | Looking up facts | Platforms, API surface, symbol coverage, build | |
| 160 | +| **[Explanation](docs/explanation.md)** | Understanding the why | Why FFM + Zig, when zero-copy pays, benchmarks | |
| 161 | + |
| 162 | +Architecture decisions are recorded as [ADRs](adr/ADR.md) (MADR 3.0) — the |
| 163 | +foundational choices and their trade-offs, one file per decision. |
| 164 | + |
87 | 165 | ## License |
88 | 166 |
|
89 | 167 | [BSD 3-Clause](LICENSE) — the same primary license as zstd, which is bundled |
|
0 commit comments