Skip to content

Commit e31290a

Browse files
dfa1claude
dfa1
and
claude
authored
docs: split README into Diátaxis docs pages (#2)
Move how-to guides, reference, and explanation out of the README into docs/how-to.md, docs/reference.md, and docs/explanation.md. README now carries the intro, doc table, quick-start, and license only. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent f703624 commit e31290a

4 files changed

Lines changed: 168 additions & 172 deletions

File tree

‎README.md‎

Lines changed: 8 additions & 172 deletions
Original file line numberDiff line numberDiff line change
@@ -17,187 +17,23 @@ straight from your own data.
1717
1818
## Documentation
1919

20-
The docs follow the [Diátaxis](https://diataxis.fr) framework — four kinds of
21-
documentation, each serving a different need:
20+
The docs follow the [Diátaxis](https://diataxis.fr) framework:
2221

2322
| | Purpose | Start here |
2423
|---|---|---|
25-
| **[Tutorial](docs/tutorial.md)** | Learning by doing | [Getting started](docs/tutorial.md) |
26-
| **[How-to guides](#how-to-guides)** | Solving a specific task | [Hot paths](#compress-on-a-hot-path), [Dictionaries](#compress-many-small-payloads-with-a-dictionary), [Zero-copy](#avoid-heap-copies-with-memorysegment), [Self-built lib](#run-against-a-self-built-libzstd) |
27-
| **[Reference](#reference)** | Looking up facts | [Platforms](#supported-platforms), [API surface](#api-surface), [Symbol coverage](docs/supported.md), [Build](#build-from-source) |
28-
| **[Explanation](#explanation)** | Understanding the why | [Why FFM + Zig](#why-ffm-and-zig), [When zero-copy pays](docs/zero-copy.md), [Benchmarks](docs/benchmarks.md) |
29-
30-
---
31-
32-
## Tutorial: Getting started
33-
34-
New here? **[docs/tutorial.md](docs/tutorial.md)** takes you from a clean checkout
35-
to your first compress/decompress round-trip, step by step.
36-
37-
## How-to guides
38-
39-
Task-focused recipes. Each assumes you have the library on the classpath (see the
40-
[tutorial](#tutorial-getting-started)).
41-
42-
### Compress on a hot path
43-
44-
Reuse a context to amortise native allocation across many calls:
45-
46-
```java
47-
try (ZstdCompressCtx cctx = new ZstdCompressCtx().level(19);
48-
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
49-
byte[] packed = cctx.compress(message);
50-
byte[] restored = dctx.decompress(packed, message.length);
51-
}
52-
```
53-
54-
Pick the level explicitly with `Zstd.maxCompressionLevel()` /
55-
`minCompressionLevel()` when you need the extreme ends.
56-
57-
### Compress many small payloads with a dictionary
58-
59-
For many small, similar payloads (log lines, JSON records, protobufs), a
60-
dictionary compresses each one far smaller than it could be alone. Train one on
61-
representative samples:
62-
63-
```java
64-
ZstdDictionary dict = ZstdDictionary.train(sampleRecords, 16 * 1024);
65-
66-
try (ZstdCompressCtx cctx = new ZstdCompressCtx();
67-
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
68-
byte[] packed = cctx.compress(record, dict);
69-
byte[] restored = dctx.decompress(packed, record.length, dict);
70-
}
71-
72-
byte[] persisted = dict.toByteArray(); // store / ship the dictionary
73-
ZstdDictionary reloaded = ZstdDictionary.of(persisted);
74-
```
75-
76-
On a hot path, digest the dictionary once to skip per-call setup:
77-
78-
```java
79-
try (ZstdCompressDict cdict = new ZstdCompressDict(dict, 19);
80-
ZstdDecompressDict ddict = new ZstdDecompressDict(dict);
81-
ZstdCompressCtx cctx = new ZstdCompressCtx();
82-
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
83-
byte[] packed = cctx.compress(record, cdict);
84-
byte[] restored = dctx.decompress(packed, record.length, ddict);
85-
}
86-
```
87-
88-
### Avoid heap copies with `MemorySegment`
89-
90-
When your data is already off-heap — an `mmap` slice in, an arena buffer out —
91-
use the `MemorySegment` overloads to skip the heap `byte[]` bounce entirely. FFM
92-
hands zstd the segment address directly: no copy in, no copy out, no GC churn.
93-
94-
```java
95-
try (Arena arena = Arena.ofConfined();
96-
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
97-
MemorySegment frame = reader.mmapSlice(); // already native
98-
long n = Zstd.decompressedSize(frame); // read header, no copy
99-
MemorySegment out = arena.allocate(n); // becomes the backing buffer
100-
dctx.decompress(out, frame); // native → native
101-
}
102-
```
103-
104-
There are matching `compress(dst, src)` / `decompress(dst, src)` overloads (plus
105-
dictionary variants) returning the number of bytes written. For *why and when*
106-
this pays off, see the [explanation](docs/zero-copy.md).
107-
108-
### Run against a self-built libzstd
109-
110-
To use a `libzstd` you built yourself instead of the bundled one, point the
111-
loader at it:
112-
113-
```bash
114-
java -Dzstd.lib.path=/path/to/libzstd.dylib --enable-native-access=ALL-UNNAMED ...
115-
```
116-
117-
Build any of the six targets from any host:
118-
119-
```bash
120-
./scripts/build-zstd.sh <output-resources-dir> <classifier>
121-
# classifier: osx-aarch64 | osx-x86_64 | linux-x86_64 | linux-aarch64
122-
# | windows-x86_64 | windows-aarch64
123-
```
124-
125-
## Reference
126-
127-
### Supported platforms
128-
129-
The library — `io.github.dfa1.zstd:zstd` — ships as a pure-Java module plus one
130-
native artifact per platform:
131-
132-
| OS | aarch64 | x86_64 |
133-
|---------|:-------:|:------:|
134-
| macOS | ✅ | ✅ |
135-
| Linux | ✅ | ✅ |
136-
| Windows | ✅ | ✅ |
137-
138-
### API surface
139-
140-
| Type | Role |
141-
|---|---|
142-
| `Zstd` | one-shot `compress` / `decompress`, level + version queries, `compressBound`, `decompressedSize` |
143-
| `ZstdCompressCtx` / `ZstdDecompressCtx` | reusable contexts; `byte[]` and `MemorySegment` overloads, dictionary variants |
144-
| `ZstdDictionary` | train (`ZDICT`), load, persist, query dict id |
145-
| `ZstdCompressDict` / `ZstdDecompressDict` | pre-digested dictionaries for hot paths |
146-
| `ZstdFrame` | frame inspection: header, sizes, dict id, skippable frames |
147-
| `ZstdException` / `ZstdErrorCode` | typed errors mapped from zstd's sentinels |
148-
149-
### Symbol coverage
150-
151-
Which zstd C symbols are bound (and which deprecated ones are intentionally not),
152-
with a per-area breakdown and a comparison against zstd-jni:
153-
[docs/supported.md](docs/supported.md).
154-
155-
### Runtime requirement
156-
157-
Native access requires `--enable-native-access=ALL-UNNAMED` (or your module name)
158-
on the JVM command line.
159-
160-
### Build from source
161-
162-
Requires JDK 25+, Maven, and [Zig](https://ziglang.org/) on `PATH`.
24+
| **[Tutorial](docs/tutorial.md)** | Learning by doing | Clean checkout → first round-trip |
25+
| **[How-to guides](docs/how-to.md)** | Solving a specific task | Hot paths, dictionaries, zero-copy, self-built lib |
26+
| **[Reference](docs/reference.md)** | Looking up facts | Platforms, API surface, symbol coverage, build |
27+
| **[Explanation](docs/explanation.md)** | Understanding the why | Why FFM + Zig, when zero-copy pays, benchmarks |
16328

16429
```bash
16530
git clone --recurse-submodules https://github.com/dfa1/zstd-java.git
166-
cd zstd-java
167-
mvn test
31+
cd zstd-java && mvn test
16832
```
16933

170-
`scripts/build-zstd.sh` compiles `libzstd.{dylib,so,dll}` from the
171-
`third_party/zstd` submodule (pinned to tag `v1.5.7`) with `zig cc`, cross-compiling
172-
any of the six targets from any host.
34+
Requires JDK 25+ and `--enable-native-access=ALL-UNNAMED` at runtime.
17335

174-
### License
36+
## License
17537

17638
[BSD 3-Clause](LICENSE) — the same primary license as zstd, which is bundled
17739
under its BSD terms (zstd is dual BSD / GPLv2, © Meta Platforms, Inc.).
178-
179-
## Explanation
180-
181-
### Why FFM and Zig
182-
183-
The bindings use the **Foreign Function & Memory API** rather than JNI: no
184-
hand-written C glue, no separate native compile step for the binding layer, and a
185-
direct path from Java to zstd's addresses — which is what makes the zero-copy
186-
`MemorySegment` API possible.
187-
188-
The native library itself is built from vendored zstd source via **`zig cc`** as
189-
a drop-in C compiler. zstd is pure C with no build-system dependencies, so the
190-
sources are compiled directly — no autotools, no CMake. Zig bundles clang and
191-
libc for every target, enabling hermetic cross-compilation without a sysroot:
192-
any host can build all six platform artifacts.
193-
194-
### When zero-copy pays off
195-
196-
The `MemorySegment` fast path eliminates the heap `byte[]` bounce and the
197-
per-call allocation it implies. The reasoning, and the cases where it does and
198-
does not matter, is in [docs/zero-copy.md](docs/zero-copy.md).
199-
200-
### Benchmarks
201-
202-
Throughput and allocation versus zstd-jni (JNI) and aircompressor (pure Java),
203-
including an async-profiler breakdown: [docs/benchmarks.md](docs/benchmarks.md).

‎docs/explanation.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Explanation
2+
3+
## Why FFM and Zig
4+
5+
The bindings use the **Foreign Function & Memory API** rather than JNI: no
6+
hand-written C glue, no separate native compile step for the binding layer, and a
7+
direct path from Java to zstd's addresses — which is what makes the zero-copy
8+
`MemorySegment` API possible.
9+
10+
The native library itself is built from vendored zstd source via **`zig cc`** as
11+
a drop-in C compiler. zstd is pure C with no build-system dependencies, so the
12+
sources are compiled directly — no autotools, no CMake. Zig bundles clang and
13+
libc for every target, enabling hermetic cross-compilation without a sysroot:
14+
any host can build all six platform artifacts.
15+
16+
## When zero-copy pays off
17+
18+
The `MemorySegment` fast path eliminates the heap `byte[]` bounce and the
19+
per-call allocation it implies. The reasoning, and the cases where it does and
20+
does not matter, is in [zero-copy.md](zero-copy.md).
21+
22+
## Benchmarks
23+
24+
Throughput and allocation versus zstd-jni (JNI) and aircompressor (pure Java),
25+
including an async-profiler breakdown: [benchmarks.md](benchmarks.md).

‎docs/how-to.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# How-to guides
2+
3+
Task-focused recipes. Each assumes you have the library on the classpath (see the
4+
[tutorial](tutorial.md)).
5+
6+
## Compress on a hot path
7+
8+
Reuse a context to amortise native allocation across many calls:
9+
10+
```java
11+
try (ZstdCompressCtx cctx = new ZstdCompressCtx().level(19);
12+
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
13+
byte[] packed = cctx.compress(message);
14+
byte[] restored = dctx.decompress(packed, message.length);
15+
}
16+
```
17+
18+
Pick the level explicitly with `Zstd.maxCompressionLevel()` /
19+
`minCompressionLevel()` when you need the extreme ends.
20+
21+
## Compress many small payloads with a dictionary
22+
23+
For many small, similar payloads (log lines, JSON records, protobufs), a
24+
dictionary compresses each one far smaller than it could be alone. Train one on
25+
representative samples:
26+
27+
```java
28+
ZstdDictionary dict = ZstdDictionary.train(sampleRecords, 16 * 1024);
29+
30+
try (ZstdCompressCtx cctx = new ZstdCompressCtx();
31+
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
32+
byte[] packed = cctx.compress(record, dict);
33+
byte[] restored = dctx.decompress(packed, record.length, dict);
34+
}
35+
36+
byte[] persisted = dict.toByteArray(); // store / ship the dictionary
37+
ZstdDictionary reloaded = ZstdDictionary.of(persisted);
38+
```
39+
40+
On a hot path, digest the dictionary once to skip per-call setup:
41+
42+
```java
43+
try (ZstdCompressDict cdict = new ZstdCompressDict(dict, 19);
44+
ZstdDecompressDict ddict = new ZstdDecompressDict(dict);
45+
ZstdCompressCtx cctx = new ZstdCompressCtx();
46+
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
47+
byte[] packed = cctx.compress(record, cdict);
48+
byte[] restored = dctx.decompress(packed, record.length, ddict);
49+
}
50+
```
51+
52+
## Avoid heap copies with `MemorySegment`
53+
54+
When your data is already off-heap — an `mmap` slice in, an arena buffer out —
55+
use the `MemorySegment` overloads to skip the heap `byte[]` bounce entirely. FFM
56+
hands zstd the segment address directly: no copy in, no copy out, no GC churn.
57+
58+
```java
59+
try (Arena arena = Arena.ofConfined();
60+
ZstdDecompressCtx dctx = new ZstdDecompressCtx()) {
61+
MemorySegment frame = reader.mmapSlice(); // already native
62+
long n = Zstd.decompressedSize(frame); // read header, no copy
63+
MemorySegment out = arena.allocate(n); // becomes the backing buffer
64+
dctx.decompress(out, frame); // native → native
65+
}
66+
```
67+
68+
There are matching `compress(dst, src)` / `decompress(dst, src)` overloads (plus
69+
dictionary variants) returning the number of bytes written. For *why and when*
70+
this pays off, see the [explanation](zero-copy.md).
71+
72+
## Run against a self-built libzstd
73+
74+
To use a `libzstd` you built yourself instead of the bundled one, point the
75+
loader at it:
76+
77+
```bash
78+
java -Dzstd.lib.path=/path/to/libzstd.dylib --enable-native-access=ALL-UNNAMED ...
79+
```
80+
81+
Build any of the six targets from any host:
82+
83+
```bash
84+
./scripts/build-zstd.sh <output-resources-dir> <classifier>
85+
# classifier: osx-aarch64 | osx-x86_64 | linux-x86_64 | linux-aarch64
86+
# | windows-x86_64 | windows-aarch64
87+
```

‎docs/reference.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Reference
2+
3+
## Supported platforms
4+
5+
The library — `io.github.dfa1.zstd:zstd` — ships as a pure-Java module plus one
6+
native artifact per platform:
7+
8+
| OS | aarch64 | x86_64 |
9+
|---------|:-------:|:------:|
10+
| macOS | ✅ | ✅ |
11+
| Linux | ✅ | ✅ |
12+
| Windows | ✅ | ✅ |
13+
14+
## API surface
15+
16+
| Type | Role |
17+
|---|---|
18+
| `Zstd` | one-shot `compress` / `decompress`, level + version queries, `compressBound`, `decompressedSize` |
19+
| `ZstdCompressCtx` / `ZstdDecompressCtx` | reusable contexts; `byte[]` and `MemorySegment` overloads, dictionary variants |
20+
| `ZstdDictionary` | train (`ZDICT`), load, persist, query dict id |
21+
| `ZstdCompressDict` / `ZstdDecompressDict` | pre-digested dictionaries for hot paths |
22+
| `ZstdFrame` | frame inspection: header, sizes, dict id, skippable frames |
23+
| `ZstdException` / `ZstdErrorCode` | typed errors mapped from zstd's sentinels |
24+
25+
## Symbol coverage
26+
27+
Which zstd C symbols are bound (and which deprecated ones are intentionally not),
28+
with a per-area breakdown and a comparison against zstd-jni:
29+
[supported.md](supported.md).
30+
31+
## Runtime requirement
32+
33+
Native access requires `--enable-native-access=ALL-UNNAMED` (or your module name)
34+
on the JVM command line.
35+
36+
## Build from source
37+
38+
Requires JDK 25+, Maven, and [Zig](https://ziglang.org/) on `PATH`.
39+
40+
```bash
41+
git clone --recurse-submodules https://github.com/dfa1/zstd-java.git
42+
cd zstd-java
43+
mvn test
44+
```
45+
46+
`scripts/build-zstd.sh` compiles `libzstd.{dylib,so,dll}` from the
47+
`third_party/zstd` submodule (pinned to tag `v1.5.7`) with `zig cc`, cross-compiling
48+
any of the six targets from any host.

0 commit comments

Comments
 (0)