Skip to content

Commit 83fce25

Browse files
dfa1claude
andcommitted
docs(benchmarks): add golden-corpus best-vs-best, fix the unfair JNI framing
The synthetic tables compared our zero-copy MemorySegment path against zstd-jni's *allocating* byte[] API — our best vs their worst — which inflated both the speed edge and the "allocation-free vs JNI" claim. Add a Golden corpus: best vs best section (our MemorySegment vs zstd-jni's own direct-ByteBuffer zero-copy path, same zstd 1.5.7, 3x3x5 + gc): allocation ties at ~0 B/op, throughput leads +9-10% compress / +23% small decompress and ties on large decompress — the expected FFM call-overhead shape. Rewrite the TL;DR to separate the two comparisons honestly, note zstd-jni's zero-copy mode in the contestant table, correct the environment block (same zstd version both sides; corpus run is publication-grade, synthetic is directional), and add the corpus reproduce command. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 6473877 commit 83fce25

1 file changed

Lines changed: 76 additions & 23 deletions

File tree

docs/benchmarks.md

Lines changed: 76 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -8,39 +8,87 @@ the [`benchmark/`](../benchmark) module; see its
88
| Contestant | Binding | Modes |
99
|------------|---------|-------|
1010
| **zstd-java** (this project) | FFM (no JNI) | `byte[]` and zero-copy `MemorySegment` |
11-
| **zstd-jni** (`com.github.luben`) | JNI | `byte[]` |
11+
| **zstd-jni** (`com.github.luben`) | JNI | `byte[]` **and** zero-copy direct `ByteBuffer` |
1212
| **aircompressor** (`io.airlift:aircompressor-v3`) | pure Java | caller-buffer `byte[]` |
1313

1414
## TL;DR
1515

16-
- **Throughput:** zstd-java's `MemorySegment` path is fastest at small/cache-resident
17-
payloads (~5-9% over `byte[]`, more over JNI). The edge **shrinks to nothing at
18-
64 MiB**, where everything is DRAM-bandwidth bound and converges. aircompressor
19-
(pure Java) trails the native bindings, ~2x on compress.
20-
- **Allocation:** this is the real `MemorySegment` win. The segment path is
21-
**allocation-free** (flat ~0 B/op at any size); the `byte[]`/JNI paths allocate
22-
~the output size **on every call**. At 64 MiB the `byte[]` path churns 67 MB/op
23-
through the young generation; JNI compress churns ~79 MB/op. Invisible in
24-
throughput, brutal under sustained load.
25-
26-
The headline isn't raw speed — it's **eliminating per-op heap allocation**, which
27-
matters most exactly where throughput converges (large, bandwidth-bound payloads
28-
under GC).
16+
- **Best vs best** (our `MemorySegment` vs zstd-jni's own zero-copy direct
17+
`ByteBuffer`, same zstd 1.5.7 both sides): **allocation is a tie** — both are
18+
~0 B/op. The throughput edge is **per-call overhead**: clearest on small,
19+
call-overhead-dominated payloads (+10–23%) and **converging to a tie** when
20+
compute or bandwidth dominates (large decompress, 64 MiB). This is the honest
21+
FFM-vs-JNI shape — biggest where the payload is smallest. See
22+
[Golden corpus: best vs best](#golden-corpus-best-vs-best).
23+
- **vs the allocating `byte[]` APIs:** the `MemorySegment` path is
24+
**allocation-free** (flat ~0 B/op at any size) while `byte[]` / JNI-`byte[]`
25+
allocate ~the output size **every call** (67–79 MB/op at 64 MiB). Real, but it
26+
compares our zero-copy path against their *heap* API — not their zero-copy one.
27+
The allocation win is over the convenient API, not over JNI per se.
28+
29+
The honest headline: against zstd-jni's *best* path we **match on allocation and
30+
lead modestly on call overhead**; against the convenient `byte[]` APIs the
31+
zero-copy path additionally eliminates per-op heap allocation.
2932

3033
## Environment
3134

3235
- Apple M5, 32 GB. P-core L2 16 MiB (Apple Silicon: shared SLC, no classic L3).
3336
The 64 MiB payload is the cache-busting case.
34-
- JDK 25 (Azul). zstd-jni 1.5.7-11, aircompressor-v3 3.6, JMH 1.37.
35-
- Level 3 (zstd default). Payloads are deterministic, ~3x-compressible text.
36-
37-
**These are a quick, low-iteration run (1 fork, 2 warmup, 3 measurement) — a
38-
directional read, not publication-grade. The 64 MiB rows in particular have wide
39-
intervals (few ops per iteration). Rerun with JMH defaults on the target host
40-
before quoting.**
37+
- JDK 25 (Azul). zstd-jni 1.5.7-11 (bundles zstd 1.5.7, matching our build),
38+
aircompressor-v3 3.6, JMH 1.37.
39+
- Level 3 (zstd default).
40+
- **Golden corpus** run: 3 forks × 3 warmup × 5 measurement, `-prof gc`, error
41+
bars are 99.9% CIs.
42+
- **Synthetic** tables below: deterministic ~3x-compressible text, and a quick,
43+
low-iteration run (1 fork, 2 warmup, 3 measurement) — directional, not
44+
publication-grade; the 64 MiB rows have wide intervals. Rerun with JMH defaults
45+
before quoting those.
46+
47+
## Golden corpus: best vs best
48+
49+
The fairest comparison: **our best zero-copy path against zstd-jni's best
50+
zero-copy path**, both reusing a context and off-heap buffers, neither allocating
51+
per call — our `MemorySegment` (`compressJavaSegment` / `decompressJavaSegment`)
52+
vs zstd-jni's direct-`ByteBuffer` API (`compressJniByteBuffer` /
53+
`decompressJniByteBuffer`). Inputs are real fixtures from zstd's own
54+
[golden corpus](../third_party/zstd/tests/golden-compression), not synthetic
55+
text, so the small/structured cases exercise per-call boundary overhead — exactly
56+
where FFM-vs-JNI differs. Both sides link **the same zstd 1.5.7**, so any gap is
57+
binding overhead, not codec version.
58+
59+
This run is publication-grade for the cut shown (3 forks × 3 warmup × 5
60+
measurement, `-prof gc`), on the environment below.
61+
62+
### Throughput (ops/ms, higher is better)
63+
64+
| file (size) | JavaSegment | JniByteBuffer | edge |
65+
|-------------|------------:|--------------:|-----:|
66+
| compress `http` (1.2 KiB) | **353.6** ±3.0 | 322.1 ±22.9 | +9.8% |
67+
| compress `large-literal` (200 KiB) | **46.1** ±1.4 | 42.2 ±0.3 | +9.4% |
68+
| decompress `http` | **922.7** ±5.9 | 750.8 ±0.9 | +22.9% |
69+
| decompress `large-literal` | 56.1 ±0.7 | 55.6 ±0.4 | +0.9% (tie) |
70+
71+
### Allocation (`gc.alloc.rate.norm`, B/op)
72+
73+
| | JavaSegment | JniByteBuffer |
74+
|-|------------:|--------------:|
75+
| every case | ~0.00 | ~0.00 |
76+
77+
**Reading it:** we lead ~+9–10% on compress and +23% on small decompress (the
78+
call-overhead-dominated cases), tie on large decompress (bandwidth-bound), and
79+
**match exactly on allocation** — both genuinely zero-copy. The earlier
80+
"allocation-free vs JNI" claim only held against JNI's `byte[]` API; against its
81+
zero-copy path the allocation advantage is gone, and the speed edge is the
82+
expected FFM call-overhead margin, largest at the smallest payloads.
4183

4284
## Throughput (ops/ms, higher is better)
4385

86+
> The tables below use the original **synthetic** payloads and compare against
87+
> zstd-jni's *allocating* `byte[]` API (`zstdJni`), not its zero-copy path. They
88+
> show the `MemorySegment`-vs-`byte[]` allocation story; for the fair
89+
> zero-copy-vs-zero-copy comparison see
90+
> [Golden corpus: best vs best](#golden-corpus-best-vs-best) above.
91+
4492
### Compress
4593

4694
| size | zstdJavaSegment | zstdJavaBytes | zstdJni | aircompressor |
@@ -137,8 +185,13 @@ removes entirely, and it dominates under sustained, allocation-sensitive load.
137185
```bash
138186
./mvnw -q -pl benchmark -am package -DskipTests
139187

140-
# throughput + allocation, all sizes
141-
java -jar benchmark/target/benchmarks.jar -prof gc
188+
# golden corpus, best vs best (our MemorySegment vs zstd-jni direct ByteBuffer)
189+
java --enable-native-access=ALL-UNNAMED -jar benchmark/target/benchmarks.jar \
190+
"GoldenCorpusBenchmark.*(Segment|JniByteBuffer)" \
191+
-p file=http,large-literal-and-match-lengths -f 3 -wi 3 -i 5 -prof gc
192+
193+
# synthetic throughput + allocation, all sizes
194+
java -jar benchmark/target/benchmarks.jar CompressBenchmark DecompressBenchmark -prof gc
142195

143196
# single size
144197
java -jar benchmark/target/benchmarks.jar -prof gc -p size=67108864

0 commit comments

Comments
 (0)