Skip to content

Commit da6663c

Browse files
dfa1claude
andcommitted
feat: cover all advanced parameters
- ZstdCompressParameter now lists every stable ZSTD_cParameter (adds targetCBlockSize, the four LDM params, jobSize, overlapLog) - ZstdDecompressParameter (ZSTD_dParameter: windowLogMax) + ZstdDecompressCtx .parameter()/.windowLogMax(); bind ZSTD_DCtx_setParameter - bounds introspection: ZstdCompressParameter.bounds() / ZstdDecompressParameter.bounds() return ZstdBounds, via ZSTD_c/dParam_getBounds (a ZSTD_bounds struct returned by value through FFM) - compatibility.md now 51/186 Tests: compression-level bounds match the library; every compression parameter reports bounds; new params round-trip at their lower bound; windowLogMax is accepted and rejects out-of-range. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 1ddd474 commit da6663c

8 files changed

Lines changed: 209 additions & 10 deletions

File tree

docs/compatibility.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ and semantics follow the [official manual](https://facebook.github.io/zstd/doc/a
55

66
- zstd version: **1.6.0** (vendored `third_party/zstd`)
77
- Public symbols exported by `libzstd`: **186**
8-
- Bound so far: **48** (~26%)
8+
- Bound so far: **51** (~27%)
99

1010
"Bound" means the symbol has a `MethodHandle` in `Bindings` and is reachable
1111
through the public Java API. The rest are reachable from native code but not yet
@@ -27,7 +27,7 @@ low-level variants that an idiomatic Java API does not need.
2727
| Dictionary training (ZDICT) | 4 / 12 | `trainFromBuffer`; cover/fastCover optimizers not bound |
2828
| Streaming — compress | 3 / 22 | `ZstdOutputStream` (compressStream2 + buffer sizes) |
2929
| Streaming — decompress | 3 / 15 | `ZstdInputStream` (decompressStream + buffer sizes) |
30-
| Advanced parameters | 4 / 38 | `CCtx_setParameter` + `compress2` (level, checksum, LDM, windowLog) and `C/DCtx_loadDictionary` (dictionary streams); MT/getBounds not bound |
30+
| Advanced parameters | 7 / 38 | all `ZSTD_cParameter` + `ZSTD_dParameter` via `ZstdCompressParameter`/`ZstdDecompressParameter`; `compress2`, `C/DCtx_setParameter`, `loadDictionary`, `c/dParam_getBounds`; MT inert on single-thread build |
3131
| Frame inspection | 9 / 13 | `ZstdFrame`: isFrame, header, compressedSize, decompressedBound, dictId, skippable read/write; getFrameProgression/_advanced not bound |
3232
| Memory sizing | 0 / 14 | `sizeof_*` / `estimate*` accounting not bound |
3333
| Low-level block | 0 / 12 | expert block/continue API not bound |
@@ -54,15 +54,17 @@ low-level variants that an idiomatic Java API does not need.
5454
| `ZDICT_isError`, `ZDICT_getErrorName` | internal error mapping in `ZstdDictionary` |
5555
| `ZSTD_compressStream2`, `ZSTD_CStreamInSize`, `ZSTD_CStreamOutSize`, `ZSTD_CCtx_setParameter` | `ZstdOutputStream` |
5656
| `ZSTD_decompressStream`, `ZSTD_DStreamInSize`, `ZSTD_DStreamOutSize` | `ZstdInputStream` |
57-
| `ZSTD_compress2`, `ZSTD_CCtx_setParameter` | `ZstdCompressCtx.parameter` / `checksum` / `longDistanceMatching` / `windowLog` (+ `ZstdCompressParameter`) |
57+
| `ZSTD_compress2`, `ZSTD_CCtx_setParameter` | `ZstdCompressCtx.parameter` / `checksum` / `longDistanceMatching` / `windowLog` (all of `ZstdCompressParameter`) |
58+
| `ZSTD_DCtx_setParameter` | `ZstdDecompressCtx.parameter` / `windowLogMax` (`ZstdDecompressParameter`) |
59+
| `ZSTD_cParam_getBounds`, `ZSTD_dParam_getBounds` | `ZstdCompressParameter.bounds()` / `ZstdDecompressParameter.bounds()` (`ZstdBounds`) |
5860
| `ZSTD_CCtx_loadDictionary`, `ZSTD_DCtx_loadDictionary` | `ZstdOutputStream` / `ZstdInputStream` dictionary constructors |
5961
| `ZSTD_isFrame`, `ZSTD_findFrameCompressedSize`, `ZSTD_decompressBound`, `ZSTD_getDictID_fromFrame`, `ZSTD_getFrameHeader`, `ZSTD_isSkippableFrame`, `ZSTD_writeSkippableFrame`, `ZSTD_readSkippableFrame` | `ZstdFrame` (+ `ZstdFrameHeader`, `ZstdFrameType`, `ZstdSkippableContent`) |
6062
| `ZSTD_getErrorCode` | `ZstdException.code()` (+ `ZstdErrorCode`) |
6163

6264
## Roadmap (priority order)
6365

6466
1. ~~**Streaming**~~ — done: `ZstdOutputStream` / `ZstdInputStream` (`compressStream2`, `decompressStream`, bounded buffers, dictionary constructors). Remaining: `MemorySegment`-buffer driver, `pledgedSrcSize`.
65-
2. **Advanced parameters** — done for compression: `CCtx_setParameter` + `compress2` via `ZstdCompressCtx` (`checksum`, `longDistanceMatching`, `windowLog`, generic `parameter`). Remaining: `cParam_getBounds`, `pledgedSrcSize`, and `nbWorkers` (needs a multithreaded native build).
67+
2. ~~**Advanced parameters**~~ — done: every `ZSTD_cParameter`/`ZSTD_dParameter` via `ZstdCompressParameter`/`ZstdDecompressParameter` (+ `bounds()`), on both contexts. Remaining: `pledgedSrcSize`; `nbWorkers` is settable but inert until the native build enables multithreading.
6668
3. ~~**Frame inspection**~~ — done: `ZstdFrame` (`isFrame`, `header`, `compressedSize`, `decompressedBound`, `dictId`, skippable read/write). Remaining: `getDictID_fromDict/CDict/DDict`, `getFrameProgression`.
6769
4. **Better dictionaries**`ZDICT_optimizeTrainFromBuffer_cover` / `_fastCover`, `finalizeDictionary`.
6870
5. ~~**Typed errors**~~ — done: `ZstdException.code()` returns `ZstdErrorCode` (via `getErrorCode`).
@@ -204,7 +206,7 @@ low-level variants that an idiomatic Java API does not need.
204206
| `ZSTD_resetDStream` ||
205207
| `ZSTD_sizeof_DStream` ||
206208

207-
### Advanced parameters (4/38)
209+
### Advanced parameters (7/38)
208210

209211
| Symbol | Bound |
210212
|---|:---:|
@@ -239,11 +241,11 @@ low-level variants that an idiomatic Java API does not need.
239241
| `ZSTD_DCtx_reset` ||
240242
| `ZSTD_DCtx_setFormat` ||
241243
| `ZSTD_DCtx_setMaxWindowSize` ||
242-
| `ZSTD_DCtx_setParameter` | |
243-
| `ZSTD_cParam_getBounds` | |
244+
| `ZSTD_DCtx_setParameter` | |
245+
| `ZSTD_cParam_getBounds` | |
244246
| `ZSTD_compress2` ||
245247
| `ZSTD_createCCtxParams` ||
246-
| `ZSTD_dParam_getBounds` | |
248+
| `ZSTD_dParam_getBounds` | |
247249
| `ZSTD_estimateCCtxSize_usingCCtxParams` ||
248250
| `ZSTD_freeCCtxParams` ||
249251

zstd/src/main/java/io/github/dfa1/zstd/Bindings.java

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package io.github.dfa1.zstd;
22

33
import java.lang.foreign.FunctionDescriptor;
4+
import java.lang.foreign.MemoryLayout;
45
import java.lang.invoke.MethodHandle;
56

67
import static java.lang.foreign.ValueLayout.ADDRESS;
@@ -133,6 +134,20 @@ final class Bindings {
133134
NativeLibrary.lookup("ZSTD_compress2",
134135
FunctionDescriptor.of(JAVA_LONG, ADDRESS, ADDRESS, JAVA_LONG, ADDRESS, JAVA_LONG));
135136

137+
// size_t ZSTD_DCtx_setParameter(ZSTD_DCtx*, ZSTD_dParameter, int value)
138+
static final MethodHandle DCTX_SET_PARAMETER =
139+
NativeLibrary.lookup("ZSTD_DCtx_setParameter",
140+
FunctionDescriptor.of(JAVA_LONG, ADDRESS, JAVA_INT, JAVA_INT));
141+
142+
// ZSTD_bounds { size_t error; int lowerBound; int upperBound; } — returned by value
143+
private static final MemoryLayout BOUNDS_LAYOUT =
144+
MemoryLayout.structLayout(JAVA_LONG, JAVA_INT, JAVA_INT);
145+
// ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter) / ZSTD_dParam_getBounds(ZSTD_dParameter)
146+
static final MethodHandle CPARAM_GET_BOUNDS =
147+
NativeLibrary.lookup("ZSTD_cParam_getBounds", FunctionDescriptor.of(BOUNDS_LAYOUT, JAVA_INT));
148+
static final MethodHandle DPARAM_GET_BOUNDS =
149+
NativeLibrary.lookup("ZSTD_dParam_getBounds", FunctionDescriptor.of(BOUNDS_LAYOUT, JAVA_INT));
150+
136151
// size_t ZSTD_compressStream2(ZSTD_CCtx*, ZSTD_outBuffer*, ZSTD_inBuffer*, ZSTD_EndDirective)
137152
static final MethodHandle COMPRESS_STREAM2 =
138153
NativeLibrary.lookup("ZSTD_compressStream2",

zstd/src/main/java/io/github/dfa1/zstd/Zstd.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ static long call(SizeCall c) {
198198
return code;
199199
}
200200

201-
private static boolean isError(long code) {
201+
static boolean isError(long code) {
202202
try {
203203
return ((int) Bindings.IS_ERROR.invokeExact(code)) != 0;
204204
} catch (Throwable t) {
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
package io.github.dfa1.zstd;
2+
3+
import java.lang.foreign.Arena;
4+
import java.lang.foreign.MemorySegment;
5+
import java.lang.foreign.SegmentAllocator;
6+
import java.lang.invoke.MethodHandle;
7+
8+
import static java.lang.foreign.ValueLayout.JAVA_INT;
9+
import static java.lang.foreign.ValueLayout.JAVA_LONG;
10+
11+
/// The inclusive range of values a compression or decompression parameter
12+
/// accepts, from `ZSTD_cParam_getBounds` / `ZSTD_dParam_getBounds`.
13+
///
14+
/// @param lowerBound the smallest accepted value, inclusive
15+
/// @param upperBound the largest accepted value, inclusive
16+
public record ZstdBounds(int lowerBound, int upperBound) {
17+
18+
/// Calls a `*_getBounds` function (which returns a `ZSTD_bounds` struct by
19+
/// value: `{ size_t error; int lowerBound; int upperBound; }`).
20+
static ZstdBounds query(MethodHandle getBounds, int parameter) {
21+
try (Arena arena = Arena.ofConfined()) {
22+
MemorySegment bounds = (MemorySegment) getBounds.invokeExact((SegmentAllocator) arena, parameter);
23+
long error = bounds.get(JAVA_LONG, 0);
24+
if (Zstd.isError(error)) {
25+
throw new ZstdException("parameter has no queryable bounds");
26+
}
27+
return new ZstdBounds(bounds.get(JAVA_INT, 8), bounds.get(JAVA_INT, 12));
28+
} catch (Throwable t) {
29+
throw rethrow(t);
30+
}
31+
}
32+
33+
@SuppressWarnings("unchecked")
34+
private static <E extends Throwable> RuntimeException rethrow(Throwable t) throws E {
35+
throw (E) t;
36+
}
37+
}

zstd/src/main/java/io/github/dfa1/zstd/ZstdCompressParameter.java

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,18 @@ public enum ZstdCompressParameter {
2525
TARGET_LENGTH(106),
2626
/// Compression strategy (1..9: fast … btultra2).
2727
STRATEGY(107),
28+
/// Target uncompressed block size, in bytes (0 = no target); trades ratio for latency.
29+
TARGET_C_BLOCK_SIZE(130),
2830
/// Enable long-distance matching (1) for better ratio on large inputs.
2931
ENABLE_LONG_DISTANCE_MATCHING(160),
32+
/// Long-distance matching hash table size, as a power of two.
33+
LDM_HASH_LOG(161),
34+
/// Minimum match length for the long-distance matcher, in bytes.
35+
LDM_MIN_MATCH(162),
36+
/// Log size of each bucket in the long-distance matcher's hash table.
37+
LDM_BUCKET_SIZE_LOG(163),
38+
/// How often to insert entries into the long-distance matcher's hash table, as a log.
39+
LDM_HASH_RATE_LOG(164),
3040
/// Write the decompressed size into the frame header (1, the default) or not (0).
3141
CONTENT_SIZE_FLAG(200),
3242
/// Append a 32-bit content checksum to the frame (1) for integrity, or not (0, the default).
@@ -36,7 +46,11 @@ public enum ZstdCompressParameter {
3646
/// Number of worker threads (0 = single-threaded). Requires a multithreaded
3747
/// build of the native library; the bundled library is single-threaded, so a
3848
/// non-zero value has no effect.
39-
NB_WORKERS(400);
49+
NB_WORKERS(400),
50+
/// Size of a compression job, in bytes, when multithreading (0 = automatic).
51+
JOB_SIZE(401),
52+
/// Overlap between consecutive multithreading jobs, as a log fraction of the window.
53+
OVERLAP_LOG(402);
4054

4155
private final int value;
4256

@@ -50,4 +64,13 @@ public enum ZstdCompressParameter {
5064
int value() {
5165
return value;
5266
}
67+
68+
/// The inclusive `[lower, upper]` range of values this parameter accepts,
69+
/// queried from zstd via `ZSTD_cParam_getBounds`.
70+
///
71+
/// @return the valid bounds for this parameter
72+
/// @throws ZstdException if zstd reports no bounds for this parameter
73+
public ZstdBounds bounds() {
74+
return ZstdBounds.query(Bindings.CPARAM_GET_BOUNDS, value);
75+
}
5376
}

zstd/src/main/java/io/github/dfa1/zstd/ZstdDecompressCtx.java

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,26 @@ private static MemorySegment create() {
2626
}
2727
}
2828

29+
/// Sets an advanced decompression parameter, sticky across subsequent calls.
30+
///
31+
/// @param parameter the parameter to set
32+
/// @param value the value, validated natively against the parameter's bounds
33+
/// @return `this`, for chaining
34+
/// @throws ZstdException if the value is out of range for the parameter
35+
public ZstdDecompressCtx parameter(ZstdDecompressParameter parameter, int value) {
36+
Zstd.call(() -> (long) Bindings.DCTX_SET_PARAMETER.invokeExact(ptr(), parameter.value(), value));
37+
return this;
38+
}
39+
40+
/// Sets the largest back-reference window the decoder will accept, as a
41+
/// power of two. Raise it to decode frames built with a large `windowLog`.
42+
///
43+
/// @param windowLogMax the base-2 log of the maximum accepted window size
44+
/// @return `this`, for chaining
45+
public ZstdDecompressCtx windowLogMax(int windowLogMax) {
46+
return parameter(ZstdDecompressParameter.WINDOW_LOG_MAX, windowLogMax);
47+
}
48+
2949
/// Decompresses a frame into a buffer of at most `maxSize` bytes.
3050
///
3151
/// @param compressed a complete zstd frame
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
package io.github.dfa1.zstd;
2+
3+
/// Advanced decompression parameters settable on a [ZstdDecompressCtx], mirroring
4+
/// `ZSTD_dParameter` from the
5+
/// [official manual](https://facebook.github.io/zstd/doc/api_manual_latest.html).
6+
///
7+
/// Set them with [ZstdDecompressCtx#parameter(ZstdDecompressParameter, int)].
8+
public enum ZstdDecompressParameter {
9+
10+
/// Largest back-reference window the decoder will accept, as a power of two.
11+
/// Frames needing a larger window are rejected; raise this to decode them.
12+
WINDOW_LOG_MAX(100);
13+
14+
private final int value;
15+
16+
ZstdDecompressParameter(int value) {
17+
this.value = value;
18+
}
19+
20+
/// The native `ZSTD_dParameter` integer value.
21+
///
22+
/// @return the enum value as defined by zstd
23+
int value() {
24+
return value;
25+
}
26+
27+
/// The inclusive `[lower, upper]` range of values this parameter accepts,
28+
/// queried from zstd via `ZSTD_dParam_getBounds`.
29+
///
30+
/// @return the valid bounds for this parameter
31+
/// @throws ZstdException if zstd reports no bounds for this parameter
32+
public ZstdBounds bounds() {
33+
return ZstdBounds.query(Bindings.DPARAM_GET_BOUNDS, value);
34+
}
35+
}

zstd/src/test/java/io/github/dfa1/zstd/ZstdParameterTest.java

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
import org.junit.jupiter.api.Nested;
44
import org.junit.jupiter.api.Test;
5+
import org.junit.jupiter.params.ParameterizedTest;
6+
import org.junit.jupiter.params.provider.EnumSource;
57

68
import java.nio.charset.StandardCharsets;
79

@@ -69,6 +71,71 @@ void explicitWindowLogRoundTrips() {
6971
}
7072
}
7173

74+
@Nested
75+
class Bounds {
76+
77+
@Test
78+
void compressionLevelBoundsMatchTheLibrary() {
79+
ZstdBounds bounds = ZstdCompressParameter.COMPRESSION_LEVEL.bounds();
80+
assertThat(bounds.lowerBound()).isEqualTo(Zstd.minCompressionLevel());
81+
assertThat(bounds.upperBound()).isEqualTo(Zstd.maxCompressionLevel());
82+
}
83+
84+
@ParameterizedTest
85+
@EnumSource(ZstdCompressParameter.class)
86+
void everyCompressionParameterReportsBounds(ZstdCompressParameter parameter) {
87+
ZstdBounds bounds = parameter.bounds();
88+
assertThat(bounds.upperBound()).isGreaterThanOrEqualTo(bounds.lowerBound());
89+
}
90+
91+
@Test
92+
void decompressionParameterReportsBounds() {
93+
ZstdBounds bounds = ZstdDecompressParameter.WINDOW_LOG_MAX.bounds();
94+
assertThat(bounds.upperBound()).isGreaterThan(bounds.lowerBound());
95+
}
96+
}
97+
98+
@Nested
99+
class MoreParameters {
100+
101+
@ParameterizedTest
102+
@EnumSource(value = ZstdCompressParameter.class,
103+
names = {"TARGET_C_BLOCK_SIZE", "LDM_HASH_LOG", "LDM_MIN_MATCH",
104+
"LDM_BUCKET_SIZE_LOG", "STRATEGY"})
105+
void settingAParameterAtItsLowerBoundRoundTrips(ZstdCompressParameter parameter) {
106+
// Given a parameter set to a valid in-range value
107+
int value = Math.max(parameter.bounds().lowerBound(), 1);
108+
byte[] frame;
109+
try (ZstdCompressCtx ctx = new ZstdCompressCtx().parameter(parameter, value)) {
110+
frame = ctx.compress(PAYLOAD);
111+
}
112+
// Then the frame still decompresses
113+
assertThat(Zstd.decompress(frame)).isEqualTo(PAYLOAD);
114+
}
115+
}
116+
117+
@Nested
118+
class Decompress {
119+
120+
@Test
121+
void windowLogMaxIsAccepted() {
122+
// Given a decompressor configured with a raised window limit
123+
byte[] frame = Zstd.compress(PAYLOAD);
124+
try (ZstdDecompressCtx ctx = new ZstdDecompressCtx().windowLogMax(31)) {
125+
// Then normal frames still decode
126+
assertThat(ctx.decompress(frame, PAYLOAD.length)).isEqualTo(PAYLOAD);
127+
}
128+
}
129+
130+
@Test
131+
void rejectsOutOfRangeValue() {
132+
try (ZstdDecompressCtx ctx = new ZstdDecompressCtx()) {
133+
assertThatThrownBy(() -> ctx.parameter(ZstdDecompressParameter.WINDOW_LOG_MAX, 99))
134+
.isInstanceOf(ZstdException.class);
135+
}
136+
}
137+
}
138+
72139
@Nested
73140
class GenericSetter {
74141

0 commit comments

Comments
 (0)