Skip to content

Commit 3263eed

Browse files
ajelenakajelenak
committed
Add encoder selector to LZ4 filter via cd_values[1]
A new signed `cd_values[1]` slot selects LZ4HC level (positive), fast-encoder acceleration (negative), or the legacy LZ4_compress_default (zero/absent), with HC level clamped to [`LZ4HC_CLEVEL_MIN`, `LZ4HC_CLEVEL_MAX`]. The on-disk chunk format is unchanged, so existing archives and the reverse filter continue to work without modification.
1 parent c06e1ea commit 3263eed

6 files changed

Lines changed: 198 additions & 8 deletions

File tree

‎LZ4/LZ4_HDF5_format.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,13 @@ The size of the data field stored in big endian signed 32 bit format.
4343

4444
If this value is equal to the known decompressed size of the block, the data is stored
4545
not compressed. Otherwise, the data is in the [LZ4 Block](https://github.com/lz4/lz4/blob/v1.10.0/doc/lz4_Block_format.md) format.
46+
47+
## Encoder choice
48+
49+
The on-disk format above is independent of the encoder the writer used.
50+
The plugin's `cd_values[1]` selects between `LZ4_compress_default`
51+
(value `0`, the default), `LZ4_compress_HC` (positive value, HC level),
52+
and accelerated `LZ4_compress_fast` (negative value, acceleration =
53+
`-cd_values[1]`). All three produce LZ4 Block payloads that the same
54+
`LZ4_decompress_safe` path reads, so files written with any encoder
55+
choice are bitstream-compatible with readers that only know the default.

‎LZ4/README.txt‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,30 @@ Example:
2020
folders:
2121
set(ENV{HDF5_ROOT} "/temp/hdf5")
2222
set(ENV{LZ4_ROOT} "/temp/lz4")
23+
24+
Filter parameters (cd_values[])
25+
-------------------------------
26+
27+
cd_values[0] -- block size (unchanged). 0 or absent selects the plugin
28+
default (1 GiB).
29+
30+
cd_values[1] -- encoder selector (optional). Stored as unsigned int but
31+
interpreted as a signed int:
32+
33+
0 : LZ4_compress_default (legacy behavior, the default)
34+
> 0 : LZ4HC level, clamped to [LZ4HC_CLEVEL_MIN, LZ4HC_CLEVEL_MAX]
35+
(currently [2, 12]); 9 is the LZ4HC default.
36+
< 0 : fast-encoder acceleration = -cd_values[1], lower-bounded at 1
37+
and clamped by liblz4 internally at LZ4_ACCELERATION_MAX
38+
(currently 65537).
39+
40+
Because the slot is unsigned on disk, a negative encoder value is stored
41+
as its two's-complement bit pattern. For example, acceleration 8 is
42+
written as cd_values[1] = -8 and "h5dump -p" displays the slot as the
43+
unsigned decimal 4294967288. The plugin casts back to int when reading.
44+
45+
cd_values[1] only affects writes. The on-disk chunk format is unchanged:
46+
LZ4HC, accelerated fast LZ4, and the default encoder all produce LZ4
47+
Block payloads that the existing reader (LZ4_decompress_safe) decodes
48+
without modification. Files written by the older plugin (cd_nelmts <= 1)
49+
are read by the new plugin identically.

‎LZ4/example/h5ex_d_lz4.c‎

Lines changed: 120 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,132 @@
2020
2121
************************************************************/
2222

23-
#include "hdf5.h"
2423
#include <stdio.h>
2524
#include <stdlib.h>
25+
#include "hdf5.h"
2626

2727
#define FILENAME "h5ex_d_lz4.h5"
28+
#define FILENAME_ENC "h5ex_d_lz4_enc.h5"
2829
#define DATASET "DS1"
2930
#define DIM0 32
3031
#define DIM1 64
3132
#define CHUNK0 4
3233
#define CHUNK1 8
3334
#define H5Z_FILTER_LZ4 32004
3435

36+
static int
37+
write_lz4_dataset(hid_t file_id, hid_t space_id, const hsize_t chunk[2], const char *name,
38+
size_t nelmts, const unsigned int *cd_values, const int *wdata)
39+
{
40+
hid_t dcpl_id = H5I_INVALID_HID;
41+
hid_t dset_id = H5I_INVALID_HID;
42+
int ret = -1;
43+
44+
if ((dcpl_id = H5Pcreate(H5P_DATASET_CREATE)) < 0)
45+
goto done;
46+
if (H5Pset_filter(dcpl_id, H5Z_FILTER_LZ4, H5Z_FLAG_MANDATORY, nelmts, cd_values) < 0)
47+
goto done;
48+
if (H5Pset_chunk(dcpl_id, 2, chunk) < 0)
49+
goto done;
50+
if ((dset_id = H5Dcreate(file_id, name, H5T_STD_I32LE, space_id, H5P_DEFAULT, dcpl_id, H5P_DEFAULT)) <
51+
0)
52+
goto done;
53+
if (H5Dwrite(dset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata) < 0)
54+
goto done;
55+
56+
ret = 0;
57+
done:
58+
if (dset_id >= 0)
59+
H5Dclose(dset_id);
60+
if (dcpl_id >= 0)
61+
H5Pclose(dcpl_id);
62+
return ret;
63+
}
64+
65+
static int
66+
read_and_check(hid_t file_id, const char *name, int expected_max, hsize_t *out_storage)
67+
{
68+
hid_t dset_id = H5I_INVALID_HID;
69+
int rdata[DIM0][DIM1];
70+
int i, j, max_seen;
71+
hsize_t storage = 0;
72+
int ret = -1;
73+
74+
if ((dset_id = H5Dopen(file_id, name, H5P_DEFAULT)) < 0)
75+
goto done;
76+
if (H5Dread(dset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata[0]) < 0)
77+
goto done;
78+
storage = H5Dget_storage_size(dset_id);
79+
max_seen = rdata[0][0];
80+
for (i = 0; i < DIM0; i++)
81+
for (j = 0; j < DIM1; j++)
82+
if (max_seen < rdata[i][j])
83+
max_seen = rdata[i][j];
84+
if (max_seen != expected_max)
85+
goto done;
86+
if (out_storage)
87+
*out_storage = storage;
88+
ret = 0;
89+
done:
90+
if (dset_id >= 0)
91+
H5Dclose(dset_id);
92+
return ret;
93+
}
94+
95+
static int
96+
run_encoder_check(const int *wdata, int expected_max, const hsize_t dims[2], const hsize_t chunk[2])
97+
{
98+
hid_t file_id = H5I_INVALID_HID;
99+
hid_t space_id = H5I_INVALID_HID;
100+
hsize_t size_hc9 = 0, size_hc12 = 0, size_hc99 = 0;
101+
const unsigned int cd_hc9[2] = {0, 9};
102+
const unsigned int cd_hc12[2] = {0, 12};
103+
const unsigned int cd_hc99[2] = {0, 99};
104+
int ret = -1;
105+
106+
if ((file_id = H5Fcreate(FILENAME_ENC, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT)) < 0)
107+
goto done;
108+
if ((space_id = H5Screate_simple(2, dims, NULL)) < 0)
109+
goto done;
110+
111+
if (write_lz4_dataset(file_id, space_id, chunk, "DS_HC9", 2, cd_hc9, wdata) < 0)
112+
goto done;
113+
if (write_lz4_dataset(file_id, space_id, chunk, "DS_HC12", 2, cd_hc12, wdata) < 0)
114+
goto done;
115+
if (write_lz4_dataset(file_id, space_id, chunk, "DS_HC99", 2, cd_hc99, wdata) < 0)
116+
goto done;
117+
118+
H5Sclose(space_id);
119+
space_id = H5I_INVALID_HID;
120+
H5Fclose(file_id);
121+
file_id = H5I_INVALID_HID;
122+
123+
if ((file_id = H5Fopen(FILENAME_ENC, H5F_ACC_RDONLY, H5P_DEFAULT)) < 0)
124+
goto done;
125+
126+
if (read_and_check(file_id, "DS_HC9", expected_max, &size_hc9) < 0)
127+
goto done;
128+
if (read_and_check(file_id, "DS_HC12", expected_max, &size_hc12) < 0)
129+
goto done;
130+
if (read_and_check(file_id, "DS_HC99", expected_max, &size_hc99) < 0)
131+
goto done;
132+
133+
printf("....Encoder selector check ........\n");
134+
printf(" DS_HC9: round-trip OK\n");
135+
printf(" DS_HC12: round-trip OK\n");
136+
printf(" DS_HC99: round-trip OK; storage equals DS_HC12: %s\n",
137+
size_hc99 == size_hc12 ? "yes" : "no");
138+
139+
if (size_hc99 == size_hc12)
140+
ret = 0;
141+
done:
142+
if (space_id >= 0)
143+
H5Sclose(space_id);
144+
if (file_id >= 0)
145+
H5Fclose(file_id);
146+
return ret;
147+
}
148+
35149
int
36150
main(void)
37151
{
@@ -215,6 +329,11 @@ main(void)
215329
if (avail)
216330
printf("lz4 filter is available now since H5Dread triggered loading of the filter.\n");
217331

332+
if (run_encoder_check(wdata[0], max, dims, chunk) < 0) {
333+
printf("Encoder selector check FAILED\n");
334+
goto done;
335+
}
336+
218337
ret_value = 0;
219338

220339
done:

‎LZ4/example/testfiles/h5ex_d_lz4.tst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,7 @@ Filter info is available from the dataset creation property
99
....Reading lz4 compressed data ................
1010
Maximum value in DS1 is 1890
1111
lz4 filter is available now since H5Dread triggered loading of the filter.
12+
....Encoder selector check ........
13+
DS_HC9: round-trip OK
14+
DS_HC12: round-trip OK
15+
DS_HC99: round-trip OK; storage equals DS_HC12: yes

‎LZ4/src/H5Zlz4.c‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454

5555
#include "H5PLextern.h"
5656
#include "lz4.h"
57+
#include "lz4hc.h"
5758

5859
static size_t H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_values[],
5960
size_t nbytes, size_t *buf_size, void **buf);
@@ -100,6 +101,29 @@ H5PLget_plugin_info(void)
100101
return H5Z_LZ4;
101102
}
102103

104+
static int
105+
lz4_encode(const char *src, char *dst, int srcSz, int dstCap, int encoderParam)
106+
{
107+
if (encoderParam > 0) {
108+
int level = encoderParam;
109+
if (level < LZ4HC_CLEVEL_MIN)
110+
level = LZ4HC_CLEVEL_MIN;
111+
if (level > LZ4HC_CLEVEL_MAX)
112+
level = LZ4HC_CLEVEL_MAX;
113+
return LZ4_compress_HC(src, dst, srcSz, dstCap, level);
114+
}
115+
else if (encoderParam < 0) {
116+
/* Widen before negating so encoderParam == INT_MIN cannot overflow. */
117+
long accel = -(long)encoderParam;
118+
if (accel < 1)
119+
accel = 1;
120+
/* No explicit upper clamp: LZ4_compress_fast() clamps internally to
121+
* LZ4_ACCELERATION_MAX (currently 65537; see lz4.c). */
122+
return LZ4_compress_fast(src, dst, srcSz, dstCap, (int)accel);
123+
}
124+
return LZ4_compress_default(src, dst, srcSz, dstCap);
125+
}
126+
103127
static size_t
104128
H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_values[], size_t nbytes,
105129
size_t *buf_size, void **buf)
@@ -173,6 +197,7 @@ H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_value
173197
size_t maxDestSize;
174198
char *rpos; /* pointer to current read position */
175199
char *roBuf; /* pointer to current write position */
200+
int encoderParam;
176201

177202
if (nbytes > INT32_MAX) {
178203
/* can only compress chunks up to 2GB */
@@ -185,6 +210,7 @@ H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_value
185210
else {
186211
blockSize = DEFAULT_BLOCK_SIZE;
187212
}
213+
encoderParam = (cd_nelmts > 1) ? (int)cd_values[1] : 0;
188214
if (blockSize > nbytes) {
189215
blockSize = nbytes;
190216
}
@@ -213,8 +239,9 @@ H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_value
213239
if (nbytes - origWritten < blockSize) /* the last block may be < blockSize */
214240
blockSize = nbytes - origWritten;
215241

216-
compBlockSize = LZ4_compress_default(
217-
rpos, roBuf + 4, blockSize, LZ4_compressBound(blockSize)); /// reserve space for compBlockSize
242+
compBlockSize = lz4_encode(
243+
rpos, roBuf + 4, (int)blockSize,
244+
LZ4_compressBound((int)blockSize), encoderParam); /// reserve space for compBlockSize
218245
if (!compBlockSize)
219246
goto error;
220247
if (compBlockSize >= blockSize) /* compression did not save any space, do a memcpy instead */

‎docs/RegisteredFilterPlugins.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -266,22 +266,25 @@ The filter's source code is available from https://github.com/lz4/lz4.
266266

267267
#### Plugin ID `32004` Information
268268

269-
Number of `cd_values[]` parameters is one (`cd_nelmts = 1`).
269+
Number of `cd_values[]` parameters is up to two (`cd_nelmts <= 2`).
270270

271271
| `cd_values[]` | Description |
272272
|---|---|
273273
| `[0]` | Block size in bytes at most 2,113,929,216 bytes. Default is 1 GiB (1,073,741,824 bytes). (optional) |
274+
| `[1]` | Encoder selector, stored unsigned but interpreted as signed `int`. `0` (default) selects `LZ4_compress_default`. A positive value selects `LZ4_compress_HC` at that level, clamped to `[LZ4HC_CLEVEL_MIN, LZ4HC_CLEVEL_MAX]` (currently `[2, 12]`; `9` is the LZ4HC default). A negative value selects `LZ4_compress_fast` with acceleration = `-cd_values[1]`, lower-bounded at `1` and clamped by liblz4 internally at `LZ4_ACCELERATION_MAX` (currently `65537`). Negative values store as their two's-complement bit pattern, so e.g. acceleration `8` prints as `4294967288` in `h5dump -p`. The on-disk chunk format is unchanged across all three encoders. (optional) |
274275

275276
The description of the chunk binary format produced by this plugin is at [LZ4_HDF5_format.md](https://github.com/HDFGroup/hdf5_plugins/blob/master/LZ4/LZ4_HDF5_format.md).
276277

277-
`h5repack` example for the `--filter` option: `<objects list>:UD=32004,0,0`.
278+
`h5repack` examples for the `--filter` option:
279+
* default encoder, default block size: `<objects list>:UD=32004,0,0`
280+
* LZ4HC level 9 (archival), default block size: `<objects list>:UD=32004,0,2,0,9`
278281

279-
https://github.com/HDFGroup/hdf5_plugins/tree/master/LZ4/src
282+
https://github.com/HDFGroup/hdf5_plugins/tree/master/LZ4/
280283

281284
##### Contact
282285

283-
Michael Rissi (Dectris Ltd.)<br/>
284-
Email: michael dot rissi at dectris dot com
286+
HDF Group Helpdesk<br/>
287+
Email: help at hdfgroup dot org
285288

286289
---
287290

0 commit comments

Comments
 (0)