Skip to content

Commit c00fb8d

Browse files
ajelenakajelenak
authored
Add encoder selector to LZ4 plugin 32004 via cd_values[1] (#267)
* Add encoder selector to LZ4 filter via cd_values[1] A new signed `cd_values[1]` slot selects LZ4HC level (positive), fast-encoder acceleration (negative), or the legacy LZ4_compress_default (zero/absent), with HC level clamped to [`LZ4HC_CLEVEL_MIN`, `LZ4HC_CLEVEL_MAX`]. The on-disk chunk format is unchanged, so existing archives and the reverse filter continue to work without modification. * Harden LZ4 forward-filter output sizing against overflow Reject blocks above LZ4_MAX_INPUT_SIZE (where LZ4_compressBound returns 0) before allocating, and guard the nBlocks × perBlock size_t multiply so a tiny blockSize on a 32-bit build can't wrap into an undersized malloc and heap overflow.
1 parent 3ac87e4 commit c00fb8d

5 files changed

Lines changed: 231 additions & 10 deletions

File tree

‎LZ4/README.txt‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,38 @@ Example:
2020
folders:
2121
set(ENV{HDF5_ROOT} "/temp/hdf5")
2222
set(ENV{LZ4_ROOT} "/temp/lz4")
23+
24+
Filter parameters (cd_values[])
25+
-------------------------------
26+
27+
cd_values[0] -- block size (unchanged). 0 or absent selects the plugin
28+
default (1 GiB).
29+
30+
cd_values[1] -- encoder selector (optional). Stored as unsigned int but
31+
interpreted as a signed int. It follows the same signed compression-level
32+
convention as liblz4's lz4frame API, so a value means the same thing here
33+
as it does everywhere else in the LZ4 ecosystem:
34+
35+
>= 2 : LZ4HC level (2..12); values above 12 are clamped to 12.
36+
9 is the LZ4HC default and the recommended archival setting.
37+
0 or 1 : default fast encoder (LZ4_compress_default; acceleration 1).
38+
< 0 : fast encoder with acceleration = -cd_values[1] + 1, so -1 is
39+
acceleration 2, -2 is acceleration 3, and so on. Values below
40+
-65536 are clamped to -65536 (acceleration LZ4_ACCELERATION_MAX,
41+
currently 65537).
42+
43+
Note the negative side is offset by one: acceleration 1 is the default
44+
(0 or 1), so the first faster-than-default step is -1 (acceleration 2).
45+
To request a specific acceleration A (A >= 2), set cd_values[1] = 1 - A.
46+
For example, acceleration 8 -> cd_values[1] = -7; acceleration 9 -> -8.
47+
48+
Because the slot is unsigned on disk, a negative value is stored as its
49+
two's-complement bit pattern. For example, cd_values[1] = -8 (acceleration
50+
9) is stored as 4294967288, which is what "h5dump -p" displays. The plugin
51+
casts the slot back to int when reading.
52+
53+
cd_values[1] only affects writes. The on-disk chunk format is unchanged:
54+
LZ4HC, accelerated fast LZ4, and the default encoder all produce LZ4
55+
Block payloads that the existing reader (LZ4_decompress_safe) decodes
56+
without modification. Files written by the older plugin (cd_nelmts <= 1)
57+
are read by the new plugin identically.

‎LZ4/example/h5ex_d_lz4.c‎

Lines changed: 124 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,136 @@
2020
2121
************************************************************/
2222

23-
#include "hdf5.h"
2423
#include <stdio.h>
2524
#include <stdlib.h>
25+
#include "hdf5.h"
2626

2727
#define FILENAME "h5ex_d_lz4.h5"
28+
#define FILENAME_ENC "h5ex_d_lz4_enc.h5"
2829
#define DATASET "DS1"
2930
#define DIM0 32
3031
#define DIM1 64
3132
#define CHUNK0 4
3233
#define CHUNK1 8
3334
#define H5Z_FILTER_LZ4 32004
3435

36+
static int
37+
write_lz4_dataset(hid_t file_id, hid_t space_id, const hsize_t chunk[2], const char *name, size_t nelmts,
38+
const unsigned int *cd_values, const int *wdata)
39+
{
40+
hid_t dcpl_id = H5I_INVALID_HID;
41+
hid_t dset_id = H5I_INVALID_HID;
42+
int ret = -1;
43+
44+
if ((dcpl_id = H5Pcreate(H5P_DATASET_CREATE)) < 0)
45+
goto done;
46+
if (H5Pset_filter(dcpl_id, H5Z_FILTER_LZ4, H5Z_FLAG_MANDATORY, nelmts, cd_values) < 0)
47+
goto done;
48+
if (H5Pset_chunk(dcpl_id, 2, chunk) < 0)
49+
goto done;
50+
if ((dset_id = H5Dcreate(file_id, name, H5T_STD_I32LE, space_id, H5P_DEFAULT, dcpl_id, H5P_DEFAULT)) < 0)
51+
goto done;
52+
if (H5Dwrite(dset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata) < 0)
53+
goto done;
54+
55+
ret = 0;
56+
done:
57+
if (dset_id >= 0)
58+
H5Dclose(dset_id);
59+
if (dcpl_id >= 0)
60+
H5Pclose(dcpl_id);
61+
return ret;
62+
}
63+
64+
static int
65+
read_and_check(hid_t file_id, const char *name, int expected_max, hsize_t *out_storage)
66+
{
67+
hid_t dset_id = H5I_INVALID_HID;
68+
int rdata[DIM0][DIM1];
69+
int i, j, max_seen;
70+
hsize_t storage = 0;
71+
int ret = -1;
72+
73+
if ((dset_id = H5Dopen(file_id, name, H5P_DEFAULT)) < 0)
74+
goto done;
75+
if (H5Dread(dset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, rdata[0]) < 0)
76+
goto done;
77+
storage = H5Dget_storage_size(dset_id);
78+
max_seen = rdata[0][0];
79+
for (i = 0; i < DIM0; i++)
80+
for (j = 0; j < DIM1; j++)
81+
if (max_seen < rdata[i][j])
82+
max_seen = rdata[i][j];
83+
if (max_seen != expected_max)
84+
goto done;
85+
if (out_storage)
86+
*out_storage = storage;
87+
ret = 0;
88+
done:
89+
if (dset_id >= 0)
90+
H5Dclose(dset_id);
91+
return ret;
92+
}
93+
94+
static int
95+
run_encoder_check(const int *wdata, int expected_max, const hsize_t dims[2], const hsize_t chunk[2])
96+
{
97+
hid_t file_id = H5I_INVALID_HID;
98+
hid_t space_id = H5I_INVALID_HID;
99+
hsize_t size_hc9 = 0, size_hc12 = 0, size_hc99 = 0;
100+
const unsigned int cd_hc9[2] = {0, 9};
101+
const unsigned int cd_hc12[2] = {0, 12};
102+
const unsigned int cd_hc99[2] = {0, 99};
103+
const unsigned int cd_fast[2] = {0, (unsigned int)(-8)}; /* fast, acceleration 9 */
104+
int ret = -1;
105+
106+
if ((file_id = H5Fcreate(FILENAME_ENC, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT)) < 0)
107+
goto done;
108+
if ((space_id = H5Screate_simple(2, dims, NULL)) < 0)
109+
goto done;
110+
111+
if (write_lz4_dataset(file_id, space_id, chunk, "DS_HC9", 2, cd_hc9, wdata) < 0)
112+
goto done;
113+
if (write_lz4_dataset(file_id, space_id, chunk, "DS_HC12", 2, cd_hc12, wdata) < 0)
114+
goto done;
115+
if (write_lz4_dataset(file_id, space_id, chunk, "DS_HC99", 2, cd_hc99, wdata) < 0)
116+
goto done;
117+
if (write_lz4_dataset(file_id, space_id, chunk, "DS_FAST", 2, cd_fast, wdata) < 0)
118+
goto done;
119+
120+
H5Sclose(space_id);
121+
space_id = H5I_INVALID_HID;
122+
H5Fclose(file_id);
123+
file_id = H5I_INVALID_HID;
124+
125+
if ((file_id = H5Fopen(FILENAME_ENC, H5F_ACC_RDONLY, H5P_DEFAULT)) < 0)
126+
goto done;
127+
128+
if (read_and_check(file_id, "DS_HC9", expected_max, &size_hc9) < 0)
129+
goto done;
130+
if (read_and_check(file_id, "DS_HC12", expected_max, &size_hc12) < 0)
131+
goto done;
132+
if (read_and_check(file_id, "DS_HC99", expected_max, &size_hc99) < 0)
133+
goto done;
134+
if (read_and_check(file_id, "DS_FAST", expected_max, NULL) < 0)
135+
goto done;
136+
137+
printf("....Encoder selector check ........\n");
138+
printf(" DS_HC9: round-trip OK\n");
139+
printf(" DS_HC12: round-trip OK\n");
140+
printf(" DS_HC99: round-trip OK; storage equals DS_HC12: %s\n", size_hc99 == size_hc12 ? "yes" : "no");
141+
printf(" DS_FAST: round-trip OK\n");
142+
143+
if (size_hc99 == size_hc12)
144+
ret = 0;
145+
done:
146+
if (space_id >= 0)
147+
H5Sclose(space_id);
148+
if (file_id >= 0)
149+
H5Fclose(file_id);
150+
return ret;
151+
}
152+
35153
int
36154
main(void)
37155
{
@@ -215,6 +333,11 @@ main(void)
215333
if (avail)
216334
printf("lz4 filter is available now since H5Dread triggered loading of the filter.\n");
217335

336+
if (run_encoder_check(wdata[0], max, dims, chunk) < 0) {
337+
printf("Encoder selector check FAILED\n");
338+
goto done;
339+
}
340+
218341
ret_value = 0;
219342

220343
done:

‎LZ4/example/testfiles/h5ex_d_lz4.tst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,8 @@ Filter info is available from the dataset creation property
99
....Reading lz4 compressed data ................
1010
Maximum value in DS1 is 1890
1111
lz4 filter is available now since H5Dread triggered loading of the filter.
12+
....Encoder selector check ........
13+
DS_HC9: round-trip OK
14+
DS_HC12: round-trip OK
15+
DS_HC99: round-trip OK; storage equals DS_HC12: yes
16+
DS_FAST: round-trip OK

‎LZ4/src/H5Zlz4.c‎

Lines changed: 54 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454

5555
#include "H5PLextern.h"
5656
#include "lz4.h"
57+
#include "lz4hc.h"
5758

5859
static size_t H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_values[],
5960
size_t nbytes, size_t *buf_size, void **buf);
@@ -100,6 +101,34 @@ H5PLget_plugin_info(void)
100101
return H5Z_LZ4;
101102
}
102103

104+
/* liblz4 caps fast-mode acceleration at LZ4_ACCELERATION_MAX internally but
105+
* does not expose the macro in lz4.h; mirror the value from lz4.c here. */
106+
#ifndef LZ4_ACCELERATION_MAX
107+
#define LZ4_ACCELERATION_MAX 65537
108+
#endif
109+
110+
static int
111+
lz4_encode(const char *src, char *dst, int srcSz, int dstCap, int encoderParam)
112+
{
113+
/* Mirror liblz4's lz4frame compressionLevel convention:
114+
* >= LZ4HC_CLEVEL_MIN (2) -> LZ4HC at that level
115+
* 0 or 1 -> default fast (acceleration 1)
116+
* < 0 -> fast, acceleration = -encoderParam + 1
117+
* Clamp to the usable range first so the acceleration arithmetic cannot
118+
* overflow. */
119+
if (encoderParam > LZ4HC_CLEVEL_MAX)
120+
encoderParam = LZ4HC_CLEVEL_MAX;
121+
if (encoderParam < -(LZ4_ACCELERATION_MAX - 1))
122+
encoderParam = -(LZ4_ACCELERATION_MAX - 1);
123+
124+
if (encoderParam >= LZ4HC_CLEVEL_MIN)
125+
return LZ4_compress_HC(src, dst, srcSz, dstCap, encoderParam);
126+
else {
127+
int acceleration = (encoderParam < 0) ? -encoderParam + 1 : 1;
128+
return LZ4_compress_fast(src, dst, srcSz, dstCap, acceleration);
129+
}
130+
}
131+
103132
static size_t
104133
H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_values[], size_t nbytes,
105134
size_t *buf_size, void **buf)
@@ -173,6 +202,9 @@ H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_value
173202
size_t maxDestSize;
174203
char *rpos; /* pointer to current read position */
175204
char *roBuf; /* pointer to current write position */
205+
int encoderParam;
206+
int blockBound;
207+
size_t perBlock;
176208

177209
if (nbytes > INT32_MAX) {
178210
/* can only compress chunks up to 2GB */
@@ -185,11 +217,29 @@ H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_value
185217
else {
186218
blockSize = DEFAULT_BLOCK_SIZE;
187219
}
220+
encoderParam = (cd_nelmts > 1) ? (int)cd_values[1] : 0;
188221
if (blockSize > nbytes) {
189222
blockSize = nbytes;
190223
}
191-
nBlocks = (nbytes - 1) / blockSize + 1;
192-
maxDestSize = nBlocks * LZ4_compressBound(blockSize) + 4 + 8 + nBlocks * 4;
224+
nBlocks = (nbytes - 1) / blockSize + 1;
225+
226+
/* LZ4_compressBound returns 0 when blockSize exceeds LZ4_MAX_INPUT_SIZE,
227+
* which sits ~32 MiB below INT32_MAX -- so a block in that gap passes the
228+
* nbytes check above. Reject it before sizing the output buffer; this
229+
* also guarantees the (int)blockSize casts below stay in range. */
230+
blockBound = LZ4_compressBound((int)blockSize);
231+
if (blockBound <= 0)
232+
goto error;
233+
234+
/* Output size = nBlocks * (blockBound + 4-byte block header) + 12-byte
235+
* chunk header. Guard the size_t arithmetic before malloc: a tiny
236+
* blockSize makes nBlocks huge, and on a 32-bit platform the product
237+
* can wrap, yielding an undersized buffer and a heap overflow in the
238+
* write loop below. blockBound > 0 here, so perBlock cannot be zero. */
239+
perBlock = (size_t)blockBound + 4;
240+
if (nBlocks > (SIZE_MAX - 12) / perBlock)
241+
goto error;
242+
maxDestSize = nBlocks * perBlock + 12;
193243
if (NULL == (outBuf = malloc(maxDestSize))) {
194244
goto error;
195245
}
@@ -213,8 +263,8 @@ H5Z_filter_lz4(unsigned int flags, size_t cd_nelmts, const unsigned int cd_value
213263
if (nbytes - origWritten < blockSize) /* the last block may be < blockSize */
214264
blockSize = nbytes - origWritten;
215265

216-
compBlockSize = LZ4_compress_default(
217-
rpos, roBuf + 4, blockSize, LZ4_compressBound(blockSize)); /// reserve space for compBlockSize
266+
compBlockSize = lz4_encode(rpos, roBuf + 4, (int)blockSize, LZ4_compressBound((int)blockSize),
267+
encoderParam); /// reserve space for compBlockSize
218268
if (!compBlockSize)
219269
goto error;
220270
if (compBlockSize >= blockSize) /* compression did not save any space, do a memcpy instead */

‎docs/RegisteredFilterPlugins.md‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -266,22 +266,30 @@ The filter's source code is available from https://github.com/lz4/lz4.
266266

267267
#### Plugin ID `32004` Information
268268

269-
Number of `cd_values[]` parameters is one (`cd_nelmts = 1`).
269+
Number of `cd_values[]` parameters is up to two (`cd_nelmts <= 2`).
270270

271271
| `cd_values[]` | Description |
272272
|---|---|
273273
| `[0]` | Block size in bytes at most 2,113,929,216 bytes. Default is 1 GiB (1,073,741,824 bytes). (optional) |
274+
| `[1]` | Encoder selector, stored unsigned but interpreted as signed `int`. Negative values store as their two's-complement bit pattern, so `-8` prints as `4294967288` in `h5dump -p`. (optional) |
274275

276+
`cd_values[1]` follows liblz4's `lz4frame` signed compression-level convention:
277+
278+
* `>= 2` selects `LZ4_compress_HC` at that level (`2..12`; higher values clamp to `12`, and `9` is the LZ4HC default).
279+
* `0` or `1` selects the default fast encoder (`LZ4_compress_default`, acceleration `1`).
280+
* A negative value selects `LZ4_compress_fast` with acceleration = `-cd_values[1] + 1` (so `-1` is acceleration `2`), clamped below at `-65536` (acceleration `LZ4_ACCELERATION_MAX`, currently `65537`). To request a specific acceleration `A >= 2`, set `cd_values[1] = 1 - A` (e.g. acceleration `9` → `-8`).
275281
The description of the chunk binary format produced by this plugin is at [LZ4_HDF5_format.md](https://github.com/HDFGroup/hdf5_plugins/blob/master/LZ4/LZ4_HDF5_format.md).
276282

277-
`h5repack` example for the `--filter` option: `<objects list>:UD=32004,0,0`.
283+
`h5repack` examples for the `--filter` option:
284+
* default encoder, default block size: `<objects list>:UD=32004,0,0`
285+
* LZ4HC level 9 (archival), default block size: `<objects list>:UD=32004,0,2,0,9`
278286

279-
https://github.com/HDFGroup/hdf5_plugins/tree/master/LZ4/src
287+
https://github.com/HDFGroup/hdf5_plugins/tree/master/LZ4/
280288

281289
##### Contact
282290

283-
Michael Rissi (Dectris Ltd.)<br/>
284-
Email: michael dot rissi at dectris dot com
291+
HDF Group Helpdesk<br/>
292+
Email: help at hdfgroup dot org
285293

286294
---
287295

0 commit comments

Comments
 (0)