@@ -2,14 +2,18 @@ streamvbyte
22===========
33[ ![ Ubuntu 22.04 CI (GCC 9, 10, 11 and 12, LLVM 12, 13, 14)] ( https://github.com/lemire/streamvbyte/actions/workflows/ubuntu22.yml/badge.svg )] ( https://github.com/lemire/streamvbyte/actions/workflows/ubuntu22.yml )
44[ ![ Ubuntu 24.04 CI (GCC 13, 14, LLVM 17, 18)] ( https://github.com/lemire/streamvbyte/actions/workflows/ubuntu24.yml/badge.svg )] ( https://github.com/lemire/streamvbyte/actions/workflows/ubuntu24.yml )
5- [ ![ macOS 11 CI (LLVM 13, GCC 10, 11, 12) ] ( https://github.com/lemire/streamvbyte/actions/workflows/macos.yml/badge.svg )] ( https://github.com/lemire/streamvbyte/actions/workflows/macos.yml )
5+ [ ![ macOS CI ] ( https://github.com/lemire/streamvbyte/actions/workflows/macos.yml/badge.svg )] ( https://github.com/lemire/streamvbyte/actions/workflows/macos.yml )
66[ ![ VS-Latest-CI] ( https://github.com/lemire/streamvbyte/actions/workflows/vs16.yml/badge.svg )] ( https://github.com/lemire/streamvbyte/actions/workflows/vs16.yml )
77[ ![ VS17-CI] ( https://github.com/lemire/streamvbyte/actions/workflows/vs.yml/badge.svg )] ( https://github.com/lemire/streamvbyte/actions/workflows/vs.yml )
88
9- StreamVByte is a new integer compression technique that applies SIMD instructions (vectorization) to
9+ StreamVByte is an integer compression technique that applies SIMD instructions (vectorization) to
1010Google's Group Varint approach. The net result is faster than other byte-oriented compression
1111techniques.
1212
13+ Both the encoder and the decoder are vectorized: there is an optimized path using SSE 4.1 on
14+ x64 processors and one using NEON on 64-bit ARM (AArch64) processors. A portable scalar
15+ fallback is used otherwise, and the appropriate path is selected at runtime.
16+
1317The approach is patent-free, the code is available under the Apache License.
1418
1519
@@ -129,13 +133,15 @@ cmake --install build
129133#### Benchmarking with CMake
130134
131135
132- After building, you may run our benchmark as follows:
136+ After building (with ` -DSTREAMVBYTE_ENABLE_TESTS=ON ` ) , you may run our benchmark as follows:
133137
134138```
135- ./build/test/ perf
139+ ./build/perf
136140```
137141
138- The benchmarks are not currently built under Windows.
142+ On Apple Silicon, run it with ` sudo ` to enable the hardware performance counters
143+ (instructions, cycles, branch and cache misses); on Linux you may instead relax
144+ ` kernel.perf_event_paranoid ` . The benchmarks are not currently built under Windows.
139145
140146
141147### 2. Building with Makefile:
@@ -180,6 +186,24 @@ zigzag_encode(mysignedints, myunsignedints, number); // mysignedints => myunsign
180186zigzag_decode(myunsignedints, mysignedints, number); // myunsignedints => mysignedints
181187```
182188
189+ Performance
190+ -----------
191+
192+ StreamVByte sustains multi-gigabyte-per-second throughput for both encoding and
193+ decoding. The exact speed depends on the data distribution and on your hardware.
194+ As an illustration, the `perf` benchmark (500,000 random 32-bit integers per run,
195+ throughput measured against the 2 MB uncompressed input) reports the following on
196+ an Apple M4 Max:
197+
198+ | data distribution | encode | decode |
199+ |------------------------------|-----------|-----------|
200+ | mixed 1–4 byte (log-uniform) | ~27 GB/s | ~41 GB/s |
201+ | full-range (mostly 4 byte) | ~26 GB/s | ~33 GB/s |
202+ | small values (1 byte) | ~27 GB/s | ~36 GB/s |
203+
204+ You can reproduce these numbers with `./build/perf` after building the tests
205+ (see the Benchmarking instructions above).
206+
183207Technical posts
184208---------------
185209
0 commit comments