Skip to content

Commit 51e1739

Browse files
dfa1claude
andcommitted
docs: explain why Vortex outperforms Hardwood Parquet in benchmark section
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 9e5b4c9 commit 51e1739

1 file changed

Lines changed: 35 additions & 0 deletions

File tree

README.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,41 @@ Source: 47.6 MB Parquet → 12.0 MB Vortex (4× compression). Both sides scalar
290290
| `parquetReadMultiColumn` — 2 cols (`fare_amount`, `PULocationID`) | 53.07 ± 1.25 | 18.8 ms | baseline |
291291
| `vortexReadMultiColumn` — 2 cols (`fare_amount`, `PULocationID`) | 140.31 ± 5.42 | 7.13 ms | **2.6×** |
292292
293+
#### Why Vortex is faster than Parquet here
294+
295+
Three compounding factors:
296+
297+
**1. Smaller file = less I/O bandwidth**
298+
ALP encodes taxi floats very compactly (fare amounts cluster around small integers, trip
299+
distances are short floats). Parquet uses PLAIN or BYTE_STREAM_SPLIT for doubles — 8 raw
300+
bytes per value. 47.6 MB → 12.0 MB means 4× less memory to read even when the file is
301+
hot in the OS page cache.
302+
303+
**2. Batch columnar API vs row-by-row cursor**
304+
Hardwood's `RowReader` requires `rows.next()` + `rows.getDouble("trip_distance")` per row
305+
— 2 virtual calls × 3 M rows = 6 M calls, plus a string-keyed column lookup on every
306+
access. `DoubleArray.fold()` is a tight loop over a flat `MemorySegment`; the JIT sees a
307+
scalar reduction over contiguous memory with no dispatch overhead.
308+
309+
**3. mmap zero-copy**
310+
Vortex reads directly from the mmap'd `MemorySegment` — the file bytes _are_ the decode
311+
input, no intermediate copies. Hardwood reads into internal page buffers and then
312+
materialises values into a row cursor (one extra copy per page).
313+
314+
Parquet also pays per-page overhead absent in Vortex: RLE-encoded definition/repetition
315+
levels for every page (even for non-null columns), page header parsing, and optional
316+
dictionary decode. Vortex's layout is a flat array of encoded values with no per-row
317+
framing.
318+
319+
```
320+
Hardwood parquetRead (per 3 M rows) Vortex vortexRead (per 3 M rows)
321+
──────────────────────────────────── ──────────────────────────────────
322+
47.6 MB read 12.0 MB read (4× less bandwidth)
323+
+ page header parse × N pages + ALP decode (branch-free ×/+)
324+
+ definition-level RLE decode × 3 M rows + fold() tight loop, no dispatch
325+
+ getDouble("col") × 3 M virtual calls
326+
```
327+
293328
### Why fewer layers = faster
294329
295330
This is my hypothesis:

0 commit comments

Comments
 (0)