perf: fair benchmark — import all 19 taxi cols, project at read time

Previous run imported only 3 columns into Vortex (smaller file = unfair advantage).
Now both formats hold all 19 columns; projection happens at read time.

Results (scalar-vs-scalar, 3 M rows):
  parquetRead 1 col:       81.26 ops/s (12.3 ms)
  vortexRead 1 col:       151.96 ops/s  (6.6 ms) → 1.9×
  parquetReadMultiColumn:  46.80 ops/s (21.4 ms)
  vortexReadMultiColumn:   57.65 ops/s (17.4 ms) → 1.2×

Vortex file is now 76 MB vs Parquet's 47.6 MB — compressor does not yet
beat ZSTD on this mixed dataset. Speedup comes from batch columnar API
and mmap zero-copy, not file size.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: dfa1

docs/explanation.md

@@ -124,53 +124,51 @@ The Java write is faster but also produces bigger files (more optimization work
 |-----------|--------------|------------------|--------------|
 | scan      | 20.4 ± 0.9   | 5.7 ± 0.6        | **3.6×**     |
 
-### Parquet vs Vortex read — NYC Yellow Taxi 2024-01, 3 M rows, 3 columns
+### Parquet vs Vortex read — NYC Yellow Taxi 2024-01, 3 M rows, 19 columns
 
-Source: 47.6 MB Parquet → 12.0 MB Vortex (4× compression). Both sides scalar decode
+Both formats store all 19 columns; projection happens at read time. Both sides scalar decode
 (Hardwood disables SIMD on JDK 25; Vortex Java uses FFM scalar reads throughout).
+File sizes: Parquet 47.6 MB, Vortex 76.0 MB (Vortex is larger here — cascading compressor
+does not yet beat Parquet's ZSTD on this mixed dataset).
 
 | Benchmark | ops/s | ms/scan | vs Parquet |
 |---|---|---|---|
-| `parquetRead` — Hardwood, 1 col (`trip_distance`) | 66.96 ± 1.99 | 14.9 ms | baseline |
-| `vortexRead` — 1 col (`trip_distance`) | 201.61 ± 1.63 | 4.96 ms | **3.0×** |
-| `parquetReadMultiColumn` — 2 cols (`fare_amount`, `PULocationID`) | 53.07 ± 1.25 | 18.8 ms | baseline |
-| `vortexReadMultiColumn` — 2 cols (`fare_amount`, `PULocationID`) | 140.31 ± 5.42 | 7.13 ms | **2.6×** |
+| `parquetRead` — Hardwood, 1 col (`trip_distance`) | 81.26 ± 0.76 | 12.3 ms | baseline |
+| `vortexRead` — 1 col (`trip_distance`) | 151.96 ± 0.84 | 6.58 ms | **1.9×** |
+| `parquetReadMultiColumn` — 2 cols (`fare_amount`, `PULocationID`) | 46.80 ± 0.90 | 21.4 ms | baseline |
+| `vortexReadMultiColumn` — 2 cols (`fare_amount`, `PULocationID`) | 57.65 ± 0.58 | 17.4 ms | **1.2×** |
 
-#### Why Vortex is faster than Parquet here
+#### Why Vortex is faster despite being a larger file
 
-Three compounding factors:
+The file size advantage is gone on this dataset — Vortex is 60% larger than the source Parquet.
+The speedup comes from two other factors:
 
-**1. Smaller file = less I/O bandwidth.**
-ALP encodes taxi floats very compactly (fare amounts cluster around small integers, trip
-distances are short floats). Parquet uses PLAIN or BYTE_STREAM_SPLIT for doubles — 8 raw
-bytes per value. 47.6 MB → 12.0 MB means 4× less memory to read even when the file is
-hot in the OS page cache.
-
-**2. Batch columnar API vs row-by-row cursor.**
+**1. Batch columnar API vs row-by-row cursor.**
 Hardwood's `RowReader` requires `rows.next()` + `rows.getDouble("trip_distance")` per row
 — 2 virtual calls × 3 M rows = 6 M calls, plus a string-keyed column lookup on every
 access. `DoubleArray.fold()` is a tight loop over a flat `MemorySegment`; the JIT sees a
 scalar reduction over contiguous memory with no dispatch overhead.
 
-**3. mmap zero-copy.**
+**2. mmap zero-copy.**
 Vortex reads directly from the mmap'd `MemorySegment` — the file bytes _are_ the decode
-input, no intermediate copies. Hardwood reads into internal page buffers and then
-materialises values into a row cursor (one extra copy per page).
-
-Parquet also pays per-page overhead absent in Vortex: RLE-encoded definition/repetition
-levels for every page (even for non-null columns), page header parsing, and optional
-dictionary decode. Vortex's layout is a flat array of encoded values with no per-row
-framing.
+input, no intermediate copies. Hardwood reads into internal page buffers and materialises
+values into a row cursor (one extra copy per page). Parquet also pays per-page framing
+overhead: RLE-encoded definition/repetition levels, page header parsing, optional dictionary
+decode. Vortex's layout is a flat array of encoded values with no per-row framing.
 
 ```
 Hardwood parquetRead (per 3 M rows)       Vortex vortexRead (per 3 M rows)
 ────────────────────────────────────      ──────────────────────────────────
-47.6 MB read                              12.0 MB read  (4× less bandwidth)
+47.6 MB on disk (smaller file)            76.0 MB on disk (larger file)
 + page header parse × N pages             + ALP decode (branch-free ×/+)
 + definition-level RLE decode × 3 M rows  + fold() tight loop, no dispatch
 + getDouble("col") × 3 M virtual calls
 ```
 
+The multi-column gap narrows to 1.2× because projecting 2 of 19 columns forces Vortex
+to skip more data per scan; Parquet's column-chunk layout already isolates each column
+on disk, so its per-column I/O is better amortised when reading multiple columns.
+
 ## Design principles
 
 - Zero-copy everywhere via FFM `MemorySegment`

performance/src/main/java/io/github/dfa1/vortex/performance/ParquetVsVortexReadBenchmark.java

@@ -8,7 +8,6 @@
 import io.github.dfa1.vortex.core.array.IntArray;
 import io.github.dfa1.vortex.encoding.EncodingRegistry;
 import io.github.dfa1.vortex.io.VortexReader;
-import io.github.dfa1.vortex.parquet.ImportOptions;
 import io.github.dfa1.vortex.parquet.ParquetImporter;
 import io.github.dfa1.vortex.scan.ScanResult;
 import org.openjdk.jmh.annotations.Benchmark;
@@ -32,7 +31,6 @@
 import java.net.http.HttpResponse;
 import java.nio.file.Files;
 import java.nio.file.Path;
-import java.util.List;
 import java.util.concurrent.TimeUnit;
 
 /// Benchmark: Hardwood Parquet read vs Java Vortex read on the same real-world dataset.
@@ -62,8 +60,6 @@ public class ParquetVsVortexReadBenchmark {
             "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet";
     private static final Path CACHE_PATH =
             Path.of(System.getProperty("java.io.tmpdir"), "yellow_tripdata_2024-01.parquet");
-    private static final List<String> COLUMNS = List.of("trip_distance", "fare_amount", "PULocationID");
-
     private static final Object SETUP_LOCK = new Object();
     private static Path sharedParquetFile;
     private static Path sharedVortexFile;
@@ -88,9 +84,8 @@ public void setup() throws Exception {
                             sharedParquetFile, Files.size(sharedParquetFile) / 1_048_576.0);
                 }
                 sharedVortexFile = Files.createTempFile("taxi-bench", ".vortex");
-                System.out.print("[ParquetVsVortexReadBenchmark] converting to vortex...");
-                ImportOptions opts = ImportOptions.defaults().withColumns(COLUMNS);
-                ParquetImporter.importParquet(sharedParquetFile, sharedVortexFile, opts);
+                System.out.print("[ParquetVsVortexReadBenchmark] converting to vortex (all columns)...");
+                ParquetImporter.importParquet(sharedParquetFile, sharedVortexFile);
                 System.out.printf(" done (%.1f MB)%n", Files.size(sharedVortexFile) / 1_048_576.0);
             }
             refCount++;