|
1 | 1 | # TODO |
2 | 2 |
|
3 | | -## Done |
4 | | - |
5 | | -- [x] **#1 Generate FlatBuffer + Protobuf sources** |
6 | | -- [x] **#2 Implement `PostscriptParser.parse()`** |
7 | | -- [x] **#3 Implement `ScanIterator.hasNext()` / `next()`** (layout traversal + flat decode; zone-map pruning pending) |
8 | | -- [x] **#4 Implement `VortexWriter.writeChunk()` + `close()`** (primitive + bool encodings, full file format) |
9 | | -- [x] **#5 Round-trip unit tests** (`VortexWriterTest`: 5 tests; `VortexFileTest`: 17 tests) |
10 | | -- [x] **#6 Zone-map pruning in `ScanIterator`** |
11 | | - - Writer embeds min/max per flat segment in `ArrayNode.stats` (Protobuf ScalarValue) |
12 | | - - Reader peeks stats from segment FlatBuffer tail, skips chunks excluded by `RowFilter` |
13 | | - - Supports `Gte`, `Lte`, `Eq`, `And` predicates on I8/I16/I32/I64, U8/U16/U32/U64, F32/F64 |
14 | | -- [x] **#7a `fastlanes.bitpacked` — spec-compliant rewrite** (all 5 steps incl. patches) |
15 | | - - Protobuf `BitPackedMetadata` (tags 1+2+3); unified FastLanes `unpack` for I8/I16/I32/I64 (signed + unsigned); |
16 | | - patches decoded from children slots 0 (indices) + 1 (values), overwritten by absolute index. |
17 | | - - Encoder writes spec-compliant protobuf metadata; round-trip + patch decode covered by tests. |
18 | | -- [x] **#7b `fastlanes.for` decoder** (reference + bitpacked residuals) |
19 | | -- [x] **#7c `vortex.sparse` decoder** (fill value + patches) |
20 | | -- [x] **#7d `vortex.alp` decoder** (ALP inverse + patches) |
21 | | -- [x] **#8 Rust writes → Java reads** (`RustWritesJavaReadsIT`, `-Pintegration`) |
22 | | - - JNI writes I64+F64 file; Java reader decodes via `DecoderRegistry.loadAll()` |
23 | | - - Added `SequenceCodec` (`vortex.sequence` = `A[i] = base + i * multiplier`) |
24 | | -- [x] **#9 Java writes → Rust reads** (`JavaWritesRustReadsIT`, `-Pintegration`) |
25 | | - - Java writer produces file; JNI reader decodes via Arrow C Data Interface |
26 | | - - `Buffer.alignment_exponent = 6` + `SegmentSpec.alignment_exponent = 6` + pre-segment 64-byte padding |
27 | | - |
28 | | -## Open |
| 3 | +## Encodings |
29 | 4 |
|
30 | 5 | - [ ] **#7 Additional encodings** |
31 | 6 | - `pcodec` — float compression (only remaining gap; bitpacked, delta, for, sparse, alp, dict, fsst, sequence, |
32 | 7 | varbin, constant, runend all landed) |
33 | 8 |
|
34 | 9 | ## Performance |
35 | 10 |
|
36 | | -- [x] **#10a Java vs JNI read benchmark** — `RustVsJavaReadBenchmark` covers volume (I64), close (F64 ALP), symbol |
37 | | - (varbin) on 10M OHLC rows. Drives README perf table. |
38 | | - |
39 | 11 | - [ ] **#10b Java vs JNI write benchmark** (`performance/` module, `-Pperformance`) |
40 | 12 | - Add `RustVsJavaWriteBenchmark` mirroring read side: same 10M-row OHLC fixture, JMH throughput, both writers. |
41 | 13 | - Old `WriteBenchmark.java` (Java-only) removed; rewrite from scratch using JNI bindings already on classpath |
|
45 | 17 | - Capture JMH JSON + JFR profile alongside README table; cite hardware (CPU model), JDK build (`java -version`), |
46 | 18 | and benchmark commit SHA so numbers don't rot silently. |
47 | 19 |
|
| 20 | +- [ ] Improve read speed |
| 21 | + - avoid switching on PType per element during a copy — use MemorySegment bulk copy |
| 22 | + - arena is a parameter: allocate once, pass down; all allocations go there |
| 23 | + - reuse buffers during decoding |
| 24 | + - don't allocate temp byte[] |
| 25 | + |
| 26 | +- [ ] Improve write speed |
| 27 | + - use the algorithm described here: https://vortex.dev/blog/btrblocks-compressor |
| 28 | + - start with compressorContext allowedCascading=3 |
| 29 | + - don't apply dict encode to dict encode |
| 30 | + |
48 | 31 | ## Large-file support |
49 | 32 |
|
50 | 33 | - [ ] **#12 Test read/write of files > 2 GB** |
|
53 | 36 | - Parquet baseline for comparison: same data fails or requires splitting when any column chunk > 2 GB |
54 | 37 | - Confirm no `int` casts silently truncate `uint64` offsets or `uint32` lengths in `SegmentSpec` |
55 | 38 |
|
56 | | -## Improve read speed |
57 | | - |
58 | | -- avoid switching on PType per element during a copy => embrace MemorySegment |
59 | | -- the arena is a parameter: it can be allocated once and pass down, all allocations should be done there |
60 | | -- reuse buffers during decoding |
61 | | -- don't allocate temp byte[] |
62 | | - |
63 | | -## Improve write speed |
64 | | - |
65 | | -- this is important but focus will be firs the reading part |
66 | | -- use the algorithm described here: https://vortex.dev/blog/btrblocks-compressor |
67 | | - - start with compressorContext allowedCascading=3 |
68 | | - - don't apply dict encode to dict encode |
69 | | - |
70 | 39 | ## Checkstyle |
71 | 40 |
|
72 | 41 | - [ ] Re-enable `TodoComment` in `checkstyle.xml` once all inline TODOs are resolved: |
|
76 | 45 | ## Array API |
77 | 46 |
|
78 | 47 | - [ ] Add typed accessors to `Array`: `getLong(i)`, `getDouble(i)`, `getInt(i)`, `getFloat(i)`, etc. |
79 | | - - All leaf codecs (primitive, bitpacked, alp, sparse, for, delta, dict) decode into `buffers[0]` = flat LE primitives — typed accessors work uniformly |
80 | | - - `vortex.struct` is never a leaf; always split into per-column arrays before element access |
81 | | - - Make `buffers`/`children` package-private once accessors cover all callers |
| 48 | + - All leaf codecs (primitive, bitpacked, alp, sparse, for, delta, dict) decode into `buffers[0]` = flat LE primitives — typed accessors work uniformly |
| 49 | + - `vortex.struct` is never a leaf; always split into per-column arrays before element access |
| 50 | + - Make `buffers`/`children` package-private once accessors cover all callers |
82 | 51 |
|
83 | 52 | - [ ] Introduce specialized typed array views: `ArrayLong`, `ArrayDouble`, etc. |
84 | | - - `ScanResult.columns().get("volume", PType.I64)` returns `ArrayLong` instead of raw `Array` |
85 | | - - `ArrayLong.forEach(LongConsumer)` — zero-allocation bulk iteration; internally hoists |
86 | | - `count = buffer.byteSize() / 8` so JIT eliminates per-element bounds check (see `javaReadVolume` benchmark) |
87 | | - - `ArrayLong.get(long i)` — random access for non-bulk callers |
88 | | - - `ArrayDouble.forEach(DoubleConsumer)` — same pattern for F64 columns |
| 53 | + - `ScanResult.columns().get("volume", PType.I64)` returns `ArrayLong` instead of raw `Array` |
| 54 | + - `ArrayLong.forEach(LongConsumer)` — zero-allocation bulk iteration; internally hoists |
| 55 | + `count = buffer.byteSize() / 8` so JIT eliminates per-element bounds check |
| 56 | + - `ArrayLong.get(long i)` — random access for non-bulk callers |
| 57 | + - `ArrayDouble.forEach(DoubleConsumer)` — same pattern for F64 columns |
89 | 58 |
|
90 | 59 | - [ ] Optional `vortex-arrow` bridge module for Arrow ecosystem interop |
91 | | - - Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe) |
92 | | - - Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need |
93 | | - Arrow Flight / DuckDB ADBC / pandas interop |
94 | | - - Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in |
95 | | - - Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means |
96 | | - the core library stays Unsafe-free |
97 | | - - Arrow JVM is actively moving to FFM (`MemorySegment`-based allocator); once that lands |
98 | | - the bridge can become zero-copy (share the mmap region directly, no copy needed) |
99 | | - |
100 | | -## Code cleanups |
101 | | - |
102 | | -- use a dedicated exception instead of IOException? |
103 | | -- runtime exception like VortexException, indicating an non-recoverable error |
104 | | -- |
105 | | - - introduce CodecType enum: this can be used to replace the string throw new IllegalStateException("$codec: |
106 | | - message") |
107 | | -- VertexException should have CodecType |
108 | | -- avoid allocating too many intermediate ByteBuffer => always use a MemorySegment from arena |
109 | | - pass the arena as part of the EncodeContext, to have more deterministic release of memory |
110 | | -- use domain primitive like RowCount or Limit/Unlimited (they cannot be zero) |
111 | | -- avoid use of IOException like: |
112 | | - if (footerSeg == null) { |
113 | | - throw new IOException("vortex: postscript missing footer segment"); |
114 | | - } |
115 | | - this is an unrecoverable exception |
| 60 | + - Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe) |
| 61 | + - Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need |
| 62 | + Arrow Flight / DuckDB ADBC / pandas interop |
| 63 | + - Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in |
| 64 | + - Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means |
| 65 | + the core library stays Unsafe-free |
| 66 | + |
| 67 | +## Code quality |
| 68 | + |
| 69 | +- [ ] Use a dedicated `VortexException` (unchecked) instead of `IOException` for unrecoverable errors |
| 70 | +- [ ] Introduce `CodecType` enum to replace stringly-typed error messages |
| 71 | +- [ ] Avoid allocating intermediate `ByteBuffer` — always use `MemorySegment` from arena |
| 72 | +- [ ] Domain primitives: `RowCount`, `Limit`/`Unlimited` (cannot be zero) |
116 | 73 |
|
117 | 74 | ## Skills |
118 | 75 |
|
119 | 76 | - [ ] Keep `.claude/skills/improve-performance.md` and `.claude/skills/review-performance.md` in sync with |
120 | | - `CLAUDE.md` and README perf notes. Re-audit whenever the memory model, allocation rule, or benchmark |
121 | | - layout changes — skills drift silently and start producing wrong guidance. |
| 77 | + `CLAUDE.md` and README perf notes. Re-audit whenever memory model, allocation rule, or benchmark layout changes. |
122 | 78 |
|
123 | 79 | ## Project |
124 | 80 |
|
125 | | -- move the project in a dedicated organization |
126 | | -- create website |
127 | | -- publish benchmarks |
128 | | -- idea is to build something like hardwood.dev but for parquet files |
129 | | - |
| 81 | +- [ ] Move project to a dedicated organization |
| 82 | +- [ ] Create website |
| 83 | +- [ ] Publish benchmarks |
| 84 | +- [ ] Build something like hardwood.dev but for vortex files |
0 commit comments