Skip to content

Commit 0623afe

Browse files
dfa1claude
andcommitted
docs: drop completed items from TODO.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 4d38299 commit 0623afe

1 file changed

Lines changed: 38 additions & 83 deletions

File tree

TODO.md

Lines changed: 38 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,13 @@
11
# TODO
22

3-
## Done
4-
5-
- [x] **#1 Generate FlatBuffer + Protobuf sources**
6-
- [x] **#2 Implement `PostscriptParser.parse()`**
7-
- [x] **#3 Implement `ScanIterator.hasNext()` / `next()`** (layout traversal + flat decode; zone-map pruning pending)
8-
- [x] **#4 Implement `VortexWriter.writeChunk()` + `close()`** (primitive + bool encodings, full file format)
9-
- [x] **#5 Round-trip unit tests** (`VortexWriterTest`: 5 tests; `VortexFileTest`: 17 tests)
10-
- [x] **#6 Zone-map pruning in `ScanIterator`**
11-
- Writer embeds min/max per flat segment in `ArrayNode.stats` (Protobuf ScalarValue)
12-
- Reader peeks stats from segment FlatBuffer tail, skips chunks excluded by `RowFilter`
13-
- Supports `Gte`, `Lte`, `Eq`, `And` predicates on I8/I16/I32/I64, U8/U16/U32/U64, F32/F64
14-
- [x] **#7a `fastlanes.bitpacked` — spec-compliant rewrite** (all 5 steps incl. patches)
15-
- Protobuf `BitPackedMetadata` (tags 1+2+3); unified FastLanes `unpack` for I8/I16/I32/I64 (signed + unsigned);
16-
patches decoded from children slots 0 (indices) + 1 (values), overwritten by absolute index.
17-
- Encoder writes spec-compliant protobuf metadata; round-trip + patch decode covered by tests.
18-
- [x] **#7b `fastlanes.for` decoder** (reference + bitpacked residuals)
19-
- [x] **#7c `vortex.sparse` decoder** (fill value + patches)
20-
- [x] **#7d `vortex.alp` decoder** (ALP inverse + patches)
21-
- [x] **#8 Rust writes → Java reads** (`RustWritesJavaReadsIT`, `-Pintegration`)
22-
- JNI writes I64+F64 file; Java reader decodes via `DecoderRegistry.loadAll()`
23-
- Added `SequenceCodec` (`vortex.sequence` = `A[i] = base + i * multiplier`)
24-
- [x] **#9 Java writes → Rust reads** (`JavaWritesRustReadsIT`, `-Pintegration`)
25-
- Java writer produces file; JNI reader decodes via Arrow C Data Interface
26-
- `Buffer.alignment_exponent = 6` + `SegmentSpec.alignment_exponent = 6` + pre-segment 64-byte padding
27-
28-
## Open
3+
## Encodings
294

305
- [ ] **#7 Additional encodings**
316
- `pcodec` — float compression (only remaining gap; bitpacked, delta, for, sparse, alp, dict, fsst, sequence,
327
varbin, constant, runend all landed)
338

349
## Performance
3510

36-
- [x] **#10a Java vs JNI read benchmark**`RustVsJavaReadBenchmark` covers volume (I64), close (F64 ALP), symbol
37-
(varbin) on 10M OHLC rows. Drives README perf table.
38-
3911
- [ ] **#10b Java vs JNI write benchmark** (`performance/` module, `-Pperformance`)
4012
- Add `RustVsJavaWriteBenchmark` mirroring read side: same 10M-row OHLC fixture, JMH throughput, both writers.
4113
- Old `WriteBenchmark.java` (Java-only) removed; rewrite from scratch using JNI bindings already on classpath
@@ -45,6 +17,17 @@
4517
- Capture JMH JSON + JFR profile alongside README table; cite hardware (CPU model), JDK build (`java -version`),
4618
and benchmark commit SHA so numbers don't rot silently.
4719

20+
- [ ] Improve read speed
21+
- avoid switching on PType per element during a copy — use MemorySegment bulk copy
22+
- arena is a parameter: allocate once, pass down; all allocations go there
23+
- reuse buffers during decoding
24+
- don't allocate temp byte[]
25+
26+
- [ ] Improve write speed
27+
- use the algorithm described here: https://vortex.dev/blog/btrblocks-compressor
28+
- start with compressorContext allowedCascading=3
29+
- don't apply dict encode to dict encode
30+
4831
## Large-file support
4932

5033
- [ ] **#12 Test read/write of files > 2 GB**
@@ -53,20 +36,6 @@
5336
- Parquet baseline for comparison: same data fails or requires splitting when any column chunk > 2 GB
5437
- Confirm no `int` casts silently truncate `uint64` offsets or `uint32` lengths in `SegmentSpec`
5538

56-
## Improve read speed
57-
58-
- avoid switching on PType per element during a copy => embrace MemorySegment
59-
- the arena is a parameter: it can be allocated once and pass down, all allocations should be done there
60-
- reuse buffers during decoding
61-
- don't allocate temp byte[]
62-
63-
## Improve write speed
64-
65-
- this is important but focus will be firs the reading part
66-
- use the algorithm described here: https://vortex.dev/blog/btrblocks-compressor
67-
- start with compressorContext allowedCascading=3
68-
- don't apply dict encode to dict encode
69-
7039
## Checkstyle
7140

7241
- [ ] Re-enable `TodoComment` in `checkstyle.xml` once all inline TODOs are resolved:
@@ -76,54 +45,40 @@
7645
## Array API
7746

7847
- [ ] Add typed accessors to `Array`: `getLong(i)`, `getDouble(i)`, `getInt(i)`, `getFloat(i)`, etc.
79-
- All leaf codecs (primitive, bitpacked, alp, sparse, for, delta, dict) decode into `buffers[0]` = flat LE primitives — typed accessors work uniformly
80-
- `vortex.struct` is never a leaf; always split into per-column arrays before element access
81-
- Make `buffers`/`children` package-private once accessors cover all callers
48+
- All leaf codecs (primitive, bitpacked, alp, sparse, for, delta, dict) decode into `buffers[0]` = flat LE primitives — typed accessors work uniformly
49+
- `vortex.struct` is never a leaf; always split into per-column arrays before element access
50+
- Make `buffers`/`children` package-private once accessors cover all callers
8251

8352
- [ ] Introduce specialized typed array views: `ArrayLong`, `ArrayDouble`, etc.
84-
- `ScanResult.columns().get("volume", PType.I64)` returns `ArrayLong` instead of raw `Array`
85-
- `ArrayLong.forEach(LongConsumer)` — zero-allocation bulk iteration; internally hoists
86-
`count = buffer.byteSize() / 8` so JIT eliminates per-element bounds check (see `javaReadVolume` benchmark)
87-
- `ArrayLong.get(long i)` — random access for non-bulk callers
88-
- `ArrayDouble.forEach(DoubleConsumer)` — same pattern for F64 columns
53+
- `ScanResult.columns().get("volume", PType.I64)` returns `ArrayLong` instead of raw `Array`
54+
- `ArrayLong.forEach(LongConsumer)` — zero-allocation bulk iteration; internally hoists
55+
`count = buffer.byteSize() / 8` so JIT eliminates per-element bounds check
56+
- `ArrayLong.get(long i)` — random access for non-bulk callers
57+
- `ArrayDouble.forEach(DoubleConsumer)` — same pattern for F64 columns
8958

9059
- [ ] Optional `vortex-arrow` bridge module for Arrow ecosystem interop
91-
- Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe)
92-
- Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need
93-
Arrow Flight / DuckDB ADBC / pandas interop
94-
- Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in
95-
- Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means
96-
the core library stays Unsafe-free
97-
- Arrow JVM is actively moving to FFM (`MemorySegment`-based allocator); once that lands
98-
the bridge can become zero-copy (share the mmap region directly, no copy needed)
99-
100-
## Code cleanups
101-
102-
- use a dedicated exception instead of IOException?
103-
- runtime exception like VortexException, indicating an non-recoverable error
104-
-
105-
- introduce CodecType enum: this can be used to replace the string throw new IllegalStateException("$codec:
106-
message")
107-
- VertexException should have CodecType
108-
- avoid allocating too many intermediate ByteBuffer => always use a MemorySegment from arena
109-
pass the arena as part of the EncodeContext, to have more deterministic release of memory
110-
- use domain primitive like RowCount or Limit/Unlimited (they cannot be zero)
111-
- avoid use of IOException like:
112-
if (footerSeg == null) {
113-
throw new IOException("vortex: postscript missing footer segment");
114-
}
115-
this is an unrecoverable exception
60+
- Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe)
61+
- Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need
62+
Arrow Flight / DuckDB ADBC / pandas interop
63+
- Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in
64+
- Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means
65+
the core library stays Unsafe-free
66+
67+
## Code quality
68+
69+
- [ ] Use a dedicated `VortexException` (unchecked) instead of `IOException` for unrecoverable errors
70+
- [ ] Introduce `CodecType` enum to replace stringly-typed error messages
71+
- [ ] Avoid allocating intermediate `ByteBuffer` — always use `MemorySegment` from arena
72+
- [ ] Domain primitives: `RowCount`, `Limit`/`Unlimited` (cannot be zero)
11673

11774
## Skills
11875

11976
- [ ] Keep `.claude/skills/improve-performance.md` and `.claude/skills/review-performance.md` in sync with
120-
`CLAUDE.md` and README perf notes. Re-audit whenever the memory model, allocation rule, or benchmark
121-
layout changes — skills drift silently and start producing wrong guidance.
77+
`CLAUDE.md` and README perf notes. Re-audit whenever memory model, allocation rule, or benchmark layout changes.
12278

12379
## Project
12480

125-
- move the project in a dedicated organization
126-
- create website
127-
- publish benchmarks
128-
- idea is to build something like hardwood.dev but for parquet files
129-
81+
- [ ] Move project to a dedicated organization
82+
- [ ] Create website
83+
- [ ] Publish benchmarks
84+
- [ ] Build something like hardwood.dev but for vortex files

0 commit comments

Comments
 (0)