Skip to content

Commit 13148d1

Browse files
committed
manual review TODO.md
1 parent 59302db commit 13148d1

1 file changed

Lines changed: 15 additions & 11 deletions

File tree

TODO.md

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,14 @@
1010

1111
## Performance
1212

13-
- [ ] **#10c Publish reproducible perf artifacts**
13+
- [ ] Publish reproducible perf artifacts
1414
- Capture JMH JSON + JFR profile alongside README table; cite hardware (CPU model), JDK build (`java -version`),
1515
and benchmark commit SHA so numbers don't rot silently.
16+
- [ ] performance tests must be peer reviewed
17+
- [ ] run performance tests on other machines (I have access only to Apple M5)
1618

1719
## Testing
1820

19-
- [ ] a lot of tests are doing "new String(array.getBytes()", let's add a method there
2021
- [ ] lots of repetitions like in every test
2122
```java
2223
private static final DType I64 = new DType.Primitive(PType.I64, false);
@@ -26,6 +27,7 @@
2627

2728
## Build
2829

30+
- [ ] prefix all modules with "vortex-"
2931
- [ ] add BOM module
3032
- [ ] deploy to maven central
3133
- [ ] drop warnings about flatbuffers
@@ -42,30 +44,32 @@
4244
## Documentation
4345
4446
- [ ] Format specification: byte-exact diagrams for file layout and each encoding, with annotated examples (Arrow spec style)
47+
- [ ] how to use the library and the cli
4548
4649
## Tooling
4750
51+
- [ ] hardwood to convert parquet files to vortex
52+
- [ ] Optional `vortex-arrow` bridge module for Arrow ecosystem interop
53+
- Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe)
54+
- Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need
55+
Arrow Flight / DuckDB ADBC / pandas interop
56+
- Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in
57+
- Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means
58+
the core library stays Unsafe-free
4859
4960
## Large-file support
5061
51-
- [ ] **#12 Test read/write of files > 2 GB**
62+
- [ ] **Test read/write of files > 2 GB**
5263
- [ ] Parquet baseline for comparison: same data should fail or require splitting when any
5364
column chunk exceeds 2 GB.
5465
55-
## Array API
66+
## API
5667
5768
- [ ] Use domain primitives (`UInt32`, `UInt64`, etc.) as value classes via Project Valhalla instead of raw `long`/`int`
5869
- See https://dfa1.github.io/articles/rethink-domain-primitives-with-valhalla
5970
- Candidates: `PType` integer kinds, buffer offsets, row indices, byte lengths
6071
- Goal: type-safety at zero cost (value class = no heap alloc, no boxing)
6172
62-
- [ ] Optional `vortex-arrow` bridge module for Arrow ecosystem interop
63-
- Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe)
64-
- Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need
65-
Arrow Flight / DuckDB ADBC / pandas interop
66-
- Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in
67-
- Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means
68-
the core library stays Unsafe-free
6973
7074
## Encodings
7175

0 commit comments

Comments
 (0)