|
10 | 10 |
|
11 | 11 | ## Performance |
12 | 12 |
|
13 | | -- [ ] **#10c Publish reproducible perf artifacts** |
| 13 | +- [ ] Publish reproducible perf artifacts |
14 | 14 | - Capture JMH JSON + JFR profile alongside README table; cite hardware (CPU model), JDK build (`java -version`), |
15 | 15 | and benchmark commit SHA so numbers don't rot silently. |
| 16 | +- [ ] performance tests must be peer reviewed |
| 17 | +- [ ] run performance tests on other machines (I have access only to Apple M5) |
16 | 18 |
|
17 | 19 | ## Testing |
18 | 20 |
|
19 | | -- [ ] a lot of tests are doing "new String(array.getBytes()", let's add a method there |
20 | 21 | - [ ] lots of repetitions like in every test |
21 | 22 | ```java |
22 | 23 | private static final DType I64 = new DType.Primitive(PType.I64, false); |
|
26 | 27 |
|
27 | 28 | ## Build |
28 | 29 |
|
| 30 | +- [ ] prefix all modules with "vortex-" |
29 | 31 | - [ ] add BOM module |
30 | 32 | - [ ] deploy to maven central |
31 | 33 | - [ ] drop warnings about flatbuffers |
|
42 | 44 | ## Documentation |
43 | 45 |
|
44 | 46 | - [ ] Format specification: byte-exact diagrams for file layout and each encoding, with annotated examples (Arrow spec style) |
| 47 | +- [ ] how to use the library and the cli |
45 | 48 |
|
46 | 49 | ## Tooling |
47 | 50 |
|
| 51 | +- [ ] hardwood to convert parquet files to vortex |
| 52 | +- [ ] Optional `vortex-arrow` bridge module for Arrow ecosystem interop |
| 53 | + - Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe) |
| 54 | + - Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need |
| 55 | + Arrow Flight / DuckDB ADBC / pandas interop |
| 56 | + - Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in |
| 57 | + - Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means |
| 58 | + the core library stays Unsafe-free |
48 | 59 |
|
49 | 60 | ## Large-file support |
50 | 61 |
|
51 | | -- [ ] **#12 Test read/write of files > 2 GB** |
| 62 | +- [ ] **Test read/write of files > 2 GB** |
52 | 63 | - [ ] Parquet baseline for comparison: same data should fail or require splitting when any |
53 | 64 | column chunk exceeds 2 GB. |
54 | 65 |
|
55 | | -## Array API |
| 66 | +## API |
56 | 67 |
|
57 | 68 | - [ ] Use domain primitives (`UInt32`, `UInt64`, etc.) as value classes via Project Valhalla instead of raw `long`/`int` |
58 | 69 | - See https://dfa1.github.io/articles/rethink-domain-primitives-with-valhalla |
59 | 70 | - Candidates: `PType` integer kinds, buffer offsets, row indices, byte lengths |
60 | 71 | - Goal: type-safety at zero cost (value class = no heap alloc, no boxing) |
61 | 72 |
|
62 | | -- [ ] Optional `vortex-arrow` bridge module for Arrow ecosystem interop |
63 | | - - Primary API stays `ArrayLong`/`ArrayDouble` (zero-copy, no deps, no Unsafe) |
64 | | - - Bridge wraps typed views into Arrow `BigIntVector`, `Float8Vector`, etc. for users who need |
65 | | - Arrow Flight / DuckDB ADBC / pandas interop |
66 | | - - Conversion involves a copy (MemorySegment → Arrow off-heap buffer) — cost is explicit and opt-in |
67 | | - - Arrow JVM uses `sun.misc.Unsafe` / Netty internally; keeping it in a separate module means |
68 | | - the core library stays Unsafe-free |
69 | 73 |
|
70 | 74 | ## Encodings |
71 | 75 |
|
|
0 commit comments