Skip to content

Commit a380a41

Browse files
dfa1claude
andcommitted
docs: document >2GB file advantage over Parquet; add TODO for large-file test
Vortex uses uint64 offsets in the footer and uint32 segment lengths, avoiding Parquet's 2GB column-chunk limit (int32 page-size field). The FFM MemorySegment API maps files of arbitrary size, unlike the legacy FileChannel.map() which caps each mapping at 2GB. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 46817aa commit a380a41

2 files changed

Lines changed: 20 additions & 1 deletion

File tree

README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,17 @@ reads, making it easier to:
1818
- build and test on any platform with a standard JDK
1919
- debug and profile with standard JVM tooling
2020

21+
## Why Vortex instead of Parquet
22+
23+
| | Parquet | Vortex |
24+
|---|---------|--------|
25+
| Max column-chunk size | 2 GB (`int32` page offsets) | 4 GB per segment (`uint32` length) |
26+
| Max file offset | 2 GB (`int32` on some implementations) | 16 EB (`uint64` in footer) |
27+
| Encoding extensibility | Fixed codec set | Plugin registry, any encoding |
28+
29+
Files larger than 2 GB are a practical problem with Parquet: the `int32` data-page size field in the page header caps individual column chunks at 2 GB.
30+
Vortex uses `uint64` offsets throughout the footer, and this library maps files with `MemorySegment` (Java FFM API), which has no per-mapping size limit — unlike the legacy `FileChannel.map()` API that caps each mapping at 2 GB.
31+
2132
## Prior art and inspiration
2233

2334
| Project | Language | Notes |

TODO.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,4 +123,12 @@
123123
- move the project in a dedicated organization
124124
- create website
125125
- publish benchmarks
126-
- idea is to build something like hardwood.dev but for parquet files
126+
- idea is to build something like hardwood.dev but for parquet files
127+
128+
## Large-file support
129+
130+
- [ ] **#12 Test read/write of files > 2 GB**
131+
- Write a vortex file whose total segment data exceeds 2 GB (e.g., several wide int64 columns × enough rows)
132+
- Verify `VortexFile.open()` maps correctly and `ScanIterator` decodes without offset truncation
133+
- Parquet baseline for comparison: same data fails or requires splitting when any column chunk > 2 GB
134+
- Confirm no `int` casts silently truncate `uint64` offsets or `uint32` lengths in `SegmentSpec`

0 commit comments

Comments
 (0)