Skip to content

Commit 4abd197

Browse files
dfa1claude
andcommitted
docs: document FlatBuffers vs Protobuf design choice
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 8a62ee9 commit 4abd197

1 file changed

Lines changed: 12 additions & 9 deletions

File tree

README.md

Lines changed: 12 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,15 +25,18 @@ reads, making it easier to:
2525
| [spiraldb/vortex](https://github.com/spiraldb/vortex) | Rust | Reference implementation + JNI bindings |
2626
| [spiraldb/vortex-go](https://github.com/spiraldb/vortex-go) | Go | Pure-language port; primary inspiration for this project's approach |
2727

28-
## Status
29-
30-
| Component | Status |
31-
|-----------|--------|
32-
| Trailer + postscript parsing | Working |
33-
| FlatBuffer footer / layout / dtype | Working |
34-
| Zone-map predicate pruning | Not started |
35-
| Scan iterator (chunked reads) | Not started |
36-
| Writer | Not started |
28+
## Serialization formats
29+
30+
The format uses two serialization libraries for different roles:
31+
32+
| Format | Used for | Why |
33+
|--------|----------|-----|
34+
| **FlatBuffers** | Footer, Layout, Array structure | Zero-copy random access — fields read directly from memory-mapped bytes, no allocation |
35+
| **Protobuf** | Codec metadata, DType, Scalar values | Schema evolution and cross-language compatibility for small blobs |
36+
37+
FlatBuffers suit the file-structure layer: the footer is parsed once at open and the layout tree is traversed during scan — both benefit from direct field access on mapped memory. Protobuf suits codec metadata: tiny blobs parsed once per chunk, where schema evolution matters more than zero-copy speed.
38+
39+
Replacing protobuf with FlatBuffers is not viable — existing `.vortex` files produced by the Rust reference implementation embed protobuf bytes in codec metadata blobs, and wire compatibility requires matching the format exactly.
3740

3841
## Requirements
3942

0 commit comments

Comments
 (0)