Skip to content

Commit 6e8ba5d

Browse files
dfa1claude
andcommitted
docs: reopen #7a — bitpacked patches (step 3) not yet implemented
Steps 1, 2, 4, 5 are done; patch decode (child slot indices+values) is missing. Integration tests pass only because JNI benchmark data doesn't trigger patches. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent c148395 commit 6e8ba5d

1 file changed

Lines changed: 15 additions & 4 deletions

File tree

TODO.md

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -30,10 +30,21 @@
3030
- `pcodec` — float compression
3131
- `fsst` — string compression (in-flight, see `FsstCodec.java`)
3232

33-
- [ ] **#7a `fastlanes.bitpacked` — patches (step 3)**
34-
- Steps 1, 2, 4, 5 landed: protobuf metadata (tags 1+2), unified `unpack(buf, bitWidth, offset, T, rowCount)` using
35-
FastLanes algorithm, encoder writes protobuf, round-trip test green.
36-
- Step 3 still open: decode child slots (indices + values), overwrite output at patch indices.
33+
- [ ] **#7a Fix `fastlanes.bitpacked` — spec-compliant rewrite**
34+
- Root cause: current code guesses format by metadata byte size (9 = Java, 2 = JNI). Wrong.
35+
The spec always uses protobuf metadata regardless of writer origin.
36+
- **Spec** (from `encodings/fastlanes/src/bitpacking/vtable/mod.rs`):
37+
- Metadata: protobuf `BitPackedMetadata``bit_width u32` (tag 1), `offset u32` (tag 2, 0≤offset<1024),
38+
`patches PatchesMetadata` (tag 3, optional)
39+
- Buffer size: `ceil((len + offset) / 1024) * 128 * bit_width` bytes
40+
- FastLanes block layout: `LANES = 1024 / T` (T = element bit-width); `FL_ORDER = [0,4,2,6,1,5,3,7]`; logical
41+
index for `(row, lane)` = `FL_ORDER[row/8]*16 + (row%8)*128 + lane` — same formula for all types
42+
- Step 1: delete `decodeJni()` and `decodeJava()`; parse metadata as protobuf (tags 1+2)
43+
- Step 2: implement single unified `unpack(buf, bitWidth, offset, T, rowCount) → long[]` using the FastLanes
44+
algorithm above
45+
- Step 3: handle patches — decode child slots (indices + values), overwrite output at patch indices (**missing**; steps 1,2,4,5 done)
46+
- Step 4: align `encode()` to write protobuf metadata (tags 1+2) instead of the 9-byte custom format
47+
- Step 5: update `BitpackedCodecTest` for round-trip with spec-compliant metadata
3748
- Reference: `spiraldb/vortex` `encodings/fastlanes/src/bitpacking/`, `spiraldb/fastlanes-rs` `src/bitpacking.rs` +
3849
`src/macros.rs`
3950

0 commit comments

Comments
 (0)