feat: native pure-Rust u64 FastPFOR codec by CommanderStorm · Pull Request #85 · fast-pack/FastPFOR-rs

CommanderStorm · 2026-07-17T10:44:15Z

Adds a native 64-bit (u64) FastPFOR codec to the pure-Rust side

Adds a 64-bit (`u64`) FastPFOR codec to the pure-Rust side, which was previously 32-bit only. `FastPForWide128`/`FastPForWide256` compress `u64` values via the (now un-gated) `BlockCodec64` trait: FastPFOR-packed aligned blocks plus a variable-byte tail for the sub-block remainder. The wire format is byte-identical to the C++ `CppFastPFor128`/`CppFastPFor256` `encode64`/`decode64` paths, verified by parity unit tests and a 30-minute differential fuzz run (13.8M executions, no crashes). - bitpacking_wide: generic scalar packer for `u64` at widths 0..=64, whose bit-ordering is cross-checked against the proven u32 kernels. - fastpfor64: `FastPForWide<N>` with 65-entry exception tables, a 2-word exception bitmap, and u64 exception values. - Un-gate `BlockCodec64` so pure-Rust builds get 64-bit support. - Add the `fastpfor_u64` differential fuzz target and a README example. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

for more information, see https://pre-commit.ci

codecov · 2026-07-17T10:46:25Z

Codecov Report

❌ Patch coverage is 97.35350% with 14 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/rust/integer_compression/fastpfor32.rs	95.94%	6 Missing ⚠️
src/rust/integer_compression/variable_byte.rs	93.65%	4 Missing ⚠️
src/test_utils.rs	76.47%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

Explain the Rust code on its own terms rather than by reference to the C++ implementation, drop a redundant test comment, and write comments as short single-sentence lines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Consolidate the separate FastPForWide128/256 types into the existing FastPFor128/FastPFor256: each now implements AnyLenCodec (u32) and BlockCodec64 (u64), mirroring the C++ codecs that expose both widths from one type. The u64 engine is retained as a private field, so buffer reuse is preserved and the public API drops the extra FastPForWide* names. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

for more information, see https://pre-commit.ci

The u32 and u64 codecs had near-identical encode_page/decode_page/best_bit implementations differing only in element width. Extract a single generic FastPForEngine<N, T> parameterized by a FastPForInt trait that abstracts the width-specific pieces (bit width, exception bitmap word count, pack/unpack, shifts). u32 keeps its hand-unrolled bit-packing kernels via trait dispatch, so there is no performance regression; u64 uses the generic wide packer. fastpfor.rs and fastpfor64.rs become thin wrappers over the engine. Net -296 lines with the page algorithm now written once. Verified byte-identical: u32 encode_compare fuzz (Rust==C++, 808K runs) and u64 differential fuzz (619K runs) both clean, plus the full test suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

for more information, see https://pre-commit.ci

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

- Use Vec::clear() instead of truncate(0) in the encode_compare fuzz target (newer clippy flags truncate-to-zero). - Apply rustfmt to the generic-engine allow-attribute and the u64 fuzz target. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

# Conflicts: # src/test_utils.rs

for more information, see https://pre-commit.ci

nyurik · 2026-07-17T18:10:27Z

why not flatten FastPForEngine into FastPFor ?

pub struct FastPFor<const N: usize, T: FastPForInt = u32> {...}

nyurik · 2026-07-18T03:12:19Z

i did a number of refactorings, and did a bit of a deep dive into benchmarks - making sure the new impl has the same speed (inlining always seems to be the key sadly). I think a bit more cleanup might be needed

The pedantic `clippy::inline_always` lint was failing CI on the thin forwarders in the `FastPForInt` impls. The `#[inline(always)]` is intentional (documented above each fn); add an explicit `#[allow]` with reason so clippy stops flagging it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CI's `just docs` denies rustdoc warnings. Fix the four broken links on the `FastPFor` doc comment: - `BlockCodec`, `FastPForBlock128`, `FastPForBlock256` now use explicit `crate::` paths (they're re-exported at the crate root, not in scope in this module). - `FastPForInt` is a private sealed trait, so drop the intra-doc link and leave it as plain code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CommanderStorm · 2026-07-18T12:13:46Z

+        pub struct $name {
+            narrow: CompositeCodec<$block, VariableByte>,
+            wide: FastPFor<$n, u64>,
+        }


not entirely happy with this.

So now that we have smushed u32 and u64 together, we have the worst of both worlds imo.
Your version means that

the code uses a bit harder to read macros (likely fine)

allocates both narrow AND wide

I would still prefer the engine abstraction because at least there, we are not allocating wide+narrow.

We can OnceCell it, but that also has costs for perf..

Collapse the two-field FastPFor codec into a single generic `FastPForCodec<N, T>` with one `inner: CompositeCodec<FastPFor<N,T>, VariableByte<T>>`. - Give `BlockCodec`/`AnyLenCodec` an associated `Elem` so the block engine and composite are one width-generic implementation instead of duplicated u32/u64 paths. - Unify `VariableByte` into a generic `VariableByte<T>` with u32 and u64 tail impls; drop the free `vbyte_encode64`/`vbyte_decode64` functions and `VariableByteWide`. - Split the public codec by width: `FastPFor128`/`FastPFor256` are u32; add `FastPForWide128`/`FastPForWide256` for u64. Each type serves one width, so it holds one codec. - Delete the redundant `BlockCodec64 for FastPFor<N,u64>`; the u64 type keeps a delegating `BlockCodec64` impl for C++ parity comparison. Wire format unchanged: C++ u64 parity tests pass. All tests, clippy, rustfmt, and rustdoc clean under both default and `cpp` features. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqJPh46kx6YzhwUjX6EPDn

Replace the hand-written operator bounds, ZERO/ONE constants, and per-type significant_bits impls on FastPForInt with a PrimInt supertrait bound. Only the width-specific members (scratch buffers, packing/bitmap kernels) remain. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqJPh46kx6YzhwUjX6EPDn

for more information, see https://pre-commit.ci

CommanderStorm and others added 2 commits July 17, 2026 12:43

[pre-commit.ci] auto fixes from pre-commit.com hooks

9d76737

for more information, see https://pre-commit.ci

CommanderStorm and others added 3 commits July 17, 2026 12:52

[pre-commit.ci] auto fixes from pre-commit.com hooks

989b9a3

for more information, see https://pre-commit.ci

CommanderStorm commented Jul 17, 2026

View reviewed changes

Comment thread fuzz/fuzz_targets/fastpfor_u64.rs Outdated

CommanderStorm and others added 3 commits July 17, 2026 13:50

Update fuzz/fuzz_targets/fastpfor_u64.rs

b3d15e2

[pre-commit.ci] auto fixes from pre-commit.com hooks

91d0be5

for more information, see https://pre-commit.ci

CommanderStorm marked this pull request as ready for review July 17, 2026 12:30

Copilot AI review requested due to automatic review settings July 17, 2026 12:30

Merge branch 'main' into feat/native-u64-fastpfor

3b5e457

Copilot AI reviewed Jul 17, 2026

CommanderStorm requested a review from nyurik July 17, 2026 12:48

nyurik and others added 2 commits July 17, 2026 14:07

Merge branch 'main' into feat/native-u64-fastpfor

e17a413

# Conflicts: # src/test_utils.rs

[pre-commit.ci] auto fixes from pre-commit.com hooks

620d824

for more information, see https://pre-commit.ci

nyurik added 8 commits July 17, 2026 19:16

wip

c6045e0

wip

dcf64f7

optimize

2a2e2da

wip

b6f4da0

wip

84d4367

wip

9b3fbed

refactor more to per-bit files

f68089f

Merge branch 'main' into feat/native-u64-fastpfor

2f9f7b4

CommanderStorm and others added 2 commits July 18, 2026 13:07

CommanderStorm commented Jul 18, 2026

View reviewed changes

CommanderStorm and others added 2 commits July 18, 2026 15:12

CommanderStorm commented Jul 18, 2026

View reviewed changes

Comment thread src/rust/integer_compression/fastpfor_int.rs Outdated

CommanderStorm and others added 2 commits July 18, 2026 15:50

Apply suggestion from @CommanderStorm

2f5fc2c

[pre-commit.ci] auto fixes from pre-commit.com hooks

056725d

for more information, see https://pre-commit.ci

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: native pure-Rust u64 FastPFOR codec#85

feat: native pure-Rust u64 FastPFOR codec#85
CommanderStorm wants to merge 26 commits into
fast-pack:mainfrom
CommanderStorm:feat/native-u64-fastpfor

CommanderStorm commented Jul 17, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

nyurik commented Jul 17, 2026

Uh oh!

nyurik commented Jul 18, 2026

Uh oh!

CommanderStorm Jul 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

CommanderStorm commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

nyurik commented Jul 17, 2026

Uh oh!

nyurik commented Jul 18, 2026

Uh oh!

CommanderStorm Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CommanderStorm commented Jul 17, 2026 •

edited

Loading

codecov Bot commented Jul 17, 2026 •

edited

Loading