Skip to content

Commit 8d9052e

Browse files
connortsui20claude
andauthored
Decompose TurboQuant into 2 ScalarFnArrays and DictArray (#7374)
## Summary Tracking issue: #7297 Decomposes `TurboQuant` into: ```text ScalarFnArray(L2Denorm, [ ScalarFnArray(SorfTransform, [ Extension<Vector>( FixedSizeListArray( DictArray(codes=Primitive<u8>, values=Primitive<f32>), padded_dim ) ) ]), norms ]) ``` This makes the implementation more modular and turns the TurboQuant-specific pieces into reusable building blocks. Also defines SORF sign generation with a frozen local `SplitMix64` implementation and cleans up related vector compute code. ## API Changes - add `SorfTransform` - remove the `TurboQuant` encoding/array type - keep `TurboQuantScheme` as the compressor entry point - simplify a all scalar fn APIs as we no longer need `ApproxOptions` Note that `ApproxOptions` doesn't actually make sense here because we are doing exact compute here, it is the encoding itself that is lossy. Until we figure out what the exact semantics are of a lossy encoding I will remove this. ## Testing More tests: - TurboQuant roundtrip / structural coverage - SORF roundtrip + norm preservation - deterministic SplitMix64 / sign generation coverage - readthrough behavior (when children are normalized) for vector similarity ops --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent 360b41d commit 8d9052e

38 files changed

Lines changed: 2836 additions & 2946 deletions

vortex-file/src/strategy.rs

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -56,8 +56,6 @@ use vortex_pco::Pco;
5656
use vortex_runend::RunEnd;
5757
use vortex_sequence::Sequence;
5858
use vortex_sparse::Sparse;
59-
#[cfg(feature = "unstable_encodings")]
60-
use vortex_tensor::encodings::turboquant::TurboQuant;
6159
use vortex_utils::aliases::hash_map::HashMap;
6260
use vortex_utils::aliases::hash_set::HashSet;
6361
use vortex_zigzag::ZigZag;
@@ -111,8 +109,6 @@ pub static ALLOWED_ENCODINGS: LazyLock<HashSet<ArrayId>> = LazyLock::new(|| {
111109
allowed.insert(RunEnd.id());
112110
allowed.insert(Sequence.id());
113111
allowed.insert(Sparse.id());
114-
#[cfg(feature = "unstable_encodings")]
115-
allowed.insert(TurboQuant.id());
116112
allowed.insert(ZigZag.id());
117113

118114
#[cfg(feature = "zstd")]

vortex-tensor/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ half = { workspace = true }
2929
itertools = { workspace = true }
3030
num-traits = { workspace = true }
3131
prost = { workspace = true }
32-
rand = { workspace = true }
3332

3433
[dev-dependencies]
34+
rand = { workspace = true }
3535
rand_distr = { workspace = true }
3636
rstest = { workspace = true }

vortex-tensor/public-api.lock

Lines changed: 83 additions & 161 deletions
Large diffs are not rendered by default.

vortex-tensor/src/encodings/mod.rs

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@
44
//! Encodings for the different tensor types.
55
66
// TODO(connor):
7-
// pub mod norm; // Unit-normalized vectors.
87
// pub mod spherical; // Spherical transform on unit-normalized vectors.
98

109
pub mod turboquant;

vortex-tensor/src/encodings/turboquant/array/data.rs

Lines changed: 0 additions & 261 deletions
This file was deleted.

vortex-tensor/src/encodings/turboquant/array/mod.rs

Lines changed: 0 additions & 11 deletions
This file was deleted.

vortex-tensor/src/encodings/turboquant/array/slots.rs

Lines changed: 0 additions & 38 deletions
This file was deleted.

0 commit comments

Comments
 (0)