Commit 8d9052e
## Summary
Tracking issue: #7297
Decomposes `TurboQuant` into:
```text
ScalarFnArray(L2Denorm, [
ScalarFnArray(SorfTransform, [
Extension<Vector>(
FixedSizeListArray(
DictArray(codes=Primitive<u8>, values=Primitive<f32>),
padded_dim
)
)
]),
norms
])
```
This makes the implementation more modular and turns the
TurboQuant-specific pieces into reusable building blocks.
Also defines SORF sign generation with a frozen local `SplitMix64`
implementation and cleans up related vector compute code.
## API Changes
- add `SorfTransform`
- remove the `TurboQuant` encoding/array type
- keep `TurboQuantScheme` as the compressor entry point
- simplify a all scalar fn APIs as we no longer need `ApproxOptions`
Note that `ApproxOptions` doesn't actually make sense here because we
are doing exact compute here, it is the encoding itself that is lossy.
Until we figure out what the exact semantics are of a lossy encoding I
will remove this.
## Testing
More tests:
- TurboQuant roundtrip / structural coverage
- SORF roundtrip + norm preservation
- deterministic SplitMix64 / sign generation coverage
- readthrough behavior (when children are normalized) for vector
similarity ops
---------
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
1 parent 360b41d commit 8d9052e
38 files changed
Lines changed: 2836 additions & 2946 deletions
File tree
- vortex-file/src
- vortex-tensor
- src
- encodings
- turboquant
- array
- compute
- scheme
- tests
- scalar_fns
- sorf_transform
- vortex/benches
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
60 | | - | |
61 | 59 | | |
62 | 60 | | |
63 | 61 | | |
| |||
111 | 109 | | |
112 | 110 | | |
113 | 111 | | |
114 | | - | |
115 | | - | |
116 | 112 | | |
117 | 113 | | |
118 | 114 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | 32 | | |
34 | 33 | | |
| 34 | + | |
35 | 35 | | |
36 | 36 | | |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
8 | 7 | | |
9 | 8 | | |
10 | 9 | | |
This file was deleted.
This file was deleted.
This file was deleted.
0 commit comments