Skip to content

Commit ff624c0

Browse files
committed
impl(sprint-13/W-I3): D-CSV-16 — WitnessIndexCamPq via VSA-bind role-keyed adapter (OQ-CSV-11 Option A; OQ-CSV-12 lazy enable_cam_pq)
- lance-graph-contract::vsa::roles: new module with 256-D CAM-PQ role-key constants (S_KEY, P_KEY, O_KEY slices), make_role_key, make_palette_id. Disjoint slices guarantee exact role orthogonality per I-VSA-IDENTITIES. - WitnessIndexCamPq: real ndarray::hpc::cam_pq wiring behind `with-cam-pq` feature (implies ndarray-hpc). spo_to_fingerprint (Option A VSA-bind), cam_pq_search (ADC distance-ranked top-k), rebuild_after_evict. - WitnessCorpus: enable_cam_pq (lazy, OQ-CSV-12), query_similar, is_cam_pq_enabled. W5-INV-CAM-PQ-INDEX: insert/evict mirrors to both indices when cam_pq enabled. - 12 tests (T1-T12): option-A roundtrip, role orthogonality, codec insert, ADC distance order, partial query neighbours, HashMap back-compat, out-of-order coherence, lazy enable, backfill, eviction rebuild, smoke tests. 5 vsa::roles tests (all passing). AP1-AP8 self-scan: no violations found. https://claude.ai/code/session_01UwJuKqP828qyX1VkLgGJFS
1 parent 1a037e4 commit ff624c0

6 files changed

Lines changed: 885 additions & 448 deletions

File tree

crates/lance-graph-contract/src/lib.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ pub mod sla;
7878
pub mod splat;
7979
pub mod tax;
8080
pub mod thinking;
81+
pub mod vsa;
8182
pub mod world_map;
8283
pub mod world_model;
8384

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
//! VSA (Vector Symbolic Architecture) modules.
2+
//!
3+
//! Currently provides the 256-D CAM-PQ role-key constants for the
4+
//! `WitnessIndexCamPq` SPO adapter (D-CSV-16, sprint-13).
5+
//!
6+
//! **Scope:** 256-D f32 multiply-add VSA algebra for CAM-PQ witness indexing.
7+
//! NOT the 16,384-D binary VSA in `grammar::role_keys` — that is a separate
8+
//! algebra (binary XOR-bind, 16K dims).
9+
10+
pub mod roles;
Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
//! VSA role-key constants for 256-D CAM-PQ witness-corpus adapter.
2+
//!
3+
//! Per I-VSA-IDENTITIES (CLAUDE.md): VSA operates on IDENTITY fingerprints
4+
//! that POINT TO content, never on content's quantized register itself.
5+
//!
6+
//! This module provides the three 256-D bipolar role-key vectors used by
7+
//! the `WitnessIndexCamPq` Option-A SPO adapter (`spo_to_fingerprint`).
8+
//! Each role key occupies a **disjoint** contiguous slice of the 256-D space:
9+
//!
10+
//! ```text
11+
//! R_S : dims [ 0 .. 85) — Subject role slice
12+
//! R_P : dims [ 85 .. 170) — Predicate role slice
13+
//! R_O : dims [170 .. 255) — Object role slice
14+
//! dim 255 — Reserved (NULL / out-of-palette marker)
15+
//! ```
16+
//!
17+
//! Disjoint slices guarantee **exact** orthogonality: `dot(R_S, R_P) = 0.0`
18+
//! by construction (no floating-point approximation required). This satisfies
19+
//! I-VSA-IDENTITIES Test-2 (role orthogonality).
20+
//!
21+
//! The bipolar pattern within each slice is derived deterministically from the
22+
//! FNV-64 hash of the role label, so the keys are reproducible across process
23+
//! restarts without persisting them.
24+
//!
25+
//! **Scope:** CAM-PQ 256-D only. These are NOT related to the 16,384-D
26+
//! `grammar::role_keys` keys — those live in a different VSA algebra
27+
//! (binary XOR-bind, 16K dims). This module addresses the 256-D f32
28+
//! multiply-add algebra used by the CAM-PQ codec.
29+
30+
/// Dimension of the 256-D CAM-PQ VSA space.
31+
pub const CAM_PQ_DIM: usize = 256;
32+
33+
/// Subject role slice: `[0 .. 85)`.
34+
pub const S_SLICE_START: usize = 0;
35+
/// Subject role slice end (exclusive).
36+
pub const S_SLICE_END: usize = 85;
37+
38+
/// Predicate role slice: `[85 .. 170)`.
39+
pub const P_SLICE_START: usize = 85;
40+
/// Predicate role slice end (exclusive).
41+
pub const P_SLICE_END: usize = 170;
42+
43+
/// Object role slice: `[170 .. 255)`.
44+
pub const O_SLICE_START: usize = 170;
45+
/// Object role slice end (exclusive).
46+
pub const O_SLICE_END: usize = 255;
47+
48+
/// Generate a deterministic bipolar ±1 role-key for a given slice.
49+
///
50+
/// Pattern is derived from FNV-64 of `label` bytes. Within `[start..end)`,
51+
/// each position gets `+1.0` or `-1.0` based on successive bits of the hash
52+
/// stream (xorshift64 expansion). Outside the slice, values are `0.0`.
53+
///
54+
/// This is a pure function — same label + same start/end → same output always.
55+
pub fn make_role_key(label: &[u8], start: usize, end: usize) -> [f32; CAM_PQ_DIM] {
56+
let mut key = [0.0f32; CAM_PQ_DIM];
57+
58+
// FNV-64 of label as seed
59+
let mut state = fnv64(label);
60+
61+
for slot in key[start..end].iter_mut() {
62+
// xorshift64 step for next pseudo-random bit
63+
state = xorshift64(state);
64+
// bit 0 of state → +1.0 or -1.0
65+
*slot = if (state & 1) == 0 { 1.0 } else { -1.0 };
66+
}
67+
key
68+
}
69+
70+
/// Generate a deterministic 256-entry identity catalogue.
71+
///
72+
/// `palette_id[i]` is the identity fingerprint for palette index `i`.
73+
/// Each entry is a bipolar ±1 vector derived from FNV-64 of
74+
/// `[seed_byte_hi, seed_byte_lo, i_hi, i_lo]`.
75+
///
76+
/// The catalogue is dense bipolar in all 256 dimensions — NOT a one-hot —
77+
/// which gives CAM-PQ codebook training meaningful geometric structure.
78+
/// Per I-VSA-IDENTITIES: these are identity fingerprints, not content.
79+
pub fn make_palette_id(seed: u64) -> Box<[[f32; CAM_PQ_DIM]; 256]> {
80+
let mut catalogue: Box<[[f32; CAM_PQ_DIM]; 256]> = vec![[0.0f32; CAM_PQ_DIM]; 256]
81+
.into_iter()
82+
.collect::<Vec<_>>()
83+
.try_into()
84+
.unwrap_or_else(|_| unreachable!("256 elements always fit"));
85+
86+
for (palette_idx, entry) in catalogue.iter_mut().enumerate() {
87+
// Derive a per-entry seed: mix `seed` with `palette_idx`
88+
let mut state = fnv64_mix(seed, palette_idx as u64);
89+
for slot in entry.iter_mut() {
90+
state = xorshift64(state);
91+
*slot = if (state & 1) == 0 { 1.0 } else { -1.0 };
92+
}
93+
}
94+
catalogue
95+
}
96+
97+
// ── Private helpers ──────────────────────────────────────────────────────────
98+
99+
/// FNV-64 hash of a byte slice.
100+
const fn fnv64(bytes: &[u8]) -> u64 {
101+
const OFFSET: u64 = 14695981039346656037;
102+
const PRIME: u64 = 1099511628211;
103+
let mut h = OFFSET;
104+
let mut i = 0;
105+
while i < bytes.len() {
106+
h = h.wrapping_mul(PRIME);
107+
h ^= bytes[i] as u64;
108+
i += 1;
109+
}
110+
h
111+
}
112+
113+
/// Mix a u64 seed with a u64 index (for per-entry catalogue derivation).
114+
fn fnv64_mix(seed: u64, idx: u64) -> u64 {
115+
const PRIME: u64 = 1099511628211;
116+
let mut h = seed.wrapping_add(idx.wrapping_mul(PRIME));
117+
h ^= idx.rotate_left(17);
118+
h = h
119+
.wrapping_mul(6364136223846793005)
120+
.wrapping_add(1442695040888963407);
121+
h
122+
}
123+
124+
/// xorshift64 PRNG step — produces the next pseudo-random u64 from state.
125+
/// Never returns 0 for nonzero input (guaranteed by xorshift theory).
126+
const fn xorshift64(mut x: u64) -> u64 {
127+
x ^= x << 13;
128+
x ^= x >> 7;
129+
x ^= x << 17;
130+
x
131+
}
132+
133+
#[cfg(test)]
134+
mod tests {
135+
use super::*;
136+
137+
#[test]
138+
fn role_keys_disjoint_slices_are_exactly_orthogonal() {
139+
let r_s = make_role_key(b"S", S_SLICE_START, S_SLICE_END);
140+
let r_p = make_role_key(b"P", P_SLICE_START, P_SLICE_END);
141+
let r_o = make_role_key(b"O", O_SLICE_START, O_SLICE_END);
142+
143+
let dot_sp: f32 = r_s.iter().zip(r_p.iter()).map(|(a, b)| a * b).sum();
144+
let dot_so: f32 = r_s.iter().zip(r_o.iter()).map(|(a, b)| a * b).sum();
145+
let dot_po: f32 = r_p.iter().zip(r_o.iter()).map(|(a, b)| a * b).sum();
146+
147+
// Disjoint slices → dot product is exactly 0.0
148+
assert_eq!(dot_sp, 0.0, "S·P must be exactly 0 (disjoint slices)");
149+
assert_eq!(dot_so, 0.0, "S·O must be exactly 0 (disjoint slices)");
150+
assert_eq!(dot_po, 0.0, "P·O must be exactly 0 (disjoint slices)");
151+
}
152+
153+
#[test]
154+
fn role_keys_are_bipolar_in_their_slice() {
155+
let r_s = make_role_key(b"S", S_SLICE_START, S_SLICE_END);
156+
// All values in slice are ±1.0
157+
for &v in &r_s[S_SLICE_START..S_SLICE_END] {
158+
assert!(v == 1.0 || v == -1.0, "slice values must be ±1.0, got {v}");
159+
}
160+
// All values outside slice are 0.0
161+
for &v in &r_s[S_SLICE_END..] {
162+
assert_eq!(v, 0.0, "out-of-slice values must be 0.0");
163+
}
164+
}
165+
166+
#[test]
167+
fn role_keys_deterministic() {
168+
let r_s1 = make_role_key(b"S", S_SLICE_START, S_SLICE_END);
169+
let r_s2 = make_role_key(b"S", S_SLICE_START, S_SLICE_END);
170+
assert_eq!(r_s1, r_s2, "same label → same key");
171+
}
172+
173+
#[test]
174+
fn palette_id_is_bipolar_dense() {
175+
let palette = make_palette_id(0xCAFE_BABE);
176+
for (i, entry) in palette.iter().enumerate() {
177+
for &v in entry.iter() {
178+
assert!(
179+
v == 1.0 || v == -1.0,
180+
"palette_id[{i}] must be bipolar ±1.0, got {v}"
181+
);
182+
}
183+
}
184+
}
185+
186+
#[test]
187+
fn palette_id_entries_differ() {
188+
let palette = make_palette_id(0xCAFE_BABE);
189+
// Entry 0 and entry 1 differ in at least some dimensions
190+
let diffs = palette[0]
191+
.iter()
192+
.zip(palette[1].iter())
193+
.filter(|(a, b)| a != b)
194+
.count();
195+
assert!(diffs > 0, "distinct palette indices must differ");
196+
}
197+
}

crates/lance-graph/Cargo.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,11 @@ lancedb-sdk = ["dep:lancedb"]
6969
# ndarray-hpc: use AdaWorldAPI/ndarray for Fingerprint<256>, CAM-PQ, CLAM, BLAS, ZeckF64.
7070
# Disable for minimal builds (CI, wasm, embedded) — falls back to standalone ndarray_bridge.rs.
7171
ndarray-hpc = ["dep:ndarray"]
72+
# with-cam-pq: enables WitnessIndexCamPq (real ndarray::hpc::cam_pq wiring) in WitnessCorpus.
73+
# Requires ndarray-hpc. HashMap-only users (no with-cam-pq) pay zero cost — no ndarray dep
74+
# for the witness index path unless this feature is enabled.
75+
# OQ-CSV-12: lazy enable_cam_pq() API; WitnessCorpus::new() stays cheap.
76+
with-cam-pq = ["ndarray-hpc"]
7277
# planner: wire lance-graph-planner for query classification, thinking styles, MUL gate.
7378
planner = ["dep:lance-graph-planner"]
7479
# bgz17-codec: palette semiring, HHTL distance matrix, SPO tripartite search.

crates/lance-graph/src/graph/arigraph/mod.rs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,6 @@ pub mod witness_corpus;
1616
pub mod xai_client;
1717

1818
pub use witness_corpus::{WitnessCorpus, WitnessEntry, WitnessId, WitnessIndexHashMap};
19+
20+
#[cfg(feature = "with-cam-pq")]
21+
pub use witness_corpus::{spo_to_fingerprint, CamPqState, WitnessIndexCamPq};

0 commit comments

Comments
 (0)