Skip to content

Commit 2f15278

Browse files
claudeclaude
committed
PR #477 review fixes: verify_layout offset-bound, target-state framing, lint
CodeRabbit / Codex review addressed: 1. soa_envelope::verify_layout() — P2 correctness fix Previously only checked that column widths *sum* to the declared stride. A column whose end offset exceeds the stride (e.g. two 4-byte columns at offsets 4 and 8 with stride 8) passed the sum check but placed data outside every row. Now each column's end offset is checked against stride directly before the pairwise overlap scan. New test: column_past_stride_caught. Total: 8 tests, all green. 2. soa-three-tier-model.md — Major: target-state vs current-state Sections that asserted "there is no baton/emission" in present tense while MailboxSoA::emit() still exists in source now carry explicit "Target state:" / "Current state:" labels. The removal is scheduled, not yet landed — the doc now says so. 3. CLAUDE.md — Major: patch-warning wording "treat it as a build error to fix" was too absolute; transitive semver mismatch is a legitimate cause. Reworded to "policy alert — verify direct deps and Cargo.lock wiring; track/resolve transitive blockers explicitly." 4. q3-standing-wave-falsification.md — Minor lint Blockquote lines with multiple spaces after `>` normalized (MD027). Unlabelled fenced code block at line 252 given `text` tag (MD040). 5. q4-hhtl-audit.md — Minor lint Blockquote spacing normalized (MD027). 6. New plan + epiphany (board hygiene for this session's findings) .claude/plans/cycle-coherent-soa-snapshot-v1.md — Arc-swap COW at column granularity; 6 deliverables D-SOA-SNAP-1..6; the byte-scale complement to temporal.rs row-scale deinterlace (PR #468). .claude/board/EPIPHANIES.md — E-DEINTERLACE-TWO-SCALES prepended. .claude/board/INTEGRATION_PLANS.md — plan entry prepended. https://claude.ai/code/session_0147hSzjmWZDuy2MSQNrhEK5
1 parent e0e016d commit 2f15278

8 files changed

Lines changed: 301 additions & 21 deletions

File tree

‎.claude/board/EPIPHANIES.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,59 @@
1+
## 2026-06-06 — E-DEINTERLACE-TWO-SCALES — deinterlace is one operation at two scales; no-cross-cycle-lag = byte-scale deinterlace
2+
3+
**Status:** FINDING (source-grounded; `temporal.rs` PR #468 confirms row-scale; byte-scale is a documented gap)
4+
**Confidence:** High
5+
6+
**The synthesis:** temporal causality in the SoA system must be enforced at two
7+
independent scales that share the same monotonic clock:
8+
9+
```text
10+
Row/query scale → HLC tick + DependsClosure → temporal.rs::deinterlace() (SHIPPED, PR #468)
11+
Byte/column scale → SoaEnvelope::cycle() stamp → MailboxSoA Arc-swap COW (GAP — plan written)
12+
```
13+
14+
Both are the SAME operation — "sort by the causal clock and project the result
15+
into the reader's reference frame" — but at different granularities.
16+
17+
**Row scale (PR #468 confirms):**
18+
`temporal.rs:18-20` defines the standing wave correctly: "merge-sort by HLC
19+
tick and every field's row lands on one timeline. The result IS the standing
20+
wave / kanban SoA." The `deinterlace()` function + `EpistemicMode` (Strict /
21+
Aware / Retro) + `DependsClosure` implement this. 8 tests pass.
22+
23+
**Byte scale (current gap):**
24+
Nothing in the codebase prevents a reader from holding column data from SoA
25+
cycle N and cycle N+1 in the same SIMD sweep. The `SoaEnvelope::cycle()` stamp
26+
exists but is not enforced as a snapshot barrier.
27+
28+
**The fix (plan: `cycle-coherent-soa-snapshot-v1.md`):**
29+
Arc-swap COW at column granularity in `MailboxSoa::advance_phase`:
30+
1. Writer increments `cycle`, then swaps the `Arc<[u8]>` of each mutated column.
31+
2. Reader snapshots all column Arcs under one cycle stamp (lock-free retry).
32+
3. The resulting `MailboxSoaSnapshot { cycle, cols }` is structurally coherent.
33+
34+
**The boundary:**
35+
`MultiLaneColumn` in ndarray stays layout-only. The Arc-swap policy lives in
36+
lance-graph's `MailboxSoa`. ndarray does not learn that cycles exist.
37+
38+
**The clock is one clock:**
39+
`SoaEnvelope::cycle()` (byte scale) and `QueryReference::ref_version` (row
40+
scale) are the same monotonic sequence. Threading `snapshot.cycle` into
41+
`QueryReference` closes the loop: row-scale and byte-scale deinterlace use
42+
the same clock.
43+
44+
**Standing wave clarification (Q3 probe result):**
45+
The "standing wave" is NOT a compute recurrence. It is the deinterlaced
46+
projection over Lance versions — provided by Lance versioning itself (O(1)
47+
90° lookup). Do not implement a standing wave in compute.
48+
49+
**Cross-ref:**
50+
- PR #468 (`temporal.rs`) — row-scale (SHIPPED)
51+
- PR #477 (`soa_envelope.rs`) — envelope contract (IN REVIEW)
52+
- `.claude/plans/cycle-coherent-soa-snapshot-v1.md` — byte-scale fix plan
53+
- `docs/probes/q3-standing-wave-falsification.md` — probe confirming no wave in compute
54+
55+
---
56+
157
## 2026-06-04 — E-AUDIT-RETENTION-CAVEAT — substrate-b consumer doc Lance-versions-as-audit claim was overstated; corrected to retention-policy-gated (codex P1 on #465)
258

359
**Status:** CORRECTION (codex P1 on PR #465, 2026-06-04; merged + immediate follow-up correction per the no-silent-edit discipline — the FIX appends; the original epiphany E-SUBSTRATE-B-CAPABILITY-ROADMAP stands as the corrected reference now reads).

‎.claude/board/INTEGRATION_PLANS.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
## 2026-06-06 — cycle-coherent-soa-snapshot-v1 (Arc-swap COW at column granularity; byte-scale deinterlace; no-cross-cycle-lag guarantee)
2+
3+
**Status:** QUEUED. Design-spec only, no code. **Plan file:** `.claude/plans/cycle-coherent-soa-snapshot-v1.md`.
4+
**Owns:** 6 deliverables D-SOA-SNAP-1..6.
5+
- D-SOA-SNAP-1: `MailboxSoaSnapshot` type in lance-graph-contract
6+
- D-SOA-SNAP-2: `SnapshotProvider` trait in lance-graph-contract
7+
- D-SOA-SNAP-3: Arc-swap write path in `MailboxSoa::advance_phase`
8+
- D-SOA-SNAP-4: `snapshot()` impl on `MailboxSoa`
9+
- D-SOA-SNAP-5: No-cross-cycle-lag falsification test (writer thread + 8 reader threads)
10+
- D-SOA-SNAP-6: Wire `snapshot.cycle` into `QueryReference` (close row-scale / byte-scale clock loop)
11+
**Epiphany:** E-DEINTERLACE-TWO-SCALES (prepended 2026-06-06).
12+
**Companion:** PR #468 (`temporal.rs`, row-scale, SHIPPED); PR #477 (`soa_envelope.rs`, IN REVIEW).
13+
**Boundary:** ndarray stays layout-only (`MultiLaneColumn`); Arc-swap policy in lance-graph only.
14+
15+
---
16+
117
## 2026-06-05 — cesium-osm-substrate-v1 (OpenStreetMap as 6th source class for the 3DGS-ArcGIS-Cesium ingestion plan; OSM PBF → Arrow → Lance → SPO → cesium tileset → splat renderer; substrate-reuse with splat-native-ultrasound-v1)
218

319
**Status:** PROPOSAL. Design-spec only, no code. **Plan file:** `.claude/plans/cesium-osm-substrate-v1.md` (~430 LOC). **Trigger:** user feasibility question on OSM × Cesium × Gaussian-splat coupling; cross-session coordination with OGAR.

‎.claude/plans/cycle-coherent-soa-snapshot-v1.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# Plan: Cycle-Coherent SoA Snapshot — No-Cross-Cycle-Lag Guarantee
2+
3+
**Version:** v1
4+
**Date:** 2026-06-06
5+
**Status:** Queued
6+
**D-ids:** D-SOA-SNAP-1 through D-SOA-SNAP-6
7+
8+
---
9+
10+
## The problem
11+
12+
`temporal.rs` (PR #468) closes the row-scale deinterlace gap: HLC tick →
13+
`classify/deinterlace` → causally-coherent row sequence. But there is a
14+
parallel byte-scale gap: nothing prevents a reader from holding a mix of
15+
column data from cycle N and cycle N+1 within the same SIMD sweep. This is
16+
the **cross-cycle lag problem** — a SIMD sweep that is not internally
17+
single-cycle is not coherent.
18+
19+
The deinterlace operation is one operation at two scales:
20+
21+
```text
22+
Row/query scale → HLC tick + DependsClosure → temporal.rs (SHIPPED, PR #468)
23+
Byte/column scale → SoaEnvelope::cycle() stamp → MailboxSoA Arc-swap (THIS PLAN)
24+
```
25+
26+
---
27+
28+
## The mechanism: Arc-swap COW at column granularity
29+
30+
The SoA mailbox carries its columns as `Arc<[u8]>` slices (via
31+
`MultiLaneColumn` in ndarray). The invariant is:
32+
33+
> **A reader that snapshots all column Arcs at the same `cycle()` stamp sees
34+
> a single coherent cycle. No column can be from a prior cycle.**
35+
36+
### Write path (in `lance-graph`, `MailboxSoa::advance_phase`)
37+
38+
On every `advance_phase(to: KanbanPhase)`:
39+
40+
1. Increment `cycle` counter on the envelope.
41+
2. For each mutated column: swap the `Arc` pointer — `Arc::make_mut` on the
42+
backing `Arc<[u8]>` of the `MultiLaneColumn`, write the new data, then
43+
publish the new Arc via an `ArcSwap` (or `RwLock<Arc<MultiLaneColumn>>`).
44+
3. The cycle increment is a `SeqCst` store (fence) BEFORE the column Arc
45+
swaps. Readers who observe the new cycle will see the new column data.
46+
47+
### Read path (in `lance-graph`, `MailboxSoaView`)
48+
49+
On `snapshot()`:
50+
51+
1. Load cycle stamp.
52+
2. Clone all column Arcs under the same cycle stamp (atomic snapshot loop:
53+
re-read cycle after loading all Arcs; retry if it changed — lock-free
54+
single-retry is sufficient because writers are serialized through
55+
`advance_phase`).
56+
3. Return `MailboxSoaSnapshot { cycle, cols: [...] }`.
57+
58+
The snapshot guarantees all column data is from the same cycle.
59+
60+
### Boundary: ndarray stays layout-only
61+
62+
`MultiLaneColumn` in ndarray is `Arc<[u8]>` with typed lane iterators —
63+
**layout-only**. The Arc-swap policy (when to swap, how to snapshot, the
64+
cycle fence) belongs in `lance-graph`'s `MailboxSoa`. ndarray never learns
65+
that cycles or snapshots exist. The boundary is:
66+
67+
```text
68+
ndarray::simd::MultiLaneColumn — Arc<[u8]>, lane iters, Send + Sync, zero-copy reads
69+
lance-graph::MailboxSoa — Arc-swap on advance_phase, cycle fence, snapshot()
70+
```
71+
72+
### Connection to temporal.rs
73+
74+
`SoaEnvelope::cycle()` is the byte-scale clock. `QueryReference::ref_version`
75+
is the row-scale clock (a Lance version). They are the same monotonic clock
76+
at different granularities — Lance version N corresponds to SoA cycle C(N).
77+
When `temporal.rs::deinterlace` runs at query time, the `V_ref` it uses should
78+
align with the `cycle()` of the snapshot being queried.
79+
80+
Wiring: `VersionScheduler::on_version(&view, at, exec)` provides the Lance
81+
version; the `MailboxSoaSnapshot` that went into that version carries its
82+
`cycle`. Threading `snapshot.cycle` into `QueryReference` closes the loop so
83+
row-scale and byte-scale deinterlace use the same clock.
84+
85+
---
86+
87+
## Deliverables
88+
89+
### D-SOA-SNAP-1 — `MailboxSoaSnapshot` type in lance-graph-contract
90+
91+
A `MailboxSoaSnapshot` struct: `cycle: u32`, `cols: Vec<Arc<MultiLaneColumn>>`.
92+
Snapshot is `Send + Sync`. No reference to the originating `MailboxSoa`.
93+
This is a point-in-time read — immutable after creation.
94+
95+
### D-SOA-SNAP-2 — `SnapshotProvider` trait in lance-graph-contract
96+
97+
```rust
98+
pub trait SnapshotProvider {
99+
fn snapshot(&self) -> MailboxSoaSnapshot;
100+
}
101+
```
102+
103+
Zero deps in contract. `MailboxSoa` in lance-graph implements it.
104+
105+
### D-SOA-SNAP-3 — Arc-swap write path in `MailboxSoa::advance_phase`
106+
107+
In lance-graph (not contract, not ndarray): implement the cycle fence +
108+
column Arc-swap on every `advance_phase`. Use `std::sync::RwLock<Arc<MultiLaneColumn>>`
109+
per column (no external arc-swap crate needed unless benchmarks show
110+
contention; add as a feature flag if needed).
111+
112+
### D-SOA-SNAP-4 — `snapshot()` implementation on `MailboxSoa`
113+
114+
Lock-free snapshot: load cycle, clone all column Arcs, re-read cycle, retry
115+
once if changed. Return `MailboxSoaSnapshot`.
116+
117+
### D-SOA-SNAP-5 — No-cross-cycle-lag falsification test
118+
119+
```rust
120+
// Spawn a writer thread: advance_phase in a loop (100 cycles).
121+
// Spawn 8 reader threads: each calls snapshot() in a loop.
122+
// Assert: every snapshot has all columns reporting the same cycle.
123+
// Assert: no snapshot mixes data from two different cycles.
124+
```
125+
126+
The test is the formal statement of the guarantee. If it passes, the
127+
invariant is mechanically enforced, not just documented.
128+
129+
### D-SOA-SNAP-6 — Wire `snapshot.cycle` into `QueryReference`
130+
131+
In the planner: when a query resolves a `MailboxSoaSnapshot`, thread
132+
`snapshot.cycle` through `QueryReference::hlc_tick` (or a new
133+
`QueryReference::soa_cycle: Option<u32>` field) so `deinterlace` at
134+
row scale uses the same cycle boundary as the snapshot at byte scale.
135+
136+
---
137+
138+
## Prerequisite gap fixes (order matters)
139+
140+
These mechanical fixes should land before or alongside D-SOA-SNAP-1
141+
(they settle the column shape):
142+
143+
1. Remove `MailboxSoA::emit()` + `CollapseGateEmission` from source.
144+
2. Rename `last_emission_cycle` → `last_active_cycle` in MailboxSoA.
145+
3. Drop `entity_type: u16` from SoA row — MailboxId IS NiblePath.
146+
4. Fix `OntologyRegistry::enumerate_first_with_entity_type_id` linear scan.
147+
5. Remove `MappingRow.thinking_style` — Kanban owns thinking styles.
148+
6. Fix `unbundle_from` in `kv_bundle.rs:29` — `wrapping_sub` is not the
149+
inverse of weighted-average `bundle_into`.
150+
151+
Items 1-5 settle the column shape before the Arc-swap schema is frozen.
152+
Item 6 is independent but should not be deferred (correctness bug).
153+
154+
---
155+
156+
## Non-goals
157+
158+
- No recurrence / standing wave implementation. The standing wave is the
159+
deinterlaced Lance version projection, provided by Lance versioning
160+
(O(1) 90° lookup). Do not implement it in compute.
161+
- No baton. No emission. No inter-mailbox handoff type. The snapshot is
162+
consumed in-place; nothing is transmitted.
163+
- ndarray does not learn about cycles, snapshots, or advance_phase.
164+
165+
---
166+
167+
## Cross-references
168+
169+
- `temporal.rs` (PR #468) — row-scale deinterlace (SHIPPED)
170+
- `soa_envelope.rs` (PR #477) — envelope LE contract (IN REVIEW)
171+
- `soa-three-tier-model.md` — three-tier lifecycle model
172+
- `q3-standing-wave-falsification.md` — falsification: standing wave = Lance
173+
versioning, not compute recurrence
174+
- `.claude/board/EPIPHANIES.md` E-DEINTERLACE-TWO-SCALES — the synthesis
175+
- `ndarray/src/simd_soa.rs` — `MultiLaneColumn` (layout-only; Arc-swap lives
176+
in lance-graph, not here)

‎CLAUDE.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,11 @@ workspace is local — prefer the local/fork source over the registry, always.
1414
- If a fork's coordinates (git URL, branch/tag, feature flag) are unknown,
1515
**STOP and ask**. Do NOT fall back to crates.io as a convenience or to make a
1616
build pass.
17-
- `"warning: Patch <crate> ... was not used in the crate graph"` means the fork
18-
is NOT actually wired — treat it as a build error to fix, never a warning to
19-
ignore.
17+
- `"warning: Patch <crate> ... was not used in the crate graph"` is a policy
18+
alert. It can indicate missing fork wiring OR a transitive semver mismatch
19+
that prevents the patch from applying. Do not ignore it: verify direct
20+
`Cargo.toml` patch entries and `Cargo.lock` wiring, then track/resolve any
21+
transitive blocker explicitly before closing the issue.
2022
- crates.io is permitted ONLY for crates that have no AdaWorldAPI fork / no local
2123
source.
2224

‎crates/lance-graph-contract/src/soa_envelope.rs‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -182,12 +182,23 @@ pub trait SoaEnvelope {
182182
found: Self::LAYOUT_VERSION,
183183
});
184184
}
185-
// 2. Columns are non-overlapping and their widths sum to the stride.
185+
// 2. Columns are non-overlapping, each fits within [0, stride), and
186+
// their widths sum to the stride.
187+
// Checking only the width sum is insufficient: two columns whose
188+
// widths sum to the stride can still have one column whose end
189+
// offset exceeds the stride (e.g. offsets 4+8 with stride 8).
186190
let cols = self.columns();
187191
let mut summed = 0usize;
192+
let stride = self.row_stride();
188193
for (i, a) in cols.iter().enumerate() {
189194
let (a_start, a_end) = a.row_byte_range();
190195
summed += a.col_bytes_per_row();
196+
if a_end > stride {
197+
return Err(EnvelopeError::StrideMismatch {
198+
declared: stride,
199+
summed: a_end,
200+
});
201+
}
191202
for b in &cols[i + 1..] {
192203
let (b_start, b_end) = b.row_byte_range();
193204
let overlap = a_start < b_end && b_start < a_end;
@@ -199,7 +210,6 @@ pub trait SoaEnvelope {
199210
}
200211
}
201212
}
202-
let stride = self.row_stride();
203213
if summed != stride {
204214
return Err(EnvelopeError::StrideMismatch {
205215
declared: stride,
@@ -325,6 +335,21 @@ mod tests {
325335
));
326336
}
327337

338+
#[test]
339+
fn column_past_stride_caught() {
340+
// Two 4-byte columns at offsets 4 and 8 with stride 8.
341+
// Width sum = 8 = stride, but column B's end (12) > stride (8).
342+
let cols = vec![
343+
ColumnDescriptor { name_id: 0, kind: ColumnKind::F32, elems_per_row: 1, row_offset: 4 },
344+
ColumnDescriptor { name_id: 1, kind: ColumnKind::F32, elems_per_row: 1, row_offset: 8 },
345+
];
346+
let env = TestEnvelope { cols, stride: 8, rows: 1, bytes: vec![0u8; 8], cycle: 0 };
347+
assert!(matches!(
348+
env.verify_layout(),
349+
Err(EnvelopeError::StrideMismatch { .. })
350+
));
351+
}
352+
328353
#[test]
329354
fn packet_size_mismatch_caught() {
330355
let mut env = two_col_envelope(4);

‎docs/architecture/soa-three-tier-model.md‎

Lines changed: 14 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,14 @@
1010

1111
**Every SoA envelope is zero-copy from creation to Lance tombstone.**
1212

13-
There is no baton. There is no emission. There is no inter-mailbox handoff type.
14-
No bytes leave the backing store until Lance's own columnar I/O writes them to
15-
disk — and even then the in-memory store is unchanged, not serialized and
16-
freed.
13+
**Target state:** there is no baton, no emission, and no inter-mailbox handoff
14+
type. No bytes leave the backing store until Lance's own columnar I/O writes
15+
them to disk — and even then the in-memory store is unchanged, not serialized
16+
and freed.
17+
18+
**Current state:** legacy `MailboxSoA::emit()` and `CollapseGateEmission`
19+
artifacts still exist in source and are scheduled for removal (see Tier 1
20+
below). Treat them as migration-only; do not call or extend them.
1721

1822
---
1923

@@ -47,9 +51,10 @@ Lance soft-delete (tombstone) ← sole lifecycle event that ends the store
4751
last written. It is a same-cycle guard, not a history column. (Rename from
4852
`last_emission_cycle` in source — the emission framing is wrong.)
4953

50-
**`MailboxSoA::emit()` and `CollapseGateEmission` in source are code artifacts
51-
from a superseded design and must be removed.** There is no inter-mailbox
52-
handoff type.
54+
**`MailboxSoA::emit()` and `CollapseGateEmission` are legacy artifacts from a
55+
superseded design and are scheduled for removal.** Until that lands, treat them
56+
as migration-only and non-canonical. There is no intended inter-mailbox handoff
57+
type.
5358

5459
---
5560

@@ -142,8 +147,8 @@ out pending resolution. **Do not fall back to crates.io surrealdb.**
142147

143148
| Concept | Status |
144149
|---|---|
145-
| `CollapseGateEmission` as cross-mailbox carrier | **WRONG** — remove from source |
146-
| `MailboxSoA::emit()` | **WRONG** — remove from source |
150+
| `CollapseGateEmission` as cross-mailbox carrier | **WRONG** — scheduled for removal |
151+
| `MailboxSoA::emit()` | **WRONG** — scheduled for removal |
147152
| "Baton" as inter-mailbox handoff | **WRONG** — superseded |
148153
| `wire_cost_bytes() = 13 + 10·baton_count` | **WRONG** — from CLAUDE.md E-BATON-1, now superseded |
149154
| `Vsa16kF32` as a cross-mailbox carrier | **WRONG** — deprecated, lives only as legacy `cycle` column in `BindSpace` |

‎docs/probes/q3-standing-wave-falsification.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33
> **Branch:** `claude/stoic-turing-M0Eiq`
44
> **Date:** 2026-06-06
55
> **Files read:** `crystal/fingerprint.rs`, `cognitive_shader.rs`, `collapse_gate.rs`,
6-
> `cycle_accumulator.rs`, `crystal/cycle.rs`, `recipe_kernels.rs`, `atoms.rs`,
7-
> `planner/src/cache/kv_bundle.rs`, `ndarray/src/hpc/vsa.rs`
6+
> `cycle_accumulator.rs`, `crystal/cycle.rs`, `recipe_kernels.rs`, `atoms.rs`,
7+
> `planner/src/cache/kv_bundle.rs`, `ndarray/src/hpc/vsa.rs`
88
> **Method:** Read all VSA, braid, permutation, bundle, and cognitive-shader source in
9-
> lance-graph-contract + lance-graph-planner. Answer 7 questions with source citations.
10-
> No narrative — executable code only.
9+
> lance-graph-contract + lance-graph-planner. Answer 7 questions with source citations.
10+
> No narrative — executable code only.
1111
1212
---
1313

@@ -249,7 +249,7 @@ The standing-wave framing incorrectly assumed that temporal persistence had to b
249249
implemented as a recurrence within the compute graph. It does not. Lance provides it
250250
structurally:
251251

252-
```
252+
```text
253253
current compute (per-mailbox, feed-forward, ephemeral)
254254
│ commit
255255
▼

0 commit comments

Comments
 (0)