|
| 1 | +# Multi-Server Cognition — Legacy-Stack Displacement + Expansion Readiness (v1) |
| 2 | + |
| 3 | +> **Status:** PROPOSAL — rationale + expansion-readiness. NOT committed deliverables. |
| 4 | +> §2 (displacement) is current-state reasoning; §3 (multi-server) is a design |
| 5 | +> hypothesis that follows from the invariants but is **unbuilt**; §4 (determinism) |
| 6 | +> is a real, **unproven** prerequisite. |
| 7 | +> **Purpose:** Kill the "for multiple servers we need JanusGraph / Cassandra / |
| 8 | +> Zitadel" argument, and record the conditional path to multi-server HA/scale-out |
| 9 | +> that does **not** require them. |
| 10 | +> **Confidence:** HIGH vs JanusGraph + Cassandra; vs Zitadel — displaced by **Ory** |
| 11 | +> (Kratos+Hydra) for authN, §5; HIGH that the log-replicated-SoA shape is |
| 12 | +> *correct*; LOW that it is *proven* (gated on §4). |
| 13 | +
|
| 14 | +--- |
| 15 | + |
| 16 | +## 1. The argument being killed |
| 17 | + |
| 18 | +"For multiple servers / HA / scale-out we should adopt a proven distributed |
| 19 | +system" → reaches for **JanusGraph** (distributed graph), **Cassandra** |
| 20 | +(wide-column), or **Zitadel** (identity, on Cockroach-Raft). Each drags a |
| 21 | +heavyweight external service + its own consistency model + a JVM/Go runtime into |
| 22 | +a stack whose entire premise is a **single Rust binary** with an in-binary, |
| 23 | +zero-copy, **no-serialization** cognitive substrate. |
| 24 | + |
| 25 | +## 2. Component-by-component displacement |
| 26 | + |
| 27 | +| Legacy | Reached for | Covered by | Verdict | |
| 28 | +|---|---|---|---| |
| 29 | +| **JanusGraph** | distributed graph = Gremlin + Cassandra/HBase + ES/Solr | SurrealDB graph + lance-graph SoA/SIMD + Cypher/Gremlin/SPARQL front-ends on a Raft KV | **Displaced.** Replaces a JVM 3-system assembly with one Rust stack + CP consistency. Its only edge is proven billion-edge cluster scale — not this regime. | |
| 30 | +| **Cassandra** | scale-out wide-column store | SurrealDB-on-TiKV (Raft, CP, transactional) | **Displaced on the right axis.** A belief/memory substrate wants CP (read-your-writes, no last-write-wins clobber), not Cassandra's AP/eventual. Cassandra wins only at extreme multi-DC write-availability under partition — not needed. | |
| 31 | +| **Zitadel** | identity / OIDC IdP (runs on Cockroach-Raft) | authZ — RBAC `Policy` + `TenantId` Chinese-wall + JWT verify + merkle audit — **already in-stack**. authN/IdP — **Ory** (decided, §5) | **Displaced by Ory, not by Raft.** Raft was the wrong axis: the value is IdP *behavior*, not storage Raft. **Ory Kratos+Hydra** (composable, headless) fills authN; binary stays verification-only. See §5. | |
| 32 | +| **TiKV** | — | (it IS the Raft engine — the displacement *mechanism*, not a thing displaced) | Role = the replicated **log** tier (§3). It is "under lance," not "instead of lance" — see §3. | |
| 33 | +| **Databend** | — | — | Orthogonal. Cold analytics edge only (export-fed; its Raft replicates only its own metadata). Not in the multi-server path. | |
| 34 | + |
| 35 | +## 3. The multi-server expansion architecture |
| 36 | + |
| 37 | +"Multiple servers" is a replication seam, and a replication seam is a |
| 38 | +serialization boundary — which the no-serialization invariant forbids in the |
| 39 | +**hot** path. So the seam goes where serialization is already legal (the |
| 40 | +durable/egress tier), **never** on the hot SoA. |
| 41 | + |
| 42 | +**Replicate the log, not the SoA:** |
| 43 | + |
| 44 | +- Raft replicates the **episodic / belief-delta log** — append-only, ordered, |
| 45 | + content-addressed; each entry = one committed belief delta. |
| 46 | +- Each server keeps its **own local zero-copy SoA**, rebuilt by *applying* the |
| 47 | + log. SoA = the Raft **state machine**; the AriGraph **Witness episodic arc = |
| 48 | + the Raft log**; the Markov belief-chain = the state machine it drives. |
| 49 | + |
| 50 | +This dissolves the apparent "TiKV **or** lance" choice at the KVS layer — a false binary: |
| 51 | + |
| 52 | +- **TiKV/Raft replicates the *log*** (committed belief-deltas; cross-node consensus). |
| 53 | +- **lance materializes the *state* locally** per node (zero-copy SoA, rebuilt from applied entries). |
| 54 | +- → TiKV-the-log **under** lance-the-state. Both present, different roles. |
| 55 | + |
| 56 | +**Maps onto the existing `surrealdb/core/src/kvs/lance` engine:** the **WAL is the |
| 57 | +replication unit** (Raft-replicate the WAL); `memtable` + the lance dataset are |
| 58 | +the per-node materialized state followers rebuild. The structure is already |
| 59 | +shaped for this. |
| 60 | + |
| 61 | +**Rubikon = the append point.** `commit_gate.rs` is where a candidate belief |
| 62 | +becomes committed → that is the Raft append. The 550 ms candidate cloud stays |
| 63 | +**node-local, uncommitted, un-replicated** (no consensus cost); only committed |
| 64 | +belief-deltas pay the Raft round-trip. Consensus load is bounded to **real |
| 65 | +commits**, not every thought-spark. (Libet/Heckhausen admission control = the |
| 66 | +gate; the gate = the consensus boundary.) |
| 67 | + |
| 68 | +## 4. Hard prerequisite — deterministic apply (the unproven part) |
| 69 | + |
| 70 | +Same log → same SoA on every node **only if** applying a belief-delta is |
| 71 | +byte-deterministic across machines. NARS truth revision is f32 math, and |
| 72 | +floating-point **non-associativity** can make two nodes diverge from an |
| 73 | +identical log. So multi-server convergence **depends on** the determinism |
| 74 | +discipline the repo already prizes — `thinking-engine::reencode_safety` (x256 |
| 75 | +byte-determinism) + the D-SDR-26 determinism rule. This is the **precondition**, |
| 76 | +not a side-quest. |
| 77 | + |
| 78 | +- Log entries are the **binary `canonical_bytes` / `CausalEdge64`** form — never |
| 79 | + JSON. Keeps the no-serialization invariant intact: hot path zero-copy, |
| 80 | + replication log in canonical binary. |
| 81 | +- **CONJECTURE (NOT PROVEN):** the full NARS/truth apply path is byte-deterministic |
| 82 | + across architectures. Per the `bf16-hhtl-terrain` probe-queue discipline, the |
| 83 | + next deliverable here is the **determinism probe**, not more synthesis. |
| 84 | + |
| 85 | +## 5. The one genuinely separate decision — authN — **DECIDED: Ory** |
| 86 | + |
| 87 | +The storage of identity isn't the gap (it's just another tenant-partitioned |
| 88 | +table); the gap is IdP *behavior* (issue tokens, login, MFA, SSO/OIDC). This was |
| 89 | +always **orthogonal to Raft**, and the decision is **Ory** — the composable, |
| 90 | +headless option, deliberately *not* Zitadel's all-in-one: |
| 91 | + |
| 92 | +- **Ory Hydra** = OAuth2 / OIDC server → token **issuance** (the piece the Raft |
| 93 | + stack structurally lacks). |
| 94 | +- **Ory Kratos** = identity lifecycle → login, registration, MFA, recovery, user store. |
| 95 | +- **Ory Keto** = **not adopted** — authZ stays in-stack (`Policy` RBAC + `TenantId` |
| 96 | + Chinese-wall + merkle audit); Keto's Zanzibar model would duplicate it. |
| 97 | + |
| 98 | +**Integration:** Ory runs the IdP (its own Postgres/Cockroach persister); the |
| 99 | +binary stays a **verification-only consumer** — `auth.rs` already verifies the |
| 100 | +JWTs Hydra issues. authN therefore lives entirely **at the request boundary, |
| 101 | +never in the belief-delta hot loop**, and touches **none** of the §3 consensus |
| 102 | +layer. The legacy "Zitadel-or-build-it" fork collapses: Ory (Kratos+Hydra) fills |
| 103 | +authN, the in-stack authZ is untouched. |
| 104 | + |
| 105 | +**Open (low-risk default = standalone):** Ory doesn't natively persist to |
| 106 | +SurrealDB, so its store stays standalone Postgres/Cockroach rather than riding |
| 107 | +the shared replicated log. Revisit only if a SurrealDB persister is ever worth writing. |
| 108 | + |
| 109 | +## 6. Phasing (only if/when multi-server is actually needed) |
| 110 | + |
| 111 | +Single-node (Stefan-style) needs **none** of this — local lance, no Raft, no |
| 112 | +TiKV. The expansion is a readiness path, sequenced: |
| 113 | + |
| 114 | +1. **Determinism proof** (§4) — byte-deterministic NARS apply probe. Gate for everything below. |
| 115 | +2. **WAL-as-Raft-log** — promote the `kvs/lance` WAL to a Raft-replicated log; followers apply to local lance + memtable. |
| 116 | +3. **Commit-gate = append point** — wire `commit_gate.rs` to the Raft append; pre-Rubikon stays node-local. |
| 117 | +4. **Follower-read semantics** — local SoA may lag the leader; route CP reads to leader or wait-for-apply. |
| 118 | +5. **authN decision** (§5) — independent; only if external auth is needed. |
| 119 | + |
| 120 | +## Cross-references |
| 121 | + |
| 122 | +- `surrealdb/core/src/kvs/lance/{wal.rs, memtable.rs, flusher.rs, commit_gate.rs, schema.rs}` — the LSM-on-lance engine this rides (schema is opaque-KV today; typed columns deferred). |
| 123 | +- `.claude/plans/cognitive-substrate-convergence-v1.md` — D-CSV-6/7 (WitnessCorpus + MailboxSoA W-slot); the belief-delta log is the episodic arc those build. |
| 124 | +- `.claude/board/TECH_DEBT.md` — TD-ARIGRAPH-EPISODIC-FIDELITY-1 (the episodic arc / Witness), TD-JSON-SERIALIZATION-SITES-1 (canonical_bytes vs JSON on the log). |
| 125 | +- `thinking-engine::reencode_safety` + D-SDR-26 — the determinism prerequisite (§4). |
| 126 | +- `CLAUDE.md` — AGI-as-glove (the SoA *is* the substrate), I-SUBSTRATE-MARKOV (the Markov state machine), the no-serialization invariant. |
| 127 | + |
| 128 | +--- |
| 129 | + |
| 130 | +*Authored 2026-05-27. PROPOSAL — §2 displacement is current-state; §3 multi-server expansion is gated on the §4 determinism probe.* |
0 commit comments