Skip to content

Commit b255f29

Browse files
varshith-Gitvarshith-Git
committed
docs: Convert architecture diagram to beautiful Mermaid format
- Interactive, color-coded layer visualization - Added crash recovery sequence diagram - Much better rendering on GitHub - Cleaner and more professional
1 parent aa5f19b commit b255f29

1 file changed

Lines changed: 137 additions & 108 deletions

File tree

‎README.md‎

Lines changed: 137 additions & 108 deletions
Original file line numberDiff line numberDiff line change
@@ -162,118 +162,147 @@ results = client.search([0.15, 0.25, ...], k=5)
162162

163163
Valori uses a **strict layered architecture** ensuring the deterministic kernel remains pure while enabling production durability and multiple deployment modes.
164164

165+
```mermaid
166+
graph TB
167+
subgraph Clients["🖥️ CLIENT APPLICATIONS"]
168+
PythonApp["Python Scripts"]
169+
RustApp["Rust Applications"]
170+
HTTPClient["HTTP Clients"]
171+
Embedded["Embedded Devices<br/>(ARM Cortex-M)"]
172+
end
173+
174+
subgraph Interface["💻 INTERFACE LAYER (std)"]
175+
direction LR
176+
FFI["Python FFI (pyo3)<br/>EmbeddedKernel<br/>• Direct in-process<br/>• Microsecond latency"]
177+
HTTP["HTTP Server (axum)<br/>REST API<br/>• /v1/memory/*<br/>• Multi-client"]
178+
end
179+
180+
subgraph Durability["💾 DURABILITY LAYER (std)"]
181+
direction TB
182+
Engine["Engine Coordinator"]
183+
184+
subgraph Persistence["Persistence Components"]
185+
WALWriter["WAL Writer<br/>• bincode serialize<br/>• fsync() durability<br/>• Length-prefixed framing"]
186+
WALReader["WAL Reader<br/>• Deserialize commands<br/>• Iterator API<br/>• replay_wal()"]
187+
SnapshotMgr["Snapshot Manager<br/>• encode_state()<br/>• decode_state()<br/>• BLAKE3 hashing"]
188+
end
189+
190+
subgraph Storage["📁 Persistent Storage"]
191+
WALFile["commands.wal<br/>[version:u8]<br/>[length:u32]<br/>[command:bytes]"]
192+
SnapshotFile["state.snapshot<br/>[Header]<br/>[Kernel]<br/>[Metadata]<br/>[Index]"]
193+
end
194+
end
195+
196+
subgraph Kernel["⚙️ VALORI KERNEL (no_std, pure Rust)"]
197+
direction TB
198+
KernelState["KernelState&lt;R,D,N,E&gt;<br/>Deterministic State Machine"]
199+
200+
subgraph CoreComponents["Core Components"]
201+
direction LR
202+
VectorStorage["📊 Vector Storage<br/>RecordPool[R]<br/>FxpVector&lt;D&gt;<br/>• insert()<br/>• delete()<br/>• get()"]
203+
Graph["🕸️ Knowledge Graph<br/>NodePool[N]<br/>EdgePool[E]<br/>AdjacencyList<br/>• create_node()<br/>• create_edge()"]
204+
FXP["🔢 Fixed-Point Math<br/>Q16.16 (i32)<br/>• add, sub, mul, div<br/>• l2_distance()<br/>• normalize()"]
205+
end
206+
207+
Verify["🔐 Cryptographic Verification<br/>kernel_state_hash() → [u8;32]<br/>BLAKE3 deterministic hashing"]
208+
end
209+
210+
%% Client connections
211+
PythonApp --> FFI
212+
RustApp --> FFI
213+
HTTPClient --> HTTP
214+
Embedded -.->|Direct Link| KernelState
215+
216+
%% Interface to Durability
217+
FFI --> Engine
218+
HTTP --> Engine
219+
220+
%% Durability components
221+
Engine --> WALWriter
222+
Engine --> WALReader
223+
Engine --> SnapshotMgr
224+
225+
WALWriter -->|Write| WALFile
226+
WALReader -->|Read| WALFile
227+
SnapshotMgr -->|Save/Load| SnapshotFile
228+
229+
%% Recovery flow
230+
WALReader -.->|Replay| KernelState
231+
SnapshotMgr -.->|Restore| KernelState
232+
233+
%% Durability to Kernel
234+
Engine --> KernelState
235+
236+
%% Kernel internals
237+
KernelState --> VectorStorage
238+
KernelState --> Graph
239+
KernelState --> FXP
240+
KernelState --> Verify
241+
242+
%% Styling
243+
classDef clientStyle fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000
244+
classDef interfaceStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#000
245+
classDef durabilityStyle fill:#fce4ec,stroke:#c2185b,stroke-width:2px,color:#000
246+
classDef kernelStyle fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#000
247+
classDef storageStyle fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
248+
249+
class PythonApp,RustApp,HTTPClient,Embedded clientStyle
250+
class FFI,HTTP interfaceStyle
251+
class Engine,WALWriter,WALReader,SnapshotMgr durabilityStyle
252+
class KernelState,VectorStorage,Graph,FXP,Verify kernelStyle
253+
class WALFile,SnapshotFile storageStyle
165254
```
166-
╔═══════════════════════════════════════════════════════════════════════════════════╗
167-
║ CLIENT APPLICATIONS ║
168-
║ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ║
169-
║ │ Python │ │ Rust App │ │ HTTP │ │ Embedded │ ║
170-
║ │ Scripts │ │ Binary │ │ Clients │ │ Devices │ ║
171-
║ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ ║
172-
╚═════════╪══════════════════╪══════════════════════╪══════════════════╪═══════════╝
173-
│ │ │ │
174-
▼ ▼ ▼ ▼
175-
┌─────────────────────────────────────────────────────────────────────────────────┐
176-
│ INTERFACE LAYER (std) │
177-
│ ┌──────────────────────┐ ┌────────────────────────────┐ │
178-
│ │ Python FFI (pyo3) │ │ HTTP Server (axum) │ │
179-
│ │ ┌────────────────┐ │ │ ┌──────────────────────┐ │ │
180-
│ │ │ EmbeddedKernel │ │ │ │ REST API Handlers │ │ │
181-
│ │ │ (in-process) │ │ │ │ /v1/memory/* │ │ │
182-
│ │ └────────┬───────┘ │ │ └──────────┬───────────┘ │ │
183-
│ └───────────┼──────────┘ └─────────────┼──────────────┘ │
184-
│ │ │ │
185-
│ └────────────────┬───────────────────────┘ │
186-
└───────────────────────────────┼──────────────────────────────────────────────────┘
187-
│
188-
▼
189-
┌─────────────────────────────────────────────────────────────────────────────────┐
190-
│ DURABILITY LAYER (std) │
191-
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
192-
│ │ Engine Coordinator │ │
193-
│ │ ┌────────────────┐ ┌────────────────┐ ┌────────────────────────────┐ │ │
194-
│ │ │ WAL Writer │ │ WAL Reader │ │ Snapshot Manager │ │ │
195-
│ │ │ │ │ │ │ │ │ │
196-
│ │ │ • Serialize │ │ • Deserialize │ │ • encode_state() │ │ │
197-
│ │ │ • fsync() │ │ • Iterator │ │ • decode_state() │ │ │
198-
│ │ │ • Length Frame │ │ • replay_wal() │ │ • BLAKE3 hash │ │ │
199-
│ │ └───────┬────────┘ └────────┬───────┘ └────────┬───────────────────┘ │ │
200-
│ └──────────┼──────────────────────┼──────────────────┼──────────────────────┘ │
201-
│ │ │ │ │
202-
│ ▼ ▼ ▼ │
203-
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
204-
│ │ Persistence │ │
205-
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
206-
│ │ │ commands.wal │ │state.snapshot│ │ Metadata │ │ │
207-
│ │ │ │ │ │ │ │ │ │
208-
│ │ │ [v:u8][l:u32]│──┐ │ [Header] │ │ Indexes, │ │ │
209-
│ │ │ [cmd_bytes] │ │ │ [Kernel] │ │ Configs │ │ │
210-
│ │ │ ... │ │ │ [Meta] │ │ │ │ │
211-
│ │ └──────────────┘ │ │ [Index] │ └──────────────┘ │ │
212-
│ │ │ └──────────────┘ │ │
213-
│ │ │ ▲ │ │
214-
│ │ │ │ │ │
215-
│ │ └────Replay──────┘ │ │
216-
│ │ (Crash Recovery) │ │
217-
│ └──────────────────────────────────────────────────────────────────────────┘ │
218-
└────────────────────────────────────┬────────────────────────────────────────────┘
219-
│
220-
▼
221-
┌─────────────────────────────────────────────────────────────────────────────────┐
222-
│ VALORI KERNEL (no_std, pure Rust) │
223-
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
224-
│ │ KernelState<R,D,N,E> │ │
225-
│ │ ┌────────────────────────────────────────────────────────────────────┐ │ │
226-
│ │ │ Deterministic State Machine │ │ │
227-
│ │ │ │ │ │
228-
│ │ │ • apply(Command) → Result<(), Error> │ │ │
229-
│ │ │ • All operations use Fixed-Point arithmetic (Q16.16) │ │ │
230-
│ │ │ • No floating point, no randomness, no timestamps │ │ │
231-
│ │ │ • Bit-identical across x86, ARM, WASM, RISC-V │ │ │
232-
│ │ └────────────────────────────────────────────────────────────────────┘ │ │
233-
│ │ │ │
234-
│ │ ┌────────────────┐ ┌────────────────┐ ┌────────────────────────────┐ │ │
235-
│ │ │ Vector Storage │ │ Knowledge Graph│ │ Fixed-Point Math │ │ │
236-
│ │ │ │ │ │ │ │ │ │
237-
│ │ │ RecordPool[R] │ │ NodePool[N] │ │ FxpScalar (i32) │ │ │
238-
│ │ │ FxpVector<D> │ │ EdgePool[E] │ │ FxpVector<D> │ │ │
239-
│ │ │ │ │ AdjacencyList │ │ │ │ │
240-
│ │ │ • insert() │ │ │ │ • add, sub, mul, div │ │ │
241-
│ │ │ • delete() │ │ • create_node()│ │ • l2_distance() │ │ │
242-
│ │ │ • get() │ │ • create_edge()│ │ • normalize() │ │ │
243-
│ │ └────────────────┘ └────────────────┘ └────────────────────────────┘ │ │
244-
│ │ │ │
245-
│ │ ┌────────────────────────────────────────────────────────────────────┐ │ │
246-
│ │ │ Cryptographic Verification │ │ │
247-
│ │ │ • kernel_state_hash() → [u8; 32] (BLAKE3) │ │ │
248-
│ │ │ • Deterministic snapshot encoding │ │ │
249-
│ │ └────────────────────────────────────────────────────────────────────┘ │ │
250-
│ └──────────────────────────────────────────────────────────────────────────┘ │
251-
└─────────────────────────────────────────────────────────────────────────────────┘
252-
253-
DATA FLOW: CRASH RECOVERY
254-
255-
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
256-
│ Snapshot │────────▶│ WAL Reader │────────▶│ Kernel │
257-
│ (State S₀) │ Load │ (Commands) │ Replay │ (State Sₙ) │
258-
└──────────────┘ └──────────────┘ └──────────────┘
259-
│ │ │
260-
│ │ │
261-
└─────────────────────────┴────────────────────────┘
262-
│
263-
verify_hash()
264-
│
265-
▼
266-
✅ Bit-Identical State
267-
(Guaranteed Determinism)
255+
256+
### 🔄 Crash Recovery Flow
257+
258+
```mermaid
259+
sequenceDiagram
260+
participant S as Snapshot File
261+
participant W as WAL File
262+
participant R as WAL Reader
263+
participant K as Kernel
264+
participant V as Verifier
265+
266+
Note over S,V: System Restart After Crash
267+
268+
S->>K: 1. Load snapshot (State S₀)
269+
activate K
270+
Note over K: Kernel at snapshot state
271+
272+
W->>R: 2. Read WAL commands
273+
activate R
274+
275+
loop For each command
276+
R->>K: 3. Replay command
277+
Note over K: Apply deterministically
278+
end
279+
deactivate R
280+
281+
K->>K: 4. Compute state hash
282+
K->>V: 5. Verify hash
283+
activate V
284+
285+
alt Hash matches expected
286+
V-->>K: ✅ Recovery successful
287+
Note over K: State Sₙ (bit-identical)
288+
else Hash mismatch
289+
V-->>K: ❌ Recovery failed
290+
Note over K: Corruption detected
291+
end
292+
deactivate V
293+
deactivate K
268294
```
269295

270-
**Key Properties**:
296+
### 🎯 Key Properties
297+
298+
| Layer | Characteristics | Guarantees |
299+
|-------|----------------|------------|
300+
| **Kernel** | `no_std`, pure functions, Q16.16 fixed-point | Bit-identical across x86/ARM/WASM |
301+
| **Durability** | WAL + Snapshots, bincode serialization | Crash recovery, deterministic replay |
302+
| **Interface** | HTTP (axum) or FFI (pyo3) | Flexible deployment, same kernel |
303+
| **Storage** | Length-prefixed WAL, structured snapshots | Durability, atomicity |
271304

272-
- **Separation of Concerns**: Core kernel stays pure (no I/O), durability wrapped outside
273-
- **Deterministic Core**: Fixed-point math ensures bit-identical results across architectures
274-
- **Crash Recovery**: Snapshot + WAL replay = mathematically proven state restoration
275-
- **Multi-Deployment**: Same kernel runs embedded (no_std) or as HTTP service (std)
276-
- **Verifiable**: Cryptographic hashes prove state integrity
305+
**Separation of Concerns**: Core kernel stays pure (no I/O) → Durability wrapped outside → Flexible interfaces
277306

278307
See [Architecture Details](architecture.md) for deep dive.
279308

0 commit comments

Comments
 (0)