Skip to content

Commit e408a54

Browse files
varshith-Gitvarshith-Git
committed
added essence
1 parent 1f36924 commit e408a54

1 file changed

Lines changed: 304 additions & 0 deletions

File tree

‎essence.rst‎

Lines changed: 304 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,304 @@
1+
Valori Kernel - Codebase Essence
2+
================================
3+
4+
This document provides a comprehensive, deep-dive analysis of the entire Valori codebase structure.
5+
It maps every file and folder (excluding build artifacts and hidden files) to its specific purpose in the system.
6+
7+
.. note::
8+
**Legend**:
9+
- **[DIR]**: Directory
10+
- **[FILE]**: Source file or Configuration
11+
- **[TEST]**: Test suite or verification script
12+
13+
Root Directory
14+
--------------
15+
High-level project configuration and documentation.
16+
17+
- ``.`` (**Root**)
18+
- ``Cargo.toml``: **[FILE]** Workspace definition. Manages shared dependencies and members (kernel, node, ffi, etc.).
19+
- ``README.md``: **[FILE]** Primary entry point. Contains architecture overview, quickstart guides, and project status.
20+
- ``architecture.md``: **[FILE]** Detailed architectural blueprint (Layered design, Determinism specs).
21+
- ``all-functions.md``: **[FILE]** Auto-generated reference of all kernel functions.
22+
- ``build.log``: **[FILE]** Log file capturing build outputs (useful for debugging CI/Build failures).
23+
- ``Dockerfile``: **[FILE]** Docker container definition for packaging the Valori Node server.
24+
- ``LICENSE``: **[FILE]** AGPLv3 License text.
25+
- ``COMMERCIAL_LICENSE.md``: **[FILE]** Commercial licensing terms.
26+
- ``test_replication_e2e.py``: **[TEST]** Top-level end-to-end Python script verifying Leader-Follower replication.
27+
- ``valori_ffi.pyd``: **[FILE]** Compiled Python extension module (shared object) for direct FFI usage.
28+
29+
crates/ (Core Libraries)
30+
------------------------
31+
The workspace holding the fundamental Rust libraries.
32+
33+
- ``crates/`` (**Workspace**)
34+
- ``kernel/``: **[DIR]** **THE HEART**. The pure, `no_std`, deterministic core.
35+
- ``Cargo.toml``: Dependencies for the kernel (minimal).
36+
- ``src/``:
37+
- ``lib.rs``: Kernel library entry point. Re-exports core modules.
38+
- ``error.rs``: Defines ``KernelError`` types (CapacityExceeded, etc.).
39+
- ``event.rs``: Defines ``KernelEvent`` enum for Event Sourcing (InsertRecord, CreateNode, etc.).
40+
- ``proof.rs``: Cryptographic proof generation logic (State Hashing).
41+
- ``verify.rs``: logic to verify proof signatures/hashes.
42+
- ``replay.rs``: Traits/Logic for replaying commands from logs.
43+
- ``replay_events.rs``: Logic for replaying *Events* (Phase 23+).
44+
- ``config.rs``: Kernel-level configuration constants/structs.
45+
- ``fxp/``: **[DIR]** Fixed-Point Arithmetic (Q16.16)
46+
- ``mod.rs``: Module definition.
47+
- ``qformat.rs``: The ``FxpScalar`` type definition (i32 wrapper).
48+
- ``ops.rs``: Deterministic implementation of Add, Sub, Mul, Div, Sqrt.
49+
- ``graph/``: **[DIR]** Knowledge Graph Components
50+
- ``mod.rs``: Module exposure.
51+
- ``node.rs``: ``Node`` struct definition (ID, Kind, pointers).
52+
- ``edge.rs``: ``Edge`` struct definition (From, To, Kind).
53+
- ``pool.rs``: Static memory pool implementation for Nodes/Edges.
54+
- ``adjacency.rs``: Adjacency list logic for graph traversal.
55+
- ``index/``: **[DIR]** Vector Indexing
56+
- ``mod.rs``: Indexing traits.
57+
- ``brute_force.rs``: Baseline sequential scan search (Deterministic reference).
58+
- ``math/``: **[DIR]** Vector Math
59+
- ``mod.rs``: Math traits.
60+
- ``l2.rs``: Euclidean distance calculation (using FXP).
61+
- ``dot.rs``: Dot product calculation.
62+
- ``quant/``: **[DIR]** Quantization (Future)
63+
- ``mod.rs``: Traits for vector quantization.
64+
- ``snapshot/``: **[DIR]** State Serialization
65+
- ``mod.rs``: Snapshot traits.
66+
- ``encode.rs``: Deterministic serialization of KernelState.
67+
- ``decode.rs``: Deserialization logic.
68+
- ``hash.rs``: Computing the cryptographic hash of the state.
69+
- ``blake3.rs``: Wrapper around BLAKE3 hasher.
70+
- ``state/``: **[DIR]** State Machine
71+
- ``mod.rs``: State module.
72+
- ``kernel.rs``: **MAIN STRUCT** ``KernelState``. Holds storage, graph, and index.
73+
- ``command.rs``: ``Command`` enum for Legacy WAL operations.
74+
- ``storage/``: **[DIR]** Vector Storage
75+
- ``mod.rs``: Storage traits.
76+
- ``record.rs``: ``Record`` struct (ID + Vector data).
77+
- ``pool.rs``: Static memory pool for Records.
78+
- ``types/``: **[DIR]** Core Types
79+
- ``mod.rs``: Type exports.
80+
- ``vector.rs``: Generic ``FxpVector`` struct.
81+
- ``scalar.rs``: Scalar type aliases.
82+
- ``id.rs``: Strongly typed IDs (RecordId, NodeId, EdgeId).
83+
- ``enums.rs``: Enumerations (NodeKind, EdgeKind).
84+
- ``tests/``: **[DIR]** **[TEST]** Kernel Unit Tests
85+
- ``mod.rs``: Test module setup.
86+
- ``determinism_tests.rs``: Tests verifying cross-arch output stability.
87+
- ``fxp_tests.rs``: Tests for fixed-point math precision/overflow.
88+
- ``graph_tests.rs``: Tests for graph operations (add node/edge).
89+
- ``index_tests.rs``: Tests for search correctness.
90+
- ``math_tests.rs``: Tests for distance functions.
91+
- ``proof_tests.rs``: Tests for state hashing.
92+
- ``quant_tests.rs``: Tests for quantization.
93+
- ``snapshot_tests.rs``: Tests for save/restore roundtrips.
94+
- ``state_tests.rs``: Tests for basic kernel state operations.
95+
- ``storage_tests.rs``: Tests for record pool management.
96+
- ``e2e_tests.rs``: Internal end-to-end flows.
97+
98+
- ``persistence/``: **[DIR]** Persistence Abstractions
99+
- ``Cargo.toml``: Dependencies.
100+
- ``src/``:
101+
- ``lib.rs``: Library entry.
102+
- ``error.rs``: Persistence-specific errors.
103+
- ``fixtures.rs``: Helpers for creating test data (WALs, Snapshots).
104+
- ``idx.rs``: Logic for handling index files.
105+
- ``snapshot.rs``: Traits for reading/writing snapshots.
106+
- ``wal.rs``: Traits for Write-Ahead Log interaction.
107+
108+
- ``cli/``: **[DIR]** Command Line Interface
109+
- ``Cargo.toml``: Dependencies (Clap, etc.).
110+
- ``tests/``: Integration tests for CLI arguments.
111+
- ``src/``:
112+
- ``main.rs``: CLI entry point.
113+
- ``commands/``: (If present) Subcommand implementations.
114+
115+
- ``demo-generator/``: **[DIR]** Data Generator
116+
- ``Cargo.toml``: Dependencies.
117+
- ``src/``:
118+
- ``main.rs``: Executable to generate synthetic vector/graph data for demos.
119+
120+
node/ (Server Implementation)
121+
-----------------------------
122+
The production-grade HTTP server and application logic wrapping the kernel.
123+
124+
- ``node/`` (**Server**)
125+
- ``Cargo.toml``: Server dependencies (Axum, Tokio, Serde, Tracing).
126+
- ``build_err.txt``: Capture of build errors (ephemeral).
127+
- ``src/``:
128+
- ``main.rs``: **ENTRY POINT**. Initializes logging, config, engine, and HTTP server.
129+
- ``lib.rs``: Exposes modules for integration testing.
130+
- ``server.rs``: **API LAYER**. Defines routes (POST /records, etc.) and handlers.
131+
- ``api.rs``: **DTOs**. Request/Response structs (Serde derived) including Batch types.
132+
- ``engine.rs``: **COORDINATOR**. Wraps ``KernelState``, manages WAL/EventLog I/O, locks, and high-level logic.
133+
- ``config.rs``: ``NodeConfig`` struct parsing (env vars / CLI args).
134+
- ``errors.rs``: ``EngineError`` enum and Actix/Axum error mapping.
135+
- ``telemetry.rs``: Prometheus metrics setup and Tracing subscriber init.
136+
- ``metadata.rs``: Key-Value store for auxiliary metadata (not in core kernel).
137+
- ``persistence.rs``: Integration with ``crates/persistence``.
138+
- ``recovery.rs``: Logic to replay WAL/Snapshots on startup.
139+
- ``replication.rs``: **REPLICATION**. Logic for Leader/Follower states and stream consumption.
140+
- ``wal_writer.rs``: Appends commands to disk with ``fsync``.
141+
- ``wal_reader.rs``: Iterates over WAL files.
142+
- ``network/``: **[DIR]** Internal Cluster Networking
143+
- ``mod.rs``: Module definitions.
144+
- ``client.rs``: (Likely) Internal client for node-to-node communication.
145+
- ``events/``: **[DIR]** Event Sourcing Implementation (Phase 23+)
146+
- ``mod.rs``: Module exports.
147+
- ``event_log.rs``: Handles I/O for the append-only Event Log. Includes ``append_batch``.
148+
- ``event_commit.rs``: **CRITICAL**. Implements the Atomic Commit Pipeline (Shadow -> Persist -> Apply).
149+
- ``event_replay.rs``: Logic to rebuild state from Event Log.
150+
- ``structure/``: **[DIR]** HNSW Indexing (Host-side)
151+
- ``mod.rs``: Module exports.
152+
- ``hnsw.rs``: HNSW Implementation (approximate search index managed by Node, not Kernel).
153+
- ``tests/``: **[DIR]** **[TEST]** Integration Test Suite
154+
- ``integration_tests.rs``: Basic HTTP API tests.
155+
- ``api_batch_ingest.rs``: **[TEST]** Verifies atomic batch ingestion and rollback.
156+
- ``api_replication.rs``: **[TEST]** Verifies Leader-Follower replication sync.
157+
- ``replication_bootstrap.rs``: Tests initial snapshot bootstrap for followers.
158+
- ``replication_cluster.rs``: Tests multi-node cluster scenarios.
159+
- ``replication_divergence.rs``: Tests detection of state divergence.
160+
- ``fuzz_crash_recovery.rs``: **[TEST]** Fuzz testing for crash consistency.
161+
- ``persistence_tests.rs``: Verifies WAL writing/reading.
162+
- ``persistence_index_tests.rs``: Verifies persistence of auxiliary indices.
163+
- ``deterministic_ivf_tests.rs``: Tests for IVF index determinism.
164+
- ``deterministic_pq_tests.rs``: Tests for PQ index determinism.
165+
- ``deterministic_kmeans_tests.rs``: Tests for K-Means clustering.
166+
- ``deterministic_edge_tests.rs``: Tests for graph edge cases.
167+
- ``hnsw_tests.rs``: Tests for HNSW index recall/precision.
168+
- ``multi_arch_determinism.rs``: **[TEST]** Critical test for cross-arch output stability.
169+
- ``fuzz/``: Directory for fuzz targets.
170+
171+
python/ (Client SDK)
172+
--------------------
173+
The usage-facing Python library.
174+
175+
- ``python/`` (**SDK**)
176+
- ``setup.py``: Build script using ``setuptools-rust`` to compile the FFI extension.
177+
- ``test_valori_integrated.py``: **[TEST]** Integration test validating the installed package works.
178+
- ``test_memory.py``: Tests for memory-related features.
179+
- ``test_protocol.py``: Tests for the wire protocol.
180+
- ``test_protocol_remote.py``: Tests specifically for remote protocol.
181+
- ``test_unified.py``: Unified test suite.
182+
- ``valori/``: **[DIR]** The Python Package
183+
- ``__init__.py``: Package root. Exports ``Valori`` factory.
184+
- ``local.py``: ``LocalClient`` implementation wrapping the FFI.
185+
- ``remote.py``: ``RemoteClient`` implementation using HTTP (requests).
186+
- ``protocol.py``: Definitions of API protocol schemas.
187+
- ``memory.py``: High-level Memory API abstraction.
188+
- ``chunking.py``: Text chunking utilities.
189+
- ``ingest.py``: data ingestion helpers.
190+
- ``kinds.py``: Enum definitions for Python.
191+
- ``adapters/``: **[DIR]** Framework Integrations
192+
- ``__init__.py``: Exports.
193+
- ``base.py``: Base adapter classes.
194+
- ``langchain.py``: LangChain integration.
195+
- ``langchain_vectorstore.py``: VectorStore implementation for LangChain.
196+
- ``llamaindex.py``: LlamaIndex integration.
197+
- ``sentence_transformers_adapter.py``: Helper for using SentenceTransformers.
198+
- ``utils.py``: Utility functions.
199+
- ``tests/``: **[DIR]** **[TEST]** Unit Tests
200+
- ``test_adapters.py``: Verifies adapters work.
201+
- ``test_protocol_errors.py``: Verifies error handling.
202+
- ``examples/``: **[DIR]** Usage Examples
203+
- ``demo_embeddings.py``: Script showing embedding generation.
204+
- ``demo_remote.py``: demo of connecting to a remote node.
205+
- ``demo_sentence_transformers.py``: Complete flow with ST.
206+
207+
ffi/ (Python Bindings)
208+
----------------------
209+
The bridge between Rust and Python.
210+
211+
- ``ffi/`` (**FFI**)
212+
- ``Cargo.toml``: Dependencies (PyO3, valori-kernel, valori-node).
213+
- ``src/``:
214+
- ``lib.rs``: **[FILE]** The PyO3 binding logic. Exposes ``ValoriEngine`` class to Python.
215+
- ``test_valori.py``: Quick verify script for the built shared object.
216+
217+
embedded/ (Bare Metal)
218+
----------------------
219+
Experiments and implementations for embedded targets.
220+
221+
- ``embedded/`` (**Embedded**)
222+
- ``Cargo.toml``: Embedded dependencies.
223+
- ``src/``:
224+
- ``main.rs``: Entry point for embedded demo.
225+
- ``flash.rs``: Flash memory storage simulation/driver.
226+
- ``checkpoint.rs``: Checkpointing logic for embedded.
227+
- ``recovery.rs``: Recovery logic tailored for low-resource environments.
228+
- ``proof.rs``: Proof generation for constrained devices.
229+
- ``shadow.rs``: Shadow execution (dual-run) logic.
230+
- ``snapshot.rs``: Snapshot handling.
231+
- ``transport.rs``: Communication logic (Serial/UART etc.).
232+
- ``wal.rs``: Wal implementation for flash.
233+
- ``wal_stream.rs``: Streaming WAL logic.
234+
235+
docs/ (Documentation)
236+
---------------------
237+
Project documentation and specifications.
238+
239+
- ``docs/``
240+
- ``api-reference.md``: HTTP API Endpoint details.
241+
- ``python-usage-guide.md``: Detailed Python SDK guide.
242+
- ``core-concepts.md``: Explanations of Fixed-Point, Determinism, Kernel.
243+
- ``getting-started.md``: General onboarding.
244+
- ``embedded-quickstart.md``: Specifics for ARM/Embedded usage.
245+
- ``remote-mode.md``: Guide for running Server/Clustered mode.
246+
- ``architecture.md``: (See root architecture.md).
247+
- ``determinism-guarantees.md``: Formal contract of what is guaranteed.
248+
- ``multi-arch-determinism.md``: Proofs and CI details for cross-arch support.
249+
- ``verifiable-replication.md``: Explain how replication is verified.
250+
- ``wal-replay-guarantees.md``: Crash recovery specifics.
251+
- ``memory_protocol_v0.md``: Legacy protocol specs.
252+
- ``memory_protocol_v1.md``: Current protocol specs.
253+
- ``authentication.md``: Auth specs.
254+
- ``adapter-improvements.md``: Notes on framework adapters.
255+
- ``publishing-pypi.md``: Guide for releasing the Python package.
256+
- ``functions.md``: Likely function reference.
257+
- ``python-reference.md``: Python API reference.
258+
259+
demo/ & examples/
260+
-----------------
261+
Scripts and resources for demonstration.
262+
263+
- ``demo/``
264+
- ``demo_adapters.py``: Demo of adapter usage.
265+
- ``demo_run.py``: Generic demo runner.
266+
- ``e2e_lifecycle.py``: **[TEST]** Lifecycle verification.
267+
- ``simple_remote.py``: Basic connection script.
268+
- ``examples/`` (Root)
269+
- ``langchain_example.py``: Example usage with LangChain.
270+
- ``llamaindex_example.py``: Example usage with LlamaIndex.
271+
272+
verify/ (Tools)
273+
---------------
274+
Standalone verification utilities.
275+
276+
- ``verify/``
277+
- ``Cargo.toml``
278+
- ``src/``
279+
- ``main.rs``: CLI tool to verify database integrity offline.
280+
281+
src/ (Root Source?)
282+
-------------------
283+
*Note: This appears to be a separate crate source directory at the root level, likely for the `src` member defined in root `Cargo.toml` if applicable. It mirrors `crates/kernel/src` structure. It serves as the primary compiled source for the `valori` crate if defined in root.*
284+
285+
- ``src/``
286+
- ``lib.rs``: Entry point.
287+
- ``config.rs``: Configuration.
288+
- ``error.rs``: Error definitions.
289+
- ``event.rs``: Event definitions.
290+
- ``proof.rs``: Proof logic.
291+
- ``verify.rs``: Verify logic.
292+
- ``replay.rs``, ``replay_events.rs``: Replay logic.
293+
- ``fxp/``, ``graph/``, ``index/``, ``math/``, ``quant/``, ``snapshot/``, ``state/``, ``storage/``, ``types/``: **[DIR]** Mirrors of Kernel modules.
294+
- ``tests/``: **[DIR]** Unit tests.
295+
296+
demo_db/
297+
--------
298+
Runtime data directory (typically in `.gitignore`, but present in listing).
299+
300+
- ``demo_db/``
301+
- ``events.log``: The Event Log file.
302+
- ``metadata.idx``: Metadata index file.
303+
- ``snapshot.val``: Snapshot file.
304+

0 commit comments

Comments
 (0)