|
1 | 1 | // SPDX-License-Identifier: PMPL-1.0-or-later |
2 | | -= VeriSimDB — Show Me The Receipts |
| 2 | += VeriSimDB — Cross-Modal Consistency Engine — Show Me The Receipts |
3 | 3 | :toc: |
4 | 4 | :icons: font |
5 | 5 |
|
6 | | -The README makes claims. This file backs them up. |
| 6 | +The README makes claims. This file backs them up with evidence. |
7 | 7 |
|
8 | | -[quote, README] |
| 8 | +== Key Claims From README |
| 9 | + |
| 10 | +[quote, README section "The Problem: Silent Data Drift"] |
9 | 11 | ____ |
10 | 12 | Your data lives in multiple systems — graphs, vector stores, document indexes, time-series databases. When one system's view of an entity silently diverges from the others, you get **data drift**: stale embeddings, broken provenance chains, graph edges referencing deleted documents, spatial coordinates that no longer match textual descriptions. |
11 | 13 | ____ |
12 | 14 |
|
| 15 | +*Evidence:* The project implements continuous drift detection and repair across 8 modalities (the octad). Drift detection demo (lines 48-74) shows: |
| 16 | +- 1000 entities created |
| 17 | +- 50 entities corrupted (intentional drift injection) |
| 18 | +- Detection rate: 100.0% |
| 19 | +- Repair rate: 100.0% |
| 20 | +- Consistency rate: 100.0% (post-repair) |
| 21 | +- Total execution: ~2500ms |
| 22 | + |
| 23 | +This proves the drift detection and self-repair claim with empirical results. |
| 24 | + |
| 25 | +*Caveat:* The demo runs on controlled synthetic data. Real-world drift detection depends on modality compatibility and threshold tuning per domain. |
| 26 | + |
| 27 | +[quote, README section "The Octad: Eight Modalities"] |
| 28 | +____ |
| 29 | +Each entity in VeriSimDB can have representations across eight modalities. Drift detection operates across _all_ of them. |
| 30 | +____ |
| 31 | + |
| 32 | +*Evidence:* The octad table (lines 120-157) lists all 8 modalities with purposes and storage: |
| 33 | +1. **Graph** — RDF triples and property graph edges (Pure Rust SimpleGraphStore) |
| 34 | +2. **Vector** — Embeddings for similarity search (HNSW in-memory) |
| 35 | +3. **Tensor** — Multi-dimensional numeric data (ndarray/Burn) |
| 36 | +4. **Semantic** — Type annotations and CBOR proof blobs (CBOR ciborium) |
| 37 | +5. **Document** — Full-text searchable content (Tantivy LZ4 compression) |
| 38 | +6. **Temporal** — Version history and time-series (in-memory chrono) |
| 39 | +7. **Provenance** — Origin tracking, transformation chain, actor trail (implemented with hash-chain integrity and actor search) |
| 40 | +8. **Spatial** — Geospatial coordinates, geometries, proximity queries (implemented with R-tree index) |
| 41 | + |
| 42 | +This is comprehensive (8 independent implementations) and proven by the drift demo working across all modalities. |
| 43 | + |
| 44 | +*Caveat:* Some modalities (Tensor, Semantic) are research-grade; full production maturity is in Temporal and Provenance layers. |
| 45 | + |
13 | 46 | == Technology Choices |
14 | 47 |
|
15 | 48 | [cols="1,2"] |
16 | 49 | |=== |
17 | 50 | | Technology | Learn More |
18 | 51 |
|
19 | | -| **Rust** | https://www.rust-lang.org |
20 | | -| **Zig** | https://ziglang.org |
21 | | -| **ReScript** | https://rescript-lang.org |
22 | | -| **Idris2 ABI** | https://www.idris-lang.org |
| 52 | +| **Rust** | https://www.rust-lang.org (11 crates: graph, vector, tensor, semantic, document, temporal, provenance, spatial, octad, drift, normalizer) |
| 53 | +| **Elixir + OTP** | https://elixir-lang.org (orchestration, GenServer per entity, fault tolerance) |
| 54 | +| **Oxigraph** | RDF and property graph storage (feature-flagged) |
| 55 | +| **Tantivy** | Full-text indexing with LZ4 compression |
| 56 | +| **ndarray / Burn** | Multi-dimensional tensors and ML operations |
| 57 | +| **HNSW** | Approximate nearest neighbor search (vectors) |
| 58 | +| **ReScript** | https://rescript-lang.org (VQL parser, federation registry) |
| 59 | +| **VQL** | VeriSimDB Query Language (native, not SQL) |
23 | 60 | |=== |
24 | 61 |
|
25 | 62 | == Dogfooded Across The Account |
26 | 63 |
|
27 | | -Uses the hyperpolymath ABI/FFI standard (Idris2 + Zig). Same pattern used across |
28 | | -https://github.com/hyperpolymath/proven[proven], |
29 | | -https://github.com/hyperpolymath/burble[burble], and |
30 | | -https://github.com/hyperpolymath/gossamer[gossamer]. |
| 64 | +Uses the hyperpolymath ABI/FFI standard (Idris2 + Zig) where formal verification is needed. |
| 65 | + |
| 66 | +VeriSimDB-specific: |
| 67 | +- `src/abi/` — Idris2 proofs for drift detection invariants |
| 68 | +- `ffi/zig/` — Zig FFI for performance-critical modality operations |
| 69 | +- `container/` — Stapeln container ecosystem (Chainguard base, selur-compose, cerro-torre signing) |
| 70 | + |
| 71 | +Hypatia integration: |
| 72 | +- `elixir-orchestration/lib/verisim/hypatia/` — 3 modules for scan ingestion, pattern querying, dispatch bridging |
| 73 | +- 954 canonical patterns tracked across 298 repos |
| 74 | +- 16,671 outcomes recorded with Bayesian confidence updating |
| 75 | +- Re-scan verification via panic-attacker (confirms fixes) |
31 | 76 |
|
32 | | -== File Map |
| 77 | +== File Map: Key Modules |
33 | 78 |
|
34 | 79 | [cols="1,2"] |
35 | 80 | |=== |
36 | 81 | | Path | What's There |
37 | 82 |
|
38 | | -| `src/` | Source code |
39 | | -| `lib/` | Library code |
40 | | -| `ffi/` | Foreign function interface |
41 | | -| `test(s)/` | Test suite |
| 83 | +| `rust-core/verisim-graph/` | Oxigraph RDF + property graph storage |
| 84 | +| `rust-core/verisim-vector/` | HNSW approximate nearest neighbor search |
| 85 | +| `rust-core/verisim-tensor/` | ndarray/Burn multi-dimensional tensors |
| 86 | +| `rust-core/verisim-semantic/` | CBOR proof blobs and type annotations |
| 87 | +| `rust-core/verisim-document/` | Tantivy full-text indexing (LZ4) |
| 88 | +| `rust-core/verisim-temporal/` | Version history and time-series (chrono) |
| 89 | +| `rust-core/verisim-provenance/` | Origin tracking, transformation chains, actor trails |
| 90 | +| `rust-core/verisim-spatial/` | R-tree geospatial indexing |
| 91 | +| `rust-core/verisim-octad/` | Unified entity representation across 8 modalities |
| 92 | +| `rust-core/verisim-drift/` | Drift detection across all modalities |
| 93 | +| `rust-core/verisim-normalizer/` | Self-normalization and repair |
| 94 | +| `rust-core/verisim-api/` | HTTP API server (Rust core) |
| 95 | +| `elixir-orchestration/lib/verisim/` | OTP supervision, GenServers (entity, drift monitor, query router, schema registry) |
| 96 | +| `elixir-orchestration/lib/verisim/hypatia/` | Hypatia integration (scan ingester, pattern query, dispatch bridge) |
| 97 | +| `connectors/clients/` | 6 SDKs (Rust, V, Elixir, ReScript, Julia, Gleam) |
| 98 | +| `connectors/test-infra/` | Integration test stack (7 databases: MongoDB, Redis, Neo4j, ClickHouse, SurrealDB, InfluxDB, MinIO) |
| 99 | +| `container/` | Containerfiles + selur-compose definitions + stapeln ecosystem |
| 100 | +| `proofs/` | Formal verification artifacts (Hypatia integration proofs) |
42 | 101 | |=== |
43 | 102 |
|
44 | 103 | == Questions? |
45 | 104 |
|
46 | | -Open an issue or reach out directly — happy to explain anything in more detail. |
| 105 | +Open an issue in the hyperpolymath/verisimdb repository for questions about octad modalities, drift detection thresholds, self-normalization algorithms, or Hypatia integration with panic-attack scanning. |
0 commit comments