Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ All notable changes to VeriSimDB are documented here. This project uses https://

== [Unreleased]

=== Changed — Docs hygiene: retire fossil specs, fix six→eight modality drift (2026-07-07)

- Stale-stamped three fossil documents with prominent superseded/historical banners: `docs/status/implementation-plan.adoc` (Week 1–16 / 10%-completion plan), `docs/releases/v0.1.0-alpha.md` ("Production-Ready" / "100% Complete" claims contradicted by KNOWN-ISSUES #29/#30/#31), and `spec/system-specs.md` (pre-octad template spec with a Textual/Numeric/Categorical modality set and `verisimdb-core`/`-nif` Rustler crate layout matching neither the octad nor the real `verisim-*` crates).
- Corrected modality-count drift ("six"/"6" → eight/octad) in current docs: `docs/deployment-modes.adoc`, `docs/federation-readiness.adoc`, `docs/challenges-standalone.adoc`, `docs/challenges-federated.adoc`, `docs/challenges-hybrid.adoc`, `docs/reversibility-design.adoc`, `docs/vcl-vs-sql.adoc`, `docs/deployment/deployment.adoc`, and `ROADMAP.md`. Where a doc enumerated only the legacy six concrete stores, added the missing Provenance (hash-chain lineage) and Spatial (R-tree) entries.
- Left deliberate references untouched: HEXAD grammar keyword, VCL "six proof types" / VCL-dt++ "six clauses", and dated historical records in CHANGELOG/KNOWN-ISSUES/design docs/papers. The "is VeriSimDB a database?" positioning prose is unchanged pending owner decision D1.

== [0.2.0] — 2026-06-13

=== Added — July-1 credibility: honesty over fake success (2026-06-13)
Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Currently in-memory only. Need pluggable backends.
### 2.4 Snapshots & Backup
**Criticality: HIGH** | Effort: Medium

- Point-in-time snapshots (consistent across all 6 modalities)
- Point-in-time snapshots (consistent across all 8 modalities)
- Incremental backup (WAL-based)
- Restore from snapshot + WAL replay
- Export/import in portable format
Expand Down
4 changes: 2 additions & 2 deletions docs/challenges-federated.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,7 @@ With N modalities: latency scales with max(Store1, Store2, ..., StoreN)

**Amplification Factors:**
- **2 modalities:** 2x network calls
- **6 modalities:** 6x network calls (if not optimized)
- **8 modalities:** 8x network calls (if not optimized)
- **Failure Probability:** P(any store fails) = 1 - ∏(1 - P(store_i fails))

**Example:**
Expand Down Expand Up @@ -628,7 +628,7 @@ Monthly bandwidth:
|Charge $0.01/query (cost recovery)

|**Modality Support**
|All 6 modalities
|All 8 modalities
|Graph and Document only (limited resources)

|**Schema Changes**
Expand Down
2 changes: 1 addition & 1 deletion docs/challenges-hybrid.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -717,7 +717,7 @@ Result: Query denied at Layer 2 despite passing Layer 1
**Problem:** Hybrid mode requires **more operational expertise** than standalone.

**Skills Required:**
- **Standalone:** Understand 6 modality stores + Elixir orchestrator (7 components)
- **Standalone:** Understand 8 modality stores + Elixir orchestrator (9 components)
- **Hybrid:** All above + Network troubleshooting + Remote store APIs + Policy coordination (10+ components)

**Hiring Impact:** May need to hire **DevOps specialist** for hybrid management (+$150k/year salary)
Expand Down
44 changes: 27 additions & 17 deletions docs/challenges-standalone.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

== Overview

Standalone deployment runs VeriSimDB as a **traditional database** with all six modalities (Graph, Vector, Tensor, Semantic, Document, Temporal) hosted locally on the same machine or cluster. While this offers simplicity and performance, it introduces specific architectural, operational, and scaling challenges.
Standalone deployment runs VeriSimDB as a **traditional database** with all eight modalities (the octad: Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, Spatial) hosted locally on the same machine or cluster. While this offers simplicity and performance, it introduces specific architectural, operational, and scaling challenges.

This document details the **technical challenges** unique to standalone deployment and provides mitigation strategies where applicable.

Expand Down Expand Up @@ -38,10 +38,10 @@ This document details the **technical challenges** unique to standalone deployme

=== 1.2. Single Point of Failure

**Problem:** Loss of the standalone node means loss of all six modalities simultaneously.
**Problem:** Loss of the standalone node means loss of all eight modalities simultaneously.

**Risk Scenarios:**
1. Hardware failure → No Graph, Vector, Tensor, Semantic, Document, OR Temporal access
1. Hardware failure → No Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, OR Spatial access
2. Kernel panic → All GenServers crash (Elixir orchestrator down)
3. Disk corruption → All modalities potentially compromised

Expand All @@ -54,7 +54,7 @@ This document details the **technical challenges** unique to standalone deployme

=== 1.3. Modality Coupling

**Problem:** All six modalities share the same **Elixir orchestration layer**, creating tight coupling.
**Problem:** All eight modalities share the same **Elixir orchestration layer**, creating tight coupling.

**Consequences:**
- **Upgrade Hell:** Updating `verisim-graph-rs` (Oxigraph) requires restarting the entire stack
Expand Down Expand Up @@ -93,7 +93,7 @@ Result: All three queries degrade due to resource contention

=== 2.2. Write Amplification

**Problem:** A single Octad update can trigger writes across **all six modalities**.
**Problem:** A single Octad update can trigger writes across **all eight modalities**.

**Example:**
1. User updates Octad `550e8400-...` with new document text
Expand All @@ -103,7 +103,7 @@ Result: All three queries degrade due to resource contention
5. **Semantic modality**: Validates CBOR proof (writes metadata)
6. **Graph modality**: May add citation edges (writes RDF triples)

**Write Amplification Factor:** ~6x per logical update
**Write Amplification Factor:** up to ~8x per logical update (worst case, all modalities)

**Consequences:**
- **SSD Wear:** Premature SSD exhaustion (TBW limits)
Expand Down Expand Up @@ -138,7 +138,7 @@ Result: All three queries degrade due to resource contention

=== 3.1. Backup and Recovery Complexity

**Problem:** Six modalities require **six separate backup strategies**.
**Problem:** Eight modalities require **eight separate backup strategies**.

[cols="1,2,2"]
|===
Expand Down Expand Up @@ -167,12 +167,20 @@ Result: All three queries degrade due to resource contention
|**Temporal**
|Merkle-tree nodes → recursive snapshot
|Tree integrity check expensive on large histories

|**Provenance**
|Hash-chain lineage → append-only log export
|Chain integrity must be re-verified on restore

|**Spatial**
|R-tree index → serialized dump
|Spatial index rebuilt from stored geometry
|===

**Operational Burden:**
- **Inconsistent Snapshots:** Modalities backed up at different times → cross-modal inconsistency
- **Restore Testing:** Must test restore for all 6 modalities (6x testing effort)
- **Storage Costs:** 6 separate backups consume significant space
- **Restore Testing:** Must test restore for all 8 modalities (8x testing effort)
- **Storage Costs:** 8 separate backups consume significant space

**Mitigation:**
- **Coordinated Snapshots:** Use `VeriSim.DriftMonitor` to trigger simultaneous backups
Expand All @@ -181,7 +189,7 @@ Result: All three queries degrade due to resource contention

=== 3.2. Monitoring and Observability

**Problem:** Need to monitor **six independent data stores** plus orchestration layer.
**Problem:** Need to monitor **eight independent data stores** plus orchestration layer.

**Metrics to Track (per modality):**
- Query latency (p50, p99, p999)
Expand All @@ -191,7 +199,7 @@ Result: All three queries degrade due to resource contention
- Cache hit rates (where applicable)

**Operational Complexity:**
- **7 dashboards** (6 modalities + orchestrator) → alert fatigue
- **9 dashboards** (8 modalities + orchestrator) → alert fatigue
- **Correlation Difficulty:** Determining which modality caused a slowdown requires cross-dashboard analysis
- **Logging Volume:** Combined logs from all modalities can reach GB/day

Expand Down Expand Up @@ -254,7 +262,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] }
- **Result:** Vector store saturated, Temporal store wasted capacity

**Consequences:**
- **Inefficient Resource Use:** Paying for 6 modalities, using 1-2 heavily
- **Inefficient Resource Use:** Paying for 8 modalities, using 1-2 heavily
- **Cannot Scale Hot Path:** Standalone mode cannot add more Vector replicas without duplicating all modalities

**Mitigation:**
Expand All @@ -268,7 +276,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] }

**Theoretical Limit:**
- Elixir GenServer: ~50k messages/sec per process (microbenchmark)
- Each write requires ~5 GenServer calls (validation, routing, 6 modality dispatches, logging)
- Each write requires ~5 GenServer calls (validation, routing, 8 modality dispatches, logging)
- **Practical Write Throughput:** ~10k writes/sec

**Symptom Under Load:**
Expand All @@ -285,7 +293,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] }

=== 5.1. Single Security Boundary

**Problem:** Compromise of standalone node exposes **all six modalities** simultaneously.
**Problem:** Compromise of standalone node exposes **all eight modalities** simultaneously.

**Attack Scenarios:**
1. **Container Escape:** Attacker gains root on host → accesses all modality stores
Expand Down Expand Up @@ -353,7 +361,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] }

=== 7.1. Over-Provisioning

**Problem:** Must provision hardware for **peak load across all six modalities**.
**Problem:** Must provision hardware for **peak load across all eight modalities**.

**Example:**
- Vector modality needs 64GB RAM for HNSW index
Expand All @@ -367,7 +375,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] }

=== 7.2. Maintenance Burden

**Problem:** Single admin team must understand **six different data stores**.
**Problem:** Single admin team must understand **eight different data stores**.

**Skill Requirements:**
- Oxigraph (RDF/SPARQL)
Expand All @@ -376,8 +384,10 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] }
- CBOR (semantic proofs, schema validation)
- Tantivy (full-text search, inverted indices)
- Merkle trees (temporal versioning, cryptographic proofs)
- Hash-chain lineage (provenance integrity)
- R-tree indexing (spatial/geospatial queries)

**Hiring Difficulty:** Finding engineers with all six skill sets is rare/expensive
**Hiring Difficulty:** Finding engineers with all eight skill sets is rare/expensive

**Mitigation:**
- **Specialization:** Split team into modality-specific sub-teams (requires larger org)
Expand Down
4 changes: 2 additions & 2 deletions docs/deployment-modes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ VeriSimDB is **both** a database and a federation coordinator—the architecture

=== Mode 1: Standalone Database (Traditional)

Deploy VeriSimDB as a **complete database system** with all six modalities local.
Deploy VeriSimDB as a **complete database system** with all eight modalities (the octad) local.

[cols="1,3"]
|===
Expand All @@ -18,7 +18,7 @@ Deploy VeriSimDB as a **complete database system** with all six modalities local
|Local in-memory namespace

|Modality Stores
|All 6 modalities running on the same machine/cluster
|All 8 modalities running on the same machine/cluster

|Use Case
|Single institution, traditional database replacement
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment/deployment.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ This guide covers all deployment modes with focus on production readiness, secur

=== Mode 1: Standalone Database

Single-node deployment with all 6 modalities local.
Single-node deployment with all 8 modalities local.

==== Use Cases

Expand Down
10 changes: 6 additions & 4 deletions docs/federation-readiness.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

== What Federation Means in VeriSimDB

Federation is the ability for multiple VeriSimDB instances to coordinate queries across their respective octad stores. Each instance maintains its own six modality stores (Graph, Vector, Tensor, Semantic, Document, Temporal) and its own set of octad entities. A federated query spans multiple instances, retrieving and correlating data from peers that may be geographically distributed, operated by different organizations, or running different versions of VeriSimDB.
Federation is the ability for multiple VeriSimDB instances to coordinate queries across their respective octad stores. Each instance maintains its own eight modality stores (the octad: Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, Spatial) and its own set of octad entities. A federated query spans multiple instances, retrieving and correlating data from peers that may be geographically distributed, operated by different organizations, or running different versions of VeriSimDB.

Federation in VeriSimDB is distinct from database replication. Replicated databases maintain identical copies of the same data. Federated VeriSimDB instances hold _different_ octad entities that may reference each other through graph edges, share semantic vocabularies, or have vector embeddings in the same latent space. The federation layer coordinates cross-instance queries while each instance retains sovereignty over its own data.

Expand Down Expand Up @@ -60,14 +60,16 @@ If VCL were removed or not used, VeriSimDB would lose the following federation-c

=== Normalizer Regeneration Strategies Are Stubs

The `verisim-normalizer` crate's regeneration strategies currently return hardcoded `[regenerated]` placeholder values. When drift is detected and normalization is triggered, the normalizer identifies the authoritative modality and calls the appropriate regeneration strategy, but that strategy does not actually regenerate the drifted modality's data. Each of the six modalities needs a real regeneration implementation:
The `verisim-normalizer` crate's regeneration strategies currently return hardcoded `[regenerated]` placeholder values. When drift is detected and normalization is triggered, the normalizer identifies the authoritative modality and calls the appropriate regeneration strategy, but that strategy does not actually regenerate the drifted modality's data. Each of the eight modalities needs a real regeneration implementation:

* **Graph**: Re-derive edges from document content and semantic annotations.
* **Vector**: Re-embed from document text using the configured embedding model.
* **Tensor**: Re-compute tensor representation from source data.
* **Semantic**: Re-extract type annotations and RDF triples from document and graph.
* **Document**: Re-generate searchable text from graph, semantic, and temporal data.
* **Temporal**: Re-build version chain from modification history.
* **Provenance**: Re-build the hash-linked provenance chain from recorded modification events.
* **Spatial**: Re-derive spatial geometry and coordinates from document and semantic references.

=== Federation Executor Always Returns Empty

Expand Down Expand Up @@ -109,7 +111,7 @@ Drift detection works: the `verisim-drift` crate computes drift scores across mo

|**Normalizer (regeneration)**
|Stub
|Returns hardcoded `[regenerated]`. Six modality-specific strategies needed.
|Returns hardcoded `[regenerated]`. Eight modality-specific strategies needed.

|**Consensus Protocol**
|Design only
Expand Down Expand Up @@ -152,7 +154,7 @@ Drift detection works: the `verisim-drift` crate computes drift scores across mo
* Add authentication and authorization for peer-to-peer communication (mTLS or signed requests).
* Implement query result merging in `FederationExecutor` with configurable merge strategies (union, intersection, ranked).
* Add timeout and fallback handling for unreachable peers.
* Implement remaining normalizer regeneration strategies for all six modalities.
* Implement remaining normalizer regeneration strategies for all eight modalities.
* Add federation-scoped drift detection: compute drift across instances, not just within one.

=== Phase 3: Verified Federation
Expand Down
2 changes: 2 additions & 0 deletions docs/releases/v0.1.0-alpha.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# VeriSimDB v0.1.0-alpha Release Notes

> ⚠️ **HISTORICAL RELEASE NOTE (2026-07-07)** — the "Production-Ready" and "100% Complete" claims in this document were inaccurate and are superseded by `KNOWN-ISSUES.adoc` (see issues #29/#30/#31). Retained for release history only.

**Release Date:** 2026-02-04

**Status:** Alpha Release - Production-Ready
Expand Down
2 changes: 1 addition & 1 deletion docs/reversibility-design.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ Each modality can be made reversible:

=== Level 3: Full Multi-Modal Reversibility (MODERATE COST)

**What:** All six modalities support revert.
**What:** All eight modalities support revert.

**Implementation:**

Expand Down
5 changes: 5 additions & 0 deletions docs/status/implementation-plan.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@

// SPDX-License-Identifier: CC-BY-SA-4.0

[WARNING]
====
*SUPERSEDED (2026-07-07)* — historical planning document. The described architecture (Week 1–16 plan, "10% completion", six modality stores) no longer reflects the repo. See `KNOWN-ISSUES.adoc` and `ROADMAP.md` for current truth.
====

[.lead]
**Bringing VeriSimDB from 10% to 70% completion**

Expand Down
6 changes: 3 additions & 3 deletions docs/vcl-vs-sql.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

VCL (VeriSim Consonance Language) is the query language for VeriSimDB, a 6-core multimodal database. While VCL borrows familiar keywords from SQL (`SELECT`, `FROM`, `WHERE`, `LIMIT`, `OFFSET`), it serves a fundamentally different purpose.

VCL is **read-only**. It does not support `INSERT`, `UPDATE`, or `DELETE` statements. All mutations go through the Octad API (Rust core or Elixir orchestration layer). This is a deliberate design choice: VeriSimDB treats writes as coordinated multi-modal operations that must maintain consistency across all six modality stores (Graph, Vector, Tensor, Semantic, Document, Temporal). A simple `INSERT` statement cannot express the cross-modal invariants that a Octad write requires.
VCL is **read-only**. It does not support `INSERT`, `UPDATE`, or `DELETE` statements. All mutations go through the Octad API (Rust core or Elixir orchestration layer). This is a deliberate design choice: VeriSimDB treats writes as coordinated multi-modal operations that must maintain consistency across all eight modality stores (Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, Spatial). A simple `INSERT` statement cannot express the cross-modal invariants that a Octad write requires.

VCL is **multimodal**. Where SQL selects columns from relational tables, VCL selects _modalities_ from _octad stores_ or _federations_. A single VCL query can retrieve graph edges, vector embeddings, tensor slices, semantic annotations, document text, and temporal versions in one pass.

Expand All @@ -23,7 +23,7 @@ VCL is **federation-aware**. Queries can target a local `STORE`, a specific `HEX
|Concept |VCL |SQL

|**Column/Field Selection**
|`SELECT` with _modalities_: `GRAPH`, `VECTOR`, `TENSOR`, `SEMANTIC`, `DOCUMENT`, `TEMPORAL`, or `*` (all six). Modalities can be combined: `SELECT GRAPH, VECTOR, SEMANTIC`.
|`SELECT` with _modalities_: `GRAPH`, `VECTOR`, `TENSOR`, `SEMANTIC`, `DOCUMENT`, `TEMPORAL`, `PROVENANCE`, `SPATIAL`, or `*` (all eight). Modalities can be combined: `SELECT GRAPH, VECTOR, SEMANTIC`.
|`SELECT` with _column names_ or `*` for all columns.

|**Data Source**
Expand Down Expand Up @@ -140,7 +140,7 @@ SELECT *
FROM HEXAD 550e8400-e29b-41d4-a716-446655440000
----

The VCL version returns all six modalities for that octad. The SQL version returns only the columns in the `entities` table.
The VCL version returns all eight modalities for that octad. The SQL version returns only the columns in the `entities` table.

=== Finding Related Entities

Expand Down
2 changes: 2 additions & 0 deletions spec/system-specs.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

# VeriSimDB System Specifications

> ⚠️ **SUPERSEDED (2026-07-07)** — this is an early template spec describing a pre-octad design (modality set Textual/Numeric/Categorical…, `verisimdb-core`/`-nif` Rustler crate layout). It matches neither the octad nor the real `verisim-*` crates. The implemented system is the eight-modality octad documented in `README.adoc`/`EXPLAINME.adoc` and `spec/grammar.ebnf`.

VeriSimDB is a multi-modal verification database with 8 modality stores and
built-in proof verification. Implementation stack: Rust core storage engine,
Elixir/OTP API layer, ReScript frontend.
Expand Down
Loading