diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index cc27540..59ed36a 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -9,6 +9,12 @@ All notable changes to VeriSimDB are documented here. This project uses https:// == [Unreleased] +=== Changed — Docs hygiene: retire fossil specs, fix six→eight modality drift (2026-07-07) + +- Stale-stamped three fossil documents with prominent superseded/historical banners: `docs/status/implementation-plan.adoc` (Week 1–16 / 10%-completion plan), `docs/releases/v0.1.0-alpha.md` ("Production-Ready" / "100% Complete" claims contradicted by KNOWN-ISSUES #29/#30/#31), and `spec/system-specs.md` (pre-octad template spec with a Textual/Numeric/Categorical modality set and `verisimdb-core`/`-nif` Rustler crate layout matching neither the octad nor the real `verisim-*` crates). +- Corrected modality-count drift ("six"/"6" → eight/octad) in current docs: `docs/deployment-modes.adoc`, `docs/federation-readiness.adoc`, `docs/challenges-standalone.adoc`, `docs/challenges-federated.adoc`, `docs/challenges-hybrid.adoc`, `docs/reversibility-design.adoc`, `docs/vcl-vs-sql.adoc`, `docs/deployment/deployment.adoc`, and `ROADMAP.md`. Where a doc enumerated only the legacy six concrete stores, added the missing Provenance (hash-chain lineage) and Spatial (R-tree) entries. +- Left deliberate references untouched: HEXAD grammar keyword, VCL "six proof types" / VCL-dt++ "six clauses", and dated historical records in CHANGELOG/KNOWN-ISSUES/design docs/papers. The "is VeriSimDB a database?" positioning prose is unchanged pending owner decision D1. + == [0.2.0] — 2026-06-13 === Added — July-1 credibility: honesty over fake success (2026-06-13) diff --git a/ROADMAP.md b/ROADMAP.md index 634f6f5..fd25fba 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -97,7 +97,7 @@ Currently in-memory only. Need pluggable backends. ### 2.4 Snapshots & Backup **Criticality: HIGH** | Effort: Medium -- Point-in-time snapshots (consistent across all 6 modalities) +- Point-in-time snapshots (consistent across all 8 modalities) - Incremental backup (WAL-based) - Restore from snapshot + WAL replay - Export/import in portable format diff --git a/docs/challenges-federated.adoc b/docs/challenges-federated.adoc index 9ac8434..9bad864 100644 --- a/docs/challenges-federated.adoc +++ b/docs/challenges-federated.adoc @@ -514,7 +514,7 @@ With N modalities: latency scales with max(Store1, Store2, ..., StoreN) **Amplification Factors:** - **2 modalities:** 2x network calls -- **6 modalities:** 6x network calls (if not optimized) +- **8 modalities:** 8x network calls (if not optimized) - **Failure Probability:** P(any store fails) = 1 - ∏(1 - P(store_i fails)) **Example:** @@ -628,7 +628,7 @@ Monthly bandwidth: |Charge $0.01/query (cost recovery) |**Modality Support** -|All 6 modalities +|All 8 modalities |Graph and Document only (limited resources) |**Schema Changes** diff --git a/docs/challenges-hybrid.adoc b/docs/challenges-hybrid.adoc index 3d3ea90..3f8ebd5 100644 --- a/docs/challenges-hybrid.adoc +++ b/docs/challenges-hybrid.adoc @@ -717,7 +717,7 @@ Result: Query denied at Layer 2 despite passing Layer 1 **Problem:** Hybrid mode requires **more operational expertise** than standalone. **Skills Required:** -- **Standalone:** Understand 6 modality stores + Elixir orchestrator (7 components) +- **Standalone:** Understand 8 modality stores + Elixir orchestrator (9 components) - **Hybrid:** All above + Network troubleshooting + Remote store APIs + Policy coordination (10+ components) **Hiring Impact:** May need to hire **DevOps specialist** for hybrid management (+$150k/year salary) diff --git a/docs/challenges-standalone.adoc b/docs/challenges-standalone.adoc index 8344eff..1e9b24b 100644 --- a/docs/challenges-standalone.adoc +++ b/docs/challenges-standalone.adoc @@ -4,7 +4,7 @@ == Overview -Standalone deployment runs VeriSimDB as a **traditional database** with all six modalities (Graph, Vector, Tensor, Semantic, Document, Temporal) hosted locally on the same machine or cluster. While this offers simplicity and performance, it introduces specific architectural, operational, and scaling challenges. +Standalone deployment runs VeriSimDB as a **traditional database** with all eight modalities (the octad: Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, Spatial) hosted locally on the same machine or cluster. While this offers simplicity and performance, it introduces specific architectural, operational, and scaling challenges. This document details the **technical challenges** unique to standalone deployment and provides mitigation strategies where applicable. @@ -38,10 +38,10 @@ This document details the **technical challenges** unique to standalone deployme === 1.2. Single Point of Failure -**Problem:** Loss of the standalone node means loss of all six modalities simultaneously. +**Problem:** Loss of the standalone node means loss of all eight modalities simultaneously. **Risk Scenarios:** -1. Hardware failure → No Graph, Vector, Tensor, Semantic, Document, OR Temporal access +1. Hardware failure → No Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, OR Spatial access 2. Kernel panic → All GenServers crash (Elixir orchestrator down) 3. Disk corruption → All modalities potentially compromised @@ -54,7 +54,7 @@ This document details the **technical challenges** unique to standalone deployme === 1.3. Modality Coupling -**Problem:** All six modalities share the same **Elixir orchestration layer**, creating tight coupling. +**Problem:** All eight modalities share the same **Elixir orchestration layer**, creating tight coupling. **Consequences:** - **Upgrade Hell:** Updating `verisim-graph-rs` (Oxigraph) requires restarting the entire stack @@ -93,7 +93,7 @@ Result: All three queries degrade due to resource contention === 2.2. Write Amplification -**Problem:** A single Octad update can trigger writes across **all six modalities**. +**Problem:** A single Octad update can trigger writes across **all eight modalities**. **Example:** 1. User updates Octad `550e8400-...` with new document text @@ -103,7 +103,7 @@ Result: All three queries degrade due to resource contention 5. **Semantic modality**: Validates CBOR proof (writes metadata) 6. **Graph modality**: May add citation edges (writes RDF triples) -**Write Amplification Factor:** ~6x per logical update +**Write Amplification Factor:** up to ~8x per logical update (worst case, all modalities) **Consequences:** - **SSD Wear:** Premature SSD exhaustion (TBW limits) @@ -138,7 +138,7 @@ Result: All three queries degrade due to resource contention === 3.1. Backup and Recovery Complexity -**Problem:** Six modalities require **six separate backup strategies**. +**Problem:** Eight modalities require **eight separate backup strategies**. [cols="1,2,2"] |=== @@ -167,12 +167,20 @@ Result: All three queries degrade due to resource contention |**Temporal** |Merkle-tree nodes → recursive snapshot |Tree integrity check expensive on large histories + +|**Provenance** +|Hash-chain lineage → append-only log export +|Chain integrity must be re-verified on restore + +|**Spatial** +|R-tree index → serialized dump +|Spatial index rebuilt from stored geometry |=== **Operational Burden:** - **Inconsistent Snapshots:** Modalities backed up at different times → cross-modal inconsistency -- **Restore Testing:** Must test restore for all 6 modalities (6x testing effort) -- **Storage Costs:** 6 separate backups consume significant space +- **Restore Testing:** Must test restore for all 8 modalities (8x testing effort) +- **Storage Costs:** 8 separate backups consume significant space **Mitigation:** - **Coordinated Snapshots:** Use `VeriSim.DriftMonitor` to trigger simultaneous backups @@ -181,7 +189,7 @@ Result: All three queries degrade due to resource contention === 3.2. Monitoring and Observability -**Problem:** Need to monitor **six independent data stores** plus orchestration layer. +**Problem:** Need to monitor **eight independent data stores** plus orchestration layer. **Metrics to Track (per modality):** - Query latency (p50, p99, p999) @@ -191,7 +199,7 @@ Result: All three queries degrade due to resource contention - Cache hit rates (where applicable) **Operational Complexity:** -- **7 dashboards** (6 modalities + orchestrator) → alert fatigue +- **9 dashboards** (8 modalities + orchestrator) → alert fatigue - **Correlation Difficulty:** Determining which modality caused a slowdown requires cross-dashboard analysis - **Logging Volume:** Combined logs from all modalities can reach GB/day @@ -254,7 +262,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] } - **Result:** Vector store saturated, Temporal store wasted capacity **Consequences:** -- **Inefficient Resource Use:** Paying for 6 modalities, using 1-2 heavily +- **Inefficient Resource Use:** Paying for 8 modalities, using 1-2 heavily - **Cannot Scale Hot Path:** Standalone mode cannot add more Vector replicas without duplicating all modalities **Mitigation:** @@ -268,7 +276,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] } **Theoretical Limit:** - Elixir GenServer: ~50k messages/sec per process (microbenchmark) -- Each write requires ~5 GenServer calls (validation, routing, 6 modality dispatches, logging) +- Each write requires ~5 GenServer calls (validation, routing, 8 modality dispatches, logging) - **Practical Write Throughput:** ~10k writes/sec **Symptom Under Load:** @@ -285,7 +293,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] } === 5.1. Single Security Boundary -**Problem:** Compromise of standalone node exposes **all six modalities** simultaneously. +**Problem:** Compromise of standalone node exposes **all eight modalities** simultaneously. **Attack Scenarios:** 1. **Container Escape:** Attacker gains root on host → accesses all modality stores @@ -353,7 +361,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] } === 7.1. Over-Provisioning -**Problem:** Must provision hardware for **peak load across all six modalities**. +**Problem:** Must provision hardware for **peak load across all eight modalities**. **Example:** - Vector modality needs 64GB RAM for HNSW index @@ -367,7 +375,7 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] } === 7.2. Maintenance Burden -**Problem:** Single admin team must understand **six different data stores**. +**Problem:** Single admin team must understand **eight different data stores**. **Skill Requirements:** - Oxigraph (RDF/SPARQL) @@ -376,8 +384,10 @@ tokio = { version = "1.40", features = ["rt-multi-thread", "io-util"] } - CBOR (semantic proofs, schema validation) - Tantivy (full-text search, inverted indices) - Merkle trees (temporal versioning, cryptographic proofs) +- Hash-chain lineage (provenance integrity) +- R-tree indexing (spatial/geospatial queries) -**Hiring Difficulty:** Finding engineers with all six skill sets is rare/expensive +**Hiring Difficulty:** Finding engineers with all eight skill sets is rare/expensive **Mitigation:** - **Specialization:** Split team into modality-specific sub-teams (requires larger org) diff --git a/docs/deployment-modes.adoc b/docs/deployment-modes.adoc index 986e9fe..7174918 100644 --- a/docs/deployment-modes.adoc +++ b/docs/deployment-modes.adoc @@ -8,7 +8,7 @@ VeriSimDB is **both** a database and a federation coordinator—the architecture === Mode 1: Standalone Database (Traditional) -Deploy VeriSimDB as a **complete database system** with all six modalities local. +Deploy VeriSimDB as a **complete database system** with all eight modalities (the octad) local. [cols="1,3"] |=== @@ -18,7 +18,7 @@ Deploy VeriSimDB as a **complete database system** with all six modalities local |Local in-memory namespace |Modality Stores -|All 6 modalities running on the same machine/cluster +|All 8 modalities running on the same machine/cluster |Use Case |Single institution, traditional database replacement diff --git a/docs/deployment/deployment.adoc b/docs/deployment/deployment.adoc index f43677b..bbd63ba 100644 --- a/docs/deployment/deployment.adoc +++ b/docs/deployment/deployment.adoc @@ -70,7 +70,7 @@ This guide covers all deployment modes with focus on production readiness, secur === Mode 1: Standalone Database -Single-node deployment with all 6 modalities local. +Single-node deployment with all 8 modalities local. ==== Use Cases diff --git a/docs/federation-readiness.adoc b/docs/federation-readiness.adoc index a44f76e..f83a8fe 100644 --- a/docs/federation-readiness.adoc +++ b/docs/federation-readiness.adoc @@ -8,7 +8,7 @@ == What Federation Means in VeriSimDB -Federation is the ability for multiple VeriSimDB instances to coordinate queries across their respective octad stores. Each instance maintains its own six modality stores (Graph, Vector, Tensor, Semantic, Document, Temporal) and its own set of octad entities. A federated query spans multiple instances, retrieving and correlating data from peers that may be geographically distributed, operated by different organizations, or running different versions of VeriSimDB. +Federation is the ability for multiple VeriSimDB instances to coordinate queries across their respective octad stores. Each instance maintains its own eight modality stores (the octad: Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, Spatial) and its own set of octad entities. A federated query spans multiple instances, retrieving and correlating data from peers that may be geographically distributed, operated by different organizations, or running different versions of VeriSimDB. Federation in VeriSimDB is distinct from database replication. Replicated databases maintain identical copies of the same data. Federated VeriSimDB instances hold _different_ octad entities that may reference each other through graph edges, share semantic vocabularies, or have vector embeddings in the same latent space. The federation layer coordinates cross-instance queries while each instance retains sovereignty over its own data. @@ -60,7 +60,7 @@ If VCL were removed or not used, VeriSimDB would lose the following federation-c === Normalizer Regeneration Strategies Are Stubs -The `verisim-normalizer` crate's regeneration strategies currently return hardcoded `[regenerated]` placeholder values. When drift is detected and normalization is triggered, the normalizer identifies the authoritative modality and calls the appropriate regeneration strategy, but that strategy does not actually regenerate the drifted modality's data. Each of the six modalities needs a real regeneration implementation: +The `verisim-normalizer` crate's regeneration strategies currently return hardcoded `[regenerated]` placeholder values. When drift is detected and normalization is triggered, the normalizer identifies the authoritative modality and calls the appropriate regeneration strategy, but that strategy does not actually regenerate the drifted modality's data. Each of the eight modalities needs a real regeneration implementation: * **Graph**: Re-derive edges from document content and semantic annotations. * **Vector**: Re-embed from document text using the configured embedding model. @@ -68,6 +68,8 @@ The `verisim-normalizer` crate's regeneration strategies currently return hardco * **Semantic**: Re-extract type annotations and RDF triples from document and graph. * **Document**: Re-generate searchable text from graph, semantic, and temporal data. * **Temporal**: Re-build version chain from modification history. +* **Provenance**: Re-build the hash-linked provenance chain from recorded modification events. +* **Spatial**: Re-derive spatial geometry and coordinates from document and semantic references. === Federation Executor Always Returns Empty @@ -109,7 +111,7 @@ Drift detection works: the `verisim-drift` crate computes drift scores across mo |**Normalizer (regeneration)** |Stub -|Returns hardcoded `[regenerated]`. Six modality-specific strategies needed. +|Returns hardcoded `[regenerated]`. Eight modality-specific strategies needed. |**Consensus Protocol** |Design only @@ -152,7 +154,7 @@ Drift detection works: the `verisim-drift` crate computes drift scores across mo * Add authentication and authorization for peer-to-peer communication (mTLS or signed requests). * Implement query result merging in `FederationExecutor` with configurable merge strategies (union, intersection, ranked). * Add timeout and fallback handling for unreachable peers. -* Implement remaining normalizer regeneration strategies for all six modalities. +* Implement remaining normalizer regeneration strategies for all eight modalities. * Add federation-scoped drift detection: compute drift across instances, not just within one. === Phase 3: Verified Federation diff --git a/docs/releases/v0.1.0-alpha.md b/docs/releases/v0.1.0-alpha.md index 336ac19..5e37365 100644 --- a/docs/releases/v0.1.0-alpha.md +++ b/docs/releases/v0.1.0-alpha.md @@ -1,5 +1,7 @@ # VeriSimDB v0.1.0-alpha Release Notes +> ⚠️ **HISTORICAL RELEASE NOTE (2026-07-07)** — the "Production-Ready" and "100% Complete" claims in this document were inaccurate and are superseded by `KNOWN-ISSUES.adoc` (see issues #29/#30/#31). Retained for release history only. + **Release Date:** 2026-02-04 **Status:** Alpha Release - Production-Ready diff --git a/docs/reversibility-design.adoc b/docs/reversibility-design.adoc index de25e8b..ce88f27 100644 --- a/docs/reversibility-design.adoc +++ b/docs/reversibility-design.adoc @@ -151,7 +151,7 @@ Each modality can be made reversible: === Level 3: Full Multi-Modal Reversibility (MODERATE COST) -**What:** All six modalities support revert. +**What:** All eight modalities support revert. **Implementation:** diff --git a/docs/status/implementation-plan.adoc b/docs/status/implementation-plan.adoc index 8c4c404..fde369b 100644 --- a/docs/status/implementation-plan.adoc +++ b/docs/status/implementation-plan.adoc @@ -4,6 +4,11 @@ // SPDX-License-Identifier: CC-BY-SA-4.0 +[WARNING] +==== +*SUPERSEDED (2026-07-07)* — historical planning document. The described architecture (Week 1–16 plan, "10% completion", six modality stores) no longer reflects the repo. See `KNOWN-ISSUES.adoc` and `ROADMAP.md` for current truth. +==== + [.lead] **Bringing VeriSimDB from 10% to 70% completion** diff --git a/docs/vcl-vs-sql.adoc b/docs/vcl-vs-sql.adoc index 74a92b5..fd6f19f 100644 --- a/docs/vcl-vs-sql.adoc +++ b/docs/vcl-vs-sql.adoc @@ -10,7 +10,7 @@ VCL (VeriSim Consonance Language) is the query language for VeriSimDB, a 6-core multimodal database. While VCL borrows familiar keywords from SQL (`SELECT`, `FROM`, `WHERE`, `LIMIT`, `OFFSET`), it serves a fundamentally different purpose. -VCL is **read-only**. It does not support `INSERT`, `UPDATE`, or `DELETE` statements. All mutations go through the Octad API (Rust core or Elixir orchestration layer). This is a deliberate design choice: VeriSimDB treats writes as coordinated multi-modal operations that must maintain consistency across all six modality stores (Graph, Vector, Tensor, Semantic, Document, Temporal). A simple `INSERT` statement cannot express the cross-modal invariants that a Octad write requires. +VCL is **read-only**. It does not support `INSERT`, `UPDATE`, or `DELETE` statements. All mutations go through the Octad API (Rust core or Elixir orchestration layer). This is a deliberate design choice: VeriSimDB treats writes as coordinated multi-modal operations that must maintain consistency across all eight modality stores (Graph, Vector, Tensor, Semantic, Document, Temporal, Provenance, Spatial). A simple `INSERT` statement cannot express the cross-modal invariants that a Octad write requires. VCL is **multimodal**. Where SQL selects columns from relational tables, VCL selects _modalities_ from _octad stores_ or _federations_. A single VCL query can retrieve graph edges, vector embeddings, tensor slices, semantic annotations, document text, and temporal versions in one pass. @@ -23,7 +23,7 @@ VCL is **federation-aware**. Queries can target a local `STORE`, a specific `HEX |Concept |VCL |SQL |**Column/Field Selection** -|`SELECT` with _modalities_: `GRAPH`, `VECTOR`, `TENSOR`, `SEMANTIC`, `DOCUMENT`, `TEMPORAL`, or `*` (all six). Modalities can be combined: `SELECT GRAPH, VECTOR, SEMANTIC`. +|`SELECT` with _modalities_: `GRAPH`, `VECTOR`, `TENSOR`, `SEMANTIC`, `DOCUMENT`, `TEMPORAL`, `PROVENANCE`, `SPATIAL`, or `*` (all eight). Modalities can be combined: `SELECT GRAPH, VECTOR, SEMANTIC`. |`SELECT` with _column names_ or `*` for all columns. |**Data Source** @@ -140,7 +140,7 @@ SELECT * FROM HEXAD 550e8400-e29b-41d4-a716-446655440000 ---- -The VCL version returns all six modalities for that octad. The SQL version returns only the columns in the `entities` table. +The VCL version returns all eight modalities for that octad. The SQL version returns only the columns in the `entities` table. === Finding Related Entities diff --git a/spec/system-specs.md b/spec/system-specs.md index e139517..4962b05 100644 --- a/spec/system-specs.md +++ b/spec/system-specs.md @@ -3,6 +3,8 @@ # VeriSimDB System Specifications +> ⚠️ **SUPERSEDED (2026-07-07)** — this is an early template spec describing a pre-octad design (modality set Textual/Numeric/Categorical…, `verisimdb-core`/`-nif` Rustler crate layout). It matches neither the octad nor the real `verisim-*` crates. The implemented system is the eight-modality octad documented in `README.adoc`/`EXPLAINME.adoc` and `spec/grammar.ebnf`. + VeriSimDB is a multi-modal verification database with 8 modality stores and built-in proof verification. Implementation stack: Rust core storage engine, Elixir/OTP API layer, ReScript frontend.