Co-developing two complementary database systems to the same maturity level
- Executive Summary
- Architecture Comparison
- Shared Elements
- Phase 1: Bring VeriSimDB to 70% (12-16 weeks)
- Phase 2: Parallel Advancement (Weeks 17-24)
- Phase 3: Synergy & Integration (Weeks 25-32)
- Development Workflow
- Success Criteria
- Risk Mitigation
- Ecosystem Repos
- Tools & Infrastructure
- References
- Next Steps
- Appendix: Query Language Comparison
FormBD and VeriSimDB are sibling database projects with complementary philosophies:
-
FormBD: Narrative-first, audit-grade, single-modal (document+graph) with deep reversibility
-
VeriSimDB: Federation-first, multi-modal (6 modalities), drift-aware
Current Status:
| Project | Completion | Phase |
|---|---|---|
FormBD |
70% |
PoC implementation |
VeriSimDB |
5% |
Architecture complete, implementation nascent |
Goal: Bring VeriSimDB to 70% completion while advancing FormBD to production readiness in parallel.
┌─────────────────────────────────────────────────────────────┐ │ Form.ControlPlane (Elixir/OTP - deferred) │ ├─────────────────────────────────────────────────────────────┤ │ Form.Normalizer (Factor + Lean 4) │ │ FD discovery, proof-carrying normalization │ ├─────────────────────────────────────────────────────────────┤ │ Form.Runtime (Factor) │ │ FBQL parse/plan/exec, EXPLAIN, introspection │ ├─────────────────────────────────────────────────────────────┤ │ Form.Bridge (Zig) │ │ Stable FFI, CBOR encoding │ ├─────────────────────────────────────────────────────────────┤ │ Form.Model (Forth) │ │ Documents, edges, schemas, migrations │ ├─────────────────────────────────────────────────────────────┤ │ Form.Blocks (Forth) │ │ 4096-byte blocks, journal, crash recovery │ └─────────────────────────────────────────────────────────────┘
Query Language: FBQL (FormBD Query Language)
┌─────────────────────────────────────────────────────────────┐ │ Elixir Orchestration Layer │ │ EntityServer, DriftMonitor, QueryRouter, SchemaRegistry │ ├─────────────────────────────────────────────────────────────┤ │ ReScript Registry (Tiny Core <5k LOC) │ │ UUID → Store Mapping, KRaft Metadata Log, Trust Window │ ├─────────────────────────────────────────────────────────────┤ │ Rust Modality Stores (10 crates) │ │ graph, vector, tensor, semantic, document, temporal │ │ + hexad, drift, normalizer, api │ └─────────────────────────────────────────────────────────────┘
Query Language: VCL (VeriSim Consonance Language)
Both projects use:
-
Elixir/OTP: Fault-tolerant orchestration layer
-
Multi-model approach: FormBD (document+graph), VeriSimDB (6 modalities)
-
Auditability-first: FormBD (narrative artefacts), VeriSimDB (drift detection)
-
Query explanation: FormBD (EXPLAIN), VeriSimDB (query plan visualization)
-
Type safety: FormBD (Lean 4 proofs), VeriSimDB (ReScript + Rust)
Priority order (based on FormBD’s document-first approach):
-
Implement Tantivy-based full-text search
-
CRUD operations (create, read, update, delete)
-
Schema validation
-
Unit tests + property-based tests
Deliverable: Document modality feature-complete
-
Version history storage (inspired by FormBD’s journal)
-
Time-travel queries
-
Snapshot management
-
Audit trail
Deliverable: Temporal modality feature-complete
-
Oxigraph integration (RDF + property graph)
-
SPARQL subset
-
Edge traversal
-
Graph algorithms (shortest path, centrality)
Deliverable: Graph modality feature-complete
-
HNSW index implementation
-
Embedding storage
-
Similarity search (cosine, euclidean, dot product)
-
Dimensionality reduction hooks
Deliverable: Vector modality feature-complete
-
verisim-hexad: Unified entity abstraction -
Cross-modal indexing (UUID → modality mappings)
-
Drift detection between modalities
-
Consistency checking
Deliverable: Hexad layer complete with cross-modal queries
-
VCL parser integration (ReScript → Rust)
-
Query planner (using existing planner design docs)
-
Execution engine with multi-modal routing
-
EXPLAIN functionality (like FormBD’s FBQL EXPLAIN)
Deliverable: VCL queries execute across all modalities
-
Property-based tests (inspired by FormBD’s test suite)
-
Fuzz testing for VCL parser
-
Integration tests for all modalities
-
E2E test suite for federated scenarios
Deliverable: Test coverage > 80%, fuzzing passes 24hrs
-
Week 17-18: HTTP API server (Milestone M11 from STATE.scm)
-
Week 19-20: Form.Normalizer implementation (currently design-complete)
-
Week 21-22: Migration artefacts (M3 pending item)
-
Week 23-24: Production deployment guide, monitoring, observability
Target: FormBD 1.0 production release
-
Week 17-18: ReScript registry implementation (KRaft metadata log)
-
Week 19-20: Federation protocol (standalone → federated transition)
-
Week 21-22: ZKP integration (
proven+sactify-php) -
Week 23-24: Adaptive learner (cache TTL tuning, query plan selection)
Target: VeriSimDB 1.0 production release (standalone mode)
-
Drift Detection: Enhance FormBD’s consistency checking with VeriSimDB’s drift detection algorithms
-
Federation: Add federated FormBD nodes using VeriSimDB’s registry approach
-
Multi-Modal Projections: Unify formbd-geo and formbd-analytics as modality layers (like VeriSimDB)
-
Self-Normalization: Add
verisim-normalizerusing FormBD’s FD discovery (DFD algorithm) -
Reversibility: Enhance all modalities with FormBD’s journal-first reversibility guarantees
-
Narrative Explanations: Add narrative drift repair explanations (inspired by FormBD’s constraint rejections)
Research Question: Can VCL and FBQL interoperate?
-
Hypothesis: VCL as multi-modal routing layer, FBQL as document/graph sublanguage
-
PoC: Translate FBQL queries to VCL document+graph modality queries
-
Evaluation: Performance, expressiveness, backward compatibility
Deliverable: Technical report on FBQL/VCL unification feasibility
| Week | FormBD Focus | VeriSimDB Focus |
|---|---|---|
1-6 |
M11 HTTP API design, Form.Normalizer planning |
Rust modality stores implementation |
7-11 |
HTTP API implementation, normalizer scaffolding |
Hexad layer + VCL execution engine |
12-16 |
Testing, documentation, performance tuning |
Testing, documentation, performance tuning |
17-24 |
Production hardening, M3 migration artefacts |
Federation layer, ReScript registry |
25-32 |
Integrate drift detection, federated FormBD PoC |
Integrate self-normalization, reversibility |
Both projects maintain synchronized checkpoint files:
-
STATE.scm: Updated weekly with progress, blockers, next actions
-
ECOSYSTEM.scm: Updated when cross-project dependencies or relationships change
-
META.scm: Updated when architectural decisions or design philosophy evolves
FormBD: - [ ] M11 HTTP API server operational - [ ] Form.Normalizer scaffolding complete - [ ] Migration artefacts implemented - [ ] Test coverage > 80% - [ ] QUICKSTART guide complete
VeriSimDB: - [ ] All 6 modality stores operational - [ ] Hexad layer complete - [ ] VCL execution engine functional - [ ] Test coverage > 80% - [ ] QUICKSTART guide complete
FormBD: - [ ] FormBD 1.0 release - [ ] Production deployment docs - [ ] Monitoring/observability setup - [ ] Self-normalization operational
VeriSimDB: - [ ] VeriSimDB 1.0 release (standalone mode) - [ ] Federation protocol operational - [ ] ZKP integration complete - [ ] Adaptive learner functional
FormBD: - [ ] Drift detection integrated - [ ] Federated FormBD PoC working - [ ] formbd-geo and formbd-analytics refactored as modality layers
VeriSimDB: - [ ] Self-normalization operational across all modalities - [ ] Reversibility guarantees for all modalities - [ ] Narrative drift explanations
| Risk | Severity | Mitigation |
|---|---|---|
VeriSimDB Rust implementation slower than expected |
High |
Start with verisim-document only; defer tensor/semantic if needed |
FBQL/VCL integration infeasible |
Medium |
Keep as separate query languages; add translation layer in Phase 3 |
Elixir/OTP integration complexity |
Medium |
Share learnings between FormBD and VeriSimDB teams weekly |
FFI complexity (FormBD Zig, VeriSimDB Rust) |
Low |
Both follow ABI/FFI Universal Standard (Idris2 + Zig) |
| Risk | Severity | Mitigation |
|---|---|---|
Developer bandwidth insufficient for parallel development |
High |
Prioritize FormBD M11 and VeriSimDB modality stores; defer federation to Phase 2 |
Specification drift between projects |
Medium |
Weekly STATE.scm sync, unified ECOSYSTEM.scm updates |
Testing infrastructure setup time |
Low |
Reuse FormBD’s property-based and fuzz testing frameworks for VeriSimDB |
All located in ~/Documents/hyperpolymath-repos/:
-
formbd- Core database engine -
formbd-geo- R-tree spatial indexing projection layer -
formbd-analytics- Columnar OLAP analytics projection layer -
formbd-studio- Zero-friction GUI for non-technical users -
formbd-debugger- Proof-carrying debugger (Lean 4 + Idris 2) -
formbd-beam- BEAM/Elixir integration layer -
formbase- Open-source Airtable alternative with provenance -
zotero-formbd- Post-truth reference manager with PROMPT scores
Relationship to VeriSimDB: formbd-geo and formbd-analytics could be reimagined as VeriSimDB-style modality layers in Phase 3.
-
verisim- Main repo with 10 Rust crates:-
verisim-graph- Graph modality (Oxigraph) -
verisim-vector- Vector modality (HNSW) -
verisim-tensor- Tensor modality (ndarray) -
verisim-semantic- Semantic modality (CBOR + ZKP) -
verisim-document- Document modality (Tantivy) -
verisim-temporal- Temporal modality (version history) -
verisim-hexad- Unified entity abstraction -
verisim-drift- Cross-modal drift detection -
verisim-normalizer- Self-normalization (to be enhanced with FormBD’s approach) -
verisim-api- HTTP API server
-
Relationship to FormBD: VeriSimDB’s drift detection and federation could extend FormBD beyond single-node in Phase 3.
-
Testing: Property-based testing (FormBD uses ReScript, VeriSimDB can use Rust proptest)
-
Fuzzing: ClusterFuzzLite (both projects already configured)
-
CI/CD: GitHub Actions with Hypatia neurosymbolic scanning
-
Documentation: AsciiDoc (both projects use this format)
FormBD:
* gforth - Forth interpreter for Form.Blocks/Form.Model
* factor - Factor runtime for Form.Runtime/Form.Normalizer
* zig - Form.Bridge FFI
* lean4 - Proof verification
VeriSimDB:
* cargo - Rust workspace build
* rescript - Registry and VCL parser
* mix - Elixir orchestration layer
-
FFI Migration Guide - ABI/FFI Universal Standard (Idris2 + Zig)
-
Update ECOSYSTEM.scm in both repos to reflect sibling relationship
-
Create detailed STATE.scm for VeriSimDB with Milestones V1-V5
-
Scaffold Rust modality stores with proper tests and documentation
-
Begin Milestone V1 (verisim-document implementation)
-- Document insert with provenance
INSERT INTO papers {
title: "Machine Learning Survey",
authors: ["Smith", "Jones"],
year: 2024
} WITH PROVENANCE;
-- Graph edge insert
INSERT EDGE cites FROM paper:123 TO paper:456 {
context: "Related work section"
};
-- Explain query
EXPLAIN SELECT * FROM papers WHERE year > 2020;
-- Introspect schema
INTROSPECT SCHEMA papers;Philosophy: Narrative-first, explainable, document+graph
-- Multi-modal query
SELECT hexad.title, hexad.embedding
FROM hexads
WHERE
DOCUMENT MATCHES "machine learning" AND
VECTOR SIMILAR TO embedding(query_text) WITHIN 0.8 AND
GRAPH CONNECTED TO hexad:456 VIA "cites";
-- Drift detection query
SELECT hexad_id, drift_score
FROM hexads
WHERE DRIFT BETWEEN (document, vector) > 0.3;
-- Time-travel query
SELECT * FROM hexads
VERSION AT timestamp('2024-01-01T00:00:00Z');
-- Federation query
SELECT * FROM hexads@remote_store
WHERE modality = 'document';Philosophy: Multi-modal, federation-aware, drift-conscious
Hypothesis: VCL as routing layer, FBQL as document/graph sublanguage
-- VCL wrapping FBQL
SELECT verisim.route(
FBQL("SELECT * FROM papers WHERE year > 2020"),
modalities: ['document', 'graph']
);
-- FBQL with VCL modality hints
SELECT * FROM papers
WHERE year > 2020
WITH MODALITY HINT vector_similarity(embedding, threshold: 0.8);Evaluation needed: Feasibility, performance, backward compatibility.