1.0.1

Author: nullure

ARCHITECTURE.md

@@ -62,30 +62,31 @@ OpenMemory is a self-hosted AI memory engine implementing **Hierarchical Memory
 
 **Key Endpoints:**
 
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `GET` | `/health` | Health check and version info |
-| `GET` | `/sectors` | List available sectors and stats |
-| `POST` | `/memory/add` | Add a new memory |
-| `POST` | `/memory/query` | Query memories by similarity |
-| `POST` | `/memory/ingest` | Ingest document (PDF/DOCX/TXT) |
-| `POST` | `/memory/ingest/url` | Ingest URL content |
-| `POST` | `/memory/reinforce` | Boost memory salience |
-| `GET` | `/memory/all` | List all memories (paginated) |
-| `GET` | `/memory/:id` | Get specific memory details |
-| `DELETE` | `/memory/:id` | Delete a memory |
+| Method   | Endpoint             | Description                      |
+| -------- | -------------------- | -------------------------------- |
+| `GET`    | `/health`            | Health check and version info    |
+| `GET`    | `/sectors`           | List available sectors and stats |
+| `POST`   | `/memory/add`        | Add a new memory                 |
+| `POST`   | `/memory/query`      | Query memories by similarity     |
+| `POST`   | `/memory/ingest`     | Ingest document (PDF/DOCX/TXT)   |
+| `POST`   | `/memory/ingest/url` | Ingest URL content               |
+| `POST`   | `/memory/reinforce`  | Boost memory salience            |
+| `GET`    | `/memory/all`        | List all memories (paginated)    |
+| `GET`    | `/memory/:id`        | Get specific memory details      |
+| `DELETE` | `/memory/:id`        | Delete a memory                  |
 
 **LangGraph Mode Endpoints** (when `OM_MODE=langgraph`):
 
-| Method | Endpoint | Description |
-|--------|----------|-------------|
-| `POST` | `/lgm/store` | Store LangGraph node output |
-| `POST` | `/lgm/retrieve` | Retrieve memories for graph session |
-| `POST` | `/lgm/context` | Get summarized multi-sector context |
-| `POST` | `/lgm/reflection` | Generate and store reflections |
-| `GET` | `/lgm/config` | Inspect LangGraph configuration |
+| Method | Endpoint          | Description                         |
+| ------ | ----------------- | ----------------------------------- |
+| `POST` | `/lgm/store`      | Store LangGraph node output         |
+| `POST` | `/lgm/retrieve`   | Retrieve memories for graph session |
+| `POST` | `/lgm/context`    | Get summarized multi-sector context |
+| `POST` | `/lgm/reflection` | Generate and store reflections      |
+| `GET`  | `/lgm/config`     | Inspect LangGraph configuration     |
 
 **Features:**
+
 - CORS support for cross-origin requests
 - Bearer token authentication (optional)
 - Scheduled decay process (every 24 hours)
@@ -134,6 +135,7 @@ SECTORS = {
 #### 2.2 Memory Operations
 
 **Add Memory Flow:**
+
 ```
 1. Content → classifyContent() → {primary, additional}
 2. For each sector → embedMultiSector() → vectors[]
@@ -144,6 +146,7 @@ SECTORS = {
 ```
 
 **Query Memory Flow:**
+
 ```
 1. Query text → classifyContent() → candidate sectors
 2. For each sector → embedForSector() → query vector
@@ -181,14 +184,17 @@ Memory A ──0.85──> Memory B
 ```
 
 **Creation:**
+
 - During add: find single best match (cosine > 0.75)
 - Bidirectional if cross-sector
 
 **Reinforcement:**
+
 - On query: boost weight by 0.05 per traversal
 - Max weight: 1.0
 
 **Pruning:**
+
 - Every 7 days: remove weights < 0.05
 
 ---
@@ -199,22 +205,24 @@ Memory A ──0.85──> Memory B
 
 #### 3.1 Supported Providers
 
-| Provider | Models | Batch Support | Cost |
-|----------|--------|---------------|------|
-| **OpenAI** | `text-embedding-3-small`, `text-embedding-3-large` | ✅ | ~$0.02/1M tokens |
-| **Gemini** | `embedding-001` | ✅ | ~$0.01/1M tokens |
-| **Ollama** | `nomic-embed-text`, `bge-small`, `bge-large` | ❌ | Free (local) |
-| **Local** | Custom models | ❌ | Free (local) |
-| **Synthetic** | Hash-based | ❌ | Free |
+| Provider      | Models                                             | Batch Support | Cost             |
+| ------------- | -------------------------------------------------- | ------------- | ---------------- |
+| **OpenAI**    | `text-embedding-3-small`, `text-embedding-3-large` | ✅            | ~$0.02/1M tokens |
+| **Gemini**    | `embedding-001`                                    | ✅            | ~$0.01/1M tokens |
+| **Ollama**    | `nomic-embed-text`, `bge-small`, `bge-large`       | ❌            | Free (local)     |
+| **Local**     | Custom models                                      | ❌            | Free (local)     |
+| **Synthetic** | Hash-based                                         | ❌            | Free             |
 
 #### 3.2 Embedding Modes
 
 **Simple Mode** (`OM_EMBED_MODE=simple`):
+
 - One batch call per memory (all sectors at once)
 - Faster for OpenAI/Gemini
 - Lower API overhead
 
 **Advanced Mode** (`OM_EMBED_MODE=advanced`):
+
 - Sector-specific model selection
 - Optional parallel embedding
 - Chunking support for long texts
@@ -223,6 +231,7 @@ Memory A ──0.85──> Memory B
 #### 3.3 Chunking Strategy
 
 For texts > 512 tokens:
+
 ```
 1. Split text into overlapping chunks (512 tokens, 50 overlap)
 2. Embed each chunk separately
@@ -239,6 +248,7 @@ For texts > 512 tokens:
 #### 4.1 Schema
 
 **memories table:**
+
 ```sql
 CREATE TABLE memories (
   id TEXT PRIMARY KEY,           -- UUID
@@ -258,6 +268,7 @@ CREATE TABLE memories (
 ```
 
 **vectors table:**
+
 ```sql
 CREATE TABLE vectors (
   id TEXT NOT NULL,              -- Memory ID
@@ -269,6 +280,7 @@ CREATE TABLE vectors (
 ```
 
 **waypoints table:**
+
 ```sql
 CREATE TABLE waypoints (
   src_id TEXT PRIMARY KEY,       -- Source memory
@@ -280,6 +292,7 @@ CREATE TABLE waypoints (
 ```
 
 **embed_logs table:**
+
 ```sql
 CREATE TABLE embed_logs (
   id TEXT PRIMARY KEY,
@@ -293,15 +306,15 @@ CREATE TABLE embed_logs (
 #### 4.2 Transaction Support
 
 ```typescript
-transaction.begin()
+transaction.begin();
 try {
   // Insert memory
   // Insert vectors
   // Create waypoints
-  transaction.commit()
+  transaction.commit();
 } catch (e) {
-  transaction.rollback()
-  throw e
+  transaction.rollback();
+  throw e;
 }
 ```
 
@@ -313,12 +326,12 @@ try {
 
 #### 5.1 Supported Formats
 
-| Format | Parser | Features |
-|--------|--------|----------|
-| **PDF** | `pdf-parse` | Text extraction, metadata |
-| **DOCX** | `mammoth` | Convert to markdown |
-| **TXT** | Native | Direct read |
-| **URL** | `fetch` + `turndown` | HTML → Markdown |
+| Format   | Parser               | Features                  |
+| -------- | -------------------- | ------------------------- |
+| **PDF**  | `pdf-parse`          | Text extraction, metadata |
+| **DOCX** | `mammoth`            | Convert to markdown       |
+| **TXT**  | Native               | Direct read               |
+| **URL**  | `fetch` + `turndown` | HTML → Markdown           |
 
 #### 5.2 Processing Flow
 
@@ -332,6 +345,7 @@ Document → Extract text → Chunk if needed →
 ```
 
 **Configuration:**
+
 ```typescript
 {
   chunk_size: 2048,        // Max tokens per chunk
@@ -350,12 +364,12 @@ Document → Extract text → Chunk if needed →
 
 ```typescript
 NODE_SECTOR_MAP = {
-  observe: 'episodic',     // Observations
-  plan: 'semantic',        // Plans and strategies
-  reflect: 'reflective',   // Reflections
-  act: 'procedural',       // Actions taken
-  emotion: 'emotional'     // Emotional state
-}
+  observe: 'episodic', // Observations
+  plan: 'semantic', // Plans and strategies
+  reflect: 'reflective', // Reflections
+  act: 'procedural', // Actions taken
+  emotion: 'emotional', // Emotional state
+};
 ```
 
 #### 6.2 Context Assembly
@@ -543,33 +557,33 @@ OM_LG_REFLECTIVE=true
 
 ### Latency (100k memories)
 
-| Operation | Latency | Notes |
-|-----------|---------|-------|
-| Add memory | 80-120 ms | Depends on embedding provider |
-| Query (simple) | 110-130 ms | Single-sector search |
-| Query (multi-sector) | 150-200 ms | 2-3 sector fusion |
-| Waypoint expansion | +30-50 ms | Per hop |
-| Decay process | ~10 sec | Background, every 24h |
+| Operation            | Latency    | Notes                         |
+| -------------------- | ---------- | ----------------------------- |
+| Add memory           | 80-120 ms  | Depends on embedding provider |
+| Query (simple)       | 110-130 ms | Single-sector search          |
+| Query (multi-sector) | 150-200 ms | 2-3 sector fusion             |
+| Waypoint expansion   | +30-50 ms  | Per hop                       |
+| Decay process        | ~10 sec    | Background, every 24h         |
 
 ### Storage (SQLite)
 
-| Item | Size | Notes |
-|------|------|-------|
-| Memory metadata | ~500 bytes | Per memory |
-| Vector (768d) | ~3 KB | Per sector |
-| Waypoint | ~100 bytes | Per link |
-| **Total per memory** | ~4-6 KB | Depends on sectors |
-| **100k memories** | ~500 MB | Typical |
-| **1M memories** | ~5 GB | With indexing |
+| Item                 | Size       | Notes              |
+| -------------------- | ---------- | ------------------ |
+| Memory metadata      | ~500 bytes | Per memory         |
+| Vector (768d)        | ~3 KB      | Per sector         |
+| Waypoint             | ~100 bytes | Per link           |
+| **Total per memory** | ~4-6 KB    | Depends on sectors |
+| **100k memories**    | ~500 MB    | Typical            |
+| **1M memories**      | ~5 GB      | With indexing      |
 
 ### Throughput
 
-| Operation | Rate | Notes |
-|-----------|------|-------|
-| Add (synthetic) | ~40 ops/s | No external API |
-| Add (OpenAI) | ~10-15 ops/s | Rate limited |
-| Add (Ollama) | ~8-12 ops/s | CPU bound |
-| Query | ~30-50 ops/s | In-memory vectors |
+| Operation       | Rate         | Notes             |
+| --------------- | ------------ | ----------------- |
+| Add (synthetic) | ~40 ops/s    | No external API   |
+| Add (OpenAI)    | ~10-15 ops/s | Rate limited      |
+| Add (Ollama)    | ~8-12 ops/s  | CPU bound         |
+| Query           | ~30-50 ops/s | In-memory vectors |
 
 ---
 
@@ -591,22 +605,26 @@ OM_LG_REFLECTIVE=true
 ```
 
 **Benefits:**
+
 - Sector-specific optimization
 - Independent scaling per sector
 - Reduced contention
 
 **Trade-offs:**
+
 - Cross-sector queries need aggregation
 - Waypoints may span instances
 
 ### Vertical Scaling
 
 **Bottlenecks:**
+
 1. Embedding API rate limits → Use batch mode
 2. SQLite write contention → Use WAL mode
 3. Vector similarity computation → Use SIMD
 
 **Optimizations:**
+
 - Enable WAL mode (write-ahead logging)
 - Use connection pooling
 - Cache mean vectors in memory
@@ -617,17 +635,20 @@ OM_LG_REFLECTIVE=true
 ## Security
 
 ### Authentication
+
 - Optional bearer token (`OM_API_KEY`)
 - All write endpoints check auth
 - Read endpoints can be public
 
 ### Data Privacy
+
 - 100% local storage (no vendor lock-in)
 - Optional content encryption at rest
 - PII scrubbing hooks available
 - Tenant isolation support
 
 ### Best Practices
+
 1. Use HTTPS in production
 2. Set `OM_API_KEY` for write protection
 3. Run behind reverse proxy (nginx/caddy)
@@ -645,6 +666,7 @@ docker compose up -d
 ```
 
 Ports:
+
 - `8080` → API server
 - Data persisted in `/data/openmemory.sqlite`
 
@@ -664,6 +686,7 @@ npm start
 ```
 
 **Systemd service:**
+
 ```ini
 [Unit]
 Description=OpenMemory Service
@@ -720,6 +743,7 @@ Response:
 ### Embedding Logs
 
 Query `embed_logs` table for:
+
 - Failed embedding attempts
 - Rate limit issues
 - Provider errors
@@ -729,16 +753,19 @@ Query `embed_logs` table for:
 ## Future Architecture Enhancements
 
 ### v1.3: Learned Sector Classifier
+
 - Replace regex patterns with Tiny Transformer
 - Train on user data for better classification
 - Adaptive sector weighting
 
 ### v1.4: Federated Multi-Node
+
 - Distributed waypoint graph
 - Consensus protocol for salience
 - Cross-node query federation
 
 ### v1.5: Pluggable Vector Backends
+
 - Support pgvector (PostgreSQL)
 - Support Weaviate/Qdrant
 - Abstraction layer for vector ops
@@ -747,16 +774,16 @@ Query `embed_logs` table for:
 
 ## Glossary
 
-| Term | Definition |
-|------|------------|
-| **HMD** | Hierarchical Memory Decomposition - the core architecture |
-| **Sector** | Memory type (episodic, semantic, procedural, emotional, reflective) |
-| **Salience** | Importance score (0-1) that decays over time |
-| **Waypoint** | Associative link between memories (single strongest only) |
-| **Decay** | Time-based reduction in salience (sector-specific) |
-| **Reinforcement** | Boosting salience/waypoint strength on recall |
-| **Mean Vector** | Weighted average of all sector vectors (for waypoint matching) |
-| **Composite Score** | 0.6×similarity + 0.2×salience + 0.1×recency + 0.1×waypoint |
+| Term                | Definition                                                          |
+| ------------------- | ------------------------------------------------------------------- |
+| **HMD**             | Hierarchical Memory Decomposition - the core architecture           |
+| **Sector**          | Memory type (episodic, semantic, procedural, emotional, reflective) |
+| **Salience**        | Importance score (0-1) that decays over time                        |
+| **Waypoint**        | Associative link between memories (single strongest only)           |
+| **Decay**           | Time-based reduction in salience (sector-specific)                  |
+| **Reinforcement**   | Boosting salience/waypoint strength on recall                       |
+| **Mean Vector**     | Weighted average of all sector vectors (for waypoint matching)      |
+| **Composite Score** | 0.6×similarity + 0.2×salience + 0.1×recency + 0.1×waypoint          |
 
 ---