feat: initial commit — RAG memory system for AI agents

Author: Aetheris-Labs

.env.example

@@ -0,0 +1,13 @@
+# Storage backend: "chroma" (production) or "memory" (dev/testing)
+STORE_BACKEND=chroma
+CHROMA_URL=http://localhost:8000
+COLLECTION_NAME=engram
+
+# API
+API_PORT=4000
+
+# Maintenance
+PRUNE_INTERVAL_HOURS=24
+
+# Logging
+LOG_LEVEL=info

.github/workflows/ci.yml

@@ -0,0 +1,19 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+      - run: bun install --frozen-lockfile
+      - run: bun run lint
+      - run: bun run test

.gitignore

@@ -0,0 +1,10 @@
+node_modules/
+dist/
+.env
+.env.local
+logs/
+*.log
+chroma-data/
+.DS_Store
+Thumbs.db
+coverage/

CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## [1.0.0] — 2026-04-03
+
+### Added
+- Two store backends: ChromaDB (production) + in-memory (dev/test)
+- Similarity-based deduplication — entries with cosine similarity > 0.92 are merged
+- TTL by category: pattern 90d · warning 60d · outcome 180d · context 30d
+- IngestionPipeline with chunking (1000 char chunks, 100 char overlap)
+- Hook system with `before:ingest` / `after:ingest` events
+- SearchEngine with recency-blended reranking (15% recency weight)
+- REST API: POST /memories · POST /search · DELETE /memories/:id · POST /prune · GET /stats
+- Zod validation on all API inputs
+- 12 unit tests — all run without Chroma using the in-memory backend

CLAUDE.md

@@ -0,0 +1,55 @@
+# CLAUDE.md
+
+## What this is
+
+Engram is a RAG memory layer. Agents POST memories to `/memories`, query them via `/search`, and get back semantically ranked results. ChromaDB stores vectors; Engram handles chunking, TTL, deduplication, hooks, and re-ranking on top.
+
+## Layout
+
+```
+index.ts          ← entry point: init store, start API, schedule prune
+schemas/          ← Zod schemas for all types (Memory, SearchRequest, etc.)
+store/
+  base.ts         ← StoreBackend interface
+  chroma.ts       ← ChromaDB backend (production)
+  memory.ts       ← In-memory backend (dev/testing, no Chroma needed)
+retrieval/
+  search.ts       ← SearchEngine: query + filter
+  ranker.ts       ← rerank: blend similarity score with recency signal
+pipeline/
+  chunker.ts      ← split long text into overlapping chunks
+  ingest.ts       ← IngestionPipeline: hooks → chunk → store
+hooks/
+  index.ts        ← hook registry + built-in sanitize hook
+api/
+  server.ts       ← Bun HTTP server: /health /stats /search /memories /prune
+lib/
+  config.ts       ← Zod env validation
+  logger.ts       ← structured JSONL logger
+examples/
+  basic.ts        ← HTTP API usage example
+tests/
+```
+
+## Dev commands
+
+```bash
+bun run dev           # hot reload
+bun run test          # vitest (uses in-memory store — no Chroma needed)
+bun run example       # run examples/basic.ts (needs server running)
+bun run lint          # tsc type check
+```
+
+## Switching backends
+
+`STORE_BACKEND=memory` — no Chroma required, uses in-memory store.
+`STORE_BACKEND=chroma` — requires `docker-compose up chromadb -d` first.
+
+## Adding a hook
+
+```ts
+import { registerHook } from "./hooks/index.js";
+registerHook("before:ingest", (req) => ({ ...req, content: req.content.toUpperCase() }));
+```
+
+Available events: `before:ingest`, `after:ingest`, `before:search`, `after:search`.

Dockerfile

@@ -0,0 +1,17 @@
+FROM oven/bun:1.2-alpine AS builder
+WORKDIR /app
+COPY package.json bun.lockb* ./
+RUN bun install --frozen-lockfile
+COPY . .
+RUN bun run build
+
+FROM node:22-alpine AS runtime
+WORKDIR /app
+RUN addgroup -g 1001 -S engram && adduser -u 1001 -S engram -G engram
+COPY --from=builder /app/dist ./dist
+COPY --from=builder /app/node_modules ./node_modules
+COPY --from=builder /app/package.json ./
+RUN mkdir -p /app/logs && chown -R engram:engram /app/logs
+USER engram
+EXPOSE 4000
+CMD ["node", "dist/index.js"]

Memory.md

@@ -0,0 +1,22 @@
+# Memory
+
+Design decisions and lessons from building Engram.
+
+## Why two backends
+
+The in-memory backend exists for one reason: tests should not require a running Chroma instance. All 12 unit tests use `MemoryStore` and run offline. The `ChromaStore` has integration tests that require `docker-compose up chromadb -d`.
+
+## Deduplication threshold
+
+0.92 cosine similarity was chosen empirically. At 0.85 too many distinct warnings were merged. At 0.95 too many true duplicates slipped through. Tune via `SIMILARITY_MERGE_THRESHOLD` in `store/chroma.ts` if your use case has denser memory clusters.
+
+## Recency weight in reranker
+
+15% recency weight. Higher values cause newer but less relevant memories to surface — bad for pattern retrieval where the most similar pattern matters more than the newest one. Lower values make the system forget to prioritize recent warnings.
+
+## TTL policy rationale
+
+- `pattern` 90d — recurring market behaviour takes time to establish and validate
+- `warning` 60d — pool-specific red flags expire faster since pool conditions change
+- `outcome` 180d — execution outcomes are the most durable training signal
+- `context` 30d — ephemeral shared state, expires quickly

README.md

@@ -0,0 +1,122 @@
+# Engram
+
+![License](https://img.shields.io/badge/license-MIT-blue)
+![Runtime](https://img.shields.io/badge/runtime-Bun_1.2-black)
+![Backend](https://img.shields.io/badge/backend-ChromaDB-orange)
+
+A RAG memory system for AI agents. Store observations, retrieve relevant context by semantic similarity, and let your agents learn from what they've seen before.
+
+<br/>
+
+![Engram semantic search interface](assets/preview.svg)
+
+<br/>
+
+---
+
+## What it solves
+
+Stateless agents repeat the same mistakes. They evaluate the same bad pool twice, miss a pattern they've seen forty times, and have no way to carry knowledge from one cycle to the next.
+
+Engram gives agents a persistent memory layer with a simple contract: POST what you observe, GET what's relevant when you need to decide. Entries expire automatically, near-duplicates merge, and results are reranked by both semantic relevance and recency.
+
+---
+
+## API
+
+```bash
+# Store a memory
+POST /memories
+{
+  "category": "warning",
+  "content": "Pool 7xKp showed 8x volume/TVL spike before 31% TVL drop",
+  "poolAddress": "7xKp...",
+  "tags": ["wash-trading", "volume-spike"]
+}
+
+# Retrieve relevant memories
+POST /search
+{
+  "query": "unusual volume before TVL drop",
+  "topK": 4,
+  "category": "warning"
+}
+
+# Prune expired entries
+POST /prune
+
+# Store statistics
+GET /stats
+```
+
+---
+
+## Memory categories & TTL
+
+| Category | TTL | Use for |
+|----------|-----|---------|
+| `pattern` | 90 days | Recurring market behaviours |
+| `warning` | 60 days | Pool-specific red flags |
+| `outcome` | 180 days | Execution results with PnL |
+| `context` | 30 days | Ephemeral shared state |
+
+Entries with cosine similarity > 0.92 are merged instead of duplicated.
+
+---
+
+## Quickstart
+
+```bash
+git clone https://github.com/YOUR_USERNAME/engram
+cd engram
+bun install
+docker-compose up chromadb -d
+bun run dev
+```
+
+No Chroma? Set `STORE_BACKEND=memory` in `.env` to use the in-memory backend — no Docker needed. Tests always use the in-memory backend.
+
+```bash
+bun run example    # run examples/basic.ts against the live server
+bun run test       # vitest — no external dependencies required
+```
+
+---
+
+## Hooks
+
+Register lifecycle hooks to transform data before/after storage:
+
+```ts
+import { registerHook } from "./hooks/index.js";
+
+// Enrich every memory with a source tag before storing
+registerHook("before:ingest", (req) => ({
+  ...req,
+  tags: [...(req.tags ?? []), "auto-tagged"],
+}));
+```
+
+Available events: `before:ingest` · `after:ingest` · `before:search` · `after:search`
+
+---
+
+## Reranking
+
+Search results blend cosine similarity (85%) with a recency signal (15%). This prevents old high-similarity patterns from dominating over fresh warnings. The weights are tunable in `retrieval/ranker.ts`.
+
+---
+
+## Stack
+
+- **Runtime**: Bun 1.2
+- **Vector store**: ChromaDB
+- **Schemas**: Zod — full validation on all inputs
+- **Deduplication**: cosine similarity merge at 0.92 threshold
+- **Chunking**: 1000-char chunks with 100-char overlap
+
+---
+
+## License
+
+MIT

api/server.ts

@@ -0,0 +1,71 @@
+import { createLogger } from "../lib/logger.js";
+import { config } from "../lib/config.js";
+import type { StoreBackend } from "../store/base.js";
+import { SearchEngine } from "../retrieval/search.js";
+import { IngestionPipeline } from "../pipeline/ingest.js";
+import { SearchRequestSchema, UpsertRequestSchema } from "../schemas/index.js";
+
+const log = createLogger("API");
+
+export function startServer(store: StoreBackend): void {
+  const search = new SearchEngine(store);
+  const pipeline = new IngestionPipeline(store);
+
+  const server = Bun.serve({
+    port: config.API_PORT,
+    async fetch(req) {
+      const url = new URL(req.url);
+      const method = req.method;
+
+      // ── Health ──────────────────────────────────────────────────────────────
+      if (url.pathname === "/health" && method === "GET") {
+        return Response.json({ status: "ok", ts: Date.now() });
+      }
+
+      // ── Stats ───────────────────────────────────────────────────────────────
+      if (url.pathname === "/stats" && method === "GET") {
+        const stats = await store.stats();
+        return Response.json(stats);
+      }
+
+      // ── Search ──────────────────────────────────────────────────────────────
+      if (url.pathname === "/search" && method === "POST") {
+        const body = await req.json();
+        const parsed = SearchRequestSchema.safeParse(body);
+        if (!parsed.success) {
+          return Response.json({ error: parsed.error.flatten() }, { status: 400 });
+        }
+        const results = await search.query(parsed.data);
+        return Response.json({ results, count: results.length });
+      }
+
+      // ── Store ───────────────────────────────────────────────────────────────
+      if (url.pathname === "/memories" && method === "POST") {
+        const body = await req.json();
+        const parsed = UpsertRequestSchema.safeParse(body);
+        if (!parsed.success) {
+          return Response.json({ error: parsed.error.flatten() }, { status: 400 });
+        }
+        const memories = await pipeline.ingest(parsed.data);
+        return Response.json({ memories, count: memories.length }, { status: 201 });
+      }
+
+      // ── Delete ──────────────────────────────────────────────────────────────
+      if (url.pathname.startsWith("/memories/") && method === "DELETE") {
+        const id = url.pathname.replace("/memories/", "");
+        await store.delete(id);
+        return new Response(null, { status: 204 });
+      }
+
+      // ── Prune ───────────────────────────────────────────────────────────────
+      if (url.pathname === "/prune" && method === "POST") {
+        const pruned = await store.pruneExpired();
+        return Response.json({ pruned });
+      }
+
+      return Response.json({ error: "Not found" }, { status: 404 });
+    },
+  });
+
+  log.info(`API running at http://localhost:${config.API_PORT}`);
+}