|
| 1 | +[](https://github.com/sponsors/hyperpolymath) |
| 2 | + |
| 3 | += Anamnesis |
| 4 | + |
| 5 | +image:https://img.shields.io/badge/License-PMPL--1.0-blue.svg[License: PMPL-1.0,link="https://github.com/hyperpolymath/palimpsest-license"] |
| 6 | +image:https://img.shields.io/badge/Philosophy-Palimpsest-indigo.svg[Palimpsest,link="https://github.com/hyperpolymath/palimpsest-license"] |
| 7 | + |
| 8 | + |
| 9 | +*A conversation knowledge extraction and reconciliation system for multi-LLM development workflows.* |
| 10 | + |
| 11 | +== Problem |
| 12 | + |
| 13 | +Modern AI-assisted development creates chaos: |
| 14 | + |
| 15 | +- *Mixed-project conversations* - Multiple projects discussed in single threads |
| 16 | +- *Dead-end tangents* - Valuable ideas lost in conversation noise |
| 17 | +- *Lost context* - Decisions and rationale buried across chat histories |
| 18 | +- *Artifact fragmentation* - Code/docs created, modified, abandoned across sessions |
| 19 | +- *Multi-LLM mayhem* - Switching between Claude, ChatGPT, Mistral with no unified view |
| 20 | + |
| 21 | +== Solution |
| 22 | + |
| 23 | +Anamnesis parses tangled multi-LLM discussion threads, tracks artifact lifecycles (created→modified→removed→evaluated), cross-links related fragments across conversations, and generates clean summaries with content properly bucketed by project/topic. |
| 24 | + |
| 25 | +== Architecture |
| 26 | + |
| 27 | +Multi-language functional system leveraging each language's strengths: |
| 28 | + |
| 29 | +``` |
| 30 | +┌─────────────────────────────────────────────────────────────────┐ |
| 31 | +│ Data Sources: Claude, ChatGPT, Mistral, Git, Browser History │ |
| 32 | +└────────────────────────┬────────────────────────────────────────┘ |
| 33 | + │ |
| 34 | + ▼ |
| 35 | +┌─────────────────────────────────────────────────────────────────┐ |
| 36 | +│ Parser (OCaml): Angstrom combinators, type-safe format parsing│ |
| 37 | +└────────────────────────┬────────────────────────────────────────┘ |
| 38 | + │ |
| 39 | + ▼ |
| 40 | +┌─────────────────────────────────────────────────────────────────┐ |
| 41 | +│ Orchestrator (Elixir): OTP supervision, port pools, HTTP API │ |
| 42 | +└────────────────────────┬────────────────────────────────────────┘ |
| 43 | + │ |
| 44 | + ┌────────────┴────────────┐ |
| 45 | + │ │ |
| 46 | + ▼ ▼ |
| 47 | +┌───────────────────────┐ ┌──────────────────────────────────────┐ |
| 48 | +│ Reasoning (λProlog) │ │ Analytics (Julia) │ |
| 49 | +│ - Artifact lifecycle │ │ - RDF generation (Serd.jl) │ |
| 50 | +│ - Contamination │ │ - SPARQL queries (HTTP.jl) │ |
| 51 | +│ - Fuzzy categorization│ │ - Reservoir computing (ESN) │ |
| 52 | +│ - Meta-reasoning │ │ - KBANN (knowledge-augmented nets) │ |
| 53 | +└───────────┬───────────┘ └──────────┬───────────────────────────┘ |
| 54 | + │ │ |
| 55 | + └────────────┬────────────┘ |
| 56 | + ▼ |
| 57 | +┌─────────────────────────────────────────────────────────────────┐ |
| 58 | +│ Storage (Virtuoso): RDF triplestore, SPARQL 1.1, named graphs │ |
| 59 | +└────────────────────────┬────────────────────────────────────────┘ |
| 60 | + │ |
| 61 | + ▼ |
| 62 | +┌─────────────────────────────────────────────────────────────────┐ |
| 63 | +│ Visualization (ReScript): Reagraph graphs, timeline charts, │ |
| 64 | +│ fuzzy category viz, type-safe functional React components │ |
| 65 | +└─────────────────────────────────────────────────────────────────┘ |
| 66 | +``` |
| 67 | + |
| 68 | +== Technology Stack |
| 69 | + |
| 70 | +| Layer | Technology | Why | |
| 71 | +|-------|-----------|-----| |
| 72 | +| *Parsing* | OCaml (Angstrom, Atdgen) | Type safety, parser combinators, ELPI integration | |
| 73 | +| *Orchestration* | Elixir/OTP | Fault tolerance, concurrency, supervision trees | |
| 74 | +| *Reasoning* | λProlog (ELPI) | Higher-order abstract syntax, meta-reasoning | |
| 75 | +| *Analytics* | Julia | Scientific computing without Python, RDF, reservoir computing | |
| 76 | +| *Storage* | Virtuoso + SPARQL | RDF native, named graphs, inference | |
| 77 | +| *Visualization* | ReScript + React | Type safety, functional paradigm, eventual JS replacement | |
| 78 | + |
| 79 | +=== Critical Constraints |
| 80 | + |
| 81 | +- ❌ *NO PYTHON* - Use Julia for all scientific computing |
| 82 | +- ✅ *Type safety* throughout (OCaml, ReScript, Elixir typespecs) |
| 83 | +- ✅ *Functional paradigm* - Immutability, pure functions, declarative |
| 84 | +- ✅ *Multi-LLM* from start (stub complexity initially) |
| 85 | +- ✅ *Academic rigor* - Research-grade documentation |
| 86 | + |
| 87 | +== Project Structure |
| 88 | + |
| 89 | +``` |
| 90 | +. |
| 91 | +├── CLAUDE.md # Claude Code guidance |
| 92 | +├── README.md # This file |
| 93 | +├── parser/ # OCaml parsing components |
| 94 | +│ ├── lib/parsers/ # Format-specific parsers |
| 95 | +│ ├── bin/ # Port executable |
| 96 | +│ └── README.md |
| 97 | +├── orchestrator/ # Elixir orchestration layer |
| 98 | +│ ├── lib/anamnesis/ # Core logic |
| 99 | +│ ├── lib/anamnesis_web/ # Phoenix HTTP API |
| 100 | +│ └── README.md |
| 101 | +├── reasoning/ # λProlog meta-reasoning |
| 102 | +│ ├── lifecycle/ # Artifact state machines |
| 103 | +│ ├── categorization/ # Fuzzy membership, contamination |
| 104 | +│ ├── linking/ # Cross-conversation references |
| 105 | +│ ├── episodic/ # Memory queries |
| 106 | +│ └── README.md |
| 107 | +├── learning/ # Julia analytics |
| 108 | +│ ├── src/rdf/ # RDF manipulation |
| 109 | +│ ├── src/reservoir/ # Reservoir computing, KBANN |
| 110 | +│ ├── src/graphs/ # Graph processing |
| 111 | +│ └── README.md |
| 112 | +├── visualization/ # ReScript UI |
| 113 | +│ ├── src/components/ # React components |
| 114 | +│ ├── src/bindings/ # JS library bindings |
| 115 | +│ ├── src/types/ # Domain types |
| 116 | +│ └── README.md |
| 117 | +├── proving-ground/ # Test workspace |
| 118 | +│ └── (zotero-voyant-export copy) |
| 119 | +└── docs/ |
| 120 | + ├── architecture/ # System architecture |
| 121 | + ├── research/ # Tech stack research |
| 122 | + ├── guides/ # Integration guides |
| 123 | + └── specifications/ # RDF schemas, protocols |
| 124 | +``` |
| 125 | + |
| 126 | +== Getting Started |
| 127 | + |
| 128 | +=== Prerequisites |
| 129 | + |
| 130 | +- *OCaml* 5.0+ (via OPAM) |
| 131 | +- *Elixir* 1.15+ (via asdf or package manager) |
| 132 | +- *Julia* 1.10+ (via juliaup) |
| 133 | +- *Node.js* 18+ (for ReScript) |
| 134 | +- *Virtuoso* 7+ (via Docker or native) |
| 135 | + |
| 136 | +=== Quick Start |
| 137 | + |
| 138 | +```bash |
| 139 | += 1. Install language runtimes (see component READMEs) |
| 140 | + |
| 141 | += 2. Build parser (OCaml) |
| 142 | + |
| 143 | +cd parser |
| 144 | +opam install --deps-only . |
| 145 | +dune build |
| 146 | + |
| 147 | += 3. Build orchestrator (Elixir) |
| 148 | + |
| 149 | +cd ../orchestrator |
| 150 | +mix deps.get |
| 151 | +mix compile |
| 152 | + |
| 153 | += 4. Build reasoning (λProlog via OCaml/ELPI) |
| 154 | + |
| 155 | +cd ../reasoning |
| 156 | +dune build |
| 157 | + |
| 158 | += 5. Setup analytics (Julia) |
| 159 | + |
| 160 | +cd ../learning |
| 161 | +julia --project=. -e 'using Pkg; Pkg.instantiate()' |
| 162 | + |
| 163 | += 6. Build visualization (ReScript) |
| 164 | + |
| 165 | +cd ../visualization |
| 166 | +npm install |
| 167 | +npm run res:build |
| 168 | + |
| 169 | += 7. Start Virtuoso (Docker) |
| 170 | + |
| 171 | +docker run -d -p 8890:8890 -p 1111:1111 \ |
| 172 | + -e DBA_PASSWORD=anamnesis \ |
| 173 | + --name virtuoso \ |
| 174 | + openlink/virtuoso-opensource-7 |
| 175 | + |
| 176 | += 8. Run Anamnesis |
| 177 | + |
| 178 | +cd ../orchestrator |
| 179 | +mix phx.server |
| 180 | += Visit http://localhost:4000 |
| 181 | + |
| 182 | +``` |
| 183 | + |
| 184 | +== Current Status |
| 185 | + |
| 186 | +*Phase: Initial Implementation* |
| 187 | + |
| 188 | +=== Completed ✅ |
| 189 | + |
| 190 | +- [x] Project architecture designed |
| 191 | +- [x] Comprehensive tech stack research (6 detailed reports) |
| 192 | +- [x] Directory structure created |
| 193 | +- [x] Component READMEs written |
| 194 | +- [x] CLAUDE.md guidance created |
| 195 | +- [x] OCaml Claude parser implemented |
| 196 | +- [x] Elixir port infrastructure |
| 197 | +- [x] Julia RDF modules (schema, serialization, SPARQL, Virtuoso) |
| 198 | +- [x] Julia reservoir computing modules (ESN, embeddings) |
| 199 | +- [x] Julia graph analysis modules |
| 200 | +- [x] ReScript domain types and color mixing |
| 201 | + |
| 202 | +=== In Progress 🚧 |
| 203 | + |
| 204 | +- [ ] End-to-end integration testing |
| 205 | +- [ ] λProlog reasoning module implementation |
| 206 | +- [ ] Virtuoso storage integration |
| 207 | +- [ ] Additional LLM format parsers (ChatGPT, Mistral) |
| 208 | + |
| 209 | +=== Phase 1 Roadmap 📋 |
| 210 | + |
| 211 | +1. ✅ Create Anamnesis repository |
| 212 | +2. ✅ Research tech options (Julia RDF, reservoir computing, ReScript viz, OCaml parsers, λProlog) |
| 213 | +3. ⬜ Copy proving ground (zotero-voyant-export) |
| 214 | +4. ✅ Setup basic infrastructure (build systems, test frameworks) |
| 215 | +5. 🚧 *Milestone 1*: Parse single Claude conversation JSON → Virtuoso RDF triples |
| 216 | + |
| 217 | +=== Phase 2: Milestone 1 🎯 |
| 218 | + |
| 219 | +*Goal*: End-to-end pipeline for one conversation format |
| 220 | + |
| 221 | +- [x] Claude JSON format parser (OCaml) |
| 222 | +- [x] Port communication (OCaml ↔ Elixir) |
| 223 | +- [x] Basic RDF generation (Julia) |
| 224 | +- [ ] Virtuoso storage integration |
| 225 | +- [ ] Simple graph visualization (ReScript) |
| 226 | + |
| 227 | +== Key Design Concepts |
| 228 | + |
| 229 | +=== Multi-Category Membership |
| 230 | + |
| 231 | +Conversations don't belong to just one project. Anamnesis supports fuzzy boundaries: |
| 232 | + |
| 233 | +```turtle |
| 234 | +anamnesis:conv:123 anamnesis:belongsTo anamnesis:proj:anamnesis ; |
| 235 | + anamnesis:membershipStrength 0.8 . |
| 236 | +
|
| 237 | +anamnesis:conv:123 anamnesis:belongsTo anamnesis:proj:rescript-evangeliser ; |
| 238 | + anamnesis:membershipStrength 0.3 . |
| 239 | +``` |
| 240 | + |
| 241 | +Visualized via color mixing: 80% green (Anamnesis) + 30% orange (ReScript) = blended color. |
| 242 | + |
| 243 | +=== Artifact Lifecycle Tracking |
| 244 | + |
| 245 | +Track state transitions over time: |
| 246 | + |
| 247 | +```prolog |
| 248 | +% λProlog rules |
| 249 | +valid_transition created modified. |
| 250 | +valid_transition modified evaluated. |
| 251 | +valid_transition modified removed. |
| 252 | +
|
| 253 | +current_state artifact_123 timestamp_456 State. |
| 254 | +% State = modified |
| 255 | +``` |
| 256 | + |
| 257 | +=== Cross-Conversation Linking |
| 258 | + |
| 259 | +Fragment references span conversations: |
| 260 | + |
| 261 | +```sparql |
| 262 | += SPARQL query |
| 263 | +
|
| 264 | +SELECT ?conv1 ?conv2 ?artifact |
| 265 | +WHERE { |
| 266 | + GRAPH ?g1 { ?conv1 anamnesis:discusses ?artifact . } |
| 267 | + GRAPH ?g2 { ?conv2 anamnesis:discusses ?artifact . } |
| 268 | + FILTER(?conv1 != ?conv2) |
| 269 | +} |
| 270 | +``` |
| 271 | + |
| 272 | +=== Episodic Memory |
| 273 | + |
| 274 | +Named graphs partition conversations by episode: |
| 275 | + |
| 276 | +``` |
| 277 | +graph:conversation:uuid-1 # Conversation metadata |
| 278 | +graph:artifacts:uuid-1 # Artifact lifecycles |
| 279 | +graph:links:uuid-1 # Cross-references |
| 280 | +graph:categories:uuid-1 # Project membership |
| 281 | +``` |
| 282 | + |
| 283 | +== Documentation |
| 284 | + |
| 285 | +- *Architecture*: `/docs/architecture/system-architecture.adoc` - Complete system design |
| 286 | +- *Research*: `/docs/research/` - Tech stack evaluations: |
| 287 | + - `julia-rdf-ecosystem.adoc` - RDF libraries (Serd.jl recommended) |
| 288 | + - `julia-reservoir-computing.adoc` - ESN, KBANN (ReservoirComputing.jl) |
| 289 | + - `puppygraph-evaluation.adoc` - Property graphs vs RDF (stick with Virtuoso) |
| 290 | + - `rescript-visualization.adoc` - Reagraph, rescript-recharts |
| 291 | + - `ocaml-parsing-elixir-bridge.adoc` - Angstrom, ports vs NIFs |
| 292 | + - `lambda-prolog-meta-reasoning.adoc` - ELPI, HOAS, integration |
| 293 | +- *Component READMEs*: Each directory has detailed setup/usage docs |
| 294 | +- *CLAUDE.md*: Guidance for Claude Code when working on this project |
| 295 | + |
| 296 | +== Testing |
| 297 | + |
| 298 | +=== Proving Ground |
| 299 | + |
| 300 | +The `zotero-voyant-export` repository is our test case - it's contaminated with Anamnesis planning discussions mixed into Zotero export tool development. This IS the problem we're solving. |
| 301 | + |
| 302 | +*Strategy*: |
| 303 | +1. Copy entire repo to `proving-ground/` |
| 304 | +2. Preserve original |
| 305 | +3. Run Anamnesis pipeline |
| 306 | +4. Verify correct untangling |
| 307 | +5. Store iterations for comparison |
| 308 | + |
| 309 | +=== Test Coverage Targets |
| 310 | + |
| 311 | +- Parser: 90%+ (property-based testing with qcheck) |
| 312 | +- Orchestrator: 85%+ (ExUnit + StreamData) |
| 313 | +- Analytics: 80%+ (Test.jl) |
| 314 | +- Visualization: 75%+ (Jest + React Testing Library) |
| 315 | +- Reasoning: Example-driven (λProlog test harness) |
| 316 | + |
| 317 | +== Performance Targets |
| 318 | + |
| 319 | +| Component | Target | Strategy | |
| 320 | +|-----------|--------|----------| |
| 321 | +| Parser throughput | 100 conversations/sec | Parallel ports, incremental parsing | |
| 322 | +| λProlog queries | <100ms | ETS caching, tabled predicates | |
| 323 | +| SPARQL queries | <50ms simple, <500ms complex | Virtuoso indexing | |
| 324 | +| Graph viz | 1000-5000 nodes interactive | WebGL (Reagraph), LOD | |
| 325 | +| RDF storage | 10M+ triples | Named graph partitioning | |
| 326 | + |
| 327 | +== Related Projects |
| 328 | + |
| 329 | +- *ReScript Evangeliser* - Learning curve synergies |
| 330 | +- *Zotero:NSAI* - Neurosymbolic AI integration |
| 331 | +- *Fogbinder* - Uncertainty-as-feature framework |
| 332 | + |
| 333 | +== Contributing |
| 334 | + |
| 335 | +This is a personal research project, but architecture decisions are documented for transparency. See component READMEs for development guidelines. |
| 336 | + |
| 337 | +== Academic Context |
| 338 | + |
| 339 | +Relevant to user's research/journalism work: |
| 340 | + |
| 341 | +- *Agnotology* - Study of culturally-induced ignorance |
| 342 | +- *Axiology* - Theory of value |
| 343 | +- *Ethics* - Moral philosophy applications |
| 344 | +- *Knowledge representation* - Formal semantics, RDF, logic programming |
| 345 | + |
| 346 | +== License |
| 347 | + |
| 348 | +To be determined. |
| 349 | + |
| 350 | +== Resources |
| 351 | + |
| 352 | +- Repository: https://github.com/Hyperpolymath/anamnesis |
| 353 | +- Test Case: zotero-voyant-export (to be copied to proving-ground) |
| 354 | +- Handover Docs: `zotero-voyant-export/docs/contamination-notice/` |
| 355 | + - ANAMNESIS_HANDOVER_MINIMAL.adoc |
| 356 | + - THREAD_CONTAMINATION_WARNING.adoc |
| 357 | + |
| 358 | +--- |
| 359 | + |
| 360 | +*Status*: Initial implementation phase. Core infrastructure complete, working toward Milestone 1. |
| 361 | + |
| 362 | +*Next Immediate Steps*: Complete end-to-end testing and Virtuoso integration. |
| 363 | + |
| 364 | + |
| 365 | +== Architecture |
| 366 | + |
| 367 | +See link:TOPOLOGY.md[TOPOLOGY.md] for a visual architecture map and completion dashboard. |
0 commit comments