[recipes] Add wiki compiler orchestration recipe

Author: justfinethanku

recipes/README.md

@@ -10,6 +10,7 @@ Step-by-step builds that add a new capability to your Open Brain. Follow the ins
 | [Email History Import](email-history-import/) | Pull your Gmail archive into searchable thoughts |
 | [ChatGPT Conversation Import](chatgpt-conversation-import/) | Ingest your ChatGPT data export |
 | [Daily Digest](daily-digest/) | Automated summary of recent thoughts via email or Slack |
+| [Wiki Compiler](wiki-compiler/) | Compiles graph-backed entity pages and topic synthesis into a regenerable wiki layer you can run on demand or on a schedule |
 | [Work Operating Model Activation](work-operating-model-activation/) | Interview-driven workflow that stores how you actually work and generates agent-ready operating files |
 | [World Model Diagnostic Activation](world-model-diagnostic-activation/) | Lightweight activation path for a 20-minute world-model diagnostic that uses the base OB1 connector and a direct-paste fallback |
 | [Research-to-Decision Workflow](research-to-decision-workflow/) | Compose canonical skills into operator and investor paths for analysis, synthesis, meetings, and decision documents |

recipes/wiki-compiler/README.md

@@ -0,0 +1,276 @@
+# Wiki Compiler
+
+> The compiled view on demand layer for Open Brain: a recipe that turns structured thoughts and graph data into regenerable wiki artifacts you can run daily, weekly, or on demand.
+
+## What This Is
+
+This recipe is the composition layer Nate described in the video.
+
+It does **not** replace the Open Brain database. It sits on top of it and produces a browsable synthesis layer:
+
+- the SQL database stays the source of truth
+- the graph and wiki pages are generated artifacts
+- if a wiki page is wrong, you fix the underlying data and regenerate
+- the wiki is never the canonical store
+
+That gives you the Karpathy-style "readable compiled understanding" layer without turning markdown pages into the system of record.
+
+## What It Does
+
+`compile-wiki.mjs` orchestrates the graph/wiki stack that now exists on `main`:
+
+1. Triggers the **entity extraction worker** so new thoughts become entities, links, and evidence rows.
+2. Runs the **typed edge classifier** so the system can capture reasoning relations like `supports`, `contradicts`, and `supersedes`.
+3. Batch-generates **entity wiki pages** from linked thoughts and graph edges.
+4. Generates **topic wiki pages** from the core `thoughts` table.
+5. Optionally backfills **Gmail thread wiki pages**.
+
+The result is a compiled wiki directory plus a manifest of what ran.
+
+## Why This Matches The Promise
+
+This is the bridge between "Open Brain as durable structured memory" and "LLM wiki as compiled understanding."
+
+It gives you:
+
+- **compiled views on demand** via a single wrapper command
+- **scheduled compilation** via cron, Claude Code scheduled tasks, or any job runner
+- **graph-backed synthesis** because entity extraction + typed edges land before wiki generation
+- **filterable synthesis** because the underlying source is SQL, not raw files
+- **regenerable outputs** because the database remains authoritative
+
+That is the architecture Nate described: structured capture first, compiled wiki second.
+
+## Contributor Credit
+
+This recipe is intentionally a **composition layer over contributor work by Alan Shurafa**. The underlying graph/wiki components it orchestrates were authored in the following merged contributions:
+
+- [#197](https://github.com/NateBJones-Projects/OB1/pull/197) — entity extraction schema
+- [#199](https://github.com/NateBJones-Projects/OB1/pull/199) — entity extraction worker
+- [#208](https://github.com/NateBJones-Projects/OB1/pull/208) — typed reasoning edges + classifier
+- [#213](https://github.com/NateBJones-Projects/OB1/pull/213) — entity wiki pages
+- [#222](https://github.com/NateBJones-Projects/OB1/pull/222) — wiki synthesis
+
+This wrapper recipe packages those pieces into one reproducible workflow. It is not a rewrite of Alan's work.
+
+## Architecture
+
+```text
+Open Brain thoughts (SQL)
+        |
+        v
+entity-extraction trigger + worker
+        |
+        v
+entities / thought_entities / edges
+        |
+        +--------------------+
+        |                    |
+        v                    v
+typed-edge-classifier   entity-wiki
+        |                    |
+        v                    v
+thought_edges         per-entity wiki pages
+        |
+        +--------------------+
+        |                    |
+        v                    v
+wiki-synthesis         Gmail thread wikis
+        |
+        v
+compiled-wiki/ + compile-manifest.json
+```
+
+## Prerequisites
+
+- A working Open Brain install
+- The merged graph/wiki stack on `main`
+- Node.js 18+
+- A valid `.env.local` or shell env for the underlying recipes
+- A deployed `entity-extraction-worker` Edge Function if you want the wrapper to trigger extraction automatically
+
+### Required environment
+
+At minimum, the downstream recipes expect:
+
+```text
+OPEN_BRAIN_URL
+OPEN_BRAIN_SERVICE_KEY
+LLM_API_KEY
+```
+
+Additional environment varies by phase:
+
+- `LLM_MODEL`, `LLM_BASE_URL` for wiki synthesis
+- `ANTHROPIC_API_KEY` for typed-edge classification
+- `EMBEDDING_API_KEY` if you want semantic expansion or thought-mode entity dossiers
+- `MCP_ACCESS_KEY` or `ENTITY_EXTRACTION_MCP_ACCESS_KEY` if you want this wrapper to trigger the entity extraction worker automatically
+- `ENTITY_EXTRACTION_WORKER_URL` if you do not want the wrapper to derive it from `OPEN_BRAIN_URL`
+
+## Install
+
+Nothing extra to install. This recipe is a wrapper around merged in-repo scripts.
+
+Run it from the Open Brain repo root so the downstream recipes can read your root `.env.local`:
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs --help
+```
+
+Done when: the help text prints and the referenced component scripts exist.
+
+## On-Demand Usage
+
+### 1. Standard compile pass
+
+This is the default "build the compiled understanding layer" run:
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs
+```
+
+By default this:
+
+- tries to trigger entity extraction
+- runs typed-edge classification
+- generates entity wiki pages
+- generates the built-in `autobiography` topic wiki
+- writes outputs under `./compiled-wiki/`
+
+### 2. Dry run
+
+Use this before your first real compile:
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs --dry-run
+```
+
+### 3. Compile entity pages only
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs \
+  --skip-extraction \
+  --skip-edges \
+  --skip-topic-wiki
+```
+
+### 4. Compile topic pages only
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs \
+  --skip-extraction \
+  --skip-edges \
+  --skip-entity-wiki \
+  --topic autobiography
+```
+
+### 5. Include Gmail thread wikis
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs --gmail --gmail-limit 10
+```
+
+## Scheduling
+
+This is designed to run on demand **or** on a schedule.
+
+### Daily light compile
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs \
+  --edge-limit 25 \
+  --entity-batch-limit 15 \
+  --topic autobiography \
+  --gmail \
+  --gmail-limit 5
+```
+
+### Weekly deep compile
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs \
+  --edge-limit 150 \
+  --entity-batch-limit 75 \
+  --topic autobiography \
+  --gmail \
+  --re-evaluate
+```
+
+### Cron example
+
+```cron
+0 6 * * * cd /path/to/OB1 && /usr/bin/env node recipes/wiki-compiler/compile-wiki.mjs --edge-limit 25 --entity-batch-limit 15 --topic autobiography >> logs/wiki-compiler.log 2>&1
+```
+
+### Agent-driven schedule
+
+If you prefer Claude Code / Codex style scheduled runs, point the agent at:
+
+```bash
+node recipes/wiki-compiler/compile-wiki.mjs --edge-limit 25 --entity-batch-limit 15 --topic autobiography
+```
+
+The important contract is the same regardless of scheduler:
+
+- write new information into SQL first
+- regenerate the compiled wiki from the source tables
+- do not manually edit generated wiki pages
+
+## Output
+
+By default the wrapper writes to:
+
+```text
+compiled-wiki/
+  entities/               # entity-wiki output when using file mode
+  topics/                 # wiki-synthesis topic output
+  compile-manifest.json   # run summary and phase statuses
+```
+
+Other outputs are owned by the underlying recipes:
+
+- `data/wiki-synthesis-state.jsonl` for Gmail thread synthesis state
+- `public.thought_edges` for typed reasoning links
+- `public.entities`, `public.edges`, `public.thought_entities` for graph extraction
+
+## Important Design Rule
+
+Do **not** treat generated wiki pages as the source of truth.
+
+This recipe is built around the opposite rule:
+
+- capture raw facts and source material in Open Brain first
+- compile the wiki from that source
+- regenerate instead of manually patching summaries
+
+That is what prevents wiki drift.
+
+## Expected Outcome
+
+After a successful run you should have:
+
+- a refreshed entity graph fed by structured Open Brain data
+- reasoning edges in `thought_edges`
+- per-entity wiki pages
+- topic synthesis pages
+- optionally Gmail thread wiki artifacts
+- a manifest recording which phases ran and whether they succeeded
+
+At that point you can browse the compiled layer like a wiki while keeping SQL as the authority underneath it.
+
+## Troubleshooting
+
+**Issue: entity extraction phase skips itself**
+Solution: set `ENTITY_EXTRACTION_WORKER_URL` plus `ENTITY_EXTRACTION_MCP_ACCESS_KEY`, or define `OPEN_BRAIN_URL` plus `MCP_ACCESS_KEY` so the wrapper can derive the worker endpoint.
+
+**Issue: typed-edge classification fails**
+Solution: confirm `ANTHROPIC_API_KEY` is available and that the `typed-reasoning-edges` schema is installed on the target brain.
+
+**Issue: entity pages fail with missing relation/table errors**
+Solution: confirm the entity extraction schema and worker have been applied and allowed to populate `entities`, `edges`, and `thought_entities`.
+
+**Issue: Gmail wiki backfill fails**
+Solution: confirm your email thoughts use the expected Gmail metadata shape and that `thought_edges` exists.
+
+**Issue: topic synthesis writes the wrong pages**
+Solution: pass explicit `--topic` and `--scope key=value` flags so the wrapper only runs the slice you want.