feat(knowledge-base): self-curating knowledge base (OKF pages + folder missions)

[nicoloboschi]

Draft / WIP — opening for review of the design and structure. A few items below are still open.

What

A server-side Knowledge Base: a hierarchy of folders and pages over mental models, with a mission-driven curator that maintains pages automatically, plus a control-plane UI (file-tree + constellation). Pages project to the Open Knowledge Format (markdown + YAML frontmatter).

Model

• Folders carry a mission. Pages are backed by a mental model (mental_model_id) and marked managed (curator-owned) vs pinned (human).
• Curator (per folder, independent): after each consolidation (and on folder/mission creation), it reads the new memories since the folder was last curated (a delta — not a recall) and emits ops: create_page / merge_pages / delete_page / create_subfolder (bounded: depth ≤ 3, ≤ 8 sub-folders).
• Runs as an async operation (curate_folder task), same machinery as the mental-model refresh — no manual trigger.
• Curator pages use trigger {mode: delta, fact_types: [observation], exclude_mental_models: true, refresh_after_consolidation: true} so they synthesize from consolidated observations and stay current.

Changes

Server (hindsight-api-slim)

• New knowledge_pages table (PG + Oracle) + mission / managed / last_curated_at columns; added to BACKUP_TABLES; partial unique index on (folder, page-name) to make concurrent curator runs dedup-safe.
• api/okf.py — OKF serializer (frontmatter + body, index.md/log.md, tag/folder constellation graph).
• engine/knowledge_curator.py — the curator (LLM op plan + safe apply).
• MemoryEngine — folder/page CRUD, tree, curate, async submit + worker handler.
• /v1/default/banks/{bank}/knowledge-base/* endpoints (tree, folders, pages, nodes, graph, export).

Control plane

• knowledge-base-view.tsx — folder/file tree (create folder/page, missions, delete) + constellation toggle, OKF page panel, OKF bundle export. Sidebar entry + proxies + i18n (all locales).

Generated — OpenAPI + SDK clients + docs-skill regenerated.

Tests

• test_okf.py (unit), test_knowledge_base.py (HTTP: tree/move/cascade/graph/export), test_knowledge_curator.py (deterministic apply ops + dedup guard), test_knowledge_curator_e2e.py (hs_llm_core, full pipeline).

Open items (why draft)

• Page content quality with a weak model: the mental-model refresh's agentic reflect occasionally doesn't recall well → thin/placeholder content. The observation-only + exclude_mental_models trigger mitigates it; a forced recall+synthesize path is the likely follow-up.
• Off-mission page creation is reduced (prompt + per-folder new-memory delta) but still LLM-judgment-dependent.
• Curator concurrency is dedup-safe at the DB level (unique index); the per-folder in-process lock is best-effort.