docs: dreamer refresh (v70 fence, lenient tier extraction)

Author: ualtinok

ARCHITECTURE.md

@@ -97,7 +97,7 @@ The long-history pipeline. Tiered compartments + deterministic decay renderer (r
 
 1. **Trigger** (`compartment-trigger.ts`): threshold-relative pressure (`context_limit × execute_threshold × 5%`, clamped 5k–50k), commit clusters, and TC-chunked unsummarized-tail size (`≥ triggerBudget × 3`), while protecting the live tail. Runs off the in-memory tail (zero `opencode.db` reads steady-state); hands the resolved boundary snapshot to the runner so the historian sees exactly what the fire decision saw. On the Rust/subc module leg, the skip discriminant (`trigger_false`, `no_models`, etc.) is written durably to `HistorianDurableState.last_no_fire` (change-gated to avoid hot-path writes, and cleared on fire), and `trigger_false` details carry quantized measurements. The transform handler recognizes the `mc-historian:` session namespace as self-owned and returns an identity pass-through to prevent self-recursive loops.
 2. **Produce** (`compartment-runner-incremental.ts`): runs the historian subagent on the raw chunk above the last compartment boundary with a **bounded** prompt (no full state dump) — 4 rotating seed compartments + the last 6 persisted compartments + the project-memory block for fact dedup. Emits each compartment with 4 paraphrase tiers (`p1` verbose → `p4` anchor-only), an `importance` (decay-rate semantics), an `episode_type`, a `<facts>` block in the 5-category taxonomy, and an `<events>` block. On the Rust/subc side, the run uses a calibrated sampling temperature of 0.1 and a generous `max_output_tokens` of 32k with a 600s await budget. Truncation is tracked via `ProducerOutput.length_capped` and validation rejections write `last_failure` with validation errors and length-cap hints. Producer session IDs (`mc-historian:`) include a bound-session hash to isolate concurrent subagent lineages and prevent `TerminalRunMismatch` collisions. During assembly, enforce a `min_chunk_tokens` substance floor to avoid spawning a producer on near-empty content, but bypass this floor when the session is in emergency (≥95% usage) or on verbatim-tail profiles where folding is the sole reclaim path (`fold_is_only_reclaim` is true).
-3. **Parse + validate**: `parseCompartmentOutput` + `validateHistorianOutput` (sparse-but-strictly-increasing ordinals for consumer legs, non-overlapping ranges, correct `unprocessed_from`). Live coverage validation absorbs mid-span `system` role ordinals safely without rejecting the chunk. Apply strict gap healing to absorb tool-only gaps while rejecting narrative-bearing gaps to trigger re-reads on repair attempts.
+3. **Parse + validate**: `parseCompartmentOutput` + `validateHistorianOutput` (sparse-but-strictly-increasing ordinals for consumer legs, non-overlapping ranges, correct `unprocessed_from`). Live coverage validation absorbs mid-span `system` role ordinals safely without rejecting the chunk. Lenient tier extraction permits mismatched closing tags (e.g. `<p1>...</p2>`), terminating by any closing tier tag or next opener, with legacy migration v70 healing stranded compartments. Apply strict gap healing to absorb tool-only gaps while rejecting narrative-bearing gaps to trigger re-reads on repair attempts.
 4. **Discard-last boundary healing**: if the historian consumed to the chunk edge with weak lookahead, the last (lookahead-free) compartment is not persisted; the next run re-reads it at the head with full lookahead. Guarded by progress (`k≥2`) and emergency-disabled. Skips unanchored fact, observation, and primer promotion on the discarded tail to prevent double-storing when the range is re-processed.
 5. **Store**: publish transaction appends compartments with tier columns. Promotable facts promote to project memory (exact-dedup); `user_observations` stored only when `dreamer.user_memories.enabled` (privacy gate). Events → `compartment_events`. Compartment-chunk embeddings generated on publish (memory-gated). Publish defers a compaction-marker move (see Subsystems) and signals a deferred history refresh — it does NOT force a bust. When reattaching, the historian subscriptions are always re-drained from the start (persisted `producer_cursor` is killed to prevent dropping output). First published compartment after an empty bootstrap forces a HARD fold (since a SOFT delta is impossible without an existing boundary).
 6. **Render (decay)**: `decay-render.ts` (shared OpenCode + Pi) picks one tier per compartment via `decay-curve.ts`: half-life `H = H50·2^((I−50)/D)/max(p,0.10)` (`H50=24`, `D=25`), log-cost tier boundaries `[0.201,0.729,1.322,2.587]`, budget pressure `p` once per pass. Older / lower-importance / higher-pressure compartments demote oldest-first; past the archive boundary they render P4/self-close or drop. Self-tunes as the context window changes — no LLM call. Legacy (pre-v2) rows render P3 (if they carry a `U:` line) else P4.
@@ -144,7 +144,7 @@ Background maintenance (V2: per-task cron scheduling). A process-wide 15-min tim
 
 ## Storage & migrations
 
-`storage-db.ts` creates the schema and runs versioned migrations (`migrations.ts`, currently v1–v67). `LATEST_SUPPORTED_VERSION` is a schema fence — it MUST be bumped with every new migration (a unit test asserts it equals the highest migration), and a stale value makes the DB refuse to open after the migration applies. Schema helpers `ensureColumn()` + `healAllNullColumns()` (defined in `storage-schema-helpers.ts` to prevent cycles between `storage-db` and `migrations`) backfill upgraded DBs even if a migration row is lost. New session-scoped tables must be added to `clearSession()`. A bulletproof `MAGIC_CONTEXT_TEST_DATA_DIR` guard keeps the test suite off the live DB (running `bun test` once migrated a live DB and fail-closed running binaries). SQLite binds must use SPREAD positional args, never the array form (`bun:sqlite` binds a lone array positionally; `node:sqlite` reads it as named params and throws). For branch forks in Pi, copy durable session state (compartments, tags, pending operations, and session metadata) to the new session via `copySessionStateForClone()` in `src/features/magic-context/storage-clone.ts`. Run this copy inside an immediate SQLite transaction, filtering and mapping message ordinals and tag composite keys to the copied branch entries, and clearing cached cache bytes to trigger fresh rematerialization.
+`storage-db.ts` creates the schema and runs versioned migrations (`migrations.ts`, currently v1–v70). `LATEST_SUPPORTED_VERSION` is a schema fence — it MUST be bumped with every new migration (a unit test asserts it equals the highest migration), and a stale value makes the DB refuse to open after the migration applies. Schema helpers `ensureColumn()` + `healAllNullColumns()` (defined in `storage-schema-helpers.ts` to prevent cycles between `storage-db` and `migrations`) backfill upgraded DBs even if a migration row is lost. New session-scoped tables must be added to `clearSession()`. A bulletproof `MAGIC_CONTEXT_TEST_DATA_DIR` guard keeps the test suite off the live DB (running `bun test` once migrated a live DB and fail-closed running binaries). SQLite binds must use SPREAD positional args, never the array form (`bun:sqlite` binds a lone array positionally; `node:sqlite` reads it as named params and throws). For branch forks in Pi, copy durable session state (compartments, tags, pending operations, and session metadata) to the new session via `copySessionStateForClone()` in `src/features/magic-context/storage-clone.ts`. Run this copy inside an immediate SQLite transaction, filtering and mapping message ordinals and tag composite keys to the copied branch entries, and clearing cached cache bytes to trigger fresh rematerialization.
 
 ## Session modes
 

STRUCTURE.md

@@ -158,7 +158,7 @@ Unless specified otherwise, TypeScript paths are relative to `packages/plugin/`
 - `src/features/magic-context/storage-schema-helpers.ts`: Implement schema-mutation and NULL-healing helpers to avoid dependency cycles between database creation and migrations.
 - `src/features/magic-context/storage-meta-persisted.ts`: Read and write per-session persisted scalars and JSON blobs.
 - `src/features/magic-context/fail-closed-block.ts`: Implement loud fail-closed blocking when Magic Context cannot operate on a session.
-- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v67 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep).
+- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v70 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep).
 - `src/features/magic-context/message-index.ts`: FTS-backed raw-message index for `ctx_search`.
 - `src/features/magic-context/search.ts`: Unified retrieval over memories, raw messages, git commits, and session/smart notes.
 - `src/features/magic-context/session-project-storage.ts`: Persist session-to-project bindings and repair mis-scoped compartment chunk embeddings.