|
| 1 | +--- |
| 2 | +name: dream |
| 3 | +description: Consolidate a markdown knowledge base — merge near-duplicates, flag contradictions, prune stale notes, ingest inbox, refresh from sources, split, and rebuild the index. Use when a KB is noisy or stale, on a schedule, or to forget notes. Diff-for-approval; supports --auto. |
| 4 | +--- |
| 5 | + |
| 6 | +# dream — KB memory consolidation |
| 7 | + |
| 8 | +## Overview |
| 9 | + |
| 10 | +Dream is the maintenance pass that keeps a markdown knowledge base compact and |
| 11 | +retrievable — the agent equivalent of sleep consolidating the day's memories. Run it when |
| 12 | +the KB feels noisy (duplicates, contradictions, sprawl, stale facts), on a schedule, or |
| 13 | +to forget specific notes. Everything is proposed as a diff first; nothing changes until |
| 14 | +you approve (or you run `--auto`). |
| 15 | + |
| 16 | +Dream is KB-agnostic and generic. It never assumes a fixed location. |
| 17 | + |
| 18 | +## Invocation |
| 19 | + |
| 20 | +- `dream [<path>]` — consolidate the KB at `<path>`. If omitted, use `$KB_DIR`; if that |
| 21 | + is unset, ask for the path. Examples: `dream ./docs/kb`, `dream $KB_DIR`. |
| 22 | +- `dream --auto [<path>]` — non-interactive: apply merges and prunes silently, skip |
| 23 | + contradictions (they need human judgment), guarded by a lock file. |
| 24 | +- `dream forget "<query>" [<path>]` — find notes matching a query and propose deletion |
| 25 | + (same diff-for-approval flow as consolidation). |
| 26 | + |
| 27 | +A note is any `*.md` file with YAML frontmatter. Map missing fields gracefully: |
| 28 | + |
| 29 | +```yaml |
| 30 | +--- |
| 31 | +name: type-short-slug # equals the filename stem |
| 32 | +description: one line, keyword-front-loaded |
| 33 | +type: tech # user | feedback | project | reference | tech | ... |
| 34 | +tags: [topic, topic] |
| 35 | +sources: ["https://example.com/llms.txt"] |
| 36 | +created: 2026-06-01 |
| 37 | +updated: 2026-06-13 |
| 38 | +pinned: false # optional — pinned notes are never pruned or merged |
| 39 | +confidence: 0.8 # optional — 0..1; low + generic => prune candidate |
| 40 | +--- |
| 41 | +``` |
| 42 | + |
| 43 | +- No `type` => `unknown`. No `created`/`updated` => use file mtime. No `confidence` => |
| 44 | + neutral. No `pinned` => false. |
| 45 | + |
| 46 | +## The pass (run strictly in order) |
| 47 | + |
| 48 | +Work in memory until the apply step; modify nothing early. |
| 49 | + |
| 50 | +1. **Resolve the KB.** Confirm `<path>` holds `memory/` (notes) and ideally `MEMORY.md` |
| 51 | + (index), `raw/inbox/` (captures), `raw/` (source docs), and a state file. If the |
| 52 | + layout differs, operate on whatever `*.md` files exist and adapt. |
| 53 | +2. **Load.** Read the index, every note, `raw/inbox/*`, source docs, and state. |
| 54 | +3. **Ingest inbox.** Promote durable, general facts from `raw/inbox/*` into `memory/` |
| 55 | + (create or merge notes). Delete inbox files once distilled (they are ephemeral). Keep |
| 56 | + `raw/` source docs (immutable ground truth); record processed ones in state. |
| 57 | +4. **Dedupe and merge (near-duplicates).** Group notes by `type`. Two notes are |
| 58 | + near-duplicates when: same `type`; significant-noun overlap above ~60% (a proxy for |
| 59 | + cosine similarity > 0.9); neither is `pinned`. Draft a merged note that is more |
| 60 | + complete and specific than either, keeping the clearest, most recent phrasing. |
| 61 | + Lossless of meaning, not of words. |
| 62 | +5. **Contradictions.** Two notes assert opposing facts on the same topic (for example |
| 63 | + "deploy to ECS" vs "deploy to Vercel"). Proposed winner: more recent with higher |
| 64 | + confidence. Present both for A/B/skip (interactive); skip in `--auto`. |
| 65 | +6. **Split.** If a note holds multiple distinct facts, split into atomic notes (one fact |
| 66 | + per file), renamed to `<type>-<slug>.md`. |
| 67 | +7. **Validate frontmatter.** Every note has `name` (equals filename stem), `description`, |
| 68 | + `type`, `tags`, `created`, `updated`. Fix the `<type>-` filename prefix if the type |
| 69 | + changed. No nested `metadata:` blocks. |
| 70 | +8. **Refresh from sources.** For notes with a `sources:` URL whose `updated:` is stale |
| 71 | + (older than ~30 days) or whose facts look outdated, fetch the source (prefer |
| 72 | + `llms.txt`), update the note, bump `updated:`. Skip silently when offline. |
| 73 | +9. **Compact.** Trim each note to essential facts; drop filler. Aim for ~25 lines or |
| 74 | + fewer. |
| 75 | +10. **Prune candidates.** A note is a prune candidate when any holds: |
| 76 | + - its `type` has a retention policy and it is older than the policy's days (compare |
| 77 | + `created`/`updated` to today); |
| 78 | + - `confidence` is below 0.3 AND it has no project-unique info (no paths, |
| 79 | + identifiers, or domain nouns); |
| 80 | + - it is wrong, obsolete, or out of scope. |
| 81 | + Never prune `pinned: true`. Prefer refreshing (step 8) over deleting merely because |
| 82 | + a note is old. |
| 83 | +11. **Relink and retag.** Fix `[[slug]]` references; ensure no orphans (each note links |
| 84 | + at least one other); merge tag sprawl into the controlled vocabulary; drop links to |
| 85 | + deleted notes. |
| 86 | +12. **Rebuild index.** Regenerate `MEMORY.md` from surviving notes, grouped by `type`, |
| 87 | + one line each: `[Title](file) — hook`. |
| 88 | +13. **Verify scope.** Confirm no secrets, hosts, or confidential facts slipped in; |
| 89 | + remove any that did. |
| 90 | +14. **Stamp.** Set `state.last_dream` to today. |
| 91 | + |
| 92 | +## Retention policies (load in step 1) |
| 93 | + |
| 94 | +Default per-type retention, in days (`null` means keep forever). Override via a |
| 95 | +`retention:` map in the KB's state or config file. |
| 96 | + |
| 97 | +- `feedback`: 180 |
| 98 | +- `project`: null |
| 99 | +- `reference`: null |
| 100 | +- `tech`: 365 |
| 101 | +- `user`: null |
| 102 | +- `unknown`: 90 |
| 103 | + |
| 104 | +## Diff report (interactive) |
| 105 | + |
| 106 | +Print exactly this before any change. Omit any section that has zero items. |
| 107 | + |
| 108 | +```text |
| 109 | +## dream — consolidation report |
| 110 | +
|
| 111 | +Merges (<N>): |
| 112 | + [note:<file1>] + [note:<file2>] -> "<merged summary, ~80 chars>" |
| 113 | +
|
| 114 | +Conflicts (<N>): |
| 115 | + [note:<fileA>] vs [note:<fileB>] — "<topic>" [A/B/skip] |
| 116 | +
|
| 117 | +Prune (<N>): |
| 118 | + [note:<file>] — <type>, <age>d old |
| 119 | +
|
| 120 | +Inbox ingested (<N>), refreshed (<N>), split (<N>), relinked (<N>). |
| 121 | +
|
| 122 | +Proposed: <N> merges, <N> prunes, <N> conflicts. Apply? [Y/n] |
| 123 | +``` |
| 124 | + |
| 125 | +If there are zero proposals, print `Dream complete. No duplicate, contradictory, or stale |
| 126 | +notes found.` and stop. |
| 127 | + |
| 128 | +## Resolve and apply (interactive) |
| 129 | + |
| 130 | +- For each conflict, collect `A`, `B`, or `skip` (empty input = skip). |
| 131 | +- Final confirm: `Apply? [Y/n]`. `n` or `no` prints `Cancelled. No changes made.` and |
| 132 | + stops. |
| 133 | +- On confirm, apply in this order: |
| 134 | + 1. **Merges** — write the merged note (`<type>-<slug>.md`; `type` and `updated` = today; |
| 135 | + `confidence` = the higher of the two originals; a `source: dream` marker), delete |
| 136 | + both originals, and rewrite inbound links to point at the merged slug. |
| 137 | + 2. **Contradictions** — delete the loser of each resolved A/B pair; skipped pairs stay |
| 138 | + untouched. |
| 139 | + 3. **Prunes** — delete the note and its index line. |
| 140 | + 4. **Split, refresh, compact, relink, index rebuild** — apply the file writes. |
| 141 | + 5. **Stamp** state, and run the KB's sync step only if one exists and only after the |
| 142 | + user confirms — never auto-push. |
| 143 | + |
| 144 | +## Auto mode (`--auto`) |
| 145 | + |
| 146 | +- **Concurrency guard.** Before working, check `${TMPDIR:-/tmp}/dream_auto.lock`. If it |
| 147 | + exists and is less than 10 minutes old, print |
| 148 | + `[dream --auto] another run in progress — skipping.` and stop. Otherwise write the lock |
| 149 | + (current timestamp); delete it on every exit path. |
| 150 | +- Run load and analyze (steps 1–3). Apply merges and prunes silently (no diff, no |
| 151 | + prompt). Skip contradictions (human judgment). Apply split, refresh, compact, relink, |
| 152 | + index rebuild, and stamp. |
| 153 | +- Print: `[dream --auto] kb=<path> merged=<N> pruned=<N> conflicts_skipped=<N>`. |
| 154 | +- If contradictions were skipped, leave a single reminder note so the human runs |
| 155 | + interactive `dream` to resolve them. First search for an existing `dream-auto` reminder |
| 156 | + (by `source: dream-auto`); skip storing if one already exists. |
| 157 | + |
| 158 | +## forget |
| 159 | + |
| 160 | +`dream forget "<query>" [<path>]` searches notes by keyword and semantic match, lists |
| 161 | +them, and proposes deletion as a diff: |
| 162 | + |
| 163 | +```text |
| 164 | +[note:<file>] — "<title>" [Y/n] |
| 165 | +``` |
| 166 | + |
| 167 | +Apply only approved deletions (and their index lines). Pinned notes are listed but not |
| 168 | +deleted unless explicitly confirmed per note. |
| 169 | + |
| 170 | +## Principles |
| 171 | + |
| 172 | +- **Lossless of meaning** — compaction removes words and redundancy, never facts. |
| 173 | +- **Atomic** — one fact per file after the pass. |
| 174 | +- **Retrieval-first** — every `description` lets an agent judge relevance from the index |
| 175 | + alone; every note is reachable via tags and `[[links]]`. |
| 176 | +- **Conservative deletes** — prune only what is wrong, redundant, or out of scope; refresh |
| 177 | + rather than delete just because old. |
| 178 | +- **Idempotent** — running twice in a row changes nothing the second time. |
| 179 | +- **No surprises** — diff before write; never auto-push or auto-sync without approval. |
| 180 | + |
| 181 | +## Checklist |
| 182 | + |
| 183 | +- [ ] Resolved KB path from the argument or `$KB_DIR`; never assumed a fixed location. |
| 184 | +- [ ] Held everything in memory; no writes before the diff. |
| 185 | +- [ ] Inbox ingested and deleted; source docs kept. |
| 186 | +- [ ] Near-duplicates merged (same type, >60% noun overlap, not pinned). |
| 187 | +- [ ] Contradictions resolved (A/B) or skipped (`--auto`). |
| 188 | +- [ ] Stale notes refreshed from sources before pruning; pinned notes untouched. |
| 189 | +- [ ] Index rebuilt; no orphans; no broken `[[links]]`. |
| 190 | +- [ ] Scope verified (no secrets, hosts, or confidential facts). |
| 191 | +- [ ] State stamped; lock removed (`--auto`). |
| 192 | + |
| 193 | +## Resources |
| 194 | + |
| 195 | +- If the KB ships its own `DREAM.md` or `AGENTS.md`, read it first and align with its note |
| 196 | + format, controlled vocabulary, and sync steps. |
0 commit comments