You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CONFIGURATION.md
+70-1Lines changed: 70 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -418,6 +418,69 @@ Cross-session memory settings. All memories are scoped to the current project (i
418
418
|`injection_budget_tokens`|`number` (500–20000) |`4000`| Token budget for memory injection into `<session-history>`. |
419
419
|`auto_promote`|`boolean`|`true`| Promote eligible session facts to project memories automatically after historian or `/ctx-recomp` runs. When `false`, historian and recomp do not write any new memories — agents can still create memories explicitly via `ctx_memory write`, and existing memories continue to be injected and searched normally. |
420
420
|`retrieval_count_promotion_threshold`|`number`|`3`| Retrievals needed before a memory is auto-promoted to permanent. |
421
+
|`external`|`object`| See below |**User-config-only.** Long-term memory backend (Hindsight) — tees curated writes OUT and recalls them BACK once per session, plus an explicit-only `ctx_search` source. A cloned repo cannot redirect the endpoint or read a user's personal memory store. |
422
+
423
+
### `memory.external`
424
+
425
+
The `memory.external` block controls the **external memory backend** (Hindsight): a long-term companion store that holds curated memories OUT of the project (so they survive across projects, harness restarts, and (in future) the user's whole fleet) and recalls them BACK once per session. The local SQLite store remains the source of truth; the external store is a long-term companion.
426
+
427
+
**Security:** this entire block is **user-config-only**. `stripUnsafeProjectConfigFields()` in `src/config/project-security.ts` drops it from project-level config (parallel to `auto_update` and `sqlite`) — a cloned repo cannot redirect the endpoint, exfiltrate a user's personal memory store, or read a user's external memory by editing `magic-context.jsonc` in a project root. Set the block in `~/.config/opencode/magic-context.jsonc` (OpenCode) or `~/.pi/agent/magic-context.jsonc` (Pi).
428
+
429
+
```jsonc
430
+
{
431
+
"memory": {
432
+
"external": {
433
+
"provider":"off", // "off" (default) or "hindsight"
434
+
"endpoint":"http://10.0.0.1:8889", // required when provider is hindsight; Hindsight base URL
"project_bank":"mc-{name}-{id8}", // project bank name template; {name}=project basename, {id8}=first 8 chars of the project identity hash
437
+
"main_bank":"user-memories", // required when provider is hindsight; bank for user + global scope. Assumed to pre-exist; never created or modified by the plugin
438
+
"retain_sources": ["historian", "agent", "dreamer"], // which creation points tee (gate write-side, not read-side)
439
+
"tags": [], // static tags attached to every retained item
440
+
"recall": { // see `memory.external.recall` below
441
+
"enabled":true,
442
+
"timeout_ms":3000,
443
+
"max_tokens":2048,
444
+
"dedup_threshold":0.85,
445
+
"global_tags": [],
446
+
"global_from_prompt":false,
447
+
"search":true,
448
+
"mental_models":true,
449
+
"profile_mental_models": ["user-preferences"]
450
+
}
451
+
}
452
+
}
453
+
}
454
+
```
455
+
456
+
| Field | Type | Default | Description |
457
+
|-------|------|---------|-------------|
458
+
|`provider`|`"hindsight"`\|`"off"`|`"off"`| Backend implementation. `"off"` disables the whole feature. `"hindsight"` activates tee-on-write + once-per-session recall + the explicit `ctx_search` source. |
459
+
|`endpoint`|`string`| — | Required when `provider: "hindsight"`. Hindsight base URL (e.g. `http://10.0.0.1:8889`). Trailing slashes are stripped. SSRF-guarded. |
|`project_bank`|`string` (template) |`"mc-{name}-{id8}"`| Project-bank name template. Placeholders: `{name}` = sanitized project basename, `{id8}` = first 8 chars of the project identity hash. Created on first retain (PUT with the project bank mission). |
462
+
|`main_bank`|`string`| — | Required when `provider: "hindsight"`. Bank for `user` and `global` scope items. **Assumed to pre-exist; the plugin never creates or modifies it.**|
463
+
|`retain_sources`|`string[]`|`["historian","agent","dreamer"]`| Which creation points tee to the external backend. `historian` = fact promotion in `src/features/magic-context/memory/promotion.ts`. `agent` = `ctx_memory``write` / `update` in `src/tools/ctx-memory/tools.ts`. `dreamer` = user-memory promotion in `src/features/magic-context/user-memory/review-user-memories.ts`. Read paths (recall, search, mental-models) and W2 correctives (archive/update/verify corrective propagation) are NOT gated by this list. |
464
+
|`tags`|`string[]`|`[]`| Static tags attached to every retained item (in addition to the always-present `source:magic-context`, `category:<X>`, and the per-scope `project:<id>` / `scope:user` / `scope:global` tags). |
|`enabled`|`boolean`|`true`| Session-start recall from the external backend, merged into the context injection. Late results ride the m[1]`<external-memory>` delta; the first m[0] render awaits up to `timeout_ms` (cache-cold path). |
472
+
|`timeout_ms`|`number` (500–15000) |`3000`| Max wait for recall at the first render of a session (when the cache is already cold). Late results arrive as an m[1] delta on later passes. |
473
+
|`max_tokens`|`number` (256–8192) |`2048`| Per-slice token budget — project / profile / global each get this cap. The external block is NOT charged to the history or local-memory budget; it is bounded solely by this × 3 slices. |
474
+
|`dedup_threshold`|`number` (0.5–0.99) |`0.85`| Cosine similarity above which a recalled item is dropped as a duplicate of a local memory (or active user memory). Hash-only fallback when embeddings are unavailable. |
475
+
|`global_tags`|`string[]`|`[]`| Tag filter for the global (main-bank) recall slice, matched with `tags_match: "any"` (untagged content INCLUDED). `[]` = no filter sent (full autoRecall replacement). |
476
+
|`global_from_prompt`|`boolean`|`false`| Include an excerpt of the session's first user prompt in the global-slice recall query (the project name is always included). The first prompt is fixed for the session, so the query — and the frozen recall snapshot — stays deterministic across crash-recovery re-fires. |
477
+
|`search`|`boolean`|`true`| Expose the `ctx_search``"external"` source (project + main bank). **Explicit-only** — the auto-search hot path (every user prompt hint) NEVER hits it, even when this is `true`. |
478
+
|`mental_models`|`boolean`|`true`| Use Hindsight mental models as the fast path for the project and profile recall slices (single GET, server-refreshed), falling back to full recall when absent/empty. The global slice always uses full recall. |
479
+
|`profile_mental_models`|`string[]`|`["user-preferences"]`| Main-bank mental-model names (case-insensitive) used for the profile slice. The main bank is never modified by the plugin — create these manually. Project-bank mental models are seeded by the plugin (`project-conventions`, `project-decisions`) on the first successful retain. |
480
+
481
+
**Engine-agnostic interface.**`external-memory-provider.ts` defines a neutral `ExternalMemoryBackend` interface (mirrors the `EmbeddingProvider` pattern). Hindsight is the only shipped implementation today. Document identity is `mc:<scopeKey>:<category>:<hash>` (content-derived → idempotent re-retains upsert on the server; retries never duplicate). Bank routing: `scope: "project"` → `project_bank` (per project); `scope: "user"` and `scope: "global"` → `main_bank`. Project items carry `project:<id>` + `project-name:<basename>` tags; globals carry `scope:global` plus `origin-project:*` provenance tags and a `context` field that names the originating project so Hindsight's fact extractor links it as an entity.
482
+
483
+
**Failure semantics.** Hindsight calls are fire-and-forget; failures are logged, never thrown. The impl carries a circuit breaker (3 fails in 60s → open 5min → half-open probe) so a hung endpoint can't drag every plugin operation through its timeout. A 422 (memory-defense rejected content) is treated as a hard no and never retried; 404 on a missing bank is benign (returns empty for the slice). The bearer token is never logged. Cross-harness: OpenCode and Pi share the same banks (the impl is a single, neutral interface and the project bank name is project-derived, so both harnesses resolve the same bank).
Copy file name to clipboardExpand all lines: README.md
+5-3Lines changed: 5 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -201,7 +201,7 @@ Because it runs during idle time, the dreamer pairs well with local models, even
201
201
202
202
*The right memory at the right moment.* Every turn, active project memories and the compacted session history are injected automatically and cache-stably. On demand, the agent reaches for:
203
203
204
-
-**`ctx_search`**: one query across three layers at once: project **memories**, raw **conversation** history, and indexed **git commits**. Semantic embeddings with full-text fallback.
204
+
-**`ctx_search`**: one query across four layers at once: project **memories**, raw **conversation** history, indexed **git commits**, and (opt-in) the long-term **external** memory backend. Semantic embeddings with full-text fallback.
205
205
206
206
```
207
207
ctx_search(query="why did we pick event sourcing for orders")
@@ -213,15 +213,17 @@ Because it runs during idle time, the dreamer pairs well with local models, even
213
213
214
214
Recall works **across sessions** (a new session inherits everything) and **across harnesses** (write a memory in OpenCode, retrieve it in Pi).
215
215
216
+
> **Long-term memory***(opt-in, off by default)* tees curated writes (historian promotions, `ctx_memory` writes, dreamer user-memory promotions) to a Hindsight backend and recalls them BACK once per session as a `<external-memory>` block — across sessions, across harnesses, and across projects for the global slice. Same `ctx_search` "external" source (explicit-only, never on the auto-search hot path). User-config-only; a repo cannot redirect the endpoint. Configure under `memory.external`; see [CONFIGURATION.md](./CONFIGURATION.md#memoryexternal).
217
+
>
216
218
> **Auto search hints***(on by default)* run a background `ctx_search` each turn and whisper a "vague recall" when something relevant exists — like almost remembering a note you took. It appends only compact fragments, never full content; set `memory.auto_search.enabled: false` to turn it off. **Git commit indexing***(opt-in)* makes your project history semantically searchable as a fourth `ctx_search` source — enable with `memory.git_commit_indexing.enabled: true`.
217
219
218
220
### Agent tools at a glance
219
221
220
222
| Tool | Section | What it does |
221
223
|------|-------|-------------|
222
224
|`ctx_reduce`| Context | Queue stale tagged content for removal, cache-aware |
223
-
|`ctx_memory`| Capture | Write or delete durable cross-session memories |
|`ctx_memory`| Capture | Write or delete durable cross-session memories (project scope, plus `global` scope to the external main bank when configured) |
226
+
|`ctx_search`| Recall | Search memories, conversation history, git commits, and (explicit-only) the external long-term memory backend|
225
227
|`ctx_expand`| Recall | Decompress a history range back to the transcript |
226
228
|`ctx_note`| Recall | Deferred intentions and dreamer-evaluated smart notes |
227
229
|`ctx_skill_note`| Recall | Write back a per-skill note (gotcha/discovery/fix/workflow) for future loads |
"description": "External long-term memory backend (tee). USER config only.",
1387
+
"type": "object",
1388
+
"properties": {
1389
+
"provider": {
1390
+
"default": "off",
1391
+
"description": "External memory backend. 'hindsight' tees memory creations to a Hindsight service; 'off' disables (default). SECURITY: this whole block only honors USER-level config.",
1392
+
"type": "string",
1393
+
"enum": [
1394
+
"hindsight",
1395
+
"off"
1396
+
]
1397
+
},
1398
+
"endpoint": {
1399
+
"description": "Backend base URL (e.g. http://10.0.0.1:8889). Required when provider is hindsight.",
1400
+
"type": "string"
1401
+
},
1402
+
"api_key": {
1403
+
"description": "Bearer token for the backend (optional).",
1404
+
"type": "string"
1405
+
},
1406
+
"project_bank": {
1407
+
"default": "mc-{name}-{id8}",
1408
+
"description": "Bank name template for project-scoped items. Placeholders: {name}=project basename, {id8}=first 8 chars of the project identity hash.",
1409
+
"type": "string"
1410
+
},
1411
+
"main_bank": {
1412
+
"description": "Bank for user- and global-scoped items. Required when provider is hindsight. Assumed to pre-exist; never created or modified.",
"description": "Session-start recall from the external backend, merged into the context injection (default: true).",
1460
+
"type": "boolean"
1461
+
},
1462
+
"timeout_ms": {
1463
+
"default": 3000,
1464
+
"description": "Max wait for recall at the first render of a session (already cache-cold). Late results ride the m[1] delta. (default: 3000)",
1465
+
"type": "number",
1466
+
"minimum": 500,
1467
+
"maximum": 15000
1468
+
},
1469
+
"max_tokens": {
1470
+
"default": 2048,
1471
+
"description": "Token budget per recall slice (project / profile / global each). (default: 2048)",
1472
+
"type": "number",
1473
+
"minimum": 256,
1474
+
"maximum": 8192
1475
+
},
1476
+
"dedup_threshold": {
1477
+
"default": 0.85,
1478
+
"description": "Cosine similarity above which a recalled item is dropped as a duplicate of a local memory. Hash-only fallback when embeddings are unavailable. (default: 0.85)",
1479
+
"type": "number",
1480
+
"minimum": 0.5,
1481
+
"maximum": 0.99
1482
+
},
1483
+
"global_tags": {
1484
+
"default": [],
1485
+
"description": "Tag filter for the global (main-bank) recall slice, matched with tags_match 'any' (untagged content INCLUDED). Empty = no filter (full autoRecall replacement).",
1486
+
"type": "array",
1487
+
"items": {
1488
+
"type": "string"
1489
+
}
1490
+
},
1491
+
"global_from_prompt": {
1492
+
"default": false,
1493
+
"description": "Include an excerpt of the session's FIRST user prompt in the global-slice recall query (the project name is always included). The first prompt is fixed for the session, so the query — and the frozen recall snapshot — stays deterministic. Default false: pure template query.",
1494
+
"type": "boolean"
1495
+
},
1496
+
"search": {
1497
+
"default": true,
1498
+
"description": "Expose the ctx_search 'external' source (project + main bank). (default: true)",
1499
+
"type": "boolean"
1500
+
},
1501
+
"mental_models": {
1502
+
"default": true,
1503
+
"description": "Use Hindsight mental models as the fast path for the project/profile recall slices (single GET, server-refreshed), falling back to recall when absent/empty. (default: true)",
1504
+
"type": "boolean"
1505
+
},
1506
+
"profile_mental_models": {
1507
+
"default": [
1508
+
"user-preferences"
1509
+
],
1510
+
"description": "Main-bank mental-model names (case-insensitive) used for the profile slice. The main bank is never modified by the plugin — create these manually.",
0 commit comments