diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index de7a0999f..c864d3e04 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -128,11 +128,158 @@ Background maintenance, one worker at a time. Eligible projects detected on `mes `storage-db.ts` creates the schema and runs versioned migrations (`migrations.ts`, currently v1–v36). `LATEST_SUPPORTED_VERSION` is a schema fence — it MUST be bumped with every new migration (a unit test asserts it equals the highest migration), and a stale value makes the DB refuse to open after the migration applies. `ensureColumn()` + `healAllNullColumns()` backfill upgraded DBs even if a migration row is lost. New session-scoped tables must be added to `clearSession()`. A bulletproof `MAGIC_CONTEXT_TEST_DATA_DIR` guard keeps the test suite off the live DB (running `bun test` once migrated a live DB and fail-closed running binaries). SQLite binds must use SPREAD positional args, never the array form (`bun:sqlite` binds a lone array positionally; `node:sqlite` reads it as named params and throws). -## Session modes - -Three effective modes; the heavier features (historian, nudges, adjunct injection) are gated, while tag/drop plumbing stays on everywhere. - -| Feature | Primary + `ctx_reduce_enabled: true` | Primary + `ctx_reduce_enabled: false` | Subagents | +**Compartment events (v2, stored-not-rendered):** +- Purpose: Persist historian-extracted `causal_incident` / `trajectory_correction` events as a corpus for future dreamer aggregation; never rendered into the prompt in v2.0. +- Location: `compartment_events` table (migration v23); `insertCompartmentEvents` / `getCompartmentEvents`. +- Pattern: Anchored to durable compartment ids (`at_compartment` → id at publish); discarded-tail events filtered; cleared on session deletion. + +**Message-history index:** +- Purpose: FTS-backed raw user/assistant message search outside the transform hot path. +- Location: `src/features/magic-context/message-index.ts`, `src/features/magic-context/message-index-async.ts` +- Pattern: Async reconciliation + live event indexing + pure-query reads. + +**Git-commit index:** +- Purpose: Per-project HEAD-only commit corpus for `ctx_search` integration. +- Location: `src/features/magic-context/git-commits/` +- Pattern: NUL-free git log reader + FTS index + embedding side table; populated by dream timer. + +**Dream queue and lease:** +- Purpose: Run at most one dream worker at a time and survive restarts. +- Location: `src/features/magic-context/dreamer/queue.ts`, `src/features/magic-context/dreamer/lease.ts`, `src/features/magic-context/dreamer/storage-dream-state.ts`, `src/features/magic-context/dreamer/storage-dream-runs.ts` +- Pattern: SQLite-backed queue plus cooperative lease lock plus durable run-history table. + +**Key-files pinning:** +- Purpose: Inject up to N project files into the system prompt as `` content for the active session. +- Location: `src/features/magic-context/key-files/identify-key-files.ts`, `src/features/magic-context/key-files/read-stats.ts`, `src/features/magic-context/key-files/storage-key-files.ts` +- Pattern: Per-session selection by Dreamer; budget-bound rendering; symlink-safe realpath check. + +**User memory pipeline:** +- Purpose: Extract user behavioral observations from historian output (the v2 `` block), collect candidates, and promote recurring patterns to stable global user memories. +- Location: `src/features/magic-context/user-memory/storage-user-memory.ts`, `src/features/magic-context/user-memory/review-user-memories.ts` +- Pattern: Historian extracts candidates **only when `dreamer.user_memories.enabled`** (privacy gate, enforced post-commit best-effort on both harnesses); dreamer reviews with a multi-session recurrence gate and promotes; the baseline set renders into m[0] `` (new promotions into m[1]). user_memories are globally scoped (no `project_path`). + +**TUI ↔ server RPC:** +- Purpose: Localhost RPC for sidebar data, status/recomp dialogs, and TUI-action consumption. +- Location: `src/shared/rpc-server.ts`, `src/shared/rpc-client.ts`, `src/shared/rpc-utils.ts`, `src/shared/rpc-types.ts`, `src/shared/rpc-notifications.ts`, `src/plugin/rpc-handlers.ts` +- Pattern: Server publishes ephemeral port; TUI plugin polls for state and pushes notifications via the message queue. + +**Plugin message bus (legacy):** +- Purpose: Historical SQLite-backed TUI ↔ server bus, retained for migration compatibility. +- Location: `src/features/magic-context/plugin-messages.ts` +- Pattern: Vestigial — superseded by RPC. Module remains for forward-compat with older TUI plugin versions that may still poll it; no active runtime callers in current code. + +**Compaction markers (deferred drain, plan v6):** +- Purpose: Inject OpenCode-compatible compaction boundaries into the message table so `filterCompacted` stops at historian's last compartment boundary, shrinking the transform-input array. Marker movement is deferred from historian publish into the next materializing transform pass so a single cache-bust cycle covers both the `` rebuild AND the marker boundary advance. +- Location: `src/features/magic-context/compaction-marker.ts`, `src/hooks/magic-context/compaction-marker-manager.ts`, `src/features/magic-context/storage-meta-persisted.ts` (pending blob helpers). +- Pattern: Historian incremental runner writes the prospective new boundary (`{ordinal, endMessageId, publishedAt}`) into `session_meta.pending_compaction_marker_state` in the same transaction that publishes new compartments. The next consuming transform pass that drains `deferredHistoryRefreshSessions` calls `applyDeferredCompactionMarker(...)`, which validates the pending target against the latest stored compartment via `getCompartmentsByEndMessageId(...)` plus an OpenCode-message existence check via `getOpenCodeMessageById(...)`, then sequences `removeCompactionMarker` → `injectCompactionMarker`. Returns a tagged `MarkerUpdateOutcome` (`applied` | `already-current` | `stale-skip` | `retryable-failure`); only `retryable-failure` preserves the deferred-history signal so the next pass retries. CAS-clear (`clearPendingCompactionMarkerStateIf`) on success guards against publish/drain races within and across processes. Eager paths (`/ctx-flush`, `/ctx-recomp`) call the marker manager directly and CAS-clear any stale pending blob. Restart-safe: hook init calls `getSessionsWithPendingMarker(...)` to rehydrate deferred sets so the next pass after restart still drains. `event-handler` CAS-clears pending state on `session.compacted` (provider already advanced the boundary) and on `session.deleted` via cascade. Raw-history readers strip `summary=true` / `finish="stop"` rows to preserve original ordinals. Stable feature, default `compaction_markers: true` since v0.16.x; deferred drain since v0.19 (plan v6). + +**Auto-update checker:** +- Purpose: Self-update the cached `@latest` plugin install once per plugin process — OpenCode's plugin cache no longer auto-updates. +- Location: `src/hooks/auto-update-checker/checker.ts`, `src/hooks/auto-update-checker/cache.ts`, `src/hooks/auto-update-checker/constants.ts` +- Pattern: Fires from plugin init with on-disk cross-process dedup; rewrites the install-directory dependency entry + `bun.lock` (or runs `npm install` under OpenCode's npm-managed cache). + +**Agent prompt pack:** +- Purpose: Keep hidden-agent identities and prompt text isolated from runtime wiring. +- Location: `src/agents/dreamer.ts`, `src/agents/historian.ts` (declares `HISTORIAN_AGENT` and `HISTORIAN_EDITOR_AGENT`), `src/agents/sidekick.ts`, `src/agents/magic-context-prompt.ts` +- Pattern: Constants plus prompt builders. + +**Content stripping and replay:** +- Purpose: Strip reasoning, inline thinking, placeholder shells, structural noise, processed images, merged-assistant reasoning, system-injected stripping, and caveman compression from messages, and replay those operations on every transform pass to maintain stable message content across OpenCode's message rebuilds. +- Location: `src/hooks/magic-context/strip-content.ts`, `src/hooks/magic-context/caveman.ts`, `src/hooks/magic-context/caveman-cleanup.ts`, `src/hooks/magic-context/sentinel.ts` +- Pattern: Stateless strip functions plus deterministic in-place sentinel replacement (preserves message-part array shape across passes); paired with persisted watermarks (`cleared_reasoning_through_tag`, `stripped_placeholder_ids`, `tags.caveman_depth`) read from `session_meta` and `tags`. Several strips are provider-aware: `stripReasoningFromMergedAssistants` runs only for `anthropic`; whole-message empty-sentinel writes a `[dropped]` placeholder for non-Anthropic providers so openai-compatible providers don't see empty assistant messages. + +**Protected-tail boundary (v3):** +- Purpose: Decide, per pass, which prefix of the raw tail is eligible for the historian and which suffix stays protected — from true-raw token sizes instead of user-turn counts, so sparse-user-turn sessions can't deadlock the historian (issue #132). +- Location: `src/hooks/magic-context/protected-tail-boundary.ts` (resolver), `src/hooks/magic-context/read-session-true-raw-tokens.ts` (ordinal-keyed token index, fed by cached per-tag token counts with live-tokenize fallback), `src/hooks/magic-context/compartment-trigger.ts` (trigger consumption). +- Pattern: Boundary offset anchors at `lastCompartmentEnd + 1`; token target `N` capped at `0.40 × usable`; pure function of (messages, usage, budget) — no persisted high-watermark (backward relaxation is the #132 fix). The trigger runs in the transform off the in-memory `args.messages` tail (zero opencode.db reads steady-state; the resolved snapshot is handed to the runner so the historian sees exactly what the fire decision saw), with content-stable range fingerprints for cross-view staleness validation. + +**ctx_reduce nudges (Channel 1 / Channel 2):** +- Purpose: Keep the agent reducing its own context without cache-busting mutations — Channel 1 appends a `` to tool outputs in `tool.execute.after` (persisted to OpenCode's DB, so the bytes are durable and replay for free); Channel 2 delivers a one-shot synthetic-user ceiling nudge near the execute threshold via the live-server client on step-boundary `message.updated` events (mid-turn "tool-calls" and final "stop") — the queued message lands at the next step so the agent is warned while the pile is still growing. +- Location: `src/hooks/magic-context/ctx-reduce-nudge.ts` (shared math: severity, `reclaimable ≥ usable/3` trigger), `src/hooks/magic-context/hook-handlers.ts` (Channel 1 injection), `src/hooks/magic-context/channel2-delivery.ts` (Channel 2 lease + delivery), `packages/pi-plugin/src/ctx-reduce-nudge-pi.ts` (Pi mirror: `tool_result` mutation + `agent_end` followUp). +- Pattern: Channel-1 baselines (per-session, in-memory) carry the measurement (`tailToolTokens`, `usableTokens`) the triggers evaluate; Channel 2 uses a cross-process CAS lease (`channel2_nudge_state`: pending → claimed → delivered) with full-predicate revalidation at delivery — unknown baseline never delivers, stale predicate cancels to re-armable, only confirmed sends consume the one-per-session cap. + +**Tiered emergency drop (≥85%):** +- Purpose: Replace need-blind routine tool drops with a target-headroom eviction at force-materialize pressure — reclaim down to `fixedFloor + 0.30 × (ceiling − fixedFloor)`, evicting tool outputs oldest-first across tiers (T3 misc → T2 edit/search → T1 navigation), with newest-20% recency reserves on T1/T2. +- Location: `src/hooks/magic-context/emergency-drop.ts` (pure planner), applied from `heuristic-cleanup.ts` / `heuristic-cleanup-pi.ts`. +- Pattern: Split tag sets — `floorTags` (FULL active live-window set, floor accounting) vs `tags` (tool-only `canDrop()` eviction candidates); `last_emergency_input_sample` is the idempotence latch (no re-drop until a fresh provider usage reading arrives). + +**Caveman text compression (experimental):** +- Purpose: Apply oldest-first age-tier text compression to user/assistant text outside the protected tail when `ctx_reduce_enabled=false`. +- Location: `src/hooks/magic-context/caveman.ts` +- Pattern: Four tiers (ultra/full/lite/untouched) keyed by raw-ordinal age within the non-protected region. Persisted per-tag `caveman_depth` enables byte-identical replay; depth escalation always recomputes from `source_contents` to avoid lossy double compression. + +**Synthetic todowrite injection:** +- Purpose: Inject a deterministic `tool_use`/`tool_result` pair so the agent sees current todo state through its native todowrite mental model, even when real todowrite tool calls have been dropped from the prefix. +- Location: `src/hooks/magic-context/todo-view.ts` (renderer + hash), `src/hooks/magic-context/transform-postprocess-phase.ts` (B7 logic), `src/features/magic-context/storage-meta-persisted.ts` (state persistence) +- Pattern: Capture-path is pure DB write; cache-busting-pass injects fresh and persists `(call_id, anchor_message_id, state_json)`; defer-pass replays from persisted state_json for byte-identical wire bytes. + +**Persisted session meta:** +- Purpose: Store per-session scalars and JSON blobs that must survive across transform passes and OpenCode restarts. +- Location: `src/features/magic-context/storage-meta-shared.ts`, `src/features/magic-context/storage-meta-persisted.ts`, `src/features/magic-context/storage-meta-session.ts`, `src/features/magic-context/storage-meta.ts` +- Pattern: `session_meta` SQLite table with `ensureColumn()` and versioned migrations; typed row interfaces with runtime guards; NULL coercion in `isSessionMetaRow()` so legacy rows don't trigger fallback-to-defaults on every read. + +**Cache-busting signals (plan v6):** +- Purpose: Surface durable per-pass facts the postprocess phase uses to decide whether the v12 deferred-history drain, the deferred-marker drain, and the deferred-materialization drain are eligible to fire — without re-reading transform state. +- Location: `src/hooks/magic-context/cache-busting-signals.ts`, threaded into `RunPostTransformPhaseArgs` (`historyRebuiltThisPass`, `historyRefreshExplicitBeforePrepare`, `compartmentInjectionRebuiltFromDb`, `canConsumeDeferredLate`, `phaseJustAwaitedPublication`, etc.). +- Pattern: Captured at well-defined points in `transform.ts` (e.g. `historyRefreshExplicitBeforePrepare` is read immediately before `prepareCompartmentInjection`, not later) so concurrent transform passes don't clobber each other's signals. The drain decision (`historyWasConsumedThisPass`) combines `historyRebuiltThisPass && (canConsumeDeferredLate || phaseJustAwaitedPublication || explicitRebuildHappened) && materializationSatisfied`. Degraded-cache state (null-boundary rebuild) is tracked by `degradedCacheCountBySession` in postprocess; entry logs in `inject-compartments.ts` and a warning at `DEGRADE_CACHE_WARNING_THRESHOLD=10` consecutive degraded rebuilds. + +**External memory backend (Hindsight):** +- Purpose: Hold curated project / user / global memories in an engine-agnostic external store so they survive across projects, harness restarts, and (in future) the user's whole fleet — with a once-per-session recall rendered into the context and explicit-only search reach. The local SQLite store remains the source of truth; the external store is a long-term companion. +- Location: `src/features/magic-context/memory/external-memory.ts` (init/tee/recall/remove/upsert facade), `src/features/magic-context/memory/external-memory-provider.ts` (engine-agnostic neutral types + `ExternalMemoryBackend` interface, mirrors the `EmbeddingProvider` pattern), `src/features/magic-context/memory/external-memory-hindsight.ts` (Hindsight impl — bank resolution, retain/recall/remove/mental-models/failed-retain, circuit breaker, SSRF guard), `src/features/magic-context/memory/external-recall.ts` (session-start orchestrator: 3-slice fan-out + dedup + trim + persist-before-settle), `src/features/magic-context/memory/external-recall-read.ts` (snapshot read + marker hash), `src/hooks/magic-context/inject-compartments.ts` (m[0]/m[1] render + `cachedM0ExternalRecallHash` marker), `src/tools/ctx-memory/tools.ts` and `src/features/magic-context/memory/promotion.ts` and `src/features/magic-context/user-memory/review-user-memories.ts` (write paths + W2 corrective propagation), `src/features/magic-context/search.ts` (explicit-only `external` source). Pi mirrors the same flow in `packages/pi-plugin/src/context-handler.ts`, `packages/pi-plugin/src/inject-compartments-pi.ts`, and the Pi ctx-search / ctx-memory tools. +- Pattern: Tee on every curated write (gated by `retain_sources`) → fire-and-forget, never blocks the local write. Recall fires once per session, fans out project / profile / global slices in parallel, semantic-dedups against local rows (cosine ≥ `dedup_threshold`, hash fallback), persists the snapshot to `session_meta` BEFORE the in-flight promise settles, and renders the `` block into m[1] only (the settled snapshot bakes into m[0] on the next HARD materialization — recall is not a materialization driver; `cached_m0_external_recall_hash` is a carried marker, never a HARD bust trigger). W2 correctives propagate archive/update/verify to the external store fire-and-forget after the local commit; `merge` is a no-op (engine's domain). Document identity `mc:::` is content-derived → idempotent re-retains upsert on the server, retries never duplicate. Security: `memory.external` is user-config-only (stripped from project config by `stripUnsafeProjectConfigFields` in `src/config/project-security.ts`, parallel to `auto_update` / `sqlite`); Hindsight impl adds a circuit breaker (3 fails/60s → open 5min → half-open probe), SSRF guard, `redirect: "error"`, 422 never-retry, token never logged. + +## Entry Points + +**CLI entry:** +- Location: `packages/cli/src/index.ts` (separate `@cortexkit/magic-context` package). +- Triggers: Executed as the unified `magic-context` bin target via `npx @cortexkit/magic-context@latest `. +- Responsibilities: Detect installed harnesses (OpenCode, Pi) and dispatch `setup` / `doctor` / `migrate` flows; print usage on unknown commands. + +**Plugin entry:** +- Location: `src/index.ts` +- Triggers: OpenCode loads the package entry declared in `package.json`. +- Responsibilities: Load config; surface config-warning toasts/ignored-messages; disable the plugin when conflicting plugins are detected (DCP, OMO context-management, OpenCode auto-compaction); register hidden agents (`historian`, `historian-editor`, `dreamer`, `sidekick`); start RPC server; start auto-update checker; start dream-schedule timer; wire hooks, commands, and tools. + +**TUI plugin entry:** +- Location: `src/tui/index.tsx` (separate `./tui` export from `package.json`). +- Triggers: OpenCode TUI loads the entry declared in `tui.json`. +- Responsibilities: Register Magic Context command-palette entries (with dual-path fallback for `api.keymap.registerLayer` vs legacy `api.command.register`); register sidebar slot; mount RPC-backed data layer. + +**Message transform entry:** +- Location: `src/plugin/messages-transform.ts` +- Triggers: `experimental.chat.messages.transform` +- Responsibilities: Defensive wrapper around the magic-context hook's transform — catches transient `SQLITE_BUSY`/`SQLITE_LOCKED` errors and other failures, persists summary to `session_meta.last_transform_error`, and falls back to unmodified messages so OpenCode's prompt loop always proceeds. + +**System-prompt transform entry:** +- Location: `src/hooks/magic-context/system-prompt-hash.ts` +- Triggers: `experimental.chat.system.transform` +- Responsibilities: Inject ``, ``, `` adjunct blocks and Magic Context guidance text; persist `system_prompt_hash` for cache-stability decisions; skip injection for OpenCode's internal `title`/`summary`/`compaction` agents and any agents matched by user-configured `system_prompt_injection.skip_signatures`. + +**Event entry:** +- Location: `src/plugin/event.ts` +- Triggers: OpenCode session and message lifecycle events. +- Responsibilities: Forward lifecycle events to the runtime event handler — `message.updated` (usage tracking, model drift detection, message-index live updates, Channel-2 ceiling-nudge delivery on step boundaries), `message.removed` (tag/index cleanup, anchor cleanup), `session.deleted` (full-session cleanup). The historian trigger decision no longer runs here — it lives in the transform, fed by the in-memory message tail (the event handler has no message array and the old per-streaming-delta DB read froze the event loop on large sessions). + +**Tool entry:** +- Location: `src/plugin/tool-registry.ts` +- Triggers: Plugin initialization. +- Responsibilities: Open storage, normalize arg schemas, and expose the supported tool set. + +**Tool definition entry:** +- Location: `src/index.ts` (`tool.definition` hook calls `recordToolDefinition`) +- Triggers: OpenCode `tool.definition` hook (per tool per flight). +- Responsibilities: Record tool description and parameter token counts per `(provider, model, agent, tool_id)` for sidebar token attribution, with content-fingerprint short-circuit to avoid re-measuring stable definitions. + +**RPC server entry:** +- Location: `src/shared/rpc-server.ts` (started from `src/index.ts`) +- Triggers: Plugin initialization. +- Responsibilities: Bind localhost RPC server on ephemeral port; publish port via `session_meta` for TUI discovery; serve sidebar/status/recomp/notification endpoints registered by `src/plugin/rpc-handlers.ts`. + +## Session Modes + +Magic Context runs in three effective modes depending on `ctx_reduce_enabled` and whether the session is a subagent. The mode decides which of the heavier features (historian, nudges, prompt-adjunct injections) run for that session, while tag/drop/heuristic plumbing stays on everywhere so any subsequent manual or automated reduction still works. + +| Feature | Primary + `ctx_reduce_enabled: true` | Primary + `ctx_reduce_enabled: false` | Subagents (any `ctx_reduce_enabled`) | |---|---|---|---| | Tag DB records | ✓ | ✓ | ✓ | | `§N§` prefix injection + `ctx_reduce` tool | ✓ | ✗ | ✓ (if `ctx_reduce` available) | @@ -152,4 +299,58 @@ Fail **closed** when storage is unavailable (better to disable than silently ove ## Tag identity -Each `tags` row is one taggable source-content unit (`message`, `file`, or `tool`). `message`/`file` tags key on `(session_id, message_id)` (synthetic content id). **`tool` tags key on a COMPOSITE `(session_id, callID, tool_owner_message_id)`** — because OpenCode reuses a `callID` counter per assistant turn, so the same `read:32` recurs across turns; including the owning assistant message id gives each invocation its own row (migration v10). Owner derivation: invocation parts own themselves; result parts pop a FIFO of unpaired invocations; a result whose invocation was compacted away falls back to the nearest prior persisted owner. The same composite keying mirrors in the drop queue and heuristic cleanup so dropped keys match what the tagger persisted. Per-tag token counts (`token_count` / `input_token_count` / `reasoning_token_count`) are computed once on tag insert and summed for sidebar / boundary / nudge math (off the hot path). + +**Provider error parsing:** `src/features/magic-context/overflow-detection.ts` parses provider-specific context-overflow errors (Anthropic, OpenAI, GitHub Copilot) and persists the detected limit to `session_meta.detected_context_limit` so subsequent passes use the lower value. `needs_emergency_recovery` is set for primary sessions; subagents skip emergency-recovery state because they don't consume that path. + +**Subagent model fallback:** `promptSyncWithModelSuggestionRetry` in `src/shared/model-suggestion-retry.ts` iterates the resolved fallback chain (user-configured `fallback_models` or builtin chain) on retryable failures. Abort, timeout, and context-overflow errors short-circuit the chain — those won't succeed on a different model and the caller's emergency-recovery path handles them. Suggestion retry ("did you mean X?") runs inside each attempt. + +## Cross-Cutting Concerns + +**Logging:** Use buffered file logging from `src/shared/logger.ts` and write to the temp-file path returned by `getLogFilePath()`. Per-session logs use `sessionLog(sessionId, message)`; module-level logs use `log(message)`. Heavy logging batches to disk to avoid blocking the transform path. + +**Caching:** Use deferred reductions, cached memory-block injection, per-session TTL tracking, per-tag cached token counts (computed once on tag insert), persisted reminder-replay state, per-session live injection cache, persisted system-prompt hash, and persisted todo-snapshot replay state — all coordinated through `src/hooks/magic-context/` and `src/features/magic-context/storage-meta-*.ts`. + +**Storage:** Use the SQLite database created by `src/features/magic-context/storage-db.ts` under the cortexkit data directory resolved by `src/shared/data-path.ts` (`~/.local/share/cortexkit/magic-context/context.db` on Linux/macOS, XDG-equivalent on Windows). Legacy OpenCode-plugin-folder DBs are migrated forward on first boot. The same DB is shared cross-harness between OpenCode and Pi; session-scoped tables include a `harness` discriminator (`'opencode'` / `'pi'`) while project-scoped tables (memories, git commits) are shared. + +**Schema migrations:** `src/features/magic-context/migrations.ts` declares versioned migrations v1–v37 (`LATEST_SUPPORTED_VERSION = 37` in `storage-db.ts` is the schema-fence ceiling and MUST be bumped with every new migration; a unit test — `schema-version-fence.test.ts` — asserts `LATEST_SUPPORTED_VERSION === LATEST_MIGRATION_VERSION` so the two can't drift). Notable: v10 `tool_owner_message_id` (composite tool-tag identity); v11 `todo_synthetic_*` (synthetic-todowrite); v12 orphan `memory_embeddings` cleanup; v13 `pending_compaction_marker_state` (deferred-marker drain); v14 project-scoped key files + version counter; v15 `deferred_execute_state` (boundary execution); v16 context-limit cache sentinels; v17 multi-anchor note-nudge/auto-search JSON storage; v18 `pending_pi_compaction_marker_state`; v19 compartment-state lease table; v20 subagent invocation token accounting; v21 session lifetime work metrics; **v22 the v2.0 cache-architecture foundation (m[0]/m[1] split tables, `project_state` epoch counter, plus per-compartment `p1`–`p4` tier columns, `importance`, `episode_type`, `p1_embedding`, and `legacy` flag); v23 `compartment_events` (historian-extracted causal_incident / trajectory_correction, stored-not-rendered in v2.0); v24 `historian_runs` telemetry (per-run chunk range, compartment/fact/event counts, importance min/max/avg, status + failure reason, FK to `subagent_invocations`); v25 `pi_stable_id_scheme` (Pi stable-id cutover watermark); v26 `memory_mutation_log` + `cached_m1_bytes` (memory supersede-delta — non-additive in-session memory mutations render as an m[1] `` delta instead of bumping the project epoch, plus the frozen-m[1]-bytes cache column); v27 `tags.entry_fingerprint` (Pi fallback-tag adoption); v28 `git_sweep_coordinator` (lease/cooldown for cross-process git-commit sweeps); v29 `notes.anchor_ordinal` (note→conversation-tail traceback); v30 `cached_m0_system_hash` / `cached_m0_tool_set_hash` / `cached_m0_model_key` (HARD-bust m[0] markers — provider-side cache-eviction detection for the materialization taxonomy; the migration clears the m[0]/m[1] cache once so pre-v30 rows re-materialize cleanly); v31 ctx_reduce-nudge state (`last_nudge_undropped`, `channel2_nudge_state`, `last_emergency_input_sample` + startup heal zeroing legacy sticky/anchor nudge state); v32 protected-tail v3 boundary state + per-tag cached token counts (`tags.token_count` / `input_token_count` / `reasoning_token_count` — computed once on tag insert, summed for sidebar/boundary/nudge math); v33 `compartment_chunk_embeddings` table for cross-session semantic search across compartment windows; v34 `workspaces` / `workspace_members` tables plus `cached_m0_workspace_fingerprint` m[0] marker (with a one-shot m[0]/m[1] cache reset so pre-v34 rows re-materialize cleanly); v35 `workspaces.share_categories` default + epoch refresh for existing members; v36 `session_projects` ownership map + seed for pre-v36 embedded sessions; **v37 the external-memory v2 unified-read foundation: per-session `external_recall_json` / `external_recall_state` / `external_recall_at` columns on `session_meta` (the frozen post-dedup, post-trim snapshot every render replays for byte stability) and the `cached_m0_external_recall_hash` m[0] marker (a carried marker that drives the m[1] `` delta comparison only — deliberately NOT a `mustMaterialize` trigger; the migration is `ensureColumn`-idempotent so a dev DB that ran it under the pre-rename v31 number re-applies harmlessly).** Migration runner uses `schema_migrations` table with version-ordered execution and sibling-startup race protection (duplicate-insert is tolerated). + +**Harness-aware behavior:** `src/shared/harness.ts` exposes `setHarness()`/`getHarness()` for the runtime to identify itself; production INSERTs into session-scoped tables tag rows with the current harness. Pi-specific session-resolution paths are skipped on OpenCode and vice versa. + +**Project-config trust boundary:** `src/config/project-security.ts` `stripUnsafeProjectConfigFields()` strips privilege-escalation / exfiltration vectors from any repo-supplied (untrusted) project config BEFORE it merges over the trusted user config — shared by OpenCode and Pi so the boundary is identical cross-harness. Strips: `auto_update` (repos must not suppress plugin self-updates), `sqlite` (process-global PRAGMAs), `memory.external` (a repo must not redirect the long-term memory endpoint or read a user's personal memory store; only user-level config supplies the Hindsight endpoint/credentials), and the per-hidden-agent `prompt` / `permission` / `tools` / `system_prompt` fields (a repo must not reprogram the historian/dreamer/sidekick). A separate post-merge pass (`dropInheritedEmbeddingKeyOnRedirect`) drops the user's inherited `embedding.api_key` when the project redirected only the embedding endpoint without supplying its own key, to prevent exfiltration of the user's secret to a repository-chosen server. + +## Tag Identity (v3.3.1+) + +**Tag types:** `message`, `file`, `tool`. Each row in the `tags` table represents one source-content unit that can be tagged with `§N§` and dropped/truncated/replayed by the runtime. + +**Identity composition by type:** + +- **`message` and `file` tags:** identified by `(session_id, message_id)`. The `message_id` for these is a synthetic content id (`:p` for text, `:fileN` for files). These ids are globally unique within a session. + +- **`tool` tags:** identified by `(session_id, message_id, tool_owner_message_id)` — a *composite* identity. For tool tags, `message_id` is the OpenCode-generated callID (e.g. `read:32`). Pre-v3.3.1 the runtime keyed tool tags by callID alone, but OpenCode reuses a callID counter per assistant turn — so two assistant turns that each invoke `read:32` produced the SAME callID for different invocations. The fix: include the *owning assistant message id* in the key so each invocation gets its own row. + +**Schema enforcement:** schema migration v10 (`src/features/magic-context/migrations.ts`) adds `tool_owner_message_id` (`TEXT NULL`), a partial UNIQUE index `idx_tags_tool_composite` on `(session_id, message_id, tool_owner_message_id) WHERE type='tool' AND tool_owner_message_id IS NOT NULL`, and a partial lookup index `idx_tags_tool_null_owner` on `(session_id, message_id) WHERE type='tool' AND tool_owner_message_id IS NULL` to back lazy adoption. + +**Helper API surface (`src/features/magic-context/storage-tags.ts`):** + +- `getToolTagNumberByOwner(db, sessionId, callId, ownerMsgId)`: composite-identity lookup. +- `getNullOwnerToolTag(db, sessionId, callId)`: find a legacy NULL-owner orphan to lazily adopt. +- `adoptNullOwnerToolTag(db, tagId, ownerMsgId)`: attempt to claim a NULL-owner row (NULL guard ensures first claim wins). +- `getPersistedToolOwnerNearestPrior(db, sessionId, callId, beforeMessageId)`: derive the most recent prior owner for a tool result whose invocation isn't in the visible window. +- `deleteToolTagsByOwner(db, sessionId, ownerMsgId)`: cascade delete on `message.removed`. + +**Owner derivation (`src/hooks/magic-context/tag-messages.ts`):** + +For each tool observation in a transform pass: + +1. **Invocation parts** (`tool-invocation` / `tool_use`): owner = the message hosting the part. +2. **Result parts** (`tool` with output / `tool_result`): pop the FIFO queue of unpaired invocations for that callId; owner = the popped invocation's message id. +3. **Result-only window** (invocation compacted away): fall back to `getPersistedToolOwnerNearestPrior` for the most recent prior persisted owner; if none found, last-resort owner = the result's own message id. + +The same logic mirrors in `src/hooks/magic-context/read-session-chunk.ts: getRawSessionTagKeysThrough` so the drop queue produces composite keys that match what the tagger persisted. + +**Cleanup paths:** + +- `deleteTagsByMessageId(db, sessionId, messageId)` (called from `event-handler.ts` on `message.removed`) deletes BOTH content-id-scoped tags (text/file on the removed message) AND owner-scoped tool tags (`tool_owner_message_id == messageId`). +- `applyHeuristicCleanup` keys both the tag-side index and fingerprint-side map by composite `\x00`. The fingerprint VALUE includes ownerMsgId too, so cross-owner pairs with same `(toolName, args)` produce DISTINCT fingerprints and are NOT merged. + +**Legacy NULL-owner handling:** rows written by pre-v3.3.1 plugin versions have `tool_owner_message_id = NULL`. The Layer B backfill (`src/features/magic-context/tool-owner-backfill.ts`) populates those rows from OpenCode's session DB on plugin upgrade (lease-based concurrency, batched commits). When backfill is skipped (no OpenCode DB attached) lazy adoption converts orphans to non-NULL on the next observation. Drop queue and heuristic cleanup gracefully fall back to bare-callId match for unbackfilled NULL-owner rows. diff --git a/CONFIGURATION.md b/CONFIGURATION.md index efafe0121..15bcdea7b 100644 --- a/CONFIGURATION.md +++ b/CONFIGURATION.md @@ -419,6 +419,69 @@ Cross-session memory settings. All memories are scoped to the current project (i | `injection_budget_tokens` | `number` (500–20000) | `4000` | Token budget for memory injection into ``. | | `auto_promote` | `boolean` | `true` | Promote eligible session facts to project memories automatically after historian or `/ctx-recomp` runs. When `false`, historian and recomp do not write any new memories — agents can still create memories explicitly via `ctx_memory write`, and existing memories continue to be injected and searched normally. | | `retrieval_count_promotion_threshold` | `number` | `3` | Retrievals needed before a memory is auto-promoted to permanent. | +| `external` | `object` | See below | **User-config-only.** Long-term memory backend (Hindsight) — tees curated writes OUT and recalls them BACK once per session, plus an explicit-only `ctx_search` source. A cloned repo cannot redirect the endpoint or read a user's personal memory store. | + +### `memory.external` + +The `memory.external` block controls the **external memory backend** (Hindsight): a long-term companion store that holds curated memories OUT of the project (so they survive across projects, harness restarts, and (in future) the user's whole fleet) and recalls them BACK once per session. The local SQLite store remains the source of truth; the external store is a long-term companion. + +**Security:** this entire block is **user-config-only**. `stripUnsafeProjectConfigFields()` in `src/config/project-security.ts` drops it from project-level config (parallel to `auto_update` and `sqlite`) — a cloned repo cannot redirect the endpoint, exfiltrate a user's personal memory store, or read a user's external memory by editing `magic-context.jsonc` in a project root. Set the block in `~/.config/opencode/magic-context.jsonc` (OpenCode) or `~/.pi/agent/magic-context.jsonc` (Pi). + +```jsonc +{ + "memory": { + "external": { + "provider": "off", // "off" (default) or "hindsight" + "endpoint": "http://10.0.0.1:8889", // required when provider is hindsight; Hindsight base URL + "api_key": "{env:HINDSIGHT_API_KEY}", // optional bearer token + "project_bank": "mc-{name}-{id8}", // project bank name template; {name}=project basename, {id8}=first 8 chars of the project identity hash + "main_bank": "user-memories", // required when provider is hindsight; bank for user + global scope. Assumed to pre-exist; never created or modified by the plugin + "retain_sources": ["historian", "agent", "dreamer"], // which creation points tee (gate write-side, not read-side) + "tags": [], // static tags attached to every retained item + "recall": { // see `memory.external.recall` below + "enabled": true, + "timeout_ms": 3000, + "max_tokens": 2048, + "dedup_threshold": 0.85, + "global_tags": [], + "global_from_prompt": false, + "search": true, + "mental_models": true, + "profile_mental_models": ["user-preferences"] + } + } + } +} +``` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `provider` | `"hindsight"` \| `"off"` | `"off"` | Backend implementation. `"off"` disables the whole feature. `"hindsight"` activates tee-on-write + once-per-session recall + the explicit `ctx_search` source. | +| `endpoint` | `string` | — | Required when `provider: "hindsight"`. Hindsight base URL (e.g. `http://10.0.0.1:8889`). Trailing slashes are stripped. SSRF-guarded. | +| `api_key` | `string` | — | Optional bearer token. Supports `{env:VAR}` substitution. Never logged. | +| `project_bank` | `string` (template) | `"mc-{name}-{id8}"` | Project-bank name template. Placeholders: `{name}` = sanitized project basename, `{id8}` = first 8 chars of the project identity hash. Created on first retain (PUT with the project bank mission). | +| `main_bank` | `string` | — | Required when `provider: "hindsight"`. Bank for `user` and `global` scope items. **Assumed to pre-exist; the plugin never creates or modifies it.** | +| `retain_sources` | `string[]` | `["historian","agent","dreamer"]` | Which creation points tee to the external backend. `historian` = fact promotion in `src/features/magic-context/memory/promotion.ts`. `agent` = `ctx_memory` `write` / `update` in `src/tools/ctx-memory/tools.ts`. `dreamer` = user-memory promotion in `src/features/magic-context/user-memory/review-user-memories.ts`. Read paths (recall, search, mental-models) and W2 correctives (archive/update/verify corrective propagation) are NOT gated by this list. | +| `tags` | `string[]` | `[]` | Static tags attached to every retained item (in addition to the always-present `source:magic-context`, `category:`, and the per-scope `project:` / `scope:user` / `scope:global` tags). | +| `recall` | `object` | See below | Unified read path — session-start recall + `ctx_search` external source. | + +### `memory.external.recall` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `enabled` | `boolean` | `true` | Session-start recall from the external backend, merged into the context injection. Late results ride the m[1] `` delta; the first m[0] render awaits up to `timeout_ms` (cache-cold path). | +| `timeout_ms` | `number` (500–15000) | `3000` | Max wait for recall at the first render of a session (when the cache is already cold). Late results arrive as an m[1] delta on later passes. | +| `max_tokens` | `number` (256–8192) | `2048` | Per-slice token budget — project / profile / global each get this cap. The external block is NOT charged to the history or local-memory budget; it is bounded solely by this × 3 slices. | +| `dedup_threshold` | `number` (0.5–0.99) | `0.85` | Cosine similarity above which a recalled item is dropped as a duplicate of a local memory (or active user memory). Hash-only fallback when embeddings are unavailable. | +| `global_tags` | `string[]` | `[]` | Tag filter for the global (main-bank) recall slice, matched with `tags_match: "any"` (untagged content INCLUDED). `[]` = no filter sent (full autoRecall replacement). | +| `global_from_prompt` | `boolean` | `false` | Include an excerpt of the session's first user prompt in the global-slice recall query (the project name is always included). The first prompt is fixed for the session, so the query — and the frozen recall snapshot — stays deterministic across crash-recovery re-fires. | +| `search` | `boolean` | `true` | Expose the `ctx_search` `"external"` source (project + main bank). **Explicit-only** — the auto-search hot path (every user prompt hint) NEVER hits it, even when this is `true`. | +| `mental_models` | `boolean` | `true` | Use Hindsight mental models as the fast path for the project and profile recall slices (single GET, server-refreshed), falling back to full recall when absent/empty. The global slice always uses full recall. | +| `profile_mental_models` | `string[]` | `["user-preferences"]` | Main-bank mental-model names (case-insensitive) used for the profile slice. The main bank is never modified by the plugin — create these manually. Project-bank mental models are seeded by the plugin (`project-conventions`, `project-decisions`) on the first successful retain. | + +**Engine-agnostic interface.** `external-memory-provider.ts` defines a neutral `ExternalMemoryBackend` interface (mirrors the `EmbeddingProvider` pattern). Hindsight is the only shipped implementation today. Document identity is `mc:::` (content-derived → idempotent re-retains upsert on the server; retries never duplicate). Bank routing: `scope: "project"` → `project_bank` (per project); `scope: "user"` and `scope: "global"` → `main_bank`. Project items carry `project:` + `project-name:` tags; globals carry `scope:global` plus `origin-project:*` provenance tags and a `context` field that names the originating project so Hindsight's fact extractor links it as an entity. + +**Failure semantics.** Hindsight calls are fire-and-forget; failures are logged, never thrown. The impl carries a circuit breaker (3 fails in 60s → open 5min → half-open probe) so a hung endpoint can't drag every plugin operation through its timeout. A 422 (memory-defense rejected content) is treated as a hard no and never retried; 404 on a missing bank is benign (returns empty for the slice). The bearer token is never logged. Cross-harness: OpenCode and Pi share the same banks (the impl is a single, neutral interface and the project bank name is project-derived, so both harnesses resolve the same bank). --- @@ -652,7 +715,13 @@ Tier boundaries are hardcoded to keep behavior predictable and prevent cache-bus "injection_budget_tokens": 4000, "auto_promote": true, "auto_search": { "enabled": true, "score_threshold": 0.6, "min_prompt_chars": 20 }, - "git_commit_indexing": { "enabled": false, "since_days": 365, "max_commits": 2000 } + "git_commit_indexing": { "enabled": false, "since_days": 365, "max_commits": 2000 }, + "external": { // USER-LEVEL ONLY — stripped from project config + "provider": "hindsight", + "endpoint": "http://10.0.0.1:8889", + "api_key": "{env:HINDSIGHT_API_KEY}", + "main_bank": "user-memories" + } }, "sidekick": { diff --git a/README.md b/README.md index e1cd04975..b4428604c 100644 --- a/README.md +++ b/README.md @@ -180,7 +180,7 @@ Because it runs during idle time, the dreamer pairs well with local models, even *The right memory at the right moment.* Every turn, active project memories and the compacted session history are injected automatically and cache-stably. On demand, the agent reaches for: -- **`ctx_search`**: one query across three layers at once: project **memories**, raw **conversation** history, and indexed **git commits**. Semantic embeddings with full-text fallback. +- **`ctx_search`**: one query across four layers at once: project **memories**, raw **conversation** history, indexed **git commits**, and (opt-in) the long-term **external** memory backend. Semantic embeddings with full-text fallback. ``` ctx_search(query="why did we pick event sourcing for orders") @@ -191,6 +191,8 @@ Because it runs during idle time, the dreamer pairs well with local models, even Recall works **across sessions** (a new session inherits everything) and **across harnesses** (write a memory in OpenCode, retrieve it in Pi). +> **Long-term memory** *(opt-in, off by default)* tees curated writes (historian promotions, `ctx_memory` writes, dreamer user-memory promotions) to a Hindsight backend and recalls them BACK once per session as a `` block — across sessions, across harnesses, and across projects for the global slice. Same `ctx_search` "external" source (explicit-only, never on the auto-search hot path). User-config-only; a repo cannot redirect the endpoint. Configure under `memory.external`; see [CONFIGURATION.md](./CONFIGURATION.md#memoryexternal). +> > **Auto search hints** *(on by default)* run a background `ctx_search` each turn and whisper a "vague recall" when something relevant exists — like almost remembering a note you took. It appends only compact fragments, never full content; set `memory.auto_search.enabled: false` to turn it off. **Git commit indexing** *(opt-in)* makes your project history semantically searchable as a fourth `ctx_search` source — enable with `memory.git_commit_indexing.enabled: true`. ### Agent tools at a glance @@ -198,8 +200,8 @@ Recall works **across sessions** (a new session inherits everything) and **acros | Tool | Section | What it does | |------|-------|-------------| | `ctx_reduce` | Context | Queue stale tagged content for removal, cache-aware | -| `ctx_memory` | Capture | Write or delete durable cross-session memories | -| `ctx_search` | Recall | Search memories, conversation history, and git commits | +| `ctx_memory` | Capture | Write or delete durable cross-session memories (project scope, plus `global` scope to the external main bank when configured) | +| `ctx_search` | Recall | Search memories, conversation history, git commits, and (explicit-only) the external long-term memory backend | | `ctx_expand` | Recall | Decompress a history range back to the transcript | | `ctx_note` | Recall | Deferred intentions and dreamer-evaluated smart notes | diff --git a/STRUCTURE.md b/STRUCTURE.md index 38176c3ad..790475238 100644 --- a/STRUCTURE.md +++ b/STRUCTURE.md @@ -66,9 +66,9 @@ **`src/features/`:** - Purpose: Group reusable subsystem logic by feature. -- Contains: Magic-context services (storage, scheduler, tagger, search, message-index, overflow detection, compaction markers), dreamer runtime, sidekick support, memory system, user-memory pipeline, key-files pinning, git-commit indexer, tool-definition token measurement, schema migrations, built-in commands. +- Contains: Magic-context services (storage, scheduler, tagger, search, message-index, overflow detection, compaction markers), dreamer runtime, sidekick support, memory system, user-memory pipeline, key-files pinning, git-commit indexer, tool-definition token measurement, schema migrations, built-in commands, **external memory backend (Hindsight)**. - Key subdirs: `src/features/magic-context/dreamer/`, `src/features/magic-context/memory/`, `src/features/magic-context/sidekick/`, `src/features/magic-context/user-memory/`, `src/features/magic-context/key-files/`, `src/features/magic-context/git-commits/`, `src/features/builtin-commands/` -- Key files: `src/features/magic-context/storage-db.ts`, `src/features/magic-context/storage.ts` (barrel), `src/features/magic-context/migrations.ts`, `src/features/magic-context/message-index.ts`, `src/features/magic-context/search.ts`, `src/features/magic-context/overflow-detection.ts`, `src/features/magic-context/dreamer/runner.ts`, `src/features/magic-context/memory/storage-memory.ts`, `src/features/magic-context/user-memory/storage-user-memory.ts`, `src/features/builtin-commands/commands.ts` +- Key files: `src/features/magic-context/storage-db.ts`, `src/features/magic-context/storage.ts` (barrel), `src/features/magic-context/migrations.ts`, `src/features/magic-context/message-index.ts`, `src/features/magic-context/search.ts`, `src/features/magic-context/overflow-detection.ts`, `src/features/magic-context/dreamer/runner.ts`, `src/features/magic-context/memory/storage-memory.ts`, `src/features/magic-context/memory/external-memory.ts` (external-memory facade: init/tee/recall/remove/upsert), `src/features/magic-context/memory/external-memory-provider.ts` (engine-agnostic neutral types + `ExternalMemoryBackend` interface, mirrors the `EmbeddingProvider` pattern), `src/features/magic-context/memory/external-memory-hindsight.ts` (Hindsight impl: bank resolution, retain/recall/remove/mental-models, circuit breaker, SSRF guard), `src/features/magic-context/memory/external-recall.ts` (session-start orchestrator: 3-slice fan-out + dedup + trim + persist-before-settle), `src/features/magic-context/memory/external-recall-read.ts` (snapshot read + marker hash for m[0]), `src/features/magic-context/user-memory/storage-user-memory.ts`, `src/features/builtin-commands/commands.ts` **`src/tools/`:** - Purpose: Define the agent-facing tool surface. @@ -104,24 +104,32 @@ - `assets/magic-context.schema.json`: Generated JSON schema, kept in sync via `scripts/build-schema.ts` and `scripts/release.sh`. **Core Logic:** -- `src/hooks/magic-context/transform.ts`: Run the turn transform; orchestrate tagging, replay paths, prepareCompartmentInjection, and downstream postprocess hand-off. +- `src/hooks/magic-context/transform.ts`: Run the turn transform; orchestrate tagging, replay paths, prepareCompartmentInjection, and downstream postprocess hand-off. Also fires `startSessionRecall()` on the first message (Pi mirror: `packages/pi-plugin/src/context-handler.ts`). - `src/hooks/magic-context/transform-postprocess-phase.ts`: Apply pending ops, heuristic cleanup, deferred-note nudges, **synthetic-todowrite injection (B7)**, and auto-search hints. - `src/hooks/magic-context/hook.ts`: Compose runtime services. - `src/hooks/magic-context/strip-content.ts`: Strip and replay reasoning, inline thinking, structural noise, dropped placeholders, merged-assistant reasoning, processed images, and system-injected messages. - `src/hooks/magic-context/caveman.ts`: Experimental age-tier text compression for primary sessions with `ctx_reduce_enabled=false`. - `src/hooks/magic-context/todo-view.ts`: Build the deterministic synthetic todowrite tool part and compute its hash-based `call_id`. -- `src/hooks/magic-context/inject-compartments.ts`: m[0]/m[1] history layout — `renderM0`/`renderM1`/`materializeM0`/`mustMaterialize` (mirrored in Pi's `inject-compartments-pi.ts`). +- `src/hooks/magic-context/inject-compartments.ts`: m[0]/m[1] history layout — `renderM0`/`renderM1`/`materializeM0`/`mustMaterialize`, plus `` block render (`renderExternalMemoryBlock` / `renderExternalMemoryDelta`) and the `M0SnapshotMarkers.externalRecallHash` carried marker (mirrored in Pi's `inject-compartments-pi.ts`). - `src/hooks/magic-context/decay-curve.ts`: Council-validated deterministic tier-decay math (half-life, log-cost tier boundaries, budget pressure). - `src/hooks/magic-context/decay-render.ts`: Shared OpenCode+Pi compartment renderer built on the decay curve (replaces the removed LLM compressor). - `src/hooks/magic-context/compartment-runner-incremental.ts`: v2 historian publish path — bounded reference blocks, tiered/scored compartments, faithful per-chunk facts, discard-last, events + `p1_embedding` on publish. - `src/hooks/magic-context/reference-retrieval.ts` (+ `reference-seeds.generated.ts`): 4 rotating seed compartments + last-6 recency references for the historian prompt. - `src/hooks/magic-context/historian-prompt.generated.ts`: Generated v8.7.3 historian system prompt (source: `.alfonso/.../historian-prompt-v8.7.3.md`; re-exported via `compartment-prompt.ts`). - `src/features/magic-context/memory/memory-migration.ts`: `/ctx-session-upgrade` 9-cat→5-cat memory re-eval (active-only, permanent-safe, epoch-bumping). +- `src/features/magic-context/memory/external-memory.ts`: External-memory facade — `initializeExternalMemory`, `teeToExternalBackend`, `recallFromExternalBackend`, `removeFromExternalBackend`, `mentalModelsFromExternalBackend`, `upsertToExternalBackend`, `getExternalMemoryStatus`. Re-exported from `src/features/magic-context/memory/index.ts`. +- `src/features/magic-context/memory/external-memory-provider.ts`: Engine-agnostic neutral types (`ExternalMemoryRetainItem`, `ExternalMemoryRecallQuery`, `ExternalMemoryRemoveItem`, `ExternalMemoryMentalModelQuery`) and the `ExternalMemoryBackend` interface (mirrors the `EmbeddingProvider` pattern; HindsightMemoryBackend is the implementation). +- `src/features/magic-context/memory/external-memory-hindsight.ts`: Hindsight impl — bank resolution (`resolveBankForScope` with `mc-{name}-{id8}` project-bank template + main bank), `documentIdFor` (`mc:::`, idempotent), retain/recall/remove/mental-models/fetchFailedRetainCount over `/v1/default/banks/...`, circuit breaker (3 fails/60s → open 5min → half-open probe), SSRF guard via `embedding-ssrf.ts`, `redirect: "error"`, 422 never-retry, 10s fetch timeout. +- `src/features/magic-context/memory/external-recall.ts`: Session-start orchestrator — `startSessionRecall` (idempotent, fire-once per session; in-flight map + `pending` row deadlock guard) → 3-slice fan-out (project / profile / global) with mental-models fast path → `dedupAndTrim` (cosine ≥ `dedup_threshold` + hash fallback against local memories + active user memories; per-slice sort + token-trim) → `persistRecallState` writes `external_recall_state` / `external_recall_json` BEFORE the in-flight promise settles. Also `waitForSessionRecall` (Promise.race with timeout) and `maybeAwaitExternalRecall` (the A-path: block only when the first m[0] render is imminent and the cache is already cold). +- `src/features/magic-context/memory/external-recall-read.ts`: `readExternalRecallSnapshot` (sanitizeSlice validates per-item to fail safe on corrupted rows), `readExternalRecallHash`, `computeRecallSnapshotHash` (deterministic 16-char SHA-256 prefix of the JSON-serialized snapshot). - `src/features/magic-context/storage-db.ts`: Create durable storage; run versioned migrations; resolve runtime SQLite backend. - `src/features/magic-context/storage-meta-persisted.ts`: Read and write per-session persisted scalars and JSON blobs. -- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v32 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep). +- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v37 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep). v37 adds `external_recall_json` / `external_recall_state` / `external_recall_at` and the `cached_m0_external_recall_hash` m[0] marker. - `src/features/magic-context/message-index.ts`: FTS-backed raw-message index for `ctx_search`. -- `src/features/magic-context/search.ts`: Unified retrieval over memories, raw messages, and git commits. +- `src/features/magic-context/search.ts`: Unified retrieval over memories, raw messages, git commits, and (explicit-only) the external long-term memory backend (`SearchSource += "external"`; `searchExternal` runs project + main bank in parallel, content-hash-dedups, drops hits already in the session's `` block, respects a 5s `AbortSignal`). +- `src/tools/ctx-memory/tools.ts`: W2 correctives (Pi mirror: `packages/pi-plugin/src/tools/ctx-memory.ts`) — `write` tees to external, `update` remove-then-tee cascade, `archive` batched remove, `merge` no-op, `verify` (dreamer-only) verbatim re-upsert. Also adds `scope: "global"` which routes ONLY to the external main bank with origin provenance and no local row. +- `src/config/project-security.ts`: `stripUnsafeProjectConfigFields()` — strips `memory.external` (alongside `auto_update`, `sqlite`, hidden-agent escalation fields) from project config so a repo cannot redirect the endpoint or read a user's personal memory store. +- **Pi mirror (`packages/pi-plugin/src/`):** `context-handler.ts` fires `startSessionRecall`; `inject-compartments-pi.ts` renders the `` block (project slice baked into m[0], profile slice appended to the `` block, global slice as a separate block — mirrors OpenCode); `tools/ctx-memory.ts` and `tools/ctx-search.ts` carry the same external + W2 + `SearchSource` plumbing. Pi no longer carries a drifted inline copy of the external code — it imports from `@magic-context/core/...` (the shared core package, which mirrors the OpenCode source path). **Tests:** Co-locate tests with source as `src/**/*.test.ts`, for example `src/hooks/magic-context/hook.test.ts`, `src/tools/ctx-memory/tools.test.ts`, and `src/features/magic-context/migrations-v11.test.ts`. End-to-end coverage lives in the separate `packages/e2e-tests/` workspace. diff --git a/assets/magic-context.schema.json b/assets/magic-context.schema.json index c94eeb387..649946edc 100644 --- a/assets/magic-context.schema.json +++ b/assets/magic-context.schema.json @@ -697,22 +697,6 @@ } }, "memory": { - "default": { - "enabled": true, - "injection_budget_tokens": 4000, - "auto_promote": true, - "retrieval_count_promotion_threshold": 3, - "auto_search": { - "enabled": true, - "score_threshold": 0.6, - "min_prompt_chars": 20 - }, - "git_commit_indexing": { - "enabled": false, - "since_days": 365, - "max_commits": 2000 - } - }, "description": "Cross-session memory configuration", "type": "object", "properties": { @@ -798,6 +782,141 @@ "maximum": 20000 } } + }, + "external": { + "description": "External long-term memory backend (tee). USER config only.", + "type": "object", + "properties": { + "provider": { + "default": "off", + "description": "External memory backend. 'hindsight' tees memory creations to a Hindsight service; 'off' disables (default). SECURITY: this whole block only honors USER-level config.", + "type": "string", + "enum": [ + "hindsight", + "off" + ] + }, + "endpoint": { + "description": "Backend base URL (e.g. http://10.0.0.1:8889). Required when provider is hindsight.", + "type": "string" + }, + "api_key": { + "description": "Bearer token for the backend (optional).", + "type": "string" + }, + "project_bank": { + "default": "mc-{name}-{id8}", + "description": "Bank name template for project-scoped items. Placeholders: {name}=project basename, {id8}=first 8 chars of the project identity hash.", + "type": "string" + }, + "main_bank": { + "description": "Bank for user- and global-scoped items. Required when provider is hindsight. Assumed to pre-exist; never created or modified.", + "type": "string" + }, + "retain_sources": { + "default": [ + "historian", + "agent", + "dreamer" + ], + "description": "Which creation points tee: historian promotion, agent ctx_memory writes, dreamer user-memory promotion.", + "type": "array", + "items": { + "type": "string", + "enum": [ + "historian", + "agent", + "dreamer" + ] + } + }, + "tags": { + "default": [], + "description": "Static tags attached to every retained item.", + "type": "array", + "items": { + "type": "string" + } + }, + "recall": { + "default": { + "enabled": true, + "timeout_ms": 3000, + "max_tokens": 2048, + "dedup_threshold": 0.85, + "global_tags": [], + "global_from_prompt": false, + "search": true, + "mental_models": true, + "profile_mental_models": [ + "user-preferences" + ] + }, + "description": "Unified read path: session-start recall + ctx_search external source.", + "type": "object", + "properties": { + "enabled": { + "default": true, + "description": "Session-start recall from the external backend, merged into the context injection (default: true).", + "type": "boolean" + }, + "timeout_ms": { + "default": 3000, + "description": "Max wait for recall at the first render of a session (already cache-cold). Late results ride the m[1] delta. (default: 3000)", + "type": "number", + "minimum": 500, + "maximum": 15000 + }, + "max_tokens": { + "default": 2048, + "description": "Token budget per recall slice (project / profile / global each). (default: 2048)", + "type": "number", + "minimum": 256, + "maximum": 8192 + }, + "dedup_threshold": { + "default": 0.85, + "description": "Cosine similarity above which a recalled item is dropped as a duplicate of a local memory. Hash-only fallback when embeddings are unavailable. (default: 0.85)", + "type": "number", + "minimum": 0.5, + "maximum": 0.99 + }, + "global_tags": { + "default": [], + "description": "Tag filter for the global (main-bank) recall slice, matched with tags_match 'any' (untagged content INCLUDED). Empty = no filter (full autoRecall replacement).", + "type": "array", + "items": { + "type": "string" + } + }, + "global_from_prompt": { + "default": false, + "description": "Include an excerpt of the session's FIRST user prompt in the global-slice recall query (the project name is always included). The first prompt is fixed for the session, so the query — and the frozen recall snapshot — stays deterministic. Default false: pure template query.", + "type": "boolean" + }, + "search": { + "default": true, + "description": "Expose the ctx_search 'external' source (project + main bank). (default: true)", + "type": "boolean" + }, + "mental_models": { + "default": true, + "description": "Use Hindsight mental models as the fast path for the project/profile recall slices (single GET, server-refreshed), falling back to recall when absent/empty. (default: true)", + "type": "boolean" + }, + "profile_mental_models": { + "default": [ + "user-preferences" + ], + "description": "Main-bank mental-model names (case-insensitive) used for the profile slice. The main bank is never modified by the plugin — create these manually.", + "type": "array", + "items": { + "type": "string" + } + } + } + } + } } } }, diff --git a/packages/dashboard/src/components/ConfigEditor/config-field-coverage.ts b/packages/dashboard/src/components/ConfigEditor/config-field-coverage.ts index 8f529079f..d079c2320 100644 --- a/packages/dashboard/src/components/ConfigEditor/config-field-coverage.ts +++ b/packages/dashboard/src/components/ConfigEditor/config-field-coverage.ts @@ -83,6 +83,11 @@ export const OMITTED_BY_DESIGN: Readonly> = { "sidekick.system_prompt": "free-form prompt override; raw JSONC", "system_prompt_injection.skip_signatures": "free-form substring array; raw JSONC (no array widget in the form yet)", + // External memory backend (Hindsight tee + recall). USER config only and + // operator-configured (endpoint/api_key, bank routing, recall tuning); no + // form widgets exist. Whole subtree is raw-JSONC by design. + "memory.external": + "external long-term memory backend (provider/endpoint/banks/retain_sources/tags + recall sub-block); USER config only, operator-configured, raw JSONC", }; /** diff --git a/packages/e2e-tests/tests/memory-injection.test.ts b/packages/e2e-tests/tests/memory-injection.test.ts index 345973296..ee2099ad1 100644 --- a/packages/e2e-tests/tests/memory-injection.test.ts +++ b/packages/e2e-tests/tests/memory-injection.test.ts @@ -63,6 +63,10 @@ function seedMemory(h: TestHarness, projectIdentity: string, content: string): v const dbPath = join(h.opencode.env.dataDir, "cortexkit", "magic-context", "context.db"); const db = new Database(dbPath); try { + // Match the plugin's own busy_timeout so a concurrent writer (historian + // checkpoint, dreamer run) inside the live opencode process can't make + // this seed INSERT throw SQLITE_BUSY. Same pattern as cache-invariants. + db.query("PRAGMA busy_timeout = 5000").run(); const now = Date.now(); // Use the production hash helper so this matches the value the plugin // stores when it promotes a memory. Plugin uses Bun.CryptoHasher("md5"), diff --git a/packages/pi-plugin/src/commands/ctx-status.ts b/packages/pi-plugin/src/commands/ctx-status.ts index aca839a68..347e4715a 100644 --- a/packages/pi-plugin/src/commands/ctx-status.ts +++ b/packages/pi-plugin/src/commands/ctx-status.ts @@ -90,7 +90,7 @@ export function registerCtxStatusCommand( const modelKey = ctx.model ? `${ctx.model.provider}/${ctx.model.id}` : undefined; - const statusText = executeStatus( + const statusText = await executeStatus( currentDeps.db, sessionId, currentDeps.protectedTags ?? 20, diff --git a/packages/pi-plugin/src/context-handler.ts b/packages/pi-plugin/src/context-handler.ts index fa6692804..edc993cc6 100644 --- a/packages/pi-plugin/src/context-handler.ts +++ b/packages/pi-plugin/src/context-handler.ts @@ -41,6 +41,11 @@ import { renewCompartmentLease, } from "@magic-context/core/features/magic-context/compartment-lease"; import { getCompartments } from "@magic-context/core/features/magic-context/compartment-storage"; +import { getExternalRecallConfig } from "@magic-context/core/features/magic-context/memory/external-memory"; +import { + maybeAwaitExternalRecall, + startSessionRecall, +} from "@magic-context/core/features/magic-context/memory/external-recall"; import { resolveProjectIdentity } from "@magic-context/core/features/magic-context/memory/project-identity"; import { clearSessionTracking, @@ -1491,6 +1496,58 @@ export function registerPiContextHandler( const isFirstContextPassForSession = !firstContextPassSeenBySession.has(sessionId); firstContextPassSeenBySession.add(sessionId); + + // Fire external-memory session recall on the first pass. Mirrors + // OpenCode transform.ts `startSessionRecall` (fires once per session + // when `!loadedSessions.has(sessionId)`). The recall is async and + // bounded by recall.timeout_ms; `maybeAwaitExternalRecall` below + // waits for it only when the first m[0] materialization is imminent. + if (isFirstContextPassForSession) { + const firstUserPrompt = getExternalRecallConfig()?.global_from_prompt + ? (() => { + // Extract the first meaningful user message text for the + // global recall query enrichment (recall.global_from_prompt). + // Mirrors OpenCode's extractFirstUserPromptText. + const msgs = event.messages as Array<{ + role?: string; + content?: unknown; + }>; + for (const msg of msgs) { + if (msg.role !== "user") continue; + const text = + typeof msg.content === "string" + ? msg.content + : Array.isArray(msg.content) + ? msg.content + .filter( + (p): p is { type: string; text: string } => + p !== null && + typeof p === "object" && + (p as { type?: unknown }).type === "text" && + typeof (p as { text?: unknown }).text === + "string", + ) + .map((p) => p.text) + .join(" ") + : ""; + const trimmed = text.trim(); + if (trimmed.length > 0) return trimmed; + } + return undefined; + })() + : undefined; + startSessionRecall({ + db: options.db, + sessionId, + projectIdentity, + projectName: projectDirectory + ? (projectDirectory.split("/").filter(Boolean).at(-1) ?? + projectIdentity) + : projectIdentity, + ...(firstUserPrompt ? { firstUserPrompt } : {}), + }); + } + const piUsage = ctx.getContextUsage?.(); const tModelDetect = performance.now(); // Seed the in-memory model key from the JSONL on the first pass after a @@ -4063,6 +4120,22 @@ async function runPipeline(args: RunPipelineArgs): Promise { } } + // External memory v2 hybrid A-path: when the FIRST m[0] render is + // imminent (no cached baseline — the provider cache is already cold), + // give the in-flight recall up to recall.timeout_ms to land so the + // first materialization bakes it in. Never fires once a baseline + // exists; late recalls ride the m[1] delta instead. Mirrors OpenCode + // transform.ts `maybeAwaitExternalRecall` call (before runPostTransformPhase). + if (args.injection) { + const hasCachedM0 = + getOrCreateSessionMeta(args.db, args.sessionId).cachedM0Bytes !== null; + await maybeAwaitExternalRecall({ + db: args.db, + sessionId: args.sessionId, + hasCachedM0, + }); + } + // 6. injection — writes compartments, facts, and // project memories into message[0]. This is the second-biggest // reduction lever after heuristic cleanup: a session that's been diff --git a/packages/pi-plugin/src/index.ts b/packages/pi-plugin/src/index.ts index ba1638190..7fa0549c9 100644 --- a/packages/pi-plugin/src/index.ts +++ b/packages/pi-plugin/src/index.ts @@ -29,6 +29,7 @@ import type { MagicContextConfig, SidekickConfig, } from "@magic-context/core/config/schema/magic-context"; +import { initializeExternalMemory } from "@magic-context/core/features/magic-context/memory/external-memory"; import { resolveProjectIdentity } from "@magic-context/core/features/magic-context/memory/project-identity"; import { scheduleIncrementalIndex } from "@magic-context/core/features/magic-context/message-index-async"; import { detectOverflow } from "@magic-context/core/features/magic-context/overflow-detection"; @@ -534,6 +535,13 @@ export default async function (pi: ExtensionAPI): Promise { return; } + // Arm the external-memory backend (Hindsight). Mirrors OpenCode's + // `initializeExternalMemory(pluginConfig.memory?.external)` call at + // plugin startup. Must run before any tool registration so that the + // ctx_memory tee and ctx_search external source are live from the + // first tool call. No-op when memory.external.provider is "off". + initializeExternalMemory(config.memory?.external); + await ensureProjectRegisteredFromPiDirectory(projectDir, db); info(`registered embedding config for project ${projectIdentity}`); diff --git a/packages/pi-plugin/src/inject-compartments-pi.test.ts b/packages/pi-plugin/src/inject-compartments-pi.test.ts index 04793c2fb..c76d2f8e7 100644 --- a/packages/pi-plugin/src/inject-compartments-pi.test.ts +++ b/packages/pi-plugin/src/inject-compartments-pi.test.ts @@ -1491,3 +1491,146 @@ describe("mustMaterializePi — SOFT/HARD taxonomy (parity with OpenCode)", () = } }); }); + +describe("Pi external m[1] delta pressure-refold exclusion (cache parity)", () => { + // RED-GREEN regression for Finding #1: a large external-recall delta in m[1] + // must NEVER trigger the pressure-refold backstop. The fix subtracts + // externalDeltaTokens + wrapper overhead from the pressure comparison so + // late recall "must NEVER cause a fold" (parity with OpenCode injectM0M1). + // + // Setup: materialize m[0] with a small compartment (m[0] tokens ≈ small). + // Then seed a large external recall snapshot whose token count exceeds + // 15% of m[0] tokens. On a cache-busting pass the pressure math must + // exclude the external delta and NOT call materializeM0PiWithRetry. + // + // RED (without fix): m1PressureTokens = m1Tokens (no subtraction) → + // large delta crosses the 15% ratio → materializeM0PiWithRetry called → + // m[0] bytes change → test FAILS. + // GREEN (with fix): m1PressureTokens = m1Tokens - externalDeltaTokens - + // wrapper → ratio not crossed → no refold → m[0] bytes unchanged → PASSES. + + it("large external m[1] delta does NOT trigger pressure refold (m[0] bytes stable)", () => { + const db = createTestDb(); + const cwd = mkdtempSync(join(tmpdir(), "pi-ext-pressure-")); + try { + const state = piState("ses-pi-ext-pressure", cwd); + + // Materialize m[0] with a compartment large enough to clear the + // M0_DRIFT_RATIO_FLOOR_TOKENS=500 gate (so the ratio test can fire). + // ~600 tokens of body content ensures m[0] > 500 tokens. + const m0Body = "word ".repeat(600); // ~600 tokens + appendCompartments(db, state.sessionId, [ + { + sequence: 0, + startMessage: 1, + endMessage: 1, + startMessageId: "entry-0", + endMessageId: "entry-0", + title: "Large", + content: `U: large turn\n${m0Body}`, + p1: `U: large turn\n${m0Body}`, + }, + ]); + const firstPass = [userMessage("hello", 10)]; + const r0 = injectM0M1Pi(state, db, firstPass as never, ["entry-0"], true); + expect(r0.m0Materialized).toBe(true); + const baselineM0 = textOf(firstPass[0] as never); + const baselineM0Bytes = baselineM0.length; + + // Seed a large external recall snapshot AFTER m[0] was materialized. + // The snapshot hash differs from the m[0] baseline hash (which is "") + // so the delta will be rendered into m[1]. Make it large enough to + // exceed 15% of m[0] tokens (m[0] is small, so even a moderate delta + // crosses the ratio without the fix). + // ~200 tokens of content — well above 15% of m[0] (~600 tokens). + // Without the fix, this delta alone would cross the ratio and trigger + // a refold. With the fix, it is subtracted from m1PressureTokens. + const largeContent = "word ".repeat(200); // ~200 tokens + db.prepare( + "UPDATE session_meta SET external_recall_state = ?, external_recall_json = ?, external_recall_at = ? WHERE session_id = ?", + ).run( + "done", + JSON.stringify({ + project: [{ content: largeContent }], + profile: [], + global: [], + }), + Date.now(), + state.sessionId, + ); + + // Cache-busting pass: recomputeM1ThisPass=true. The external delta + // will be rendered into m[1]. The pressure backstop must NOT fire. + const secondPass = [userMessage("hello", 11)]; + const r1 = injectM0M1Pi( + state, + db, + secondPass as never, + ["entry-0"], + true, + ); + + // (a) m[0] NOT re-materialized — external delta must not trigger refold. + expect(r1.m0Materialized).toBe(false); + // (b) m[0] bytes byte-identical to the baseline (cache-stable). + const m0After = textOf(secondPass[0] as never); + expect(m0After.length).toBe(baselineM0Bytes); + expect(m0After).toBe(baselineM0); + // (c) m[1] contains the external delta. + expect(textOf(secondPass[1] as never)).toContain(largeContent); + } finally { + rmSync(cwd, { recursive: true, force: true }); + closeQuietly(db); + } + }); +}); + +describe("Pi external m[1] delta includes profile slice (Finding #2)", () => { + // Regression: Pi's m[1] delta previously called renderExternalMemoryBlock + // (project+global only), dropping profile-slice items. The fix uses + // renderExternalMemoryDelta (project+global+profile), matching OpenCode. + + it("profile-slice item from recall snapshot appears in Pi m[1] external delta", () => { + const db = createTestDb(); + const cwd = mkdtempSync(join(tmpdir(), "pi-ext-profile-delta-")); + try { + const state = piState("ses-pi-ext-profile", cwd); + + // Materialize m[0] first (no external recall yet). + const firstPass = [userMessage("hello", 10)]; + const r0 = injectM0M1Pi(state, db, firstPass as never, [], true); + expect(r0.m0Materialized).toBe(true); + + // Seed an external recall snapshot with a profile-slice item AFTER + // m[0] was materialized. The snapshot hash differs from the m[0] + // baseline hash (which is ""), so the delta will be rendered into m[1]. + const profileItem = "user prefers concise answers"; + db.prepare( + "UPDATE session_meta SET external_recall_state = ?, external_recall_json = ?, external_recall_at = ? WHERE session_id = ?", + ).run( + "done", + JSON.stringify({ + project: [], + profile: [{ content: profileItem }], + global: [], + }), + Date.now(), + state.sessionId, + ); + + // Cache-busting pass: the external delta must include the profile item. + const secondPass = [userMessage("hello", 11)]; + injectM0M1Pi(state, db, secondPass as never, [], true); + + // m[1] must contain the profile-slice item. + const m1Text = textOf(secondPass[1] as never); + expect(m1Text).toContain(profileItem); + // m[0] must NOT contain it (profile merges into at next + // HARD fold, not into the external block at m[0]). + expect(textOf(secondPass[0] as never)).not.toContain(profileItem); + } finally { + rmSync(cwd, { recursive: true, force: true }); + closeQuietly(db); + } + }); +}); diff --git a/packages/pi-plugin/src/inject-compartments-pi.ts b/packages/pi-plugin/src/inject-compartments-pi.ts index 256511d68..ff8ced970 100644 --- a/packages/pi-plugin/src/inject-compartments-pi.ts +++ b/packages/pi-plugin/src/inject-compartments-pi.ts @@ -25,6 +25,12 @@ * historyRefreshSessions signal. */ +import { + computeRecallSnapshotHash, + type ExternalRecallSnapshot, + readExternalRecallHash, + readExternalRecallSnapshot, +} from "@magic-context/core/features/magic-context/memory/external-recall-read"; import { getMaxMemoryIdForProjects, getMemoriesByProject, @@ -71,6 +77,8 @@ import { type MemoryRenderOptions, type PreparedCompartmentInjection, prepareCompartmentInjection, + renderExternalMemoryBlock, + renderExternalMemoryDelta, renderMemoryBlockV2, trimMemoriesToBudgetV2, trimUserMemoriesToBudget, @@ -459,6 +467,8 @@ interface FrozenM0Inputs { memories: Memory[]; userProfile: UserMemory[]; workspace: WorkspaceRenderContext; + /** External recall snapshot baked into m[0], or null when none/pending. */ + externalRecall: ExternalRecallSnapshot | null; } /** @@ -610,6 +620,11 @@ export interface PiM0SnapshotMarkers { // Captured from PiM0HardSignals at the injection call site. systemHash: string; modelKey: string; + /** Hash of the persisted external-recall snapshot baked into m[0] ('' = none). + * NOT a HARD bust trigger (external recall is not a materialization driver) — + * drives the m[1] delta comparison only. Mirrors OpenCode + * M0SnapshotMarkers.externalRecallHash. */ + externalRecallHash: string; } /** @@ -770,6 +785,7 @@ function getCachedMarkers( lastBaselineEndMessageId: cachedBoundary, systemHash: meta.cachedM0SystemHash ?? "", modelKey: meta.cachedM0ModelKey ?? "", + externalRecallHash: meta.cachedM0ExternalRecallHash ?? "", }; } @@ -859,6 +875,11 @@ function readCurrentMarkersFromCompartments( lastBaselineEndMessageId: lastBaselineEndMessageId(compartments), systemHash: (state.hardSignals ?? EMPTY_PI_HARD_SIGNALS).systemHash, modelKey: (state.hardSignals ?? EMPTY_PI_HARD_SIGNALS).modelKey, + // externalRecallHash is NOT a mustMaterializePi trigger — it rides the + // m[1] external delta only. Mirrors OpenCode M0SnapshotMarkers comment. + // Read the live hash here for marker capture; materializeM0Pi overrides + // it from the in-transaction read (TOCTOU-safe, same as projectDocsHash). + externalRecallHash: readExternalRecallHash(db, state.sessionId), }; } @@ -969,12 +990,18 @@ function renderUserProfileBlock( db: ContextDatabase, wrapper = "user-profile", memoriesOverride?: UserMemory[], + externalProfileLines: readonly { content: string }[] = [], ): string { const memories = memoriesOverride ?? safeGetActiveUserMemoriesPi(db); - if (memories.length === 0) return ""; - return `<${wrapper}>\n${memories - .map((memory) => `- ${escapeXmlContent(memory.content)}`) - .join("\n")}\n`; + const localLines = memories.map( + (memory) => `- ${escapeXmlContent(memory.content)}`, + ); + const externalLines = externalProfileLines.map( + (item) => `- ${escapeXmlContent(item.content)}`, + ); + const allLines = [...localLines, ...externalLines]; + if (allLines.length === 0) return ""; + return `<${wrapper}>\n${allLines.join("\n")}\n`; } export function renderM0Pi( @@ -991,6 +1018,8 @@ export function renderM0Pi( compartmentsOverride?: PiCompartment[], userProfileOverride?: UserMemory[], workspaceOverride?: WorkspaceRenderContext, + /** External recall snapshot to bake into m[0]. Null = no external block. */ + externalRecallOverride?: ExternalRecallSnapshot | null, ): string { const memPath = memoryProjectPath(state); const workspace = @@ -1078,10 +1107,13 @@ export function renderM0Pi( userProfileOverride ?? safeGetActiveUserMemoriesPi(db), state.userProfileBudgetTokens ?? DEFAULT_USER_PROFILE_BUDGET_TOKENS, ); + // Merge external profile slice into (mirrors OpenCode renderM0: + // renderUserProfileBlock(trimmedProfile, "user-profile", externalRecall?.profile ?? [])). const userProfile = renderUserProfileBlock( db, "user-profile", trimmedProfile, + externalRecallOverride?.profile ?? [], ); if (userProfile.length > 0) sections.push(userProfile); sections.push( @@ -1090,6 +1122,11 @@ export function renderM0Pi( : "", ); if (memoryBlock) sections.push(memoryBlock); + // Render external memory block after (mirrors OpenCode renderM0). + if (externalRecallOverride) { + const externalBlock = renderExternalMemoryBlock(externalRecallOverride); + if (externalBlock) sections.push(externalBlock); + } return sections.join("\n\n").trim(); } @@ -1188,6 +1225,13 @@ function readFrozenM0InputsPi( const userProfile = safeGetActiveUserMemoriesPi(db); const projectState = memPath ? getProjectState(db, memPath) : undefined; const globalState = getProjectState(db, GLOBAL_USER_PROFILE_PROJECT_PATH); + // In-transaction read of the persisted external recall snapshot. Overrides + // the externalRecallHash set in readCurrentMarkersFromCompartments so render + // and marker derive from the SAME read (no TOCTOU) — mirrors how + // projectDocsHash is overwritten from readProjectDocsCanonical in OpenCode. + const recallRead = readExternalRecallSnapshot(db, state.sessionId); + const externalRecall = + recallRead.state === "done" ? recallRead.snapshot : null; const markers: PiM0SnapshotMarkers = { maxCompartmentSeq: compartments.reduce( (max, compartment) => @@ -1227,8 +1271,17 @@ function readFrozenM0InputsPi( lastBaselineEndMessageId: lastBaselineEndMessageId(compartments), systemHash: (state.hardSignals ?? EMPTY_PI_HARD_SIGNALS).systemHash, modelKey: (state.hardSignals ?? EMPTY_PI_HARD_SIGNALS).modelKey, + externalRecallHash: computeRecallSnapshotHash(externalRecall), + }; + return { + docs, + markers, + compartments, + memories, + userProfile, + workspace, + externalRecall, }; - return { docs, markers, compartments, memories, userProfile, workspace }; }); return read(); } @@ -1261,6 +1314,7 @@ function renderFreshM0PiNonPersisted( frozen.compartments, frozen.userProfile, frozen.workspace, + frozen.externalRecall, ); let attempts = 0; while ( @@ -1278,6 +1332,7 @@ function renderFreshM0PiNonPersisted( frozen.compartments, frozen.userProfile, frozen.workspace, + frozen.externalRecall, ); attempts += 1; } @@ -1320,6 +1375,7 @@ export function materializeM0Pi( // rendered m[0] exceeds the history budget, escalate the decay pressure and // re-render up to 3x so tight budgets demote more aggressively. Without this, // Pi would select different (looser) tiers than OpenCode under budget pressure. + const snapshotExternalRecall = frozen.externalRecall; let decayPressureMultiplier = 1; let m0 = renderM0Pi( state, @@ -1330,6 +1386,7 @@ export function materializeM0Pi( snapshotCompartments, snapshotUserProfile, frozen.workspace, + snapshotExternalRecall, ); const historyBudget = state.historyBudgetTokens ?? DEFAULT_HISTORY_BUDGET_TOKENS; @@ -1349,6 +1406,7 @@ export function materializeM0Pi( snapshotCompartments, snapshotUserProfile, frozen.workspace, + snapshotExternalRecall, ); attempts += 1; } @@ -1428,6 +1486,10 @@ export function materializeM0Pi( upgradeState: snapshotMarkers.upgradeState, systemHash: snapshotMarkers.systemHash, modelKey: snapshotMarkers.modelKey, + // Persist the external recall hash so the next pass can compare it + // against the live hash for the m[1] delta check. NOT a hard-bust + // trigger — rides the m[1] external delta only. + externalRecallHash: snapshotMarkers.externalRecallHash, }); // Persist the rendered-memory identity in the SAME transaction as the m[0] // snapshot (parity with OpenCode materializeM0). `memory_block_ids` / @@ -1584,6 +1646,14 @@ function renderMemoryUpdatesBlockPi(args: { interface RenderM1PiResult { text: string; memoryUpdateCount: number; + /** The delta block (late-arrival snapshot) when present, + * "" otherwise. Excluded from the injectM0M1Pi pressure-refold token math + * so a large recall can NEVER cause an m[0] refold (parity with OpenCode). */ + externalDeltaText: string; + /** True when freshly rendered from current DB state. False when replayed + * from a sibling-adoption row. The pressure-refold backstop must only fire + * on recomputed bytes (parity with OpenCode RenderM1Result.recomputed). */ + recomputed: boolean; } function renderM1PiWithMetadata( @@ -1704,10 +1774,35 @@ function renderM1PiWithMetadata( if (profileBlock) sections.push(profileBlock); } + // External memory delta: when the live recall hash differs from the m[0] + // baseline hash, surface the current recall snapshot as a delta. Mirrors + // OpenCode renderM1WithMetadata's renderExternalMemoryDelta path. The delta + // carries ALL slices including profile (profile lines reconcile into + // at the next HARD fold). Captured separately so the caller + // can subtract its tokens from the pressure-refold math — recall is NOT a + // bust trigger (parity with OpenCode RenderM1Result.externalDeltaText). + let externalDeltaText = ""; + const recallRead = readExternalRecallSnapshot(db, state.sessionId); + if (recallRead.state === "done" && recallRead.snapshot) { + const currentRecallHash = computeRecallSnapshotHash(recallRead.snapshot); + if ( + currentRecallHash !== "" && + currentRecallHash !== markers.externalRecallHash + ) { + const delta = renderExternalMemoryDelta(recallRead.snapshot); + if (delta) { + externalDeltaText = delta; + sections.push(delta); + } + } + } + if (sections.length === 0) { return { text: PI_M1_PLACEHOLDER, memoryUpdateCount: memoryUpdates.count, + externalDeltaText: "", + recomputed: true, }; } // Join with "\n" (single newline) to match OpenCode renderM1 exactly — the @@ -1715,6 +1810,8 @@ function renderM1PiWithMetadata( return { text: `\n${sections.join("\n")}\n`, memoryUpdateCount: memoryUpdates.count, + externalDeltaText, + recomputed: true, }; } @@ -1743,6 +1840,7 @@ interface CachedPiM0M1Row { cached_m0_upgrade_state: string | null; cached_m0_system_hash: string | null; cached_m0_model_key: string | null; + cached_m0_external_recall_hash: string | null; cached_m0_last_baseline_end_message_id: string | null; memory_block_ids: string | null; } @@ -1779,23 +1877,24 @@ function readCachedPiM0M1Row( return db .prepare( `SELECT cached_m0_bytes, cached_m1_bytes, - cached_m0_project_memory_epoch, - cached_m0_workspace_fingerprint, - cached_m0_project_user_profile_version, - cached_m0_max_compartment_seq, - cached_m0_max_memory_id, - cached_m0_max_mutation_id, - cached_m0_max_memory_mutation_id, - cached_m0_project_docs_hash, - cached_m0_materialized_at, - cached_m0_session_facts_version, - cached_m0_upgrade_state, - cached_m0_system_hash, - cached_m0_model_key, - cached_m0_last_baseline_end_message_id, - memory_block_ids - FROM session_meta - WHERE session_id = ?`, + cached_m0_project_memory_epoch, + cached_m0_workspace_fingerprint, + cached_m0_project_user_profile_version, + cached_m0_max_compartment_seq, + cached_m0_max_memory_id, + cached_m0_max_mutation_id, + cached_m0_max_memory_mutation_id, + cached_m0_project_docs_hash, + cached_m0_materialized_at, + cached_m0_session_facts_version, + cached_m0_upgrade_state, + cached_m0_system_hash, + cached_m0_model_key, + cached_m0_external_recall_hash, + cached_m0_last_baseline_end_message_id, + memory_block_ids + FROM session_meta + WHERE session_id = ?`, ) .get(sessionId) as CachedPiM0M1Row | null; } @@ -1836,6 +1935,7 @@ function markersFromCachedPiRow( : null, systemHash: row.cached_m0_system_hash ?? "", modelKey: row.cached_m0_model_key ?? "", + externalRecallHash: row.cached_m0_external_recall_hash ?? "", }; } @@ -1938,6 +2038,7 @@ function softRefreshCachedM1Pi(args: { markers: PiM0SnapshotMarkers; memoryUpdateCount: number; recomputed: boolean; + externalDeltaText: string; } { const preRenderedKeyFilesBlock = preRenderKeyFilesBlockPi( args.state, @@ -1974,6 +2075,11 @@ function softRefreshCachedM1Pi(args: { }), memoryUpdateCount: 0, recomputed: false, + // Sibling-adoption replay: the bytes are persisted, not freshly + // rendered. The external delta is unknown from the persisted row; + // use "" so the pressure backstop (which only fires on recomputed + // bytes) is never triggered by a replayed sibling m[1]. + externalDeltaText: "", }; } @@ -2026,6 +2132,7 @@ function softRefreshCachedM1Pi(args: { markers: { ...markers, lastBaselineEndMessageId: advancedBoundary }, memoryUpdateCount: rendered.memoryUpdateCount, recomputed: true, + externalDeltaText: rendered.externalDeltaText, }; } catch (error) { try { @@ -2087,6 +2194,10 @@ export function injectM0M1Pi( let memoryUpdateCount = 0; let m1Recomputed = false; let freshFallbackRenderedMemoryIds: number[] | null = null; + // Tracks the external-recall delta text from the freshly rendered m[1] so + // the pressure backstop can subtract its tokens — recall must NEVER cause a + // fold (parity with OpenCode injectM0M1 externalDeltaText subtraction). + let m1ExternalDeltaText = ""; if (decision.value) { // On contention exhaustion, reuse the cached m[0]/m[1] pair rather than @@ -2182,6 +2293,7 @@ export function injectM0M1Pi( m1 = freshM1.text; memoryUpdateCount = freshM1.memoryUpdateCount; m1Recomputed = true; + m1ExternalDeltaText = freshM1.externalDeltaText; } else if (contentionExhausted) { // m[1] was replayed with the cached m[0] pair above. } else if (recomputeM1ThisPass) { @@ -2197,6 +2309,7 @@ export function injectM0M1Pi( markers = refreshed.markers; memoryUpdateCount = refreshed.memoryUpdateCount; m1Recomputed = refreshed.recomputed; + m1ExternalDeltaText = refreshed.externalDeltaText; } else { const replayed = replayCachedM1Pi(db, state, currentCompartments); m0 = replayed.m0; @@ -2217,6 +2330,15 @@ export function injectM0M1Pi( // Token counts (NOT char lengths) on both sides of the ratio — parity with // OpenCode. The documented intent is "m[1] exceeds ~15% of m[0] tokens"; // char length diverges from token count on XML-heavy / non-Latin content. + // + // External recall content must NEVER CAUSE a fold (spec: not a bust trigger); + // it rides along when a fold fires for other reasons. Two layers of + // subtraction from m1Tokens: the delta itself (late recall) AND a small + // wrapper overhead (every m[1] carries the wrapper, empty or not — not a + // drift signal). The wrapper tokens are also subtracted from the absolute + // cap budget for symmetry, so a tiny m[0] baseline (where the wrapper + // alone would exceed the cap) does not falsely fire a refold when the + // only m[1] content is the recall delta. (Parity with OpenCode injectM0M1.) const M0_DRIFT_RATIO_FLOOR_TOKENS = 500; const M1_DRIFT_RATIO = 0.15; const M1_ABSOLUTE_CAP_RATIO = 0.2; @@ -2225,8 +2347,21 @@ export function injectM0M1Pi( M1_ABSOLUTE_CAP_RATIO; const m1HasContent = m1 !== PI_M1_PLACEHOLDER; const m1Tokens = m1HasContent ? estimateTokens(m1) : 0; + const M1_PRESSURE_WRAPPER_TOKENS = 20; + const externalDeltaTokens = m1ExternalDeltaText + ? estimateTokens(m1ExternalDeltaText) + : 0; + const m1PressureTokens = Math.max( + 0, + m1Tokens - externalDeltaTokens - M1_PRESSURE_WRAPPER_TOKENS, + ); + const m1AbsoluteContentBudget = Math.max( + 0, + m1AbsoluteBudget - M1_PRESSURE_WRAPPER_TOKENS, + ); const m0Tokens = estimateTokens(m0); - const m1OverAbsoluteCap = m1HasContent && m1Tokens > m1AbsoluteBudget; + const m1OverAbsoluteCap = + m1HasContent && m1PressureTokens > m1AbsoluteContentBudget; if ( !materialized && !contentionExhausted && @@ -2236,7 +2371,7 @@ export function injectM0M1Pi( m1OverAbsoluteCap || (m1HasContent && m0Tokens >= M0_DRIFT_RATIO_FLOOR_TOKENS && - m1Tokens > m0Tokens * M1_DRIFT_RATIO)) + m1PressureTokens > m0Tokens * M1_DRIFT_RATIO)) ) { decision = { value: true, reason: "drift" }; try { diff --git a/packages/pi-plugin/src/pi-historian-runner.ts b/packages/pi-plugin/src/pi-historian-runner.ts index 1a95895ff..b74eea7b4 100644 --- a/packages/pi-plugin/src/pi-historian-runner.ts +++ b/packages/pi-plugin/src/pi-historian-runner.ts @@ -38,6 +38,8 @@ */ import * as crypto from "node:crypto"; +import { basename } from "node:path"; + import { embedAndStoreCompartmentChunks } from "@magic-context/core/features/magic-context/compartment-embedding"; import { insertCompartmentEvents } from "@magic-context/core/features/magic-context/compartment-events"; import { isCompartmentLeaseHeld } from "@magic-context/core/features/magic-context/compartment-lease"; @@ -991,11 +993,17 @@ export async function runPiHistorian(deps: PiHistorianDeps): Promise { await ensureProjectRegisteredFromPiDirectory(directory, db); } if (promotionActive && !discardedLast) { + // projectName mirrors the OpenCode call site: the human-readable + // bank-template label for the external-memory tee. Without it a + // future Pi-side external config would resolve the fallback bank + // ("mc-project-") and silently split the project's external + // bank across harnesses. promoteSessionFactsToMemory( db, sessionId, projectPath, validatedPass.facts ?? [], + { projectName: basename(directory) }, ); } diff --git a/packages/pi-plugin/src/tools/ctx-memory.ts b/packages/pi-plugin/src/tools/ctx-memory.ts index b4b63d03d..fab174d8c 100644 --- a/packages/pi-plugin/src/tools/ctx-memory.ts +++ b/packages/pi-plugin/src/tools/ctx-memory.ts @@ -30,9 +30,12 @@ * versa). */ +import { basename } from "node:path"; + import type { ToolDefinition } from "@earendil-works/pi-coding-agent"; import { archiveMemory, + getExternalMemoryStatus, getMemoriesByProject, getMemoryByHash, getMemoryById, @@ -40,16 +43,21 @@ import { type Memory, type MemoryCategory, mergeMemoryStats, + removeFromExternalBackend, saveEmbedding, supersededMemory, + teeToExternalBackend, updateMemoryContent, updateMemorySeenCount, + updateMemoryVerification, + upsertToExternalBackend, V2_MEMORY_CATEGORIES, } from "@magic-context/core/features/magic-context/memory"; import { embedTextForProject, getProjectEmbeddingSnapshot, } from "@magic-context/core/features/magic-context/memory/embedding"; +import type { ExternalMemoryRemoveItem } from "@magic-context/core/features/magic-context/memory/external-memory-provider"; import { computeNormalizedHash } from "@magic-context/core/features/magic-context/memory/normalize-hash"; import { normalizeStoredProjectPath, @@ -77,11 +85,22 @@ const DEFAULT_LIST_LIMIT = 10; // exact alias of `archive` (both soft-archive); `archive` is the single // soft-remove action. Primary agents get write/archive/update/merge on the // memories they already see (with ids) in the injected project-memory block; -// only `list` (bulk enumeration) stays dreamer-only. -const ALL_ACTIONS = ["write", "archive", "update", "merge", "list"] as const; +// `list` (bulk enumeration) and `verify` (dreamer-only verification) stay +// dreamer-only. +const ALL_ACTIONS = [ + "write", + "archive", + "update", + "merge", + "list", + "verify", +] as const; type CtxMemoryAction = (typeof ALL_ACTIONS)[number]; -const DREAMER_ONLY_ACTIONS: ReadonlySet = new Set(["list"]); +const DREAMER_ONLY_ACTIONS: ReadonlySet = new Set([ + "list", + "verify", +]); const ParamsSchema = Type.Object({ action: Type.Union( @@ -119,6 +138,12 @@ const ParamsSchema = Type.Object({ description: "Why the memory is being archived (optional, recommended)", }), ), + scope: Type.Optional( + Type.Union([Type.Literal("project"), Type.Literal("global")], { + description: + 'Write only. "project" (default): this project\'s memory store. "global": a cross-project fact stored ONLY in the external long-term memory backend — use when the fact is true regardless of which project you are in. Requires an external backend; recallable from the next session onward.', + }), + ), }); type CtxMemoryParams = Static; @@ -328,6 +353,20 @@ export function createCtxMemoryTool( } const sessionId = ctx.sessionManager.getSessionId(); + // Helper to build an ExternalMemoryRemoveItem from a memory row. + // Identity derives from the original content hash + project identity, + // so a corrective remove needs the row AS IT STOOD before mutation. + const buildRemoveItem = ( + memory: { content: string; category: Memory["category"] }, + memProjectIdentity: string, + ): ExternalMemoryRemoveItem => ({ + content: memory.content, + category: memory.category as MemoryCategory, + scope: "project", + projectIdentity: memProjectIdentity, + ...(ctx.cwd ? { projectName: basename(ctx.cwd) } : {}), + }); + if (params.action === "write") { const content = params.content?.trim(); if (!content) @@ -338,6 +377,31 @@ export function createCtxMemoryTool( return err("Error: 'category' is required when action is 'write'."); } + // Global scope: cross-project knowledge goes to the external + // long-term store's main bank ONLY — no local row. Mirrors + // OpenCode's ctx-memory/tools.ts global scope branch. + if (params.scope === "global") { + if (!getExternalMemoryStatus()) { + return err( + "Error: scope 'global' requires an external memory backend (memory.external) — none is configured. Use the default project scope instead.", + ); + } + void teeToExternalBackend("agent", [ + { + content, + category: rawCategory, + scope: "global", + projectIdentity, + ...(ctx.cwd ? { projectName: basename(ctx.cwd) } : {}), + sourceType: dreamerAllowed ? "dreamer" : "agent", + sessionId, + }, + ]); + return ok( + `Queued global memory in ${rawCategory} for the long-term store (origin: this project). It has no local ID; it surfaces via the session-start global recall slice and ctx_search source "external" from the next session onward.`, + ); + } + const existing = getMemoryByHash( deps.db, projectIdentity, @@ -368,6 +432,21 @@ export function createCtxMemoryTool( // session_meta), defeating the whole additive/non-additive split and // busting unrelated projects. Matches OpenCode's write path, which // likewise does no cache invalidation. + + // Tee to external backend (project scope). Fire-and-forget; never + // blocks the local write. Mirrors OpenCode's write tee. + void teeToExternalBackend("agent", [ + { + content, + category: rawCategory, + scope: "project", + projectIdentity, + ...(ctx.cwd ? { projectName: basename(ctx.cwd) } : {}), + sourceType: dreamerAllowed ? "dreamer" : "agent", + sessionId, + }, + ]); + return ok(`Saved memory [ID: ${memory.id}] in ${rawCategory}.`); } @@ -436,6 +515,26 @@ export function createCtxMemoryTool( memoryId: memory.id, content, }); + // W2 corrective propagation: drop the STALE external document (old + // content hash) and tee the corrected fact as a new document. The + // local row's content rewrite already happened in the transaction + // above, so `memory.content` is still the OLD content and `content` + // is the NEW content — both needed for the remove-then-tee cascade. + // Mirrors OpenCode's update W2 path. + void removeFromExternalBackend([ + buildRemoveItem(memory, targetIdentity), + ]); + void teeToExternalBackend("agent", [ + { + content, + category: memory.category as MemoryCategory, + scope: "project", + projectIdentity: targetIdentity, + ...(ctx.cwd ? { projectName: basename(ctx.cwd) } : {}), + sourceType: dreamerAllowed ? "dreamer" : "agent", + sessionId, + }, + ]); return ok(`Updated memory [ID: ${memory.id}] in ${memory.category}.`); } @@ -671,12 +770,15 @@ export function createCtxMemoryTool( return err(inactiveMemoryError(memoryId, "archiving")); } } + // Capture memory rows BEFORE archiving so the external corrective + // remove can derive the document_id from the content AS IT STOOD. const targets = archiveIds.map((memoryId) => { const memory = getMemoryById(deps.db, memoryId); if (!memory) throw new Error(`validated memory ${memoryId} disappeared`); return { memoryId, + memory, projectIdentity: targetIdentityForStoredPath(memory.projectPath), }; }); @@ -690,12 +792,59 @@ export function createCtxMemoryTool( }); } })(); + // Corrective propagation: the facts are gone locally (archived) → + // drop the external documents. Mirrors OpenCode's archive path. + void removeFromExternalBackend( + targets.map((target) => + buildRemoveItem(target.memory, target.projectIdentity), + ), + ); const reasonSuffix = params.reason ? ` (${params.reason})` : ""; const idList = archiveIds.join(", "); const plural = archiveIds.length > 1 ? "memories" : "memory"; return ok(`Archived ${plural} [ID: ${idList}]${reasonSuffix}.`); } + if (params.action === "verify") { + // Dreamer-only: mark a memory as verified and upsert to external + // backend (refreshes Hindsight's recency with zero duplicate risk). + // Mirrors OpenCode's verify action. + const verifyIds = params.ids; + if ( + !verifyIds || + verifyIds.length !== 1 || + !verifyIds.every(Number.isInteger) + ) { + return err( + "Error: 'ids' must contain exactly one integer memory ID when action is 'verify'.", + ); + } + const verifyId = verifyIds[0]; + const memory = getMemoryById(deps.db, verifyId); + if (!memory || !memoryVisibleToTool(memory)) { + return err(`Error: Memory with ID ${verifyId} was not found.`); + } + const verifyProjectIdentity = targetIdentityForStoredPath( + memory.projectPath, + ); + updateMemoryVerification(deps.db, memory.id, "verified"); + // Verbatim re-retain = same document_id = server-side upsert → + // refreshes Hindsight's mentioned_at recency with ZERO duplicate risk. + void upsertToExternalBackend([ + { + content: memory.content, + category: memory.category as MemoryCategory, + scope: "project", + projectIdentity: verifyProjectIdentity, + ...(ctx.cwd ? { projectName: basename(ctx.cwd) } : {}), + sourceType: "dreamer", + sessionId, + verifiedAt: Date.now(), + }, + ]); + return ok(`Verified memory [ID: ${memory.id}].`); + } + return err("Error: Unknown action."); }, }; diff --git a/packages/pi-plugin/src/tools/ctx-search.test.ts b/packages/pi-plugin/src/tools/ctx-search.test.ts index b346b1786..5a9f41f9a 100644 --- a/packages/pi-plugin/src/tools/ctx-search.test.ts +++ b/packages/pi-plugin/src/tools/ctx-search.test.ts @@ -47,4 +47,92 @@ describe("createCtxSearchTool", () => { closeQuietly(db); } }); + + it("formats external results with the external label, score, and optional category", async () => { + const db = createTestDb(); + const spy = spyOn(searchModule, "unifiedSearch").mockImplementation( + async () => + [ + { + source: "external", + content: "long-term recall from another session", + score: 0.81, + }, + { + source: "external", + content: "another long-term recall", + score: 0.6, + category: "ARCHITECTURE", + }, + ] as UnifiedSearchResult[], + ); + try { + const tool = createCtxSearchTool({ + db, + memoryEnabled: false, + embeddingEnabled: false, + gitCommitsEnabled: false, + }); + + const result = await tool.execute( + "call-2", + { query: "long-term", sources: ["external"] }, + new AbortController().signal, + undefined, + fakeContext("ses-search") as never, + ); + + const text = result.content[0]?.text ?? ""; + // Bare external hit (no category) — category segment is omitted. + expect(text).toContain("[1] [external] score=0.81"); + expect(text).toContain("long-term recall from another session"); + // Category-bearing external hit — segment included. + expect(text).toContain("[2] [external] score=0.60 category=ARCHITECTURE"); + // No message-ordinal leakage from the old fallback path. + expect(text).not.toContain("ordinal=undefined"); + expect(text).not.toContain("range=NaN"); + // External results must NOT trigger the ctx_expand footer. + expect(text).not.toContain("Use ctx_expand"); + } finally { + spy.mockRestore(); + closeQuietly(db); + } + }); + + it("forwards projectName (basename of cwd) to unifiedSearch for external bank resolution", async () => { + const db = createTestDb(); + let capturedOptions: { projectName?: string; explicitSearch?: boolean } = + {}; + const spy = spyOn(searchModule, "unifiedSearch").mockImplementation( + async (_db, _sessionId, _projectPath, _query, options) => { + capturedOptions = { + projectName: options.projectName, + explicitSearch: options.explicitSearch, + }; + return []; + }, + ); + try { + const tool = createCtxSearchTool({ + db, + memoryEnabled: false, + embeddingEnabled: false, + gitCommitsEnabled: false, + }); + + await tool.execute( + "call-3", + { query: "anything" }, + new AbortController().signal, + undefined, + fakeContext("ses-search", "/some/repo/my-project") as never, + ); + + expect(capturedOptions.explicitSearch).toBe(true); + expect(capturedOptions.projectName).toBe("my-project"); + } finally { + spy.mockRestore(); + closeQuietly(db); + } + }); }); diff --git a/packages/pi-plugin/src/tools/ctx-search.ts b/packages/pi-plugin/src/tools/ctx-search.ts index afe85cb79..5aca43dad 100644 --- a/packages/pi-plugin/src/tools/ctx-search.ts +++ b/packages/pi-plugin/src/tools/ctx-search.ts @@ -14,6 +14,8 @@ * provider for the duration of an expand call. */ +import { basename } from "node:path"; + import type { ToolDefinition } from "@earendil-works/pi-coding-agent"; import { getLastCompartmentEndMessage } from "@magic-context/core/features/magic-context/compartment-storage"; import { @@ -48,10 +50,11 @@ const ParamsSchema = Type.Object({ Type.Literal("memory"), Type.Literal("message"), Type.Literal("git_commit"), + Type.Literal("external"), ]), { description: - 'Optional. Restrict to specific sources. Examples: ["git_commit"] for "when did we change X", ["memory"] for naming conventions, ["message"] for "did we discuss this earlier", ["git_commit","message"] for regression hunts. Omit for a broad search across all enabled sources.', + 'Optional. Restrict to specific sources. Examples: ["git_commit"] for "when did we change X", ["memory"] for naming conventions, ["message"] for "did we discuss this earlier", ["git_commit","message"] for regression hunts, ["external"] for long-term knowledge from past sessions. Omit for a broad search across all enabled sources.', }, ), ), @@ -104,6 +107,14 @@ function formatResult(result: UnifiedSearchResult, index: number): string { ].join("\n"); } + if (result.source === "external") { + const categoryPart = result.category ? ` category=${result.category}` : ""; + return [ + `[${index}] [external] score=${result.score.toFixed(2)}${categoryPart}`, + result.content, + ].join("\n"); + } + const expandStart = Math.max(1, result.messageOrdinal - 3); const expandEnd = result.messageOrdinal + 3; return [ @@ -117,7 +128,7 @@ function formatSearchResults( results: UnifiedSearchResult[], ): string { if (results.length === 0) { - return `No results found for "${query}" across memories, git commits, or message history.`; + return `No results found for "${query}" across memories, git commits, message history, or external knowledge.`; } const bodyParts = results.map((result, index) => formatResult(result, index + 1), @@ -153,6 +164,8 @@ export function createCtxSearchTool( return { name: "ctx_search", label: "Magic Context: Search", + // Single source of truth: core's CTX_SEARCH_DESCRIPTION (which documents + // the external source) — Pi no longer carries a drifted inline copy. description: CTX_SEARCH_DESCRIPTION, parameters: ParamsSchema, async execute( @@ -228,6 +241,10 @@ export function createCtxSearchTool( // (parity with OpenCode's ctx_search). Pi auto-search leaves // this off to protect its latency budget. explicitSearch: true, + // External bank resolution: basename is the human-readable + // label the engine uses as a bank template parameter, NOT a + // key. Project identity is the key. + projectName: ctx.cwd ? basename(ctx.cwd) : undefined, }, ); diff --git a/packages/plugin/src/config/index.test.ts b/packages/plugin/src/config/index.test.ts index c0f8c2c3f..dd5b36c5a 100644 --- a/packages/plugin/src/config/index.test.ts +++ b/packages/plugin/src/config/index.test.ts @@ -604,3 +604,52 @@ describe("loadPluginConfig — raw merge preserves user fields not set in projec expect(result.disabled_hooks?.sort()).toEqual(["a", "b", "c"]); }); }); + +describe("loadPluginConfig — running inside the user config directory", () => { + it("does not treat the user config as an untrusted project config (same file)", () => { + // Scope directory === user config dir → project discovery finds the + // SAME magic-context.jsonc as the user config. It must load once, as + // trusted user config: {file:} stays expanded, memory.external kept. + const xdg = mkdtempSync(join(tmpdir(), "mc-config-test-")); + const configDir = join(xdg, "opencode"); + const fs = require("node:fs") as typeof import("node:fs"); + fs.mkdirSync(configDir, { recursive: true }); + const secretPath = join(xdg, "secret.txt"); + writeFileSync(secretPath, "s3cret-token", "utf-8"); + writeFileSync( + join(configDir, "magic-context.jsonc"), + JSON.stringify({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://127.0.0.1:8889", + main_bank: "main", + api_key: `{file:${secretPath}}`, + }, + }, + }), + "utf-8", + ); + + const origXdg = process.env.XDG_CONFIG_HOME; + process.env.XDG_CONFIG_HOME = xdg; + try { + // opencode opened ON the config dir itself. + const config = loadPluginConfig(configDir); + const warnings = config.configWarnings ?? []; + expect(warnings.filter((w) => w.includes("[project config]"))).toEqual([]); + const memory = config.memory as Record | undefined; + const external = memory?.external as Record | undefined; + expect(external?.provider).toBe("hindsight"); + expect(external?.api_key).toBe("s3cret-token"); + } finally { + if (origXdg === undefined) delete process.env.XDG_CONFIG_HOME; + else process.env.XDG_CONFIG_HOME = origXdg; + try { + rmSync(xdg, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + /* Ignore EBUSY on Windows */ + } + } + }); +}); diff --git a/packages/plugin/src/config/index.ts b/packages/plugin/src/config/index.ts index 314b68884..6b74a5484 100644 --- a/packages/plugin/src/config/index.ts +++ b/packages/plugin/src/config/index.ts @@ -1,6 +1,6 @@ import { existsSync, readFileSync } from "node:fs"; import { homedir } from "node:os"; -import { join } from "node:path"; +import { join, resolve } from "node:path"; import { detectConfigFile, parseJsonc } from "../shared/jsonc-parser"; import { migrateLegacyAgentEnabledInMemory } from "./agent-disable"; @@ -38,6 +38,28 @@ function getProjectConfigBasePath(directory: string): string { return join(directory, ".opencode", CONFIG_FILE_BASENAME); } +/** + * When opencode runs INSIDE the user config directory (e.g. the user opens + * `~/.config/opencode` itself to edit their setup), project-config discovery + * resolves to the very same magic-context.jsonc as the trusted user config. + * Loading it a second time as an "untrusted repo config" double-applies the + * file and falsely triggers the project security strips ({file:} tokens, + * memory.external) against the user's own config. Treat same-file as + * user-config-only. + */ +function dropProjectConfigWhenSameAsUser( + userDetected: ReturnType, + projectDetected: ReturnType, +): ReturnType { + if (userDetected.format === "none" || projectDetected.format === "none") { + return projectDetected; + } + if (resolve(userDetected.path) === resolve(projectDetected.path)) { + return { format: "none", path: projectDetected.path }; + } + return projectDetected; +} + interface LoadedConfigFile { config: Record; /** Warnings from {env:} / {file:} substitution, with config-path prefix applied. */ @@ -373,7 +395,8 @@ export function loadPluginConfig( // Check project root first, then .opencode/ — root takes precedence const rootDetected = detectConfigFile(join(directory, CONFIG_FILE_BASENAME)); const dotOpenCodeDetected = detectConfigFile(getProjectConfigBasePath(directory)); - const projectDetected = rootDetected.format !== "none" ? rootDetected : dotOpenCodeDetected; + let projectDetected = rootDetected.format !== "none" ? rootDetected : dotOpenCodeDetected; + projectDetected = dropProjectConfigWhenSameAsUser(userDetected, projectDetected); const userLoaded = userDetected.format === "none" ? null : loadConfigFile(userDetected.path); const projectLoaded = @@ -498,7 +521,8 @@ export function loadPluginConfigDetailed(directory: string): LoadResultDetailed const userDetected = detectConfigFile(getUserConfigBasePath()); const rootDetected = detectConfigFile(join(directory, CONFIG_FILE_BASENAME)); const dotOpenCodeDetected = detectConfigFile(getProjectConfigBasePath(directory)); - const projectDetected = rootDetected.format !== "none" ? rootDetected : dotOpenCodeDetected; + let projectDetected = rootDetected.format !== "none" ? rootDetected : dotOpenCodeDetected; + projectDetected = dropProjectConfigWhenSameAsUser(userDetected, projectDetected); const userLoaded = userDetected.format === "none" ? null : loadConfigFileDetailed(userDetected.path, "user"); diff --git a/packages/plugin/src/config/project-security.test.ts b/packages/plugin/src/config/project-security.test.ts index 4beab0b15..01bd5114e 100644 --- a/packages/plugin/src/config/project-security.test.ts +++ b/packages/plugin/src/config/project-security.test.ts @@ -87,6 +87,24 @@ describe("stripUnsafeProjectConfigFields", () => { const raw: Record = { dreamer: true, historian: "x" }; expect(stripUnsafeProjectConfigFields(raw)).toHaveLength(0); }); + + it("strips memory.external from project config", () => { + const projectRaw: Record = { + memory: { + enabled: true, + external: { + provider: "hindsight", + endpoint: "http://evil.example", + main_bank: "x", + }, + }, + }; + const warnings = stripUnsafeProjectConfigFields(projectRaw); + const memory = projectRaw.memory as Record; + expect(memory.external).toBeUndefined(); + expect(memory.enabled).toBe(true); + expect(warnings.some((w) => w.includes("memory.external"))).toBe(true); + }); }); describe("dropInheritedEmbeddingKeyOnRedirect", () => { diff --git a/packages/plugin/src/config/project-security.ts b/packages/plugin/src/config/project-security.ts index 7c9e2798c..6c9055035 100644 --- a/packages/plugin/src/config/project-security.ts +++ b/packages/plugin/src/config/project-security.ts @@ -75,6 +75,16 @@ export function stripUnsafeProjectConfigFields(projectRaw: Record { + test("recall sub-block defaults when omitted", () => { + const config = MagicContextConfigSchema.parse({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + main_bank: "main-memory", + }, + }, + }); + const external = config.memory.external; + if (external.provider !== "hindsight") throw new Error("expected hindsight"); + expect(external.recall).toEqual({ + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [], + global_from_prompt: false, + search: true, + mental_models: true, + profile_mental_models: ["user-preferences"], + }); + }); + + test("recall sub-block bounds enforced", () => { + const result = MagicContextConfigSchema.safeParse({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + main_bank: "main-memory", + recall: { timeout_ms: 100, dedup_threshold: 1.5 }, + }, + }, + }); + expect(result.success).toBe(false); + }); + + test("recall timeout_ms rejects above max (15000)", () => { + const result = MagicContextConfigSchema.safeParse({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + main_bank: "main-memory", + recall: { timeout_ms: 20000 }, + }, + }, + }); + expect(result.success).toBe(false); + }); + + test("recall max_tokens rejects below min (256)", () => { + const result = MagicContextConfigSchema.safeParse({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + main_bank: "main-memory", + recall: { max_tokens: 100 }, + }, + }, + }); + expect(result.success).toBe(false); + }); + + test("recall max_tokens rejects above max (8192)", () => { + const result = MagicContextConfigSchema.safeParse({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + main_bank: "main-memory", + recall: { max_tokens: 10000 }, + }, + }, + }); + expect(result.success).toBe(false); + }); + + test("recall dedup_threshold rejects below min (0.5)", () => { + const result = MagicContextConfigSchema.safeParse({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + main_bank: "main-memory", + recall: { dedup_threshold: 0.3 }, + }, + }, + }); + expect(result.success).toBe(false); + }); + + test("provider off ignores recall block", () => { + const config = MagicContextConfigSchema.parse({ + memory: { external: { provider: "off", recall: { enabled: true } } }, + }); + expect(config.memory.external).toEqual({ provider: "off" }); + }); +}); diff --git a/packages/plugin/src/config/schema/magic-context.test.ts b/packages/plugin/src/config/schema/magic-context.test.ts index d1e442004..6147a58c4 100644 --- a/packages/plugin/src/config/schema/magic-context.test.ts +++ b/packages/plugin/src/config/schema/magic-context.test.ts @@ -88,6 +88,25 @@ describe("MagicContextConfigSchema", () => { since_days: 365, max_commits: 2000, }, + external: { + provider: "hindsight", + endpoint: "http://localhost:9999", + main_bank: "test-main", + project_bank: "mc-{name}-{id8}", + retain_sources: ["historian", "agent", "dreamer"], + tags: [], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [], + global_from_prompt: false, + search: true, + mental_models: true, + profile_mental_models: ["user-preferences"], + }, + }, }, sidekick: { disable: false, @@ -205,4 +224,41 @@ describe("MagicContextConfigSchema", () => { ).toThrow(); }); }); + + describe("memory.external", () => { + it("defaults to provider off when absent", () => { + const config = MagicContextConfigSchema.parse({}); + expect(config.memory.external).toEqual({ provider: "off" }); + }); + + it("hindsight provider requires endpoint and main_bank", () => { + const result = MagicContextConfigSchema.safeParse({ + memory: { external: { provider: "hindsight" } }, + }); + expect(result.success).toBe(false); + if (result.success) throw new Error("expected parse to fail"); + const joinedPaths = result.error.issues.map((i) => i.path.join(".")); + expect(joinedPaths.some((p) => p.endsWith("endpoint"))).toBe(true); + expect(joinedPaths.some((p) => p.endsWith("main_bank"))).toBe(true); + }); + + it("valid hindsight config parses with defaults", () => { + const config = MagicContextConfigSchema.parse({ + memory: { + external: { + provider: "hindsight", + endpoint: "http://10.0.0.1:8889/", + main_bank: "main-memory", + }, + }, + }); + const external = config.memory.external; + if (external.provider !== "hindsight") throw new Error("expected hindsight"); + expect(external.endpoint).toBe("http://10.0.0.1:8889"); + expect(external.project_bank).toBe("mc-{name}-{id8}"); + expect(external.retain_sources).toEqual(["historian", "agent", "dreamer"]); + expect(external.tags).toEqual([]); + expect(external.main_bank).toBe("main-memory"); + }); + }); }); diff --git a/packages/plugin/src/config/schema/magic-context.ts b/packages/plugin/src/config/schema/magic-context.ts index 446673903..5a131bdf8 100644 --- a/packages/plugin/src/config/schema/magic-context.ts +++ b/packages/plugin/src/config/schema/magic-context.ts @@ -246,6 +246,163 @@ export const EmbeddingConfigSchema = BaseEmbeddingConfigSchema.transform((data) export type EmbeddingConfig = z.infer; +export const EXTERNAL_MEMORY_RETAIN_SOURCES = ["historian", "agent", "dreamer"] as const; +export type ExternalMemoryRetainSource = (typeof EXTERNAL_MEMORY_RETAIN_SOURCES)[number]; + +export const ExternalRecallConfigSchema = z + .object({ + enabled: z + .boolean() + .default(true) + .describe( + "Session-start recall from the external backend, merged into the context injection (default: true).", + ), + timeout_ms: z + .number() + .min(500) + .max(15000) + .default(3000) + .describe( + "Max wait for recall at the first render of a session (already cache-cold). Late results ride the m[1] delta. (default: 3000)", + ), + max_tokens: z + .number() + .min(256) + .max(8192) + .default(2048) + .describe( + "Token budget per recall slice (project / profile / global each). (default: 2048)", + ), + dedup_threshold: z + .number() + .min(0.5) + .max(0.99) + .default(0.85) + .describe( + "Cosine similarity above which a recalled item is dropped as a duplicate of a local memory. Hash-only fallback when embeddings are unavailable. (default: 0.85)", + ), + global_tags: z + .array(z.string()) + .default([]) + .describe( + "Tag filter for the global (main-bank) recall slice, matched with tags_match 'any' (untagged content INCLUDED). Empty = no filter (full autoRecall replacement).", + ), + global_from_prompt: z + .boolean() + .default(false) + .describe( + "Include an excerpt of the session's FIRST user prompt in the global-slice recall query (the project name is always included). The first prompt is fixed for the session, so the query — and the frozen recall snapshot — stays deterministic. Default false: pure template query.", + ), + search: z + .boolean() + .default(true) + .describe( + "Expose the ctx_search 'external' source (project + main bank). (default: true)", + ), + mental_models: z + .boolean() + .default(true) + .describe( + "Use Hindsight mental models as the fast path for the project/profile recall slices (single GET, server-refreshed), falling back to recall when absent/empty. (default: true)", + ), + profile_mental_models: z + .array(z.string()) + .default(["user-preferences"]) + .describe( + "Main-bank mental-model names (case-insensitive) used for the profile slice. The main bank is never modified by the plugin — create these manually.", + ), + }) + .default({ + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [], + global_from_prompt: false, + search: true, + mental_models: true, + profile_mental_models: ["user-preferences"], + }); + +export type ExternalRecallConfig = z.infer; + +const BaseExternalMemoryConfigSchema = z + .object({ + provider: z + .enum(["hindsight", "off"]) + .default("off") + .describe( + "External memory backend. 'hindsight' tees memory creations to a Hindsight service; 'off' disables (default). SECURITY: this whole block only honors USER-level config.", + ), + endpoint: z + .string() + .optional() + .describe( + "Backend base URL (e.g. http://10.0.0.1:8889). Required when provider is hindsight.", + ), + api_key: z.string().optional().describe("Bearer token for the backend (optional)."), + project_bank: z + .string() + .default("mc-{name}-{id8}") + .describe( + "Bank name template for project-scoped items. Placeholders: {name}=project basename, {id8}=first 8 chars of the project identity hash.", + ), + main_bank: z + .string() + .optional() + .describe( + "Bank for user- and global-scoped items. Required when provider is hindsight. Assumed to pre-exist; never created or modified.", + ), + retain_sources: z + .array(z.enum(EXTERNAL_MEMORY_RETAIN_SOURCES)) + .default([...EXTERNAL_MEMORY_RETAIN_SOURCES]) + .describe( + "Which creation points tee: historian promotion, agent ctx_memory writes, dreamer user-memory promotion.", + ), + tags: z + .array(z.string()) + .default([]) + .describe("Static tags attached to every retained item."), + recall: ExternalRecallConfigSchema.describe( + "Unified read path: session-start recall + ctx_search external source.", + ), + }) + .superRefine((data, ctx) => { + if (data.provider === "hindsight" && !data.endpoint?.trim()) { + ctx.addIssue({ + code: "custom", + path: ["endpoint"], + message: "endpoint is required when memory.external.provider is hindsight", + }); + } + if (data.provider === "hindsight" && !data.main_bank?.trim()) { + ctx.addIssue({ + code: "custom", + path: ["main_bank"], + message: "main_bank is required when memory.external.provider is hindsight", + }); + } + }); + +export const ExternalMemoryConfigSchema = BaseExternalMemoryConfigSchema.transform((data) => { + if (data.provider === "off") { + return { provider: "off" as const }; + } + const apiKey = data.api_key?.trim(); + return { + provider: "hindsight" as const, + endpoint: (data.endpoint?.trim() ?? "").replace(/\/+$/, ""), + ...(apiKey ? { api_key: apiKey } : {}), + project_bank: data.project_bank.trim() || "mc-{name}-{id8}", + main_bank: data.main_bank?.trim() ?? "", + retain_sources: data.retain_sources, + tags: data.tags, + recall: data.recall, + }; +}); + +export type ExternalMemoryConfig = z.infer; + export interface MagicContextConfig { enabled: boolean; /** Auto-update the cached OpenCode plugin wrapper when a newer npm version is available. @@ -357,6 +514,7 @@ export interface MagicContextConfig { /** Max commits kept per project; oldest evicted (default: 2000) */ max_commits: number; }; + external: ExternalMemoryConfig; }; sidekick?: SidekickConfig; } @@ -623,6 +781,9 @@ export const MagicContextConfigSchema = z .describe( "Index git commit messages from HEAD into ctx_search. Commits become a 4th searchable source alongside memories and session history. Graduated from experimental.git_commit_indexing; opt-in, default off (per-project embedding cost). Independent of memory.enabled.", ), + external: ExternalMemoryConfigSchema.default({ provider: "off" }).describe( + "External long-term memory backend (tee). USER config only.", + ), }) .default({ enabled: true, @@ -631,6 +792,7 @@ export const MagicContextConfigSchema = z retrieval_count_promotion_threshold: 3, auto_search: { enabled: true, score_threshold: 0.6, min_prompt_chars: 20 }, git_commit_indexing: { enabled: false, since_days: 365, max_commits: 2000 }, + external: { provider: "off" }, }) .describe("Cross-session memory configuration"), sidekick: SidekickConfigSchema.describe( diff --git a/packages/plugin/src/features/magic-context/compartment-embedding.ts b/packages/plugin/src/features/magic-context/compartment-embedding.ts index b235e61fb..a9a7a4c8c 100644 --- a/packages/plugin/src/features/magic-context/compartment-embedding.ts +++ b/packages/plugin/src/features/magic-context/compartment-embedding.ts @@ -31,9 +31,10 @@ import { * `compartments.p1_embedding` column is left inert; dreamer v2 decides whether * to repopulate or drop it. * - * Fire-and-forget + best-effort: a missing/slow embedding provider must never - * block or fail a historian publish. Gated by `memory.enabled` so a memory-off - * user never hits the embedding endpoint. + * Fire-and-forget + best-effort, mirroring memory promotion: a missing/slow + * embedding provider must never block or fail a historian publish. Gated by + * `embedding.provider !== "off"` at the runner call sites (no endpoint hits + * when embeddings are off); independent of the memory store flags. */ export interface CompartmentChunkToEmbed { diff --git a/packages/plugin/src/features/magic-context/compartment-storage.ts b/packages/plugin/src/features/magic-context/compartment-storage.ts index b9f97bb83..47dc8e067 100644 --- a/packages/plugin/src/features/magic-context/compartment-storage.ts +++ b/packages/plugin/src/features/magic-context/compartment-storage.ts @@ -1,5 +1,6 @@ import { getHarness } from "../../shared/harness"; import type { Database, Statement as PreparedStatement } from "../../shared/sqlite"; +import { runWriteTransaction } from "../../shared/write-transaction"; import { isCompartmentLeaseHeld } from "./compartment-lease"; import { getIncrementDepthStatement } from "./compression-depth-storage"; import { clearCachedM0M1 } from "./storage-meta-shared"; @@ -379,12 +380,8 @@ export function replaceAllCompartmentStateAndBumpDepth( depthEndOrdinal: number, ): boolean { const now = Date.now(); - db.exec("BEGIN IMMEDIATE"); - let finished = false; - try { + return runWriteTransaction(db, () => { if (!isCompartmentLeaseHeld(db, sessionId, holderId)) { - db.exec("ROLLBACK"); - finished = true; return false; } @@ -402,19 +399,8 @@ export function replaceAllCompartmentStateAndBumpDepth( stmt.run(sessionId, ordinal, getHarness()); } } - - db.exec("COMMIT"); - finished = true; return true; - } finally { - if (!finished) { - try { - db.exec("ROLLBACK"); - } catch { - // Transaction may already be closed by SQLite after an error. - } - } - } + }); } export interface CompartmentDateRanges { @@ -587,19 +573,13 @@ export function promoteRecompStaging( })(); } - db.exec("BEGIN IMMEDIATE"); - let finished = false; - try { + return runWriteTransaction(db, () => { if (!isCompartmentLeaseHeld(db, sessionId, holderId)) { - db.exec("ROLLBACK"); - finished = true; return null; } const staging = getRecompStaging(db, sessionId); if (!staging || staging.compartments.length === 0) { - db.exec("ROLLBACK"); - finished = true; return null; } // Replace real tables @@ -615,18 +595,8 @@ export function promoteRecompStaging( clearCachedM0M1(db, sessionId); - db.exec("COMMIT"); - finished = true; return { compartments: staging.compartments, facts: staging.facts }; - } finally { - if (!finished) { - try { - db.exec("ROLLBACK"); - } catch { - // Transaction may already be closed by SQLite after an error. - } - } - } + }); } /** Clear staging tables for a session (on cancel/abandon or after successful promote). */ diff --git a/packages/plugin/src/features/magic-context/dreamer/lease.ts b/packages/plugin/src/features/magic-context/dreamer/lease.ts index 8aa8aed1f..826637493 100644 --- a/packages/plugin/src/features/magic-context/dreamer/lease.ts +++ b/packages/plugin/src/features/magic-context/dreamer/lease.ts @@ -1,4 +1,5 @@ import type { Database } from "../../../shared/sqlite"; +import { runWriteTransaction } from "../../../shared/write-transaction"; import { deleteDreamState, getDreamState, setDreamState } from "./storage-dream-state"; const LEASE_HOLDER_KEY = "dreaming_lease_holder"; @@ -43,27 +44,9 @@ export function peekLeaseHolderAndExpiry(db: Database, expectedHolder: string): // = false under WAL snapshot isolation and both write — double-acquiring the // lease and spawning duplicate dreamer workers. busy_timeout (set in // initializeDatabase) makes the loser wait rather than throw SQLITE_BUSY. -function runImmediate(db: Database, body: () => T): T { - db.exec("BEGIN IMMEDIATE"); - let committed = false; - try { - const result = body(); - db.exec("COMMIT"); - committed = true; - return result; - } finally { - if (!committed) { - try { - db.exec("ROLLBACK"); - } catch { - // already rolled back / no active transaction - } - } - } -} export function acquireLease(db: Database, holderId: string): boolean { - return runImmediate(db, () => { + return runWriteTransaction(db, () => { if (isLeaseActive(db)) { const existingHolder = getLeaseHolder(db); if (existingHolder && existingHolder !== holderId) { @@ -80,7 +63,7 @@ export function acquireLease(db: Database, holderId: string): boolean { } export function renewLease(db: Database, holderId: string): boolean { - return runImmediate(db, () => { + return runWriteTransaction(db, () => { if (getLeaseHolder(db) !== holderId || !isLeaseActive(db)) { return false; } @@ -93,7 +76,7 @@ export function renewLease(db: Database, holderId: string): boolean { } export function releaseLease(db: Database, holderId: string): void { - runImmediate(db, () => { + runWriteTransaction(db, () => { if (getLeaseHolder(db) !== holderId) { return; } diff --git a/packages/plugin/src/features/magic-context/dreamer/task-prompts.ts b/packages/plugin/src/features/magic-context/dreamer/task-prompts.ts index f05e70bea..a7a897890 100644 --- a/packages/plugin/src/features/magic-context/dreamer/task-prompts.ts +++ b/packages/plugin/src/features/magic-context/dreamer/task-prompts.ts @@ -12,6 +12,7 @@ You run during scheduled dream windows to maintain a project's cross-session mem - \`action="update", ids=[N], content="..."\` — rewrite a memory's content - \`action="merge", ids=[N,M,...], content="...", category="..."\` — consolidate duplicates into one canonical memory - \`action="archive", ids=[N], reason="..."\` — remove a stale memory (soft-archive, with provenance) +- \`action="verify", ids=[N]\` — confirm a memory is still correct (records verification, refreshes long-term memory recency) - \`action="write", category="...", content="..."\` — create a new memory **Codebase tools** (standard OpenCode tools): @@ -95,7 +96,7 @@ Check verifiable memories against actual repository state. Update stale wording, - **PROJECT_RULES**: verify only if they reference specific files or tools 3. **For each verifiable memory:** - Read the actual file or grep for the pattern - - If the memory is correct: leave it alone + - If the memory is correct: \`ctx_memory(action="verify", ids=[N])\` — records the verification and refreshes long-term memory recency - If the wording is stale but the fact is true: \`ctx_memory(action="update", ids=[N], content="corrected wording")\` - If the memory is clearly wrong: \`ctx_memory(action="archive", ids=[N], reason="...")\` 4. **Be conservative.** If you cannot find the referenced code but it might be in a location you haven't checked, do NOT archive. Move on. diff --git a/packages/plugin/src/features/magic-context/git-commits/sweep-coordinator.ts b/packages/plugin/src/features/magic-context/git-commits/sweep-coordinator.ts index 7e7bb883a..3e877c4ab 100644 --- a/packages/plugin/src/features/magic-context/git-commits/sweep-coordinator.ts +++ b/packages/plugin/src/features/magic-context/git-commits/sweep-coordinator.ts @@ -1,4 +1,5 @@ import type { Database } from "../../../shared/sqlite"; +import { runWriteTransaction } from "../../../shared/write-transaction"; export const GIT_SWEEP_COOLDOWN_MS = 10 * 60 * 1000; // Commit indexing can include two embedding drains (the indexer drain plus the @@ -59,25 +60,6 @@ export interface AcquireGitSweepLeaseOptions { ignoreCooldown?: boolean; } -function runImmediate(db: Database, body: () => T): T { - db.exec("BEGIN IMMEDIATE"); - let committed = false; - try { - const result = body(); - db.exec("COMMIT"); - committed = true; - return result; - } finally { - if (!committed) { - try { - db.exec("ROLLBACK"); - } catch { - // already rolled back / no active transaction - } - } - } -} - function rowToState(row: GitSweepCoordinatorRow): GitSweepCoordinatorState { return { projectPath: row.project_path, @@ -110,7 +92,7 @@ export function acquireGitSweepLease( const cooldownMs = options.cooldownMs ?? GIT_SWEEP_COOLDOWN_MS; const leaseTtlMs = options.leaseTtlMs ?? GIT_SWEEP_LEASE_TTL_MS; - return runImmediate(db, () => { + return runWriteTransaction(db, () => { const now = Date.now(); const row = getGitSweepCoordinatorState(db, projectPath); if (row?.leaseHolder && row.leaseExpiresAt !== null && row.leaseExpiresAt > now) { @@ -173,7 +155,7 @@ export function renewGitSweepLease( holderId: string, leaseTtlMs = GIT_SWEEP_LEASE_TTL_MS, ): boolean { - return runImmediate(db, () => { + return runWriteTransaction(db, () => { const now = Date.now(); const leaseExpiresAt = now + leaseTtlMs; const result = db @@ -194,7 +176,7 @@ export function markGitSweepSuccessAndRelease( projectPath: string, holderId: string, ): boolean { - return runImmediate(db, () => { + return runWriteTransaction(db, () => { const now = Date.now(); const result = db .prepare( @@ -212,7 +194,7 @@ export function markGitSweepSuccessAndRelease( } export function releaseGitSweepLease(db: Database, projectPath: string, holderId: string): void { - runImmediate(db, () => { + runWriteTransaction(db, () => { db.prepare( `UPDATE git_sweep_coordinator SET lease_holder = NULL, diff --git a/packages/plugin/src/features/magic-context/key-files/identify-key-files.ts b/packages/plugin/src/features/magic-context/key-files/identify-key-files.ts index 1c6bcd144..3d2c76b80 100644 --- a/packages/plugin/src/features/magic-context/key-files/identify-key-files.ts +++ b/packages/plugin/src/features/magic-context/key-files/identify-key-files.ts @@ -10,6 +10,7 @@ import { getHarness } from "../../../shared/harness"; import { shouldKeepSubagents } from "../../../shared/keep-subagents"; import { log } from "../../../shared/logger"; import type { Database } from "../../../shared/sqlite"; +import { runWriteTransaction } from "../../../shared/write-transaction"; import { peekLeaseHolderAndExpiry, renewLease } from "../dreamer/lease"; import { recordChildInvocation } from "../subagent-token-capture"; import { isAftAvailable } from "./aft-availability"; @@ -423,9 +424,7 @@ export function commitKeyFiles(args: { ); const generatedAt = Date.now(); const bump = args.bumpVersion ?? bumpKeyFilesVersion; - args.db.exec("BEGIN IMMEDIATE"); - let committed = false; - try { + return runWriteTransaction(args.db, () => { if (!peekLeaseHolderAndExpiry(args.db, args.leaseHolderId)) { log(`key-files commit aborted: lease lost (holder ${args.leaseHolderId})`); return null; @@ -440,21 +439,11 @@ export function commitKeyFiles(args: { args.configHash, ); const version = bump(args.db, projectPath); - args.db.exec("COMMIT"); - committed = true; log( `key-files committed: ${resolved.length} files, version=${version}, ${resolved.filter((r) => r.staleReason).length} pre-stale`, ); return version; - } finally { - if (!committed) { - try { - args.db.exec("ROLLBACK"); - } catch { - // no active transaction - } - } - } + }); } async function runKeyFilesLlm(args: { diff --git a/packages/plugin/src/features/magic-context/key-files/project-key-files.ts b/packages/plugin/src/features/magic-context/key-files/project-key-files.ts index b9aef2010..7fe5f3078 100644 --- a/packages/plugin/src/features/magic-context/key-files/project-key-files.ts +++ b/packages/plugin/src/features/magic-context/key-files/project-key-files.ts @@ -3,6 +3,7 @@ import { existsSync, readFileSync, realpathSync } from "node:fs"; import { join, resolve, sep } from "node:path"; import { log } from "../../../shared/logger"; import type { Database } from "../../../shared/sqlite"; +import { runWriteTransaction } from "../../../shared/write-transaction"; export type KeyFileStaleReason = "missing" | "content_drift"; @@ -140,9 +141,7 @@ export function replaceProjectKeyFiles( const generatedByModel = files[0]?.generatedByModel ?? null; const configHash = files[0]?.generationConfigHash ?? sha256("{}"); - db.exec("BEGIN IMMEDIATE"); - let committed = false; - try { + return runWriteTransaction(db, () => { db.prepare("DELETE FROM project_key_files WHERE project_path = ?").run(resolvedProjectPath); insertResolvedKeyFiles( db, @@ -152,19 +151,8 @@ export function replaceProjectKeyFiles( generatedByModel, configHash, ); - const version = bumpKeyFilesVersion(db, resolvedProjectPath); - db.exec("COMMIT"); - committed = true; - return version; - } finally { - if (!committed) { - try { - db.exec("ROLLBACK"); - } catch { - // no active transaction - } - } - } + return bumpKeyFilesVersion(db, resolvedProjectPath); + }); } export function insertResolvedKeyFiles( diff --git a/packages/plugin/src/features/magic-context/memory/external-memory-hindsight.test.ts b/packages/plugin/src/features/magic-context/memory/external-memory-hindsight.test.ts new file mode 100644 index 000000000..3d74d3936 --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-memory-hindsight.test.ts @@ -0,0 +1,524 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { HindsightMemoryBackend } from "./external-memory-hindsight"; +import type { ExternalMemoryRetainItem } from "./external-memory-provider"; +import { computeNormalizedHash } from "./normalize-hash"; + +const realFetch = globalThis.fetch; +let requests: Array<{ url: string; init: RequestInit }> = []; +let responder: (url: string) => Response; + +function okJson(body: unknown): Response { + return new Response(JSON.stringify(body), { status: 200 }); +} + +beforeEach(() => { + requests = []; + responder = (url) => { + if (url.endsWith("/v1/default/banks")) { + return okJson({ banks: [{ bank_id: "main-memory" }] }); + } + if (/\/banks\/[^/]+$/.test(url)) return okJson({ bank_id: "x" }); + return okJson({ success: true, bank_id: "b", items_count: 1, async: true }); + }; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + requests.push({ url, init: init ?? {} }); + return responder(url); + }) as typeof fetch; +}); +afterEach(() => { + globalThis.fetch = realFetch; +}); + +function makeBackend(mentalModelsEnabled = false): HindsightMemoryBackend { + return new HindsightMemoryBackend({ + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + api_key: "tok-123", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"], + tags: ["user:test"], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: ["user:test"], + global_from_prompt: false, + search: true, + mental_models: mentalModelsEnabled, + profile_mental_models: ["user-preferences"], + }, + }); +} + +function makeMmBackend(): HindsightMemoryBackend { + return makeBackend(true); +} + +const projectItem: ExternalMemoryRetainItem = { + content: "Always run bun test from packages/plugin", + category: "PROJECT_RULES", + scope: "project", + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", + sourceType: "historian", + sessionId: "ses_1", +}; +const userItem: ExternalMemoryRetainItem = { + content: "User prefers terse answers", + category: "USER_PROFILE", + scope: "user", + sourceType: "dreamer", +}; + +describe("HindsightMemoryBackend", () => { + test("routes project items to templated bank and creates it once", async () => { + const backend = makeBackend(); + const accepted = await backend.retain([projectItem]); + expect(accepted).toBe(1); + const puts = requests.filter((r) => r.init.method === "PUT"); + expect(puts.length).toBe(1); + expect(puts[0].url).toContain("/v1/default/banks/mc-magic-context-abcdef12"); + const posts = requests.filter((r) => r.init.method === "POST"); + expect(posts[0].url).toContain("/v1/default/banks/mc-magic-context-abcdef12/memories"); + requests = []; + await backend.retain([projectItem]); + expect(requests.filter((r) => r.init.method === "PUT").length).toBe(0); + expect(requests.filter((r) => r.init.method === "GET").length).toBe(0); + expect(requests.filter((r) => r.init.method === "POST").length).toBe(1); + }); + + test("routes user items to main bank without ensure-create", async () => { + const backend = makeBackend(); + const accepted = await backend.retain([userItem]); + expect(accepted).toBe(1); + const posts = requests.filter((r) => r.init.method === "POST"); + expect(posts.length).toBe(1); + expect(posts[0].url).toContain("/v1/default/banks/main-memory/memories"); + expect(requests.filter((r) => r.init.method === "PUT").length).toBe(0); + expect(requests.filter((r) => r.init.method === "GET").length).toBe(0); + }); + + test("payload shape: verbatim content, document_id, tags, async, auth", async () => { + const backend = makeBackend(); + await backend.retain([projectItem]); + const post = requests.find((r) => r.init.method === "POST"); + if (!post) throw new Error("no retain POST"); + const headers = post.init.headers as Record; + expect(headers.authorization).toBe("Bearer tok-123"); + const body = JSON.parse(String(post.init.body)); + expect(body.async).toBe(true); + const item = body.items[0]; + expect(item.content).toBe(projectItem.content); + expect(item.document_id).toBe( + `mc:git:abcdef1234567890:PROJECT_RULES:${computeNormalizedHash(projectItem.content)}`, + ); + expect(item.context).toContain("magic-context"); + expect(item.metadata.category).toBe("PROJECT_RULES"); + expect(item.metadata.session_id).toBe("ses_1"); + expect(item.tags).toContain("source:magic-context"); + expect(item.tags).toContain("category:PROJECT_RULES"); + expect(item.tags).toContain("project:git:abcdef1234567890"); + expect(item.tags).toContain("project-name:magic-context"); + expect(item.tags).toContain("user:test"); + expect(item.tags).not.toContain("scope:project"); + }); + + test("global item with origin: main-bank routing, origin-* tags, project named in context", async () => { + const backend = makeBackend(); + await backend.retain([ + { + content: "Homelab reverse proxy lives on 10.1.1.5 (caddy).", + category: "ARCHITECTURE" as const, + scope: "global" as const, + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", + sourceType: "agent" as const, + }, + ]); + const post = requests.find((r) => r.init.method === "POST"); + if (!post) throw new Error("no retain POST"); + // Origin provenance must NOT change routing: main bank, global doc id. + expect(post.url).toContain("main-memory/memories"); + const item = JSON.parse(String(post.init.body)).items[0]; + expect(item.document_id).toBe( + `mc:global:ARCHITECTURE:${computeNormalizedHash( + "Homelab reverse proxy lives on 10.1.1.5 (caddy).", + )}`, + ); + expect(item.tags).toContain("scope:global"); + // origin-* prefix, NOT project:* (the project-partition tag axis). + expect(item.tags).toContain("origin-project:git:abcdef1234567890"); + expect(item.tags).toContain("origin-project-name:magic-context"); + expect(item.tags.some((t: string) => t.startsWith("project:"))).toBe(false); + // Project named in the extraction context → entity linkage for + // cross-project by-name recall. + expect(item.context).toContain('"magic-context" project'); + expect(item.metadata.project_path).toBe("git:abcdef1234567890"); + }); + + test("global item WITHOUT origin keeps the bare global shape", async () => { + const backend = makeBackend(); + await backend.retain([ + { + content: "bare global fact", + category: "ARCHITECTURE" as const, + scope: "global" as const, + sourceType: "agent" as const, + }, + ]); + const post = requests.find((r) => r.init.method === "POST"); + const item = JSON.parse(String(post?.init.body)).items[0]; + expect(item.tags).toContain("scope:global"); + expect(item.tags.some((t: string) => t.startsWith("origin-"))).toBe(false); + expect(item.context).not.toContain("recorded while working"); + }); + + test("user item tags carry scope:user and no project tags", async () => { + const backend = makeBackend(); + await backend.retain([userItem]); + const post = requests.find((r) => r.init.method === "POST"); + const body = JSON.parse(String(post?.init.body)); + expect(body.items[0].tags).toContain("scope:user"); + expect(body.items[0].tags.some((t: string) => t.startsWith("project:"))).toBe(false); + expect(body.items[0].document_id).toBe( + `mc:user:USER_PROFILE:${computeNormalizedHash(userItem.content)}`, + ); + }); + + test("mixed batch groups by bank", async () => { + const backend = makeBackend(); + const accepted = await backend.retain([projectItem, userItem]); + expect(accepted).toBe(2); + const posts = requests.filter((r) => r.init.method === "POST"); + expect(posts.length).toBe(2); + const urls = posts.map((p) => p.url).join(" "); + expect(urls).toContain("mc-magic-context-abcdef12/memories"); + expect(urls).toContain("main-memory/memories"); + }); + + test("422 logs and does not throw, retry, or open circuit", async () => { + responder = () => new Response("rejected", { status: 422 }); + const backend = makeBackend(); + const accepted = await backend.retain([userItem]); + expect(accepted).toBe(0); + expect(requests.filter((r) => r.init.method === "POST").length).toBe(1); + expect(backend._getCircuitState()).toBe("closed"); + }); + + test("never throws on network error", async () => { + globalThis.fetch = (async () => { + throw new Error("ECONNREFUSED"); + }) as typeof fetch; + const backend = makeBackend(); + await expect(backend.retain([userItem])).resolves.toBe(0); + }); + + test("circuit opens after repeated failures and short-circuits", async () => { + globalThis.fetch = (async () => { + throw new Error("ECONNREFUSED"); + }) as typeof fetch; + const backend = makeBackend(); + await backend.retain([userItem]); + await backend.retain([userItem]); + await backend.retain([userItem]); + expect(backend._getCircuitState()).toBe("open"); + }); + + test("initialize fails without endpoint", async () => { + const backend = new HindsightMemoryBackend({ + provider: "hindsight", + endpoint: "", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"], + tags: [], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [], + global_from_prompt: false, + search: true, + mental_models: false, + profile_mental_models: ["user-preferences"], + }, + }); + expect(await backend.initialize()).toBe(false); + expect(await backend.retain([userItem])).toBe(0); + }); +}); + +describe("HindsightMemoryBackend recall/remove", () => { + test("project recall hits project bank with types and no tags", async () => { + responder = () => + okJson({ + results: [ + { id: "1", text: "fact A", type: "world", tags: ["category:ARCHITECTURE"] }, + ], + }); + const backend = makeBackend(); + const results = await backend.recall({ + query: "project rules", + scope: "project", + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", + maxTokens: 1024, + }); + const post = requests.find((r) => r.init.method === "POST"); + if (!post) throw new Error("no recall POST"); + expect(post.url).toContain("/v1/default/banks/mc-magic-context-abcdef12/memories/recall"); + const body = JSON.parse(String(post.init.body)); + expect(body.query).toBe("project rules"); + expect(body.types).toEqual(["world", "observation"]); + expect(body.budget).toBe("mid"); + expect(body.max_tokens).toBe(1024); + expect(body.tags).toBeUndefined(); + expect(results).toEqual([ + { content: "fact A", score: undefined, category: "ARCHITECTURE" }, + ]); + }); + + test("user recall hits main bank with scope:user any_strict", async () => { + responder = () => okJson({ results: [] }); + const backend = makeBackend(); + await backend.recall({ query: "user prefs", scope: "user" }); + const body = JSON.parse(String(requests[0].init.body)); + expect(requests[0].url).toContain("/v1/default/banks/main-memory/memories/recall"); + expect(body.tags).toEqual(["scope:user"]); + expect(body.tags_match).toBe("any_strict"); + }); + + test("global recall uses config global_tags with any match", async () => { + responder = () => okJson({ results: [] }); + const backend = makeBackend(); + await backend.recall({ query: "homelab", scope: "global" }); + const body = JSON.parse(String(requests[0].init.body)); + expect(requests[0].url).toContain("/v1/default/banks/main-memory/memories/recall"); + expect(body.tags).toEqual(["user:test"]); + expect(body.tags_match).toBe("any"); + }); + + test("recall 404 (missing project bank) returns [] without opening circuit", async () => { + responder = () => new Response("not found", { status: 404 }); + const backend = makeBackend(); + const results = await backend.recall({ + query: "q", + scope: "project", + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", + }); + expect(results).toEqual([]); + expect(backend._getCircuitState()).toBe("closed"); + }); + + test("recall never throws on network error", async () => { + globalThis.fetch = (async () => { + throw new Error("ECONNREFUSED"); + }) as typeof fetch; + const backend = makeBackend(); + await expect(backend.recall({ query: "q" })).resolves.toEqual([]); + }); + + test("remove DELETEs document by derived id; 404 counts as removed", async () => { + responder = (url) => + url.includes("/documents/") ? new Response("gone", { status: 404 }) : okJson({}); + const backend = makeBackend(); + const removed = await backend.remove([ + { + content: "Always run bun test from packages/plugin", + category: "PROJECT_RULES", + scope: "project", + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", + }, + ]); + expect(removed).toBe(1); + const del = requests.find((r) => r.init.method === "DELETE"); + if (!del) throw new Error("no DELETE"); + const expectedId = `mc:git:abcdef1234567890:PROJECT_RULES:${computeNormalizedHash( + "Always run bun test from packages/plugin", + )}`; + expect(del.url).toContain( + `/v1/default/banks/mc-magic-context-abcdef12/documents/${encodeURIComponent(expectedId)}`, + ); + expect(backend._getCircuitState()).toBe("closed"); + }); + + test("retain item verifiedAt lands in metadata.verified_at", async () => { + const backend = makeBackend(); + await backend.retain([{ ...userItem, verifiedAt: 1750000000000 }]); + const post = requests.find((r) => r.init.method === "POST"); + const body = JSON.parse(String(post?.init.body)); + expect(body.items[0].metadata.verified_at).toBe(1750000000000); + }); + + test("recall never throws when results is not an array (malformed 200 body)", async () => { + responder = () => okJson({ results: {} }); + const backend = makeBackend(); + await expect(backend.recall({ query: "q" })).resolves.toEqual([]); + }); + + test("recall never throws when an item's tags is not an array", async () => { + responder = () => + okJson({ + results: [{ id: "1", text: "fact A", type: "world", tags: "not-an-array" }], + }); + const backend = makeBackend(); + await expect(backend.recall({ query: "q" })).resolves.toEqual([{ content: "fact A" }]); + }); + + test("project scope without projectIdentity short-circuits to [] with zero fetch requests", async () => { + const backend = makeBackend(); + const results = await backend.recall({ query: "q", scope: "project" }); + expect(results).toEqual([]); + expect(requests.length).toBe(0); + }); + + test("remove skips project items without projectIdentity (no DELETE, returns 0)", async () => { + const backend = makeBackend(); + const removed = await backend.remove([ + { content: "x", category: "PROJECT_RULES", scope: "project" }, + ]); + expect(removed).toBe(0); + expect(requests.filter((r) => r.init.method === "DELETE").length).toBe(0); + }); + + test("fetchFailedRetainCount returns total from the operations envelope", async () => { + responder = () => okJson({ total: 7, operations: [] }); + const backend = makeBackend(); + expect(await backend.fetchFailedRetainCount()).toBe(7); + const get = requests.find((r) => (r.init.method ?? "GET") === "GET"); + expect(get?.url).toContain("/v1/default/banks/main-memory/operations"); + expect(get?.url).toContain("type=retain"); + expect(get?.url).toContain("status=failed"); + }); + + test("fetchFailedRetainCount returns null when envelope has no total (no operations.length fallback)", async () => { + // A paginated `limit=N` slice can be smaller than the true total, so + // using `operations.length` as a fallback would understate the count + // and silently mask real backend failures. The hook is strict: only + // an explicit `total: number` is treated as a valid count. + responder = () => + okJson({ + operations: [ + { id: "op1", status: "failed", task_type: "retain" }, + { id: "op2", status: "failed", task_type: "retain" }, + ], + }); + const backend = makeBackend(); + expect(await backend.fetchFailedRetainCount()).toBeNull(); + }); + + test("fetchFailedRetainCount returns null when envelope is malformed", async () => { + responder = () => okJson({ not: "the right shape" }); + const backend = makeBackend(); + expect(await backend.fetchFailedRetainCount()).toBeNull(); + }); +}); + +describe("HindsightMemoryBackend mental models", () => { + test("mentalModels fetches project bank MMs with content", async () => { + responder = (url) => { + if (url.includes("/mental-models")) { + return okJson({ + items: [ + { id: "mm1", name: "project-conventions", content: "Conventions doc" }, + { id: "mm2", name: "project-decisions", content: "" }, + ], + }); + } + return okJson({}); + }; + const backend = makeMmBackend(); + const results = await backend.mentalModels({ + scope: "project", + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", + }); + const get = requests.find((r) => (r.init.method ?? "GET") === "GET"); + expect(get?.url).toContain("/v1/default/banks/mc-magic-context-abcdef12/mental-models"); + expect(get?.url).toContain("detail=content"); + // empty-content MM excluded + expect(results).toEqual([{ content: "Conventions doc", category: "project-conventions" }]); + }); + + test("mentalModels user scope filters main-bank MMs by configured names", async () => { + responder = () => + okJson({ + items: [ + { id: "a", name: "User-Preferences", content: "Prefers terse" }, + { id: "b", name: "unrelated-model", content: "Noise" }, + ], + }); + const backend = makeMmBackend(); + const results = await backend.mentalModels({ scope: "user" }); + expect(requests[0].url).toContain("/v1/default/banks/main-memory/mental-models"); + expect(results).toEqual([{ content: "Prefers terse", category: "User-Preferences" }]); + }); + + test("mentalModels never throws; [] on failure", async () => { + globalThis.fetch = (async () => { + throw new Error("ECONNREFUSED"); + }) as typeof fetch; + const backend = makeMmBackend(); + await expect( + backend.mentalModels({ scope: "project", projectIdentity: "git:a", projectName: "x" }), + ).resolves.toEqual([]); + }); + + test("first successful project retain seeds missing MMs once", async () => { + const seeded: string[] = []; + responder = (url) => { + if (url.includes("/mental-models")) { + const isPost = false; // overwritten by wrapped fetch + void isPost; + return okJson({ items: [] }); + } + if (url.endsWith("/v1/default/banks")) { + return okJson({ banks: [{ bank_id: "main-memory" }] }); + } + return okJson({ success: true }); + }; + const origFetch = globalThis.fetch; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = String(input); + requests.push({ url, init: init ?? {} }); + if (url.includes("/mental-models") && init?.method === "POST") { + seeded.push(JSON.parse(String(init.body)).name); + return okJson({ mental_model_id: "new", operation_id: "op" }); + } + return responder(url); + }) as typeof fetch; + const backend = makeMmBackend(); + await backend.retain([projectItem]); + await Bun.sleep(10); // seeding is fire-and-forget after retain + expect(seeded.sort()).toEqual(["project-conventions", "project-decisions"]); + seeded.length = 0; + await backend.retain([projectItem]); + await Bun.sleep(10); + expect(seeded).toEqual([]); // cached, no re-seed + globalThis.fetch = origFetch; + }); + + test("main bank retains never seed MMs", async () => { + const backend = makeMmBackend(); + await backend.retain([userItem]); + await Bun.sleep(10); + expect(requests.some((r) => r.url.includes("/mental-models"))).toBe(false); + }); + + test("mental_models false disables fetch and seeding", async () => { + const backend = makeBackend(); // mental_models: false + const results = await backend.mentalModels({ + scope: "project", + projectIdentity: "git:a", + projectName: "x", + }); + expect(results).toEqual([]); + expect(requests.length).toBe(0); + }); +}); diff --git a/packages/plugin/src/features/magic-context/memory/external-memory-hindsight.ts b/packages/plugin/src/features/magic-context/memory/external-memory-hindsight.ts new file mode 100644 index 000000000..cb487c23d --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-memory-hindsight.ts @@ -0,0 +1,645 @@ +import type { ExternalMemoryConfig } from "../../../config/schema/magic-context"; +import { log } from "../../../shared/logger"; +import { blockedEmbeddingEndpointReason } from "./embedding-ssrf"; +import type { + ExternalMemoryBackend, + ExternalMemoryMentalModelQuery, + ExternalMemoryRecallQuery, + ExternalMemoryRecallResult, + ExternalMemoryRemoveItem, + ExternalMemoryRetainItem, + ExternalMemoryScope, +} from "./external-memory-provider"; +import { computeNormalizedHash } from "./normalize-hash"; + +type HindsightConfig = Extract; + +// Circuit breaker constants — same shape as embedding-openai.ts so a hung +// Hindsight endpoint can't drag every plugin operation through its timeout. +const FAILURE_THRESHOLD = 3; +const FAILURE_WINDOW_MS = 60_000; +const OPEN_DURATION_MS = 5 * 60_000; +const FETCH_TIMEOUT_MS = 10_000; + +type CircuitState = "closed" | "open" | "half_open"; + +const RETAIN_CONTEXT = + "magic-context curated fact store (structured project/user memory, not conversation)"; +const PROJECT_BANK_MISSION = + "Curated long-term memory for one software project, fed by the magic-context plugin: " + + "project rules, architecture decisions, configuration values, constraints, and naming " + + "conventions extracted from coding sessions. Facts are pre-deduplicated and pre-curated; " + + "extract them faithfully without speculation."; + +// Project-scoped mental models seeded once per project bank (after the first +// successful retain). Source queries are REFLECT prompts — Hindsight runs them +// on each refresh to keep the document current. mode "delta" preserves stable +// prose; refresh_after_consolidation re-runs after each ingest cycle. +const PROJECT_MENTAL_MODELS: ReadonlyArray<{ + name: string; + maxTokens: number; + sourceQuery: string; +}> = [ + { + name: "project-conventions", + maxTokens: 800, + sourceQuery: + "Project conventions and rules — naming, structure, configuration, " + + "constraints, and tooling choices extracted from recent coding sessions. " + + "Surface only items that recur across multiple sessions or are explicitly " + + "asserted by the user.", + }, + { + name: "project-decisions", + maxTokens: 800, + sourceQuery: + "Key architectural and design decisions for this project — what was chosen, " + + "what was rejected, and the rationale. Focus on durable choices that affect " + + "future work; ignore one-off trade-offs.", + }, +]; + +function sanitizeBankSegment(value: string): string { + return ( + value + .toLowerCase() + .replace(/[^a-z0-9-]+/g, "-") + .replace(/^-+|-+$/g, "") + .slice(0, 40) || "project" + ); +} + +export class HindsightMemoryBackend implements ExternalMemoryBackend { + readonly backendId: string; + + private readonly endpoint: string; + private readonly apiKey: string; + private readonly projectBankTemplate: string; + private readonly mainBank: string; + private readonly staticTags: readonly string[]; + private readonly recallGlobalTags: readonly string[]; + private readonly mentalModelsEnabled: boolean; + private readonly profileMentalModelNames: Set; + private initialized = false; + private readonly ensuredBanks = new Set(); + private readonly seededMentalModelBanks = new Set(); + + // Circuit breaker state — copied from OpenAICompatibleEmbeddingProvider. + private failureTimes: number[] = []; + private circuitOpenUntil = 0; + private openLogged = false; + private halfOpenProbeInFlight = false; + + constructor(config: HindsightConfig) { + this.endpoint = config.endpoint.replace(/\/+$/, ""); + this.apiKey = config.api_key ?? ""; + this.projectBankTemplate = config.project_bank; + this.mainBank = config.main_bank; + this.staticTags = config.tags; + this.recallGlobalTags = config.recall?.global_tags ?? []; + this.mentalModelsEnabled = config.recall?.mental_models ?? true; + this.profileMentalModelNames = new Set( + (config.recall?.profile_mental_models ?? ["user-preferences"]).map((n) => + n.toLowerCase(), + ), + ); + this.backendId = `hindsight:${this.endpoint}:${this.mainBank}:${this.projectBankTemplate}`; + } + + async initialize(): Promise { + if (this.initialized) return true; + if (!this.endpoint || !this.mainBank) { + log("[magic-context] hindsight backend missing endpoint or main_bank"); + return false; + } + const blockedReason = blockedEmbeddingEndpointReason(this.endpoint); + if (blockedReason) { + log(`[magic-context] hindsight endpoint blocked: ${blockedReason}`); + return false; + } + this.initialized = true; + return true; + } + + resolveBank(item: ExternalMemoryRetainItem): string { + return this.resolveBankForScope(item.scope, item.projectIdentity, item.projectName); + } + + private resolveBankForScope( + scope: ExternalMemoryScope, + projectIdentity?: string, + projectName?: string, + ): string { + if (scope === "project" && projectIdentity) { + const id8 = projectIdentity.replace(/^(git:|dir:)/, "").slice(0, 8); + const name = sanitizeBankSegment(projectName ?? "project"); + return this.projectBankTemplate.replace("{name}", name).replace("{id8}", id8); + } + return this.mainBank; + } + + private documentIdFor(item: { + scope: ExternalMemoryScope; + projectIdentity?: string; + category: string; + content: string; + }): string { + const scopeKey = + item.scope === "project" ? (item.projectIdentity ?? "project") : item.scope; + return `mc:${scopeKey}:${item.category}:${computeNormalizedHash(item.content)}`; + } + + buildMemoryItem(item: ExternalMemoryRetainItem): Record { + const scopeTags = + item.scope === "project" && item.projectIdentity + ? [ + `project:${item.projectIdentity}`, + ...(item.projectName ? [`project-name:${item.projectName}`] : []), + ] + : [ + `scope:${item.scope}`, + // Origin provenance for globals: distinct origin-* prefix + // (NOT project:*, which stays the project-partition axis) + // so future filtered recalls can target "globals learned + // in project X" without colliding with project items. + ...(item.scope === "global" && item.projectIdentity + ? [`origin-project:${item.projectIdentity}`] + : []), + ...(item.scope === "global" && item.projectName + ? [`origin-project-name:${item.projectName}`] + : []), + ]; + // For globals with a known origin, name the project in the extraction + // context: Hindsight's fact extractor links the project as an ENTITY, + // so graph retrieval surfaces this memory whenever any session — in + // any project — recalls with that project's name in the query (the + // global slice query always carries the current project name). + const context = + item.scope === "global" && item.projectName + ? `${RETAIN_CONTEXT}; recorded while working on the "${item.projectName}" project` + : RETAIN_CONTEXT; + return { + content: item.content, + context, + document_id: this.documentIdFor(item), + metadata: { + source: "magic-context", + category: item.category, + ...(item.projectIdentity ? { project_path: item.projectIdentity } : {}), + ...(item.sessionId ? { session_id: item.sessionId } : {}), + ...(item.verifiedAt ? { verified_at: item.verifiedAt } : {}), + }, + tags: [ + "source:magic-context", + `category:${item.category}`, + ...scopeTags, + ...this.staticTags, + ], + }; + } + + async retain(items: ExternalMemoryRetainItem[], signal?: AbortSignal): Promise { + if (items.length === 0) return 0; + if (!(await this.initialize())) return 0; + const byBank = new Map(); + for (const item of items) { + const bank = this.resolveBank(item); + const group = byBank.get(bank); + if (group) group.push(item); + else byBank.set(bank, [item]); + } + let accepted = 0; + for (const [bank, group] of byBank) { + try { + if (!(await this.ensureBank(bank, signal))) continue; + const ok = await this.postRetain(bank, group, signal); + if (ok) { + accepted += group.length; + // Seed missing project-bank mental models after a successful + // retain. The MAIN bank is NEVER seeded — those MMs are + // user-curated and the plugin must not modify them. + if (bank !== this.mainBank) { + void this.ensureProjectMentalModels(bank, group[0]); + } + } + } catch (error) { + log(`[magic-context] hindsight retain failed for bank ${bank}:`, error); + } + } + return accepted; + } + + async recall( + query: ExternalMemoryRecallQuery, + signal?: AbortSignal, + ): Promise { + const scope = query.scope ?? "global"; + if (scope === "project" && !query.projectIdentity) { + log("[magic-context] hindsight recall: project scope without identity — skipping"); + return []; + } + if (!(await this.initialize())) return []; + const bank = this.resolveBankForScope(scope, query.projectIdentity, query.projectName); + const filter = + scope === "user" + ? { tags: ["scope:user"], tags_match: "any_strict" } + : scope === "global" && this.recallGlobalTags.length > 0 + ? { tags: [...this.recallGlobalTags], tags_match: "any" } + : {}; + const response = await this.request( + "POST", + `/v1/default/banks/${encodeURIComponent(bank)}/memories/recall`, + { + query: query.query, + types: ["world", "observation"], + budget: "mid", + ...(query.maxTokens ? { max_tokens: query.maxTokens } : {}), + ...filter, + }, + signal, + { benign404: true }, + ); + if (!response || response.status === 404) return []; + const body = (await response.json().catch(() => null)) as { + results?: unknown; + } | null; + const rawResults = Array.isArray(body?.results) ? body.results : []; + const results: ExternalMemoryRecallResult[] = []; + for (const r of rawResults) { + if (!r || typeof r !== "object") continue; + const item = r as { text?: unknown; score?: unknown; tags?: unknown }; + if (typeof item.text !== "string" || item.text.length === 0) continue; + const tags = Array.isArray(item.tags) ? item.tags : []; + const categoryTag = tags.find( + (t): t is string => typeof t === "string" && t.startsWith("category:"), + ); + results.push({ + content: item.text, + ...(typeof item.score === "number" ? { score: item.score } : {}), + ...(categoryTag ? { category: categoryTag.slice("category:".length) } : {}), + }); + if (query.limit && results.length >= query.limit) break; + } + return results; + } + + async remove(items: ExternalMemoryRemoveItem[], signal?: AbortSignal): Promise { + if (items.length === 0) return 0; + if (!(await this.initialize())) return 0; + let removed = 0; + for (const item of items) { + if (item.scope === "project" && !item.projectIdentity) { + log("[magic-context] hindsight remove: project scope without identity — skipping"); + continue; + } + try { + const bank = this.resolveBankForScope( + item.scope, + item.projectIdentity, + item.projectName, + ); + const documentId = this.documentIdFor(item); + const response = await this.request( + "DELETE", + `/v1/default/banks/${encodeURIComponent(bank)}/documents/${encodeURIComponent(documentId)}`, + undefined, + signal, + { benign404: true }, + ); + if (response) removed += 1; // 2xx or benign 404 (already gone) + } catch (error) { + log("[magic-context] hindsight remove failed:", error); + } + } + return removed; + } + + async mentalModels( + query: ExternalMemoryMentalModelQuery, + signal?: AbortSignal, + ): Promise { + if (!this.mentalModelsEnabled) return []; + try { + if (!(await this.initialize())) return []; + if (query.scope === "project" && !query.projectIdentity) { + log( + "[magic-context] hindsight mental-models: project scope without identity — skipping", + ); + return []; + } + const bank = this.resolveBankForScope( + query.scope, + query.projectIdentity, + query.projectName, + ); + const response = await this.request( + "GET", + `/v1/default/banks/${encodeURIComponent(bank)}/mental-models?detail=content`, + undefined, + signal, + { benign404: true }, + ); + if (!response) return []; + if (response.status === 404) return []; // bank missing → no MMs + const body = (await response.json().catch(() => null)) as { + items?: unknown; + } | null; + const rawItems = Array.isArray(body?.items) ? body.items : []; + const results: ExternalMemoryRecallResult[] = []; + for (const raw of rawItems) { + if (!raw || typeof raw !== "object") continue; + const model = raw as { + name?: unknown; + content?: unknown; + }; + if (typeof model.name !== "string" || model.name.length === 0) continue; + if (typeof model.content !== "string") continue; // null/unpopulated + const trimmed = model.content.trim(); + if (trimmed.length === 0) continue; + // For non-project scopes, gate by the configured profile names + // (case-insensitive). Project scope returns everything non-empty. + if (query.scope !== "project") { + if (!this.profileMentalModelNames.has(model.name.toLowerCase())) continue; + } + results.push({ content: trimmed, category: model.name }); + } + return results; + } catch (error) { + log("[magic-context] hindsight mental-models failed:", error); + return []; + } + } + + async fetchFailedRetainCount(signal?: AbortSignal): Promise { + try { + if (!(await this.initialize())) return null; + // benign404: a missing operations route (older Hindsight build) + // must not feed the circuit breaker — this is a pure status check + // and opening the circuit here would suppress real retains. + const response = await this.request( + "GET", + `/v1/default/banks/${encodeURIComponent(this.mainBank)}/operations?type=retain&status=failed&exclude_parents=true&limit=5`, + undefined, + signal, + { benign404: true }, + ); + if (!response || response.status === 404) return null; + const body = (await response.json().catch(() => null)) as { + total?: unknown; + } | null; + if (!body) return null; + // The live operations envelope (OperationsListResponse) exposes + // `total` as the authoritative failed-count. We do NOT fall back + // to `operations.length` — a paginated `limit=N` slice can be + // smaller than the true total, so that heuristic would understate + // the count and silently mask real backend failures. + return typeof body.total === "number" ? body.total : null; + } catch { + return null; + } + } + + /** Seed missing project-bank mental models (idempotent, fire-and-forget). + * Called after the first successful retain to a non-main bank. Seeding + * pre-existing project banks with a `project:X` MM would create a permanent + * empty document (the reflect loop sees no memories tagged for that + * project until the first retain), so we always seed AFTER retain — and + * gate on a per-bank claim so transient failures don't retry forever. */ + private async ensureProjectMentalModels( + bank: string, + sample: ExternalMemoryRetainItem, + ): Promise { + if (!this.mentalModelsEnabled) return; + if (this.seededMentalModelBanks.has(bank)) return; + this.seededMentalModelBanks.add(bank); // claim first; transient failures retry next process + try { + const listResponse = await this.request( + "GET", + `/v1/default/banks/${encodeURIComponent(bank)}/mental-models`, + undefined, + undefined, + { benign404: true }, + ); + if (!listResponse) return; + const body = (await listResponse.json().catch(() => null)) as { + items?: unknown; + } | null; + const existing = new Set(); + for (const raw of Array.isArray(body?.items) ? body.items : []) { + if (!raw || typeof raw !== "object") continue; + const name = (raw as { name?: unknown }).name; + if (typeof name === "string" && name.length > 0) { + existing.add(name.toLowerCase()); + } + } + const projectTag = sample.projectIdentity + ? `project:${sample.projectIdentity}` + : "project:unknown"; + for (const model of PROJECT_MENTAL_MODELS) { + if (existing.has(model.name)) continue; + await this.request( + "POST", + `/v1/default/banks/${encodeURIComponent(bank)}/mental-models`, + { + name: model.name, + source_query: model.sourceQuery, + tags: [projectTag], + max_tokens: model.maxTokens, + trigger: { mode: "delta", refresh_after_consolidation: true }, + }, + ); + } + } catch (error) { + log(`[magic-context] mental-model seeding failed for bank ${bank}:`, error); + } + } + + private async ensureBank(bank: string, signal?: AbortSignal): Promise { + if (bank === this.mainBank) return true; + if (this.ensuredBanks.has(bank)) return true; + const listResponse = await this.request("GET", "/v1/default/banks", undefined, signal); + if (!listResponse) return false; + const listBody = (await listResponse.json().catch(() => null)) as { + banks?: Array<{ bank_id?: string }>; + } | null; + const exists = (listBody?.banks ?? []).some((b) => b.bank_id === bank); + if (!exists) { + const created = await this.request( + "PUT", + `/v1/default/banks/${encodeURIComponent(bank)}`, + { name: bank, mission: PROJECT_BANK_MISSION }, + signal, + ); + if (!created) return false; + } + this.ensuredBanks.add(bank); + return true; + } + + private async postRetain( + bank: string, + group: ExternalMemoryRetainItem[], + signal?: AbortSignal, + ): Promise { + const response = await this.request( + "POST", + `/v1/default/banks/${encodeURIComponent(bank)}/memories`, + { items: group.map((item) => this.buildMemoryItem(item)), async: true }, + signal, + ); + if (!response) return false; + const body = (await response.json().catch(() => null)) as { success?: boolean } | null; + return body?.success === true; + } + + private async request( + method: string, + path: string, + body?: unknown, + signal?: AbortSignal, + opts?: { benign404?: boolean }, + ): Promise { + if (signal?.aborted) return null; + let isProbe = false; + let internalController: AbortController | undefined; + let timeoutHandle: ReturnType | undefined; + let onOuterAbort: (() => void) | undefined; + try { + const claim = this.claimProbeOrShortCircuit(); + if (claim === "short_circuit") return null; + isProbe = claim === "probe"; + internalController = new AbortController(); + timeoutHandle = setTimeout(() => internalController?.abort(), FETCH_TIMEOUT_MS); + onOuterAbort = () => internalController?.abort(); + if (signal) signal.addEventListener("abort", onOuterAbort, { once: true }); + + const response = await fetch(`${this.endpoint}${path}`, { + method, + headers: { + "content-type": "application/json", + ...(this.apiKey ? { authorization: `Bearer ${this.apiKey}` } : {}), + }, + ...(body !== undefined ? { body: JSON.stringify(body) } : {}), + redirect: "error", + signal: internalController.signal, + }); + + if (response.status === 404 && opts?.benign404) { + this.recordSuccess(); + return response; + } + if (response.status === 422) { + log( + `[magic-context] hindsight memory defense rejected content (${method} ${path}) — not retrying`, + ); + this.recordSuccess(); + return null; + } + if (!response.ok) { + log( + `[magic-context] hindsight request failed: ${method} ${path} → ${response.status} ${response.statusText}`, + ); + this.recordFailure(isProbe); + return null; + } + this.recordSuccess(); + return response; + } catch (error) { + const isAbort = + error instanceof Error && + (error.name === "AbortError" || error.message.includes("aborted")); + if (isAbort && signal?.aborted) { + // Caller gave up — don't penalize the endpoint. + } else { + log(`[magic-context] hindsight request error: ${method} ${path}:`, error); + this.recordFailure(isProbe); + } + return null; + } finally { + if (timeoutHandle !== undefined) clearTimeout(timeoutHandle); + if (signal && onOuterAbort) signal.removeEventListener("abort", onOuterAbort); + if (isProbe) this.halfOpenProbeInFlight = false; + } + } + + async dispose(): Promise { + this.initialized = false; + this.ensuredBanks.clear(); + } + + // ── Circuit breaker — copied verbatim from OpenAICompatibleEmbeddingProvider + // (embedding-openai.ts lines 291-372) with log prefixes "openai-compatible + // embedding" → "hindsight". Same three-state machine, same rolling window, + // same probe-claim semantics. Documented in embedding-openai.ts. + + private claimProbeOrShortCircuit(): "allow" | "probe" | "short_circuit" { + if (this.circuitOpenUntil === 0) { + return "allow"; + } + if (Date.now() < this.circuitOpenUntil) { + return "short_circuit"; + } + if (this.halfOpenProbeInFlight) { + return "short_circuit"; + } + this.halfOpenProbeInFlight = true; + log("[magic-context] hindsight: circuit half-open, probing endpoint"); + return "probe"; + } + + private recordFailure(isProbe: boolean): void { + if (isProbe) { + this.circuitOpenUntil = Date.now() + OPEN_DURATION_MS; + if (!this.openLogged) { + log( + `[magic-context] hindsight: probe failed, re-opening circuit for ${OPEN_DURATION_MS / 60_000}min`, + ); + this.openLogged = true; + } + this.failureTimes = []; + return; + } + + const now = Date.now(); + const cutoff = now - FAILURE_WINDOW_MS; + this.failureTimes = this.failureTimes.filter((t) => t > cutoff); + this.failureTimes.push(now); + + if (this.failureTimes.length >= FAILURE_THRESHOLD) { + this.circuitOpenUntil = now + OPEN_DURATION_MS; + if (!this.openLogged) { + log( + `[magic-context] hindsight: opening circuit for ${OPEN_DURATION_MS / 60_000}min after ${this.failureTimes.length} failures in ${FAILURE_WINDOW_MS / 1_000}s`, + ); + this.openLogged = true; + } + this.failureTimes = []; + } + } + + private recordSuccess(): void { + if (this.failureTimes.length > 0 || this.circuitOpenUntil > 0 || this.openLogged) { + log("[magic-context] hindsight: endpoint recovered, circuit closed"); + } + this.failureTimes = []; + this.circuitOpenUntil = 0; + this.openLogged = false; + } + + _getCircuitState(): CircuitState { + if (this.circuitOpenUntil === 0) return "closed"; + if (Date.now() < this.circuitOpenUntil) { + return this.halfOpenProbeInFlight ? "half_open" : "open"; + } + return "half_open"; + } + _getFailureCount(): number { + return this.failureTimes.length; + } + _resetCircuit(): void { + this.failureTimes = []; + this.circuitOpenUntil = 0; + this.openLogged = false; + this.halfOpenProbeInFlight = false; + } +} diff --git a/packages/plugin/src/features/magic-context/memory/external-memory-provider.ts b/packages/plugin/src/features/magic-context/memory/external-memory-provider.ts new file mode 100644 index 000000000..c4db10300 --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-memory-provider.ts @@ -0,0 +1,84 @@ +import type { MemoryCategory, MemorySourceType } from "./types"; + +/** "project" = tied to a project identity. "user" = user-level (dreamer user + * memories). "global" = neither — automatic fallback scope (nothing emits it + * in v1; reserved for corrective retains / session streams). */ +export type ExternalMemoryScope = "project" | "user" | "global"; + +export interface ExternalMemoryRetainItem { + content: string; + /** MemoryCategory or the pseudo-category "USER_PROFILE" (user memories). */ + category: MemoryCategory | "USER_PROFILE"; + /** "project" items carry projectIdentity/projectName (partition key). + * "global" items MAY carry them as ORIGIN provenance — they still route + * to the main bank, but the engine records which project the fact was + * learned in (tags + extraction context) so cross-project recall can + * link the fact to its project by name. "user" items carry neither. */ + scope: ExternalMemoryScope; + /** resolveProjectIdentity() result — "git:" or "dir:". */ + projectIdentity?: string; + /** Human-readable basename — secondary label only, never a key. */ + projectName?: string; + sourceType: MemorySourceType; + sessionId?: string; + /** Set on verify-confirmed corrective upserts — maps to engine metadata. */ + verifiedAt?: number; +} + +/** v2 neutral recall shapes — engine maps scope to its own partitions/filters. */ +export interface ExternalMemoryRecallQuery { + query: string; + scope?: ExternalMemoryScope; + projectIdentity?: string; + /** Needed by bank-template resolution for scope "project". */ + projectName?: string; + limit?: number; + maxTokens?: number; +} +export interface ExternalMemoryRecallResult { + content: string; + score?: number; + category?: string; +} + +/** Corrective removal — document identity derives from the ORIGINAL content. */ +export interface ExternalMemoryRemoveItem { + content: string; + category: MemoryCategory | "USER_PROFILE"; + scope: ExternalMemoryScope; + projectIdentity?: string; + projectName?: string; +} + +export interface ExternalMemoryMentalModelQuery { + scope: ExternalMemoryScope; + projectIdentity?: string; + projectName?: string; +} + +export interface ExternalMemoryBackend { + /** Identity string (provider + endpoint + banks) — drives singleton + * re-creation on config change, like EmbeddingProvider.modelId. */ + readonly backendId: string; + initialize(): Promise; + /** Best-effort batch retain. Never throws; returns count accepted. */ + retain(items: ExternalMemoryRetainItem[], signal?: AbortSignal): Promise; + /** v2 unified read. Never throws; [] on failure. */ + recall?( + query: ExternalMemoryRecallQuery, + signal?: AbortSignal, + ): Promise; + /** v2 corrective removal. Never throws; returns count removed (404 counts: already gone). */ + remove?(items: ExternalMemoryRemoveItem[], signal?: AbortSignal): Promise; + /** v2 optional fast path: pre-synthesized briefing documents (e.g. + * Hindsight mental models). Never throws; [] when unsupported/empty. */ + mentalModels?( + query: ExternalMemoryMentalModelQuery, + signal?: AbortSignal, + ): Promise; + /** Best-effort failed-retain count from the operations endpoint (status + * report). Returns null when the backend is offline, the endpoint is + * missing, or the response is malformed. Never throws. */ + fetchFailedRetainCount?(signal?: AbortSignal): Promise; + dispose(): Promise; +} diff --git a/packages/plugin/src/features/magic-context/memory/external-memory.test.ts b/packages/plugin/src/features/magic-context/memory/external-memory.test.ts new file mode 100644 index 000000000..cd2ab45a0 --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-memory.test.ts @@ -0,0 +1,298 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { + _resetExternalMemoryForTests, + _setTestExternalBackendFactory, + fetchExternalFailedRetains, + getExternalMemoryStatus, + getExternalRecallConfig, + initializeExternalMemory, + isExternalSearchEnabled, + recallFromExternalBackend, + removeFromExternalBackend, + teeToExternalBackend, + upsertToExternalBackend, +} from "./external-memory"; +import type { + ExternalMemoryBackend, + ExternalMemoryRecallQuery, + ExternalMemoryRemoveItem, + ExternalMemoryRetainItem, +} from "./external-memory-provider"; + +function makeFakeBackend(calls: ExternalMemoryRetainItem[][]): ExternalMemoryBackend { + return { + backendId: "fake:test", + initialize: async () => true, + retain: async (items) => { + calls.push([...items]); + return items.length; + }, + dispose: async () => {}, + }; +} + +const HINDSIGHT_TEST_CONFIG = { + provider: "hindsight" as const, + endpoint: "http://10.0.0.1:8889", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"] as ("historian" | "agent" | "dreamer")[], + tags: [] as string[], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [] as string[], + global_from_prompt: false, + search: true, + mental_models: false, + profile_mental_models: ["user-preferences"], + }, +}; + +const item: ExternalMemoryRetainItem = { + content: "Use bun test for all packages", + category: "PROJECT_RULES", + scope: "project", + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", + sourceType: "historian", + sessionId: "ses_1", +}; + +afterEach(() => _resetExternalMemoryForTests()); + +describe("teeToExternalBackend", () => { + test("no-op when provider off", async () => { + const calls: ExternalMemoryRetainItem[][] = []; + _setTestExternalBackendFactory(() => makeFakeBackend(calls)); + initializeExternalMemory({ provider: "off" }); + await teeToExternalBackend("historian", [item]); + expect(calls.length).toBe(0); + }); + + test("retains when provider configured and source allowed", async () => { + const calls: ExternalMemoryRetainItem[][] = []; + _setTestExternalBackendFactory(() => makeFakeBackend(calls)); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + await teeToExternalBackend("historian", [item]); + expect(calls.length).toBe(1); + expect(calls[0][0].content).toBe(item.content); + }); + + test("filters by retain_sources", async () => { + const calls: ExternalMemoryRetainItem[][] = []; + _setTestExternalBackendFactory(() => makeFakeBackend(calls)); + initializeExternalMemory({ ...HINDSIGHT_TEST_CONFIG, retain_sources: ["historian"] }); + await teeToExternalBackend("agent", [item]); + expect(calls.length).toBe(0); + }); + + test("never throws when backend retain rejects", async () => { + _setTestExternalBackendFactory(() => ({ + backendId: "fake:boom", + initialize: async () => true, + retain: async () => { + throw new Error("boom"); + }, + dispose: async () => {}, + })); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + await expect(teeToExternalBackend("historian", [item])).resolves.toBeUndefined(); + }); + + test("empty items is a no-op", async () => { + const calls: ExternalMemoryRetainItem[][] = []; + _setTestExternalBackendFactory(() => makeFakeBackend(calls)); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + await teeToExternalBackend("historian", []); + expect(calls.length).toBe(0); + }); + + test("config change re-creates backend; same identity keeps it", async () => { + let created = 0; + const calls: ExternalMemoryRetainItem[][] = []; + _setTestExternalBackendFactory(() => { + created += 1; + return makeFakeBackend(calls); + }); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + await teeToExternalBackend("historian", [item]); + // same identity → backend kept + initializeExternalMemory({ ...HINDSIGHT_TEST_CONFIG, retain_sources: ["historian"] }); + await teeToExternalBackend("historian", [item]); + expect(created).toBe(1); + // different endpoint → re-created + initializeExternalMemory({ ...HINDSIGHT_TEST_CONFIG, endpoint: "http://10.1.1.2:8889" }); + await teeToExternalBackend("historian", [item]); + expect(created).toBe(2); + }); +}); + +function makeRecallBackend(captured: { + recalls: ExternalMemoryRecallQuery[]; + removes: ExternalMemoryRemoveItem[][]; + retains: ExternalMemoryRetainItem[][]; +}): ExternalMemoryBackend { + return { + backendId: "fake:recall", + initialize: async () => true, + retain: async (items) => { + captured.retains.push([...items]); + return items.length; + }, + recall: async (query) => { + captured.recalls.push(query); + return [{ content: "ext fact", category: "ARCHITECTURE" }]; + }, + remove: async (items) => { + captured.removes.push([...items]); + return items.length; + }, + dispose: async () => {}, + }; +} + +describe("ungated v2 orchestrator paths", () => { + test("recallFromExternalBackend returns results when provider on", async () => { + const captured = { + recalls: [], + removes: [], + retains: [], + } as Parameters[0]; + _setTestExternalBackendFactory(() => makeRecallBackend(captured)); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + const results = await recallFromExternalBackend({ query: "q", scope: "project" }); + expect(results).toEqual([{ content: "ext fact", category: "ARCHITECTURE" }]); + expect(captured.recalls.length).toBe(1); + }); + + test("recallFromExternalBackend returns [] when provider off", async () => { + initializeExternalMemory({ provider: "off" }); + expect(await recallFromExternalBackend({ query: "q" })).toEqual([]); + }); + + test("recallFromExternalBackend never throws", async () => { + _setTestExternalBackendFactory(() => ({ + backendId: "fake:boom", + initialize: async () => true, + retain: async () => 0, + recall: async () => { + throw new Error("boom"); + }, + dispose: async () => {}, + })); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + await expect(recallFromExternalBackend({ query: "q" })).resolves.toEqual([]); + }); + + test("removeFromExternalBackend ignores retain_sources filter", async () => { + const captured = { + recalls: [], + removes: [], + retains: [], + } as Parameters[0]; + _setTestExternalBackendFactory(() => makeRecallBackend(captured)); + initializeExternalMemory({ ...HINDSIGHT_TEST_CONFIG, retain_sources: [] }); + await removeFromExternalBackend([ + { content: "x", category: "PROJECT_RULES", scope: "project", projectIdentity: "git:a" }, + ]); + expect(captured.removes.length).toBe(1); + }); + + test("upsertToExternalBackend ignores retain_sources filter", async () => { + const captured = { + recalls: [], + removes: [], + retains: [], + } as Parameters[0]; + _setTestExternalBackendFactory(() => makeRecallBackend(captured)); + initializeExternalMemory({ ...HINDSIGHT_TEST_CONFIG, retain_sources: [] }); + await upsertToExternalBackend([ + { + content: "x", + category: "PROJECT_RULES", + scope: "project", + projectIdentity: "git:a", + sourceType: "dreamer", + verifiedAt: 123, + }, + ]); + expect(captured.retains.length).toBe(1); + expect(captured.retains[0][0].verifiedAt).toBe(123); + }); + + test("getExternalRecallConfig reflects provider state", () => { + initializeExternalMemory({ provider: "off" }); + expect(getExternalRecallConfig()).toBeNull(); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + expect(getExternalRecallConfig()?.enabled).toBe(true); + expect(isExternalSearchEnabled()).toBe(true); + }); +}); + +describe("getExternalMemoryStatus", () => { + test("null when provider off", () => { + initializeExternalMemory({ provider: "off" }); + expect(getExternalMemoryStatus()).toBeNull(); + }); + + test("populated with provider + endpoint when on", () => { + _setTestExternalBackendFactory(() => makeFakeBackend([])); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + const status = getExternalMemoryStatus(); + expect(status).not.toBeNull(); + expect(status?.provider).toBe("hindsight"); + expect(status?.endpoint).toBe("http://10.0.0.1:8889"); + }); + + test("circuitState included when backend exposes _getCircuitState", () => { + _setTestExternalBackendFactory(() => ({ + backendId: "fake:circuit", + initialize: async () => true, + retain: async () => 0, + dispose: async () => {}, + _getCircuitState: () => "half_open", + })); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + expect(getExternalMemoryStatus()?.circuitState).toBe("half_open"); + }); + + test("fetchExternalFailedRetains returns null when provider off", async () => { + initializeExternalMemory({ provider: "off" }); + expect(await fetchExternalFailedRetains()).toBeNull(); + }); + + test("fetchExternalFailedRetains returns null when backend lacks hook", async () => { + _setTestExternalBackendFactory(() => makeFakeBackend([])); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + expect(await fetchExternalFailedRetains()).toBeNull(); + }); + + test("fetchExternalFailedRetains returns count from backend hook", async () => { + _setTestExternalBackendFactory(() => ({ + backendId: "fake:ops", + initialize: async () => true, + retain: async () => 0, + dispose: async () => {}, + fetchFailedRetainCount: async () => 7, + })); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + expect(await fetchExternalFailedRetains()).toBe(7); + }); + + test("fetchExternalFailedRetains swallows backend throw", async () => { + _setTestExternalBackendFactory(() => ({ + backendId: "fake:ops-boom", + initialize: async () => true, + retain: async () => 0, + dispose: async () => {}, + fetchFailedRetainCount: async () => { + throw new Error("boom"); + }, + })); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + expect(await fetchExternalFailedRetains()).toBeNull(); + }); +}); diff --git a/packages/plugin/src/features/magic-context/memory/external-memory.ts b/packages/plugin/src/features/magic-context/memory/external-memory.ts new file mode 100644 index 000000000..d84da2103 --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-memory.ts @@ -0,0 +1,229 @@ +import type { + ExternalMemoryConfig, + ExternalMemoryRetainSource, + ExternalRecallConfig, +} from "../../../config/schema/magic-context"; +import { log } from "../../../shared/logger"; +import { HindsightMemoryBackend } from "./external-memory-hindsight"; +import type { + ExternalMemoryBackend, + ExternalMemoryMentalModelQuery, + ExternalMemoryRecallQuery, + ExternalMemoryRecallResult, + ExternalMemoryRemoveItem, + ExternalMemoryRetainItem, +} from "./external-memory-provider"; + +const OFF_CONFIG: ExternalMemoryConfig = { provider: "off" }; + +let externalConfig: ExternalMemoryConfig = OFF_CONFIG; +let backend: ExternalMemoryBackend | null = null; +let testBackendFactory: ((config: ExternalMemoryConfig) => ExternalMemoryBackend | null) | null = + null; + +export function createExternalMemoryBackend( + config: ExternalMemoryConfig, +): ExternalMemoryBackend | null { + if (testBackendFactory) { + return config.provider === "off" ? null : testBackendFactory(config); + } + if (config.provider === "hindsight") { + return new HindsightMemoryBackend(config); + } + return null; +} + +function configIdentity(config: ExternalMemoryConfig): string { + if (config.provider === "off") return "external-memory:off"; + return `external-memory:${config.provider}:${config.endpoint}:${config.main_bank}:${config.project_bank}`; +} + +export function initializeExternalMemory(config?: ExternalMemoryConfig): void { + const next = config ?? OFF_CONFIG; + if (configIdentity(next) === configIdentity(externalConfig)) { + externalConfig = next; // pick up retain_sources/tags changes cheaply + return; + } + const previous = backend; + externalConfig = next; + backend = null; + if (previous) { + void previous.dispose().catch((error) => { + log("[magic-context] external memory backend dispose failed:", error); + }); + } +} + +function getOrCreateBackend(): ExternalMemoryBackend | null { + if (backend) return backend; + backend = createExternalMemoryBackend(externalConfig); + return backend; +} + +/** + * Fire-and-forget tee of memory creations to the external backend. + * Best-effort: NEVER throws, never blocks the caller's local write. + * `source` identifies the creation point for retain_sources filtering. + */ +export async function teeToExternalBackend( + source: ExternalMemoryRetainSource, + items: ExternalMemoryRetainItem[], +): Promise { + try { + if (items.length === 0) return; + if (externalConfig.provider === "off") return; + if (!externalConfig.retain_sources.includes(source)) return; + const current = getOrCreateBackend(); + if (!current) return; + if (!(await current.initialize())) return; + const accepted = await current.retain(items); + if (accepted > 0) { + log(`[magic-context] external memory: retained ${accepted}/${items.length} item(s)`); + } + } catch (error) { + log("[magic-context] external memory tee failed:", error); + } +} + +/** Resolved recall config, or null when the provider is off. */ +export function getExternalRecallConfig(): ExternalRecallConfig | null { + if (externalConfig.provider === "off") return null; + return externalConfig.recall; +} + +export function isExternalSearchEnabled(): boolean { + const recall = getExternalRecallConfig(); + return recall !== null && recall.search === true; +} + +/** + * Direct recall against the external backend. UNGATED by retain_sources + * (read path). Never throws; [] when off/unsupported/failing. + */ +export async function recallFromExternalBackend( + query: ExternalMemoryRecallQuery, + signal?: AbortSignal, +): Promise { + try { + if (externalConfig.provider === "off") return []; + const current = getOrCreateBackend(); + if (!current?.recall) return []; + if (!(await current.initialize())) return []; + return await current.recall(query, signal); + } catch (error) { + log("[magic-context] external memory recall failed:", error); + return []; + } +} + +/** + * Corrective removal. UNGATED by retain_sources (consistency propagation, + * not a retain source). Fire-and-forget; never throws. + */ +export async function removeFromExternalBackend(items: ExternalMemoryRemoveItem[]): Promise { + try { + if (items.length === 0) return; + if (externalConfig.provider === "off") return; + const current = getOrCreateBackend(); + if (!current?.remove) return; + if (!(await current.initialize())) return; + const removed = await current.remove(items); + if (removed > 0) { + log(`[magic-context] external memory: removed ${removed}/${items.length} item(s)`); + } + } catch (error) { + log("[magic-context] external memory remove failed:", error); + } +} + +/** + * Mental-model fast path (single GET vs full recall). UNGATED by + * retain_sources. Never throws; [] when off/unsupported/failing. + */ +export async function mentalModelsFromExternalBackend( + query: ExternalMemoryMentalModelQuery, + signal?: AbortSignal, +): Promise { + try { + if (externalConfig.provider === "off") return []; + const current = getOrCreateBackend(); + if (!current?.mentalModels) return []; + if (!(await current.initialize())) return []; + return await current.mentalModels(query, signal); + } catch (error) { + log("[magic-context] external memory mental-models failed:", error); + return []; + } +} + +/** + * Corrective upsert (verify-confirmed verbatim re-retain). UNGATED by + * retain_sources. Fire-and-forget; never throws. + */ +export async function upsertToExternalBackend(items: ExternalMemoryRetainItem[]): Promise { + try { + if (items.length === 0) return; + if (externalConfig.provider === "off") return; + const current = getOrCreateBackend(); + if (!current) return; + if (!(await current.initialize())) return; + await current.retain(items); + } catch (error) { + log("[magic-context] external memory upsert failed:", error); + } +} + +/** Status snapshot for the ctx-status / RPC surface. Sync, no network. */ +export interface ExternalMemoryStatus { + provider: string; + endpoint?: string; + circuitState?: string; +} + +export function getExternalMemoryStatus(): ExternalMemoryStatus | null { + if (externalConfig.provider === "off") return null; + const current = getOrCreateBackend(); + const circuitState = + current && "_getCircuitState" in current + ? (current as { _getCircuitState(): string })._getCircuitState() + : undefined; + return { + provider: externalConfig.provider, + endpoint: externalConfig.endpoint, + ...(circuitState ? { circuitState } : {}), + }; +} + +/** Best-effort failed-retain count from the operations endpoint (doctor). + * Returns null when the backend is offline, the endpoint is missing, or the + * response is malformed. Never throws. */ +export async function fetchExternalFailedRetains(signal?: AbortSignal): Promise { + try { + if (externalConfig.provider === "off") return null; + const current = getOrCreateBackend(); + if (!current || !("fetchFailedRetainCount" in current)) return null; + return await ( + current as { fetchFailedRetainCount(s?: AbortSignal): Promise } + ).fetchFailedRetainCount(signal); + } catch { + return null; + } +} + +export async function disposeExternalMemoryBackend(): Promise { + const current = backend; + backend = null; + if (current) await current.dispose(); +} + +// Test-only hooks (mirror embedding.ts naming). +export function _setTestExternalBackendFactory( + factory: ((config: ExternalMemoryConfig) => ExternalMemoryBackend | null) | null, +): void { + testBackendFactory = factory; +} +export function _resetExternalMemoryForTests(): void { + externalConfig = OFF_CONFIG; + backend = null; + testBackendFactory = null; +} diff --git a/packages/plugin/src/features/magic-context/memory/external-recall-read.ts b/packages/plugin/src/features/magic-context/memory/external-recall-read.ts new file mode 100644 index 000000000..0cb122420 --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-recall-read.ts @@ -0,0 +1,90 @@ +import { createHash } from "node:crypto"; +import type { Database } from "../../../shared/sqlite"; + +export interface ExternalRecallSliceItem { + content: string; + category?: string; +} +export interface ExternalRecallSnapshot { + project: ExternalRecallSliceItem[]; + profile: ExternalRecallSliceItem[]; + global: ExternalRecallSliceItem[]; +} +export type ExternalRecallState = "pending" | "done" | "failed"; + +export function isSnapshotEmpty(snapshot: ExternalRecallSnapshot): boolean { + return ( + snapshot.project.length === 0 && + snapshot.profile.length === 0 && + snapshot.global.length === 0 + ); +} + +/** Deterministic fingerprint of a snapshot; '' for empty/none. */ +export function computeRecallSnapshotHash(snapshot: ExternalRecallSnapshot | null): string { + if (!snapshot || isSnapshotEmpty(snapshot)) return ""; + return createHash("sha256").update(JSON.stringify(snapshot)).digest("hex").slice(0, 16); +} + +export function readExternalRecallSnapshot( + db: Database, + sessionId: string, +): { state: ExternalRecallState | null; snapshot: ExternalRecallSnapshot | null } { + let row: { state?: unknown; json?: unknown } | null = null; + try { + row = db + .prepare( + "SELECT external_recall_state AS state, external_recall_json AS json FROM session_meta WHERE session_id = ?", + ) + .get(sessionId) as { state?: unknown; json?: unknown } | null; + } catch { + return { state: null, snapshot: null }; // pre-migration DB — behave as never-started + } + const state = + row?.state === "pending" || row?.state === "done" || row?.state === "failed" + ? row.state + : null; + if (state !== "done" || typeof row?.json !== "string" || row.json.length === 0) { + return { state, snapshot: null }; + } + try { + const parsed = JSON.parse(row.json) as Partial; + return { + state, + snapshot: { + project: sanitizeSlice(parsed.project), + profile: sanitizeSlice(parsed.profile), + global: sanitizeSlice(parsed.global), + }, + }; + } catch { + return { state, snapshot: null }; + } +} + +/** Per-item validation: the JSON is self-written, but a corrupted row must + * degrade to "fewer items", never to a render-path throw (a non-string + * content would explode inside materializeM0's renderExternalLines). */ +function sanitizeSlice(value: unknown): ExternalRecallSliceItem[] { + if (!Array.isArray(value)) return []; + const items: ExternalRecallSliceItem[] = []; + for (const raw of value) { + if (!raw || typeof raw !== "object") continue; + const item = raw as { content?: unknown; category?: unknown }; + if (typeof item.content !== "string" || item.content.length === 0) continue; + items.push({ + content: item.content, + ...(typeof item.category === "string" && item.category.length > 0 + ? { category: item.category } + : {}), + }); + } + return items; +} + +/** Marker-capture helper: hash of the persisted DONE snapshot, '' otherwise. */ +export function readExternalRecallHash(db: Database, sessionId: string): string { + const { state, snapshot } = readExternalRecallSnapshot(db, sessionId); + if (state !== "done") return ""; + return computeRecallSnapshotHash(snapshot); +} diff --git a/packages/plugin/src/features/magic-context/memory/external-recall.test.ts b/packages/plugin/src/features/magic-context/memory/external-recall.test.ts new file mode 100644 index 000000000..f2e5cc3c8 --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-recall.test.ts @@ -0,0 +1,535 @@ +/// + +import { afterEach, beforeEach, describe, expect, mock, test } from "bun:test"; +import { Database } from "../../../shared/sqlite"; +import { closeQuietly } from "../../../shared/sqlite-helpers"; +import { runMigrations } from "../migrations"; +import { initializeDatabase } from "../storage-db"; +import { ensureSessionMetaRow } from "../storage-meta-shared"; +import { resetEmbeddingCacheForTests } from "./embedding-cache"; +import { + _resetExternalMemoryForTests, + _setTestExternalBackendFactory, + initializeExternalMemory, +} from "./external-memory"; +import type { ExternalMemoryBackend, ExternalMemoryRecallQuery } from "./external-memory-provider"; +import { + _resetExternalRecallForTests, + maybeAwaitExternalRecall, + normalizePromptExcerpt, + startSessionRecall, + waitForSessionRecall, +} from "./external-recall"; +import { computeRecallSnapshotHash, readExternalRecallSnapshot } from "./external-recall-read"; + +const mockEmbedBatch = mock(async () => null); +const mockLog = mock(() => {}); + +mock.module("../project-embedding-registry", () => ({ + embedBatchForProject: mockEmbedBatch, + getProjectEmbeddingSnapshot: () => null, +})); + +mock.module("../../../shared/logger", () => ({ + log: mockLog, + sessionLog: mockLog, + getLogFilePath: () => "/tmp/test.log", +})); + +const { insertMemory } = await import("./storage-memory"); +const { saveEmbedding } = await import("./storage-memory-embeddings"); + +let db: Database | null = null; + +function makeDb(): Database { + const d = new Database(":memory:"); + initializeDatabase(d); + runMigrations(d); + return d; +} + +const HINDSIGHT_TEST_CONFIG = { + provider: "hindsight" as const, + endpoint: "http://10.0.0.1:8889", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"] as ("historian" | "agent" | "dreamer")[], + tags: [] as string[], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [] as string[], + global_from_prompt: false, + search: true, + mental_models: false, + profile_mental_models: ["user-preferences"], + }, +}; + +function recallBackend( + resultsByScope: Record>, + capture?: ExternalMemoryRecallQuery[], + delayMs = 0, +): ExternalMemoryBackend { + return { + backendId: "fake:recall", + initialize: async () => true, + retain: async (items) => items.length, + recall: async (query) => { + capture?.push(query); + if (delayMs > 0) await new Promise((r) => setTimeout(r, delayMs)); + return resultsByScope[query.scope ?? "global"] ?? []; + }, + dispose: async () => {}, + }; +} + +const ARGS = { + sessionId: "ses_recall_1", + projectIdentity: "git:abcdef1234567890", + projectName: "magic-context", +}; + +beforeEach(() => { + mockEmbedBatch.mockReset(); + mockEmbedBatch.mockImplementation(async () => null); + mockLog.mockReset(); + mockLog.mockImplementation(() => {}); + db = makeDb(); + ensureSessionMetaRow(db, ARGS.sessionId); +}); + +afterEach(() => { + if (db) { + try { + closeQuietly(db); + } catch { + } finally { + db = null; + } + } + _resetExternalMemoryForTests(); + _resetExternalRecallForTests(); +}); + +describe("startSessionRecall", () => { + test("fans out 3 slices and persists a done snapshot (persist before settle)", async () => { + const captured: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => + recallBackend( + { + project: [{ content: "proj fact", category: "ARCHITECTURE" }], + user: [{ content: "user pref" }], + global: [{ content: "homelab fact" }], + }, + captured, + ), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const { state, snapshot } = readExternalRecallSnapshot(db!, ARGS.sessionId); + expect(state).toBe("done"); + expect(snapshot?.project).toEqual([{ content: "proj fact", category: "ARCHITECTURE" }]); + expect(snapshot?.profile).toEqual([{ content: "user pref" }]); + expect(snapshot?.global).toEqual([{ content: "homelab fact" }]); + expect(captured.map((q) => q.scope).sort()).toEqual(["global", "project", "user"]); + const projectQuery = captured.find((q) => q.scope === "project"); + expect(projectQuery?.projectIdentity).toBe(ARGS.projectIdentity); + expect(projectQuery?.projectName).toBe(ARGS.projectName); + expect(projectQuery?.maxTokens).toBe(2048); + }); + + test("global_from_prompt=false ignores firstUserPrompt (pure template query)", async () => { + const captured: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => recallBackend({}, captured)); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS, firstUserPrompt: "fix the flaky auth test" }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const globalQuery = captured.find((q) => q.scope === "global"); + expect(globalQuery?.query).toContain(ARGS.projectName); + expect(globalQuery?.query).not.toContain("fix the flaky auth test"); + expect(globalQuery?.query).not.toContain("current task:"); + }); + + test("global_from_prompt=true enriches the global query with a normalized prompt excerpt", async () => { + const captured: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => recallBackend({}, captured)); + initializeExternalMemory({ + ...HINDSIGHT_TEST_CONFIG, + recall: { ...HINDSIGHT_TEST_CONFIG.recall, global_from_prompt: true }, + }); + startSessionRecall({ + db: db!, + ...ARGS, + firstUserPrompt: " fix the flaky\n auth test in project-zeta ", + }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const globalQuery = captured.find((q) => q.scope === "global"); + // Project name ALWAYS stays in the query (cross-project by-name links). + expect(globalQuery?.query).toContain(ARGS.projectName); + // Whitespace collapsed, prompt content present. + expect(globalQuery?.query).toContain( + "current task: fix the flaky auth test in project-zeta", + ); + // Project + profile slices stay deterministic templates regardless. + const projectQuery = captured.find((q) => q.scope === "project"); + expect(projectQuery?.query).not.toContain("current task:"); + }); + + test("global_from_prompt=true without a prompt falls back to the template query", async () => { + const captured: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => recallBackend({}, captured)); + initializeExternalMemory({ + ...HINDSIGHT_TEST_CONFIG, + recall: { ...HINDSIGHT_TEST_CONFIG.recall, global_from_prompt: true }, + }); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const globalQuery = captured.find((q) => q.scope === "global"); + expect(globalQuery?.query).toContain(ARGS.projectName); + expect(globalQuery?.query).not.toContain("current task:"); + }); + + test("normalizePromptExcerpt collapses whitespace and caps length", () => { + expect(normalizePromptExcerpt(" a\n\n b\tc ")).toBe("a b c"); + expect(normalizePromptExcerpt(undefined)).toBe(""); + expect(normalizePromptExcerpt("x".repeat(1000)).length).toBe(400); + }); + + test("single-fire: second start joins, no duplicate recalls", async () => { + const captured: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => recallBackend({ project: [] }, captured, 20)); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + expect(captured.length).toBe(3); // one fan-out, not two + }); + + test("done state short-circuits re-fire (restart replay)", async () => { + _setTestExternalBackendFactory(() => recallBackend({ project: [{ content: "v1" }] })); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + _resetExternalRecallForTests(); // simulate process restart (in-flight map cleared) + const captured: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => recallBackend({ project: [] }, captured)); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 1000); + expect(captured.length).toBe(0); // persisted done → no re-fire + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot?.project).toEqual([ + { content: "v1" }, + ]); + }); + + test("stuck pending after crash re-fires", async () => { + db! + .prepare( + "UPDATE session_meta SET external_recall_state = 'pending' WHERE session_id = ?", + ) + .run(ARGS.sessionId); + const captured: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => recallBackend({ project: [] }, captured)); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + expect(captured.length).toBe(3); + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).state).toBe("done"); + }); + + test("backend failure persists failed state, never throws", async () => { + _setTestExternalBackendFactory(() => ({ + backendId: "fake:boom", + initialize: async () => true, + retain: async () => 0, + recall: async () => { + throw new Error("boom"); + }, + dispose: async () => {}, + })); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + // recallFromExternalBackend swallows → empty slices → done-but-empty is + // also acceptable; assert it SETTLED and snapshot is empty either way. + const { state, snapshot } = readExternalRecallSnapshot(db!, ARGS.sessionId); + expect(state === "done" || state === "failed").toBe(true); + expect(snapshot === null || computeRecallSnapshotHash(snapshot) === "").toBe(true); + }); + + test("provider off is a no-op (state stays null)", async () => { + initializeExternalMemory({ provider: "off" }); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 200); + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).state).toBeNull(); + }); + + test("hash dedup drops exact-match duplicates of local memories", async () => { + // seed a local memory with identical normalized content + insertMemory(db!, { + projectPath: ARGS.projectIdentity, + category: "ARCHITECTURE", + content: "Proj Fact", // normalizes equal to "proj fact" + sourceType: "historian", + }); + _setTestExternalBackendFactory(() => + recallBackend({ + project: [{ content: "proj fact" }, { content: "unique fact" }], + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot?.project).toEqual([ + { content: "unique fact" }, + ]); + }); + + test("cross-slice dedup: global duplicate of project hit dropped", async () => { + _setTestExternalBackendFactory(() => + recallBackend({ + project: [{ content: "shared fact" }], + global: [{ content: "shared fact" }, { content: "global only" }], + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const snapshot = readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot; + expect(snapshot?.project).toEqual([{ content: "shared fact" }]); + expect(snapshot?.global).toEqual([{ content: "global only" }]); + }); + + test("trim respects max_tokens per slice", async () => { + const big = "x".repeat(400); // ~100 tokens per line + _setTestExternalBackendFactory(() => + recallBackend({ + project: Array.from({ length: 50 }, (_, i) => ({ content: `${big} ${i}` })), + }), + ); + initializeExternalMemory({ + ...HINDSIGHT_TEST_CONFIG, + recall: { ...HINDSIGHT_TEST_CONFIG.recall, max_tokens: 256 }, + }); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const snapshot = readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot; + expect(snapshot).not.toBeNull(); + if (!snapshot) throw new Error("no snapshot"); + expect(snapshot.project.length).toBeGreaterThan(0); + expect(snapshot.project.length).toBeLessThan(50); + }); + + test("mental models replace project slice when available", async () => { + const recallCalls: ExternalMemoryRecallQuery[] = []; + _setTestExternalBackendFactory(() => ({ + ...recallBackend({ project: [{ content: "recall fallback" }] }, recallCalls), + mentalModels: async (query) => + query.scope === "project" + ? [{ content: "MM doc\nline 2", category: "project-conventions" }] + : [], + })); + initializeExternalMemory({ + ...HINDSIGHT_TEST_CONFIG, + recall: { ...HINDSIGHT_TEST_CONFIG.recall, mental_models: true }, + }); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot?.project).toEqual([ + { content: "MM doc\nline 2", category: "project-conventions" }, + ]); + // project recall is short-circuited → no project recall POST + expect(recallCalls.some((q) => q.scope === "project")).toBe(false); + }); + + test("empty mental models fall back to recall", async () => { + _setTestExternalBackendFactory(() => ({ + ...recallBackend({ project: [{ content: "recall fallback" }] }), + mentalModels: async () => [], + })); + initializeExternalMemory({ + ...HINDSIGHT_TEST_CONFIG, + recall: { ...HINDSIGHT_TEST_CONFIG.recall, mental_models: true }, + }); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot?.project).toEqual([ + { content: "recall fallback" }, + ]); + }); + + test("mental_models false skips fast path and always uses recall", async () => { + let mentalModelsCalled = false; + _setTestExternalBackendFactory(() => ({ + ...recallBackend({ project: [{ content: "recall only" }] }), + mentalModels: async () => { + mentalModelsCalled = true; + return [{ content: "should not be used" }]; + }, + })); + initializeExternalMemory({ + ...HINDSIGHT_TEST_CONFIG, + recall: { ...HINDSIGHT_TEST_CONFIG.recall, mental_models: false }, + }); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot?.project).toEqual([ + { content: "recall only" }, + ]); + expect(mentalModelsCalled).toBe(false); + }); +}); + +describe("embedding model-guard (regression: cross-model cosine dedup)", () => { + // Regression for the over-permissive filter: + // !queryModelId || queryModelId === "off" || e.modelId === queryModelId + // which admitted ALL stored vectors when the query model was unknown/"off", + // producing meaningless cosine scores across different embedding spaces. + // Correct behavior: only cosine-dedup when query model is known AND matches; + // otherwise fall back to hash-only dedup (localVectors stays empty). + // + // NOTE: The embedBatchForProject mock in this test file targets + // "../../project-embedding-registry" (relative to the test file), which + // resolves to a different path than the actual import in external-recall.ts + // ("../project-embedding-registry" relative to external-recall.ts). As a + // result, the mock is NOT called during the recall flow. Tests 2 and 3 + // therefore verify the model-guard filter expression directly (unit-level), + // since the embedding mock cannot be injected through the current test + // module boundary. Test 1 confirms the hash-dedup pipeline runs end-to-end. + + beforeEach(() => { + resetEmbeddingCacheForTests(); + }); + + test("hash dedup still drops recalled items that match a local memory (baseline)", async () => { + // Insert a local memory. The recalled item has the same normalized content. + insertMemory(db!, { + projectPath: ARGS.projectIdentity, + category: "ARCHITECTURE", + content: "shared fact", + sourceType: "historian", + }); + _setTestExternalBackendFactory(() => + recallBackend({ + project: [ + { content: "shared fact" }, // hash-duplicate → dropped + { content: "unique recalled fact" }, // no match → kept + ], + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const snapshot = readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot; + // Only the unique item survives hash dedup. + expect(snapshot?.project).toEqual([{ content: "unique recalled fact" }]); + }); + + test("model-guard: unknown/off query model → localVectors empty → no cosine dedup (real dedupAndTrim)", async () => { + // RED-GREEN regression: the old filter + // !queryModelId || queryModelId === "off" || e.modelId === queryModelId + // included ALL stored vectors when queryModelId was "off", producing + // meaningless cosine scores. The fix uses an empty localVectors when the + // query model is unknown/off. + // + // Setup: a local memory with a stored embedding (model-A, vector [1, 0]). + // The recalled item has different text (not a hash-dup) but the same + // direction vector. mockEmbedBatch returns modelId="off" for the recalled + // items, so the model guard must suppress cosine dedup entirely. + // + // OLD BUG: localVectors = [model-A vector] → cosine sim = 1.0 ≥ 0.85 → + // recalled item dropped → snapshot.project = [] → test FAILS. + // FIX: localVectors = [] → no cosine dedup → item survives → PASSES. + const localMemory = insertMemory(db!, { + projectPath: ARGS.projectIdentity, + category: "ARCHITECTURE", + content: "local memory content", + sourceType: "historian", + }); + // Store a model-A embedding for the local memory. + saveEmbedding(db!, localMemory.id, new Float32Array([1, 0]), "model-A"); + resetEmbeddingCacheForTests(); + + // mockEmbedBatch returns modelId="off" — unknown model, cosine dedup must + // be suppressed regardless of vector similarity. + mockEmbedBatch.mockImplementation(async () => ({ + vectors: [new Float32Array([1, 0])], // same direction as local — would be a cosine dup + modelId: "off", + })); + + _setTestExternalBackendFactory(() => + recallBackend({ + project: [{ content: "recalled item with different text" }], + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const snapshot = readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot; + // Item must survive: modelId="off" → localVectors empty → no cosine dedup. + expect(snapshot?.project).toEqual([{ content: "recalled item with different text" }]); + }); + + test("model-guard: known query model → same-model stored vectors participate; near-dup dropped (real dedupAndTrim)", async () => { + // Complement of the test above: when the query model IS known and matches + // the stored embedding model, cosine dedup fires and drops near-duplicates. + // + // Setup: same local memory + model-A embedding. mockEmbedBatch returns + // modelId="model-A" for the recalled item (same model as stored). + // + // FIX: localVectors = [model-A vector] → cosine sim = 1.0 ≥ 0.85 → + // recalled item dropped → snapshot.project = [] → PASSES. + const localMemory = insertMemory(db!, { + projectPath: ARGS.projectIdentity, + category: "ARCHITECTURE", + content: "local memory content", + sourceType: "historian", + }); + saveEmbedding(db!, localMemory.id, new Float32Array([1, 0]), "model-A"); + resetEmbeddingCacheForTests(); + + mockEmbedBatch.mockImplementation(async () => ({ + vectors: [new Float32Array([1, 0])], // cosine sim = 1.0 with local + modelId: "model-A", + })); + + _setTestExternalBackendFactory(() => + recallBackend({ + project: [{ content: "recalled item with different text" }], + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await waitForSessionRecall(ARGS.sessionId, 5000); + const snapshot = readExternalRecallSnapshot(db!, ARGS.sessionId).snapshot; + // Item must be dropped: model-A matches → cosine sim = 1.0 ≥ 0.85 → dedup. + expect(snapshot?.project).toEqual([]); + }); +}); + +describe("maybeAwaitExternalRecall", () => { + test("waits when first render imminent and recall pending", async () => { + _setTestExternalBackendFactory(() => + recallBackend({ project: [{ content: "late" }] }, undefined, 30), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + await maybeAwaitExternalRecall({ db: db!, sessionId: ARGS.sessionId, hasCachedM0: false }); + expect(readExternalRecallSnapshot(db!, ARGS.sessionId).state).toBe("done"); + }); + + test("does not wait when m0 already cached", async () => { + _setTestExternalBackendFactory(() => + recallBackend({ project: [{ content: "late" }] }, undefined, 5000), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + startSessionRecall({ db: db!, ...ARGS }); + const before = Date.now(); + await maybeAwaitExternalRecall({ db: db!, sessionId: ARGS.sessionId, hasCachedM0: true }); + expect(Date.now() - before).toBeLessThan(100); + }); +}); diff --git a/packages/plugin/src/features/magic-context/memory/external-recall.ts b/packages/plugin/src/features/magic-context/memory/external-recall.ts new file mode 100644 index 000000000..73b6f30a6 --- /dev/null +++ b/packages/plugin/src/features/magic-context/memory/external-recall.ts @@ -0,0 +1,346 @@ +import { estimateTokens } from "../../../hooks/magic-context/read-session-formatting"; +import { log } from "../../../shared/logger"; +import type { Database } from "../../../shared/sqlite"; +import { embedBatchForProject } from "../project-embedding-registry"; +import { getActiveUserMemories } from "../user-memory/storage-user-memory"; +import { cosineSimilarity } from "./cosine-similarity"; +import { getProjectEmbeddings } from "./embedding-cache"; +import { + getExternalRecallConfig, + mentalModelsFromExternalBackend, + recallFromExternalBackend, +} from "./external-memory"; +import type { + ExternalMemoryMentalModelQuery, + ExternalMemoryRecallResult, +} from "./external-memory-provider"; +import { + computeRecallSnapshotHash, + type ExternalRecallSliceItem, + type ExternalRecallSnapshot, + readExternalRecallSnapshot, +} from "./external-recall-read"; +import { computeNormalizedHash } from "./normalize-hash"; +import { getMemoriesByProject } from "./storage-memory"; + +const inFlight = new Map>(); + +export function _resetExternalRecallForTests(): void { + inFlight.clear(); +} + +function persistRecallState( + db: Database, + sessionId: string, + state: "pending" | "done" | "failed", + snapshot?: ExternalRecallSnapshot, +): void { + try { + db.prepare( + "UPDATE session_meta SET external_recall_state = ?, external_recall_json = ?, external_recall_at = ? WHERE session_id = ?", + ).run(state, snapshot ? JSON.stringify(snapshot) : null, Date.now(), sessionId); + } catch (error) { + log("[magic-context] external recall: persist failed:", error); + } +} + +/** Fire the once-per-session external recall. Idempotent; never throws. */ +export function startSessionRecall(args: { + db: Database; + sessionId: string; + projectIdentity: string; + projectName: string; + /** Excerpt of the session's FIRST user prompt — enriches the global-slice + * query when `recall.global_from_prompt` is enabled. The first prompt is + * immutable for the session, so the query (and therefore the frozen + * snapshot) stays deterministic across crash-recovery re-fires. */ + firstUserPrompt?: string; +}): void { + try { + const config = getExternalRecallConfig(); + if (!config?.enabled) return; + if (inFlight.has(args.sessionId)) return; + const { state } = readExternalRecallSnapshot(args.db, args.sessionId); + // done/failed → settled for this session. pending WITH no in-flight + // promise = previous process died mid-recall → re-fire (deadlock guard). + if (state === "done" || state === "failed") return; + persistRecallState(args.db, args.sessionId, "pending"); + const promise = runSessionRecall(args, config).catch((error) => { + log("[magic-context] external recall failed:", error); + persistRecallState(args.db, args.sessionId, "failed"); + }); + inFlight.set(args.sessionId, promise); + void promise.finally(() => inFlight.delete(args.sessionId)); + } catch (error) { + log("[magic-context] external recall start failed:", error); + } +} + +/** Resolve when the session's recall settles, or after timeoutMs. Never throws. */ +export async function waitForSessionRecall(sessionId: string, timeoutMs: number): Promise { + const promise = inFlight.get(sessionId); + if (!promise) return; + let timer: ReturnType | undefined; + try { + await Promise.race([ + promise, + new Promise((r) => { + timer = setTimeout(r, timeoutMs); + }), + ]); + } finally { + // Clear the timer on the fast-resolve path so the node timer queue does + // not hold the callback (and any closure) past the function return. + if (timer !== undefined) clearTimeout(timer); + } +} + +/** + * Hybrid A-path: block ONLY when the first m[0] render is imminent (cache + * already cold) and a recall is in flight. Called from the async transform + * right before the postprocess phase. + */ +export async function maybeAwaitExternalRecall(args: { + db: Database; + sessionId: string; + hasCachedM0: boolean; +}): Promise { + const config = getExternalRecallConfig(); + if (!config?.enabled) return; + if (args.hasCachedM0) return; + if (!inFlight.has(args.sessionId)) return; + await waitForSessionRecall(args.sessionId, config.timeout_ms); +} + +async function sliceWithMentalModelFastPath( + config: { mental_models: boolean }, + query: ExternalMemoryMentalModelQuery, + recallFallback: () => Promise, +): Promise { + if (config.mental_models) { + const models = await mentalModelsFromExternalBackend(query); + if (models.length > 0) return models; + } + return recallFallback(); +} + +/** Max characters of first-prompt text folded into the global recall query. + * Long prompts dilute the semantic signal and bloat the recall request. */ +const GLOBAL_QUERY_PROMPT_EXCERPT_CHARS = 400; + +/** Normalize a first-prompt excerpt for query embedding: collapse whitespace + * (multi-line prompts must not break the query shape) and cap length. */ +export function normalizePromptExcerpt(prompt: string | undefined): string { + if (!prompt) return ""; + return prompt.replace(/\s+/g, " ").trim().slice(0, GLOBAL_QUERY_PROMPT_EXCERPT_CHARS); +} + +function buildGlobalQuery(args: { projectName: string; firstUserPrompt?: string }): string { + const base = `infrastructure, environment, tooling, gotchas, and conventions relevant to working on ${args.projectName}`; + const excerpt = normalizePromptExcerpt(args.firstUserPrompt); + // Project name stays in the query either way — cross-project globals that + // mention THIS project by name (origin provenance, entity links) must keep + // surfacing even when the prompt is about something else entirely. + return excerpt ? `${base}; current task: ${excerpt}` : base; +} + +async function runSessionRecall( + args: { + db: Database; + sessionId: string; + projectIdentity: string; + projectName: string; + firstUserPrompt?: string; + }, + config: NonNullable>, +): Promise { + const [project, profile, global] = await Promise.all([ + sliceWithMentalModelFastPath( + config, + { + scope: "project", + projectIdentity: args.projectIdentity, + projectName: args.projectName, + }, + () => + recallFromExternalBackend({ + query: `project rules, architecture decisions, configuration, constraints, conventions for ${args.projectName}`, + scope: "project", + projectIdentity: args.projectIdentity, + projectName: args.projectName, + maxTokens: config.max_tokens, + }), + ), + sliceWithMentalModelFastPath(config, { scope: "user" }, () => + recallFromExternalBackend({ + query: "user preferences, working style, communication habits", + scope: "user", + maxTokens: config.max_tokens, + }), + ), + // Global slice always uses full recall — no fast path. The query is + // optionally enriched with the session's first user prompt + // (recall.global_from_prompt) so cross-project knowledge relevant to + // the task at hand — e.g. globals that name ANOTHER project the + // prompt mentions — surfaces without an explicit ctx_search. + recallFromExternalBackend({ + query: buildGlobalQuery({ + projectName: args.projectName, + ...(config.global_from_prompt && args.firstUserPrompt + ? { firstUserPrompt: args.firstUserPrompt } + : {}), + }), + scope: "global", + maxTokens: config.max_tokens, + }), + ]); + + const snapshot = await dedupAndTrim(args.db, args.projectIdentity, config, { + project: project.map(toSliceItem), + profile: profile.map(toSliceItem), + global: global.map(toSliceItem), + }); + + // PERSIST FIRST, then settle (the promise resolution is the A-path's + // signal that materializeM0 can read the snapshot). + persistRecallState(args.db, args.sessionId, "done", snapshot); +} + +function toSliceItem(result: { content: string; category?: string }): ExternalRecallSliceItem { + return { content: result.content, ...(result.category ? { category: result.category } : {}) }; +} + +function safeActiveUserMemoryContents(db: Database): string[] { + try { + return getActiveUserMemories(db).map((m) => m.content); + } catch { + return []; // table missing in minimal fixtures + } +} + +async function dedupAndTrim( + db: Database, + projectIdentity: string, + config: { dedup_threshold: number; max_tokens: number }, + raw: ExternalRecallSnapshot, +): Promise { + // ── Hash sets (always available) ── + const localMemories = getMemoriesByProject(db, projectIdentity); + const localHashes = new Set(localMemories.map((m) => m.normalizedHash)); + const userContents = safeActiveUserMemoryContents(db); + for (const content of userContents) localHashes.add(computeNormalizedHash(content)); + + // ── Embedding side (best-effort) ── + const allRecalled = [...raw.project, ...raw.profile, ...raw.global]; + const recalledVectors: (Float32Array | null)[] = allRecalled.map(() => null); + let localVectors: Float32Array[] = []; + try { + const recalledResult = + allRecalled.length > 0 + ? await embedBatchForProject( + projectIdentity, + allRecalled.map((item) => item.content), + ) + : null; + if (recalledResult) { + for (let i = 0; i < recalledResult.vectors.length; i += 1) { + recalledVectors[i] = recalledResult.vectors[i] ?? null; + } + const stored = getProjectEmbeddings(db, projectIdentity); + // Honor the model guard: only compare against local vectors that were + // embedded with the same model as the recalled items. Mismatched + // vectors (different dimensionality or space) produce meaningless + // cosine scores and must be excluded. + // When the query model is unknown ("off" / falsy), cosine dedup is + // meaningless across potentially different embedding spaces — fall + // back to hash-only dedup by keeping localVectors empty. + const queryModelId = recalledResult.modelId; + localVectors = + queryModelId && queryModelId !== "off" + ? [...stored.values()] + .filter((e) => e.modelId === queryModelId) + .map((e) => e.embedding) + : []; + if (userContents.length > 0) { + const userResult = await embedBatchForProject(projectIdentity, userContents); + if (userResult) { + for (const vector of userResult.vectors) { + if (vector) localVectors.push(vector); + } + } + } + } + } catch (error) { + log("[magic-context] external recall: dedup embedding unavailable, hash-only:", error); + } + + const keptVectors: Float32Array[] = []; + const keptHashes = new Set(); + let flatIndex = 0; + const isDuplicate = (item: ExternalRecallSliceItem): boolean => { + const hash = computeNormalizedHash(item.content); + if (localHashes.has(hash) || keptHashes.has(hash)) return true; + const vector = recalledVectors[flatIndex]; + if (vector) { + for (const localVector of localVectors) { + if (cosineSimilarity(vector, localVector) >= config.dedup_threshold) return true; + } + for (const keptVector of keptVectors) { + if (cosineSimilarity(vector, keptVector) >= config.dedup_threshold) return true; + } + } + return false; + }; + const keep = (item: ExternalRecallSliceItem): void => { + keptHashes.add(computeNormalizedHash(item.content)); + const vector = recalledVectors[flatIndex]; + if (vector) keptVectors.push(vector); + }; + + const dedupSlice = (slice: ExternalRecallSliceItem[]): ExternalRecallSliceItem[] => { + const result: ExternalRecallSliceItem[] = []; + for (const item of slice) { + if (!isDuplicate(item)) { + keep(item); + result.push(item); + } + flatIndex += 1; + } + return result; + }; + + // Order matters: project wins over global on cross-slice duplicates. + const projectSlice = dedupSlice(raw.project); + const profileSlice = dedupSlice(raw.profile); + const globalSlice = dedupSlice(raw.global); + + return { + project: trimSlice(sortSlice(projectSlice), config.max_tokens), + profile: trimSlice(sortSlice(profileSlice), config.max_tokens), + global: trimSlice(sortSlice(globalSlice), config.max_tokens), + }; +} + +/** Deterministic render order — recall result order is not guaranteed stable. */ +function sortSlice(slice: ExternalRecallSliceItem[]): ExternalRecallSliceItem[] { + return [...slice].sort((a, b) => a.content.localeCompare(b.content)); +} + +function trimSlice(slice: ExternalRecallSliceItem[], maxTokens: number): ExternalRecallSliceItem[] { + const result: ExternalRecallSliceItem[] = []; + let used = 0; + for (const item of slice) { + // Price the item the way the injection renders it: single-line items + // as "- content" (+newline), multi-line documents (mental models) + // verbatim with blank-line separators on both sides. + const multiLine = item.content.includes("\n"); + const rendered = multiLine ? item.content : `- ${item.content}`; + const tokens = estimateTokens(rendered) + (multiLine ? 2 : 1); + if (used + tokens > maxTokens) continue; + result.push(item); + used += tokens; + } + return result; +} + +export { computeRecallSnapshotHash, readExternalRecallSnapshot }; diff --git a/packages/plugin/src/features/magic-context/memory/index.ts b/packages/plugin/src/features/magic-context/memory/index.ts index 553d8e50c..a3b0cfec6 100644 --- a/packages/plugin/src/features/magic-context/memory/index.ts +++ b/packages/plugin/src/features/magic-context/memory/index.ts @@ -2,6 +2,10 @@ export * from "./constants"; export * from "./embedding"; export * from "./embedding-backfill"; export * from "./embedding-cache"; +export * from "./external-memory"; +export * from "./external-memory-provider"; +export * from "./external-recall"; +export * from "./external-recall-read"; export * from "./normalize-hash"; export * from "./project-identity"; export { promoteSessionFactsToMemory } from "./promotion"; diff --git a/packages/plugin/src/features/magic-context/memory/promotion.test.ts b/packages/plugin/src/features/magic-context/memory/promotion.test.ts index a6c336359..40b1ebbe1 100644 --- a/packages/plugin/src/features/magic-context/memory/promotion.test.ts +++ b/packages/plugin/src/features/magic-context/memory/promotion.test.ts @@ -4,6 +4,12 @@ import { afterEach, beforeEach, describe, expect, it, mock, spyOn } from "bun:te import { Database } from "../../../shared/sqlite"; import { closeQuietly } from "../../../shared/sqlite-helpers"; import { CATEGORY_DEFAULT_TTL } from "./constants"; +import { + _resetExternalMemoryForTests, + _setTestExternalBackendFactory, + initializeExternalMemory, +} from "./external-memory"; +import type { ExternalMemoryBackend, ExternalMemoryRetainItem } from "./external-memory-provider"; import { computeNormalizedHash } from "./normalize-hash"; const mockEmbedText = mock(async () => null); @@ -107,8 +113,46 @@ afterEach(() => { db = null; } } + _resetExternalMemoryForTests(); }); +const HINDSIGHT_TEST_CONFIG = { + provider: "hindsight" as const, + endpoint: "http://10.0.0.1:8889", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"] as ("historian" | "agent" | "dreamer")[], + tags: [] as string[], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [] as string[], + global_from_prompt: false, + search: true, + mental_models: false, + profile_mental_models: ["user-preferences"], + }, +}; + +function captureTee(): ExternalMemoryRetainItem[][] { + const calls: ExternalMemoryRetainItem[][] = []; + _setTestExternalBackendFactory( + (): ExternalMemoryBackend => ({ + backendId: "fake:test", + initialize: async () => true, + retain: async (items) => { + calls.push([...items]); + return items.length; + }, + dispose: async () => {}, + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + return calls; +} + describe("promotion", () => { describe("#given promotable facts", () => { it("promotes a new ARCHITECTURE_DECISIONS fact", () => { @@ -400,4 +444,59 @@ describe("promotion", () => { expect(getMemoriesByProject(db, "/repo/project")).toHaveLength(0); }); }); + + describe("#given external memory tee", () => { + it("tees newly inserted facts with project scope", async () => { + db = makeMemoryDatabase(); + const calls = captureTee(); + + promoteSessionFactsToMemory( + db, + "ses_1", + "git:rootsha", + [{ category: "PROJECT_RULES", content: "tee me" }], + { projectName: "myproj" }, + ); + await Bun.sleep(10); + + expect(calls.length).toBe(1); + expect(calls[0]?.[0]).toMatchObject({ + content: "tee me", + category: "PROJECT_RULES", + scope: "project", + projectIdentity: "git:rootsha", + projectName: "myproj", + sourceType: "historian", + sessionId: "ses_1", + }); + }); + + it("does NOT tee dedup hits", async () => { + db = makeMemoryDatabase(); + const calls = captureTee(); + + promoteSessionFactsToMemory(db, "ses_1", "git:rootsha", [ + { category: "PROJECT_RULES", content: "dup fact" }, + ]); + await Bun.sleep(10); + promoteSessionFactsToMemory(db, "ses_2", "git:rootsha", [ + { category: "PROJECT_RULES", content: "dup fact" }, + ]); + await Bun.sleep(10); + + expect(calls.length).toBe(1); + }); + + it("does NOT tee non-promotable categories", async () => { + db = makeMemoryDatabase(); + const calls = captureTee(); + + promoteSessionFactsToMemory(db, "ses_1", "git:rootsha", [ + { category: "NOT_A_CATEGORY", content: "skip me" }, + ]); + await Bun.sleep(10); + + expect(calls.length).toBe(0); + }); + }); }); diff --git a/packages/plugin/src/features/magic-context/memory/promotion.ts b/packages/plugin/src/features/magic-context/memory/promotion.ts index d109bfa93..9a42eaeee 100644 --- a/packages/plugin/src/features/magic-context/memory/promotion.ts +++ b/packages/plugin/src/features/magic-context/memory/promotion.ts @@ -2,6 +2,8 @@ import { sessionLog } from "../../../shared/logger"; import type { Database } from "../../../shared/sqlite"; import { CATEGORY_DEFAULT_TTL, PROMOTABLE_CATEGORIES } from "./constants"; import { embedTextForProject } from "./embedding"; +import { teeToExternalBackend } from "./external-memory"; +import type { ExternalMemoryRetainItem } from "./external-memory-provider"; import { computeNormalizedHash } from "./normalize-hash"; import { getMemoryByHash, insertMemory, updateMemorySeenCount } from "./storage-memory"; import { saveEmbedding } from "./storage-memory-embeddings"; @@ -31,7 +33,9 @@ export function promoteSessionFactsToMemory( sessionId: string, projectPath: string, facts: SessionFact[], + options?: { projectName?: string }, ): void { + const teedItems: ExternalMemoryRetainItem[] = []; for (const fact of facts) { if (!isPromotableCategory(fact.category)) { continue; @@ -59,6 +63,15 @@ export function promoteSessionFactsToMemory( // Intentional: fire-and-forget embedding — promotion runs infrequently (after historian passes) // and the number of new facts per pass is small. Batching adds complexity for negligible benefit. void embedAndStoreMemory(db, sessionId, projectPath, memory.id, memory.content); + teedItems.push({ + content: memory.content, + category: fact.category, + scope: "project", + projectIdentity: projectPath, + ...(options?.projectName ? { projectName: options.projectName } : {}), + sourceType: "historian", + sessionId, + }); } catch (error) { sessionLog( sessionId, @@ -67,6 +80,11 @@ export function promoteSessionFactsToMemory( ); } } + + // Fire-and-forget batched tee — never blocks or fails promotion. + if (teedItems.length > 0) { + void teeToExternalBackend("historian", teedItems); + } } async function embedAndStoreMemory( diff --git a/packages/plugin/src/features/magic-context/message-index.ts b/packages/plugin/src/features/magic-context/message-index.ts index 37887773a..966090bec 100644 --- a/packages/plugin/src/features/magic-context/message-index.ts +++ b/packages/plugin/src/features/magic-context/message-index.ts @@ -7,6 +7,7 @@ import type { RawMessage } from "../../hooks/magic-context/read-session-raw"; import { getHarness } from "../../shared/harness"; import type { Database, Statement as PreparedStatement } from "../../shared/sqlite"; import { removeSystemReminders } from "../../shared/system-directive"; +import { runWriteTransaction } from "../../shared/write-transaction"; import { clearCompressionDepth } from "./compression-depth-storage"; interface MessageHistoryIndexRow { @@ -218,9 +219,7 @@ export function indexMessagesAfterOrdinal( // is reflected, so the second skips those ordinals and inserts nothing // duplicate. The bulk SELECT of existing message-ids is still avoided (it // held the writer lock too long on ~30k-row sessions). - db.exec("BEGIN IMMEDIATE"); - let committed = false; - try { + return runWriteTransaction(db, () => { // Re-read under the lock: another process may have advanced the // watermark between the caller's out-of-transaction read and now. const effectiveWatermark = Math.max( @@ -245,18 +244,8 @@ export function indexMessagesAfterOrdinal( // Never regress a higher watermark a concurrent writer may have set. const newWatermark = Math.max(effectiveWatermark, finalWatermark); getUpsertIndexStatement(db).run(sessionId, newWatermark, now, getHarness()); - db.exec("COMMIT"); - committed = true; - } finally { - if (!committed) { - try { - db.exec("ROLLBACK"); - } catch { - // already rolled back / no active transaction - } - } - } - return inserted; + return inserted; + }); } export function ensureMessagesIndexed( diff --git a/packages/plugin/src/features/magic-context/migrations-v39.test.ts b/packages/plugin/src/features/magic-context/migrations-v39.test.ts new file mode 100644 index 000000000..5880a4717 --- /dev/null +++ b/packages/plugin/src/features/magic-context/migrations-v39.test.ts @@ -0,0 +1,62 @@ +/// + +import { describe, expect, test } from "bun:test"; +import { Database } from "../../shared/sqlite"; +import { closeQuietly } from "../../shared/sqlite-helpers"; +import { LATEST_MIGRATION_VERSION, runMigrations } from "./migrations"; +import { initializeDatabase } from "./storage-db"; + +function columnNames(db: Database, table: string): string[] { + return (db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>).map( + (column) => column.name, + ); +} + +describe("migration v39 — external recall snapshot + m[0] marker", () => { + test("adds external recall columns to session_meta on a fresh DB, idempotently", () => { + const db = new Database(":memory:"); + try { + initializeDatabase(db); + runMigrations(db); + runMigrations(db); + + const columns = columnNames(db, "session_meta"); + expect(columns).toContain("external_recall_json"); + expect(columns).toContain("external_recall_state"); + expect(columns).toContain("external_recall_at"); + expect(columns).toContain("cached_m0_external_recall_hash"); + expect( + db + .prepare("SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1") + .get(), + ).toEqual({ version: LATEST_MIGRATION_VERSION }); + } finally { + closeQuietly(db); + } + }); + + test("external recall columns store and round-trip nulls (default absent state)", () => { + const db = new Database(":memory:"); + try { + initializeDatabase(db); + runMigrations(db); + + db.prepare("INSERT INTO session_meta (session_id, harness) VALUES (?, ?)").run( + "ses_v39", + "test-harness", + ); + const row = db + .prepare( + "SELECT external_recall_json, external_recall_state, external_recall_at, cached_m0_external_recall_hash FROM session_meta WHERE session_id = ?", + ) + .get("ses_v39") as Record; + + expect(row.external_recall_json).toBeNull(); + expect(row.external_recall_state).toBeNull(); + expect(row.external_recall_at).toBeNull(); + expect(row.cached_m0_external_recall_hash).toBeNull(); + } finally { + closeQuietly(db); + } + }); +}); diff --git a/packages/plugin/src/features/magic-context/migrations.ts b/packages/plugin/src/features/magic-context/migrations.ts index 3dc6bc896..cd07d15b6 100644 --- a/packages/plugin/src/features/magic-context/migrations.ts +++ b/packages/plugin/src/features/magic-context/migrations.ts @@ -1492,6 +1492,35 @@ const MIGRATIONS: Migration[] = [ `); }, }, + { + // Was v31 on the pre-v0.23 external-memory-backend branch; renumbered + // to v33 pre-v0.24-rebase, then to v37 when upstream v0.24 shipped its + // own v33/34/35/36, then to v38 when upstream v0.25 took v37. Now v39 + // after upstream v0.26 shipped its own v37 (drain latch) and v38 + // (transform_decisions). The body is ensureColumn-idempotent, so a dev + // DB that already ran it under an old number re-applies harmlessly. + version: 39, + description: "External memory v2: session recall snapshot + m[0] recall marker", + up: (db: Database) => { + // session_meta existence guard — see v30's comment (partial test fixtures). + const hasSessionMeta = db + .prepare( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='session_meta' LIMIT 1", + ) + .get(); + if (!hasSessionMeta) return; + // Per-session external recall snapshot (post-dedup, post-trim) — the + // frozen content every render replays for byte stability. + ensureColumn(db, "session_meta", "external_recall_json", "TEXT"); + ensureColumn(db, "session_meta", "external_recall_state", "TEXT"); + ensureColumn(db, "session_meta", "external_recall_at", "INTEGER"); + // m[0] marker: hash of the external content baked into the cached m[0] + // ('' = none). NOT a mustMaterialize trigger — drives only the m[1] + // delta comparison. No cache-clear needed: the + // cachedRowMatchesState comparison normalizes NULL and '' to equal. + ensureColumn(db, "session_meta", "cached_m0_external_recall_hash", "TEXT"); + }, + }, ]; /** diff --git a/packages/plugin/src/features/magic-context/search.test.ts b/packages/plugin/src/features/magic-context/search.test.ts index bc8b07df2..a4c0ccb83 100644 --- a/packages/plugin/src/features/magic-context/search.test.ts +++ b/packages/plugin/src/features/magic-context/search.test.ts @@ -17,10 +17,17 @@ import { } from "./compartment-chunk-embedding"; import { appendCompartments, getCompartments, replaceSessionFacts } from "./compartment-storage"; import { getMemoryById, insertMemory, resetEmbeddingCacheForTests, saveEmbedding } from "./memory"; +import { + _resetExternalMemoryForTests, + _setTestExternalBackendFactory, + initializeExternalMemory, +} from "./memory/external-memory"; +import type { ExternalMemoryBackend } from "./memory/external-memory-provider"; import { ensureMessagesIndexed } from "./message-index"; import { runMigrations } from "./migrations"; import { unifiedSearch } from "./search"; import { initializeDatabase } from "./storage-db"; +import { ensureSessionMetaRow } from "./storage-meta-shared"; const readMessages = (sessionId: string) => rawMessagesBySession.get(sessionId) ?? []; const embedQuery = async (text: string) => { @@ -84,6 +91,13 @@ afterEach(() => { embeddingQueries.length = 0; rawMessagesBySession.clear(); resetEmbeddingCacheForTests(); + // Module-level external-memory state (factory + cached backend) must be + // wiped after every test in this file. The "external search source" + // describe block sets up a stub factory whose last-wins closure would + // otherwise be visible to the next test file in the suite (e.g. + // transform.test.ts's "injects empty m[0]" test, which would see a + // phantom block). + _resetExternalMemoryForTests(); }); describe("unifiedSearch", () => { @@ -801,3 +815,160 @@ describe("unifiedSearch", () => { expect(embeddingQueries).toEqual([]); }); }); + +const HINDSIGHT_TEST_CONFIG = { + provider: "hindsight" as const, + endpoint: "http://10.1.0.99:8889", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"] as ("historian" | "agent" | "dreamer")[], + tags: [] as string[], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [] as string[], + global_from_prompt: false, + search: true, + mental_models: false, + profile_mental_models: ["user-preferences"], + }, +}; + +describe("external search source", () => { + let db: Database; + const sessionId = "ses-external"; + const projectPath = "/repo/project"; + + beforeEach(() => { + // Wipe any cached backend instance from a previous test. The + // production configIdentity only spans (provider, endpoint, main_bank, + // project_bank) — recall.search and retain_sources changes are NOT + // identity changes — so a different recall.search in test N does not + // invalidate the backend created in test N-1, and the snapshot-seeding + // test would otherwise see the prior test's recall() closure. + _resetExternalMemoryForTests(); + db = createTestDb(); + // v31 columns require a session_meta row before the UPDATE in the + // snapshot-seeding test can land. createTestDb already runs migrations. + ensureSessionMetaRow(db, sessionId); + }); + + afterEach(() => { + closeQuietly(db); + }); + + it("explicit search with external enabled returns external hits", async () => { + _setTestExternalBackendFactory( + (): ExternalMemoryBackend => ({ + backendId: "fake:search", + initialize: async () => true, + retain: async () => 0, + recall: async (query) => + query.scope === "project" + ? [{ content: "external project hit" }] + : [{ content: "external global hit" }], + dispose: async () => {}, + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + + const results = await unifiedSearch(db, sessionId, projectPath, "query", { + explicitSearch: true, + // Stub the embedding seam like every other test in this file — + // without it the memory source falls back to the module-level + // embedder and pays a multi-second local-model load that has + // nothing to do with what these tests assert (and flirts with + // bun's 5s per-test timeout under load). + embedQuery: async () => null, + isEmbeddingRuntimeEnabled: () => false, + }); + const external = results.filter((r) => r.source === "external"); + expect(external.length).toBeGreaterThan(0); + expect(external.map((r) => r.content)).toContain("external project hit"); + }); + + it("non-explicit search never calls external", async () => { + let called = 0; + _setTestExternalBackendFactory( + (): ExternalMemoryBackend => ({ + backendId: "fake:search", + initialize: async () => true, + retain: async () => 0, + recall: async () => { + called += 1; + return []; + }, + dispose: async () => {}, + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + + await unifiedSearch(db, sessionId, projectPath, "query", { + explicitSearch: false, + embedQuery: async () => null, + isEmbeddingRuntimeEnabled: () => false, + }); + expect(called).toBe(0); + }); + + it("external excluded when recall.search false", async () => { + let called = 0; + _setTestExternalBackendFactory( + (): ExternalMemoryBackend => ({ + backendId: "fake:search", + initialize: async () => true, + retain: async () => 0, + recall: async () => { + called += 1; + return []; + }, + dispose: async () => {}, + }), + ); + initializeExternalMemory({ + ...HINDSIGHT_TEST_CONFIG, + recall: { ...HINDSIGHT_TEST_CONFIG.recall, search: false }, + }); + + await unifiedSearch(db, sessionId, projectPath, "query", { + explicitSearch: true, + embedQuery: async () => null, + isEmbeddingRuntimeEnabled: () => false, + }); + expect(called).toBe(0); + }); + + it("external hits already injected this session are filtered out", async () => { + db.prepare( + "UPDATE session_meta SET external_recall_state='done', external_recall_json=? WHERE session_id = ?", + ).run( + JSON.stringify({ + project: [{ content: "already injected" }], + profile: [], + global: [], + }), + sessionId, + ); + _setTestExternalBackendFactory( + (): ExternalMemoryBackend => ({ + backendId: "fake:search", + initialize: async () => true, + retain: async () => 0, + recall: async () => [{ content: "already injected" }, { content: "fresh hit" }], + dispose: async () => {}, + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + + const results = await unifiedSearch(db, sessionId, projectPath, "query", { + explicitSearch: true, + embedQuery: async () => null, + isEmbeddingRuntimeEnabled: () => false, + }); + const contents = results.filter((r) => r.source === "external").map((r) => r.content); + expect(contents).not.toContain("already injected"); + expect(contents).toContain("fresh hit"); + }); +}); diff --git a/packages/plugin/src/features/magic-context/search.ts b/packages/plugin/src/features/magic-context/search.ts index d5bfb86ef..95100db66 100644 --- a/packages/plugin/src/features/magic-context/search.ts +++ b/packages/plugin/src/features/magic-context/search.ts @@ -19,6 +19,9 @@ import { } from "./memory"; import { cosineSimilarity } from "./memory/cosine-similarity"; import { embedText, getProjectEmbeddingSnapshot, isEmbeddingEnabled } from "./memory/embedding"; +import { isExternalSearchEnabled, recallFromExternalBackend } from "./memory/external-memory"; +import { readExternalRecallSnapshot } from "./memory/external-recall-read"; +import { computeNormalizedHash } from "./memory/normalize-hash"; import { sanitizeFtsQuery } from "./memory/storage-memory-fts"; import { expandWorkspaceIdentitySetWithAliases, @@ -56,7 +59,7 @@ interface MessageSearchRow { const messageSearchStatements = new WeakMap(); -export type SearchSource = "memory" | "message" | "git_commit"; +export type SearchSource = "memory" | "message" | "git_commit" | "external"; export interface UnifiedSearchOptions { limit?: number; @@ -102,6 +105,10 @@ export interface UnifiedSearchOptions { * in; the auto-search hot path stays single-probe to protect its latency * budget. NL queries with no extractable probes are unaffected either way. */ explicitSearch?: boolean; + /** Override for tests; defaults to module-level isExternalSearchEnabled(). */ + externalSearchEnabled?: boolean; + /** Project name (basename) for external bank resolution. */ + projectName?: string; } export interface MemorySearchResult { @@ -147,11 +154,19 @@ export interface GitCommitSearchResult { matchType: "semantic" | "fts" | "hybrid"; } +export interface ExternalSearchResult { + source: "external"; + content: string; + score: number; + category?: string; +} + export type UnifiedSearchResult = | MemorySearchResult | MessageSearchResult | CompartmentSearchResult - | GitCommitSearchResult; + | GitCommitSearchResult + | ExternalSearchResult; function normalizeLimit(limit?: number): number { if (typeof limit !== "number" || !Number.isFinite(limit)) { @@ -934,6 +949,12 @@ function getSourceBoost(result: UnifiedSearchResult): number { return MESSAGE_SOURCE_BOOST; case "git_commit": return GIT_COMMIT_SOURCE_BOOST; + case "external": + // Below curated memories (1.3) — local rows outrank external + // matches when the two are even close, which is the safe default + // (a verified local fact is more authoritative than an external + // recall of an older, possibly stale statement). + return 1.0; } } @@ -962,6 +983,10 @@ function compareUnifiedResults(left: UnifiedSearchResult, right: UnifiedSearchRe return right.committedAtMs - left.committedAtMs; } + if (left.source === "external" && right.source === "external") { + return left.content.localeCompare(right.content); + } + return 0; } @@ -997,16 +1022,94 @@ function searchGitCommits(args: { return hits.map(toGitCommitResult); } +const EXTERNAL_SEARCH_TIMEOUT_MS = 5_000; + +async function searchExternal(args: { + db: Database; + sessionId: string; + projectPath: string; + projectName?: string; + query: string; + limit: number; + signal?: AbortSignal; +}): Promise { + // Tighter bound than the backend's 10s fetch timeout: an explicit tool + // call shouldn't hang on a slow Hindsight. + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), EXTERNAL_SEARCH_TIMEOUT_MS); + const onOuterAbort = () => controller.abort(); + args.signal?.addEventListener("abort", onOuterAbort, { once: true }); + try { + const [project, global] = await Promise.all([ + recallFromExternalBackend( + { + query: args.query, + scope: "project", + projectIdentity: args.projectPath, + ...(args.projectName ? { projectName: args.projectName } : {}), + limit: args.limit, + }, + controller.signal, + ), + recallFromExternalBackend( + { query: args.query, scope: "global", limit: args.limit }, + controller.signal, + ), + ]); + + // Drop hits already visible in this session's injected + // block, and cross-bank duplicates. + const { snapshot } = readExternalRecallSnapshot(args.db, args.sessionId); + const injectedHashes = new Set(); + for (const slice of [snapshot?.project, snapshot?.profile, snapshot?.global]) { + for (const item of slice ?? []) { + injectedHashes.add(computeNormalizedHash(item.content)); + } + } + const seen = new Set(); + const merged: Array<{ content: string; category?: string }> = []; + for (const hit of [...project, ...global]) { + const hash = computeNormalizedHash(hit.content); + if (injectedHashes.has(hash) || seen.has(hash)) continue; + seen.add(hash); + merged.push(hit); + } + + // Rank-based scoring (Hindsight result order is its relevance order). + const top = merged.slice(0, args.limit); + return top.map( + (hit, rank) => + ({ + source: "external" as const, + content: previewText(hit.content), + score: linearDecayScore(rank, top.length), + ...(hit.category ? { category: hit.category } : {}), + }) satisfies ExternalSearchResult, + ); + } finally { + clearTimeout(timeout); + args.signal?.removeEventListener("abort", onOuterAbort); + } +} + function resolveSources(sources: SearchSource[] | undefined): Set { if (sources === undefined) { - // Default: search all three sources. Facts are deliberately NOT a - // source — they're always rendered in so searching - // them returns content the agent already sees. + // Default: search the three local sources. Facts are deliberately NOT + // a source — they're always rendered in so searching + // them returns content the agent already sees. External is opt-in via + // the `sources` arg; the runExternal gate downstream also requires + // explicitSearch=true so the auto-search hot path never fires an + // external roundtrip even if a caller lists "external" in the array. return new Set(["memory", "message", "git_commit"]); } const set = new Set(); for (const source of sources) { - if (source === "memory" || source === "message" || source === "git_commit") { + if ( + source === "memory" || + source === "message" || + source === "git_commit" || + source === "external" + ) { set.add(source); } } @@ -1040,6 +1143,17 @@ export async function unifiedSearch( const runGitCommits = activeSources.has("git_commit") && gitCommitsEnabled; const runCompartmentChunks = runMessages && memoryFeatureEnabled && embeddingEnabled; + // External recall is opt-in AND explicit-only: the auto-search hot path + // (every user prompt hint) MUST NOT fire an external roundtrip. The + // `options.sources === undefined` clause means callers who pass no + // `sources` arg still get external on explicit searches; the auto-search + // caller never sets `explicitSearch`, so the gate short-circuits there. + const externalEnabled = options.externalSearchEnabled ?? isExternalSearchEnabled(); + const runExternal = + externalEnabled && + options.explicitSearch === true && + (options.sources === undefined || activeSources.has("external")); + // Embed the query ONCE at the top — both memory and git-commit searches // need the same vector. Previously each search called `embedQuery` // independently, producing two parallel HTTP requests for the same @@ -1120,7 +1234,7 @@ export async function unifiedSearch( limit: tierLimit, }); - const [memoryResults, gitCommitResults] = await Promise.all([ + const [memoryResults, gitCommitResults, externalResults] = await Promise.all([ runMemory ? searchMemories({ db, @@ -1146,9 +1260,25 @@ export async function unifiedSearch( }), ) : Promise.resolve([] as GitCommitSearchResult[]), + runExternal + ? searchExternal({ + db, + sessionId, + projectPath, + projectName: options.projectName, + query: trimmedQuery, + limit: tierLimit, + signal: options.signal, + }) + : Promise.resolve([] as ExternalSearchResult[]), ]); - const results = [...memoryResults, ...messageLikeResults, ...gitCommitResults] + const results = [ + ...memoryResults, + ...messageLikeResults, + ...gitCommitResults, + ...externalResults, + ] .sort(compareUnifiedResults) .slice(0, limit); diff --git a/packages/plugin/src/features/magic-context/session-parent-registry.test.ts b/packages/plugin/src/features/magic-context/session-parent-registry.test.ts new file mode 100644 index 000000000..47865ba3f --- /dev/null +++ b/packages/plugin/src/features/magic-context/session-parent-registry.test.ts @@ -0,0 +1,51 @@ +import { afterEach, describe, expect, it } from "bun:test"; +import { + _resetSessionParentRegistryForTests, + registerSessionParent, + resolveRootSessionId, + unregisterSessionParent, +} from "./session-parent-registry"; + +describe("session-parent-registry", () => { + afterEach(() => { + _resetSessionParentRegistryForTests(); + }); + + it("resolves unregistered sessions to themselves", () => { + expect(resolveRootSessionId("ses-main")).toBe("ses-main"); + }); + + it("resolves a registered child to its parent", () => { + registerSessionParent("ses-child", "ses-parent"); + expect(resolveRootSessionId("ses-child")).toBe("ses-parent"); + expect(resolveRootSessionId("ses-parent")).toBe("ses-parent"); + }); + + it("walks nested children to the root", () => { + registerSessionParent("ses-grandchild", "ses-child"); + registerSessionParent("ses-child", "ses-parent"); + expect(resolveRootSessionId("ses-grandchild")).toBe("ses-parent"); + }); + + it("ignores self-parenting and empty ids", () => { + registerSessionParent("ses-a", "ses-a"); + registerSessionParent("", "ses-parent"); + registerSessionParent("ses-b", ""); + expect(resolveRootSessionId("ses-a")).toBe("ses-a"); + expect(resolveRootSessionId("ses-b")).toBe("ses-b"); + }); + + it("terminates on a registration cycle (depth cap)", () => { + registerSessionParent("ses-x", "ses-y"); + registerSessionParent("ses-y", "ses-x"); + // Must return SOME id without looping forever. + const resolved = resolveRootSessionId("ses-x"); + expect(resolved === "ses-x" || resolved === "ses-y").toBe(true); + }); + + it("unregister removes the linkage", () => { + registerSessionParent("ses-child", "ses-parent"); + unregisterSessionParent("ses-child"); + expect(resolveRootSessionId("ses-child")).toBe("ses-child"); + }); +}); diff --git a/packages/plugin/src/features/magic-context/session-parent-registry.ts b/packages/plugin/src/features/magic-context/session-parent-registry.ts new file mode 100644 index 000000000..d7a9bebf0 --- /dev/null +++ b/packages/plugin/src/features/magic-context/session-parent-registry.ts @@ -0,0 +1,65 @@ +/** + * In-memory child→parent session registry. + * + * Child sessions (sidekick, dreamer, user task subagents) execute tools with + * their OWN session id, but several session-scoped reads only make sense + * against the conversation the child was spawned FROM: + * + * - ctx_search's message-history source: the child session has no indexed + * messages and no compartment boundary, so searching by the child id + * returns nothing — the user-visible conversation history lives on the + * root session. + * - ctx_search's "already visible" filters (memory_block_ids, the injected + * snapshot): both are persisted on the root session's + * session_meta; reading them by the child id silently disables the + * filter and re-surfaces content the parent conversation already shows. + * + * Population: the `session.created` event carries `parentID` for every child + * session, so the event handler registers all parentage centrally. Lookups + * walk to the root (depth-capped, cycle-safe) so nested children resolve to + * the user's conversation. + * + * Posture: best-effort, in-memory only (parentage never spans a restart — + * a restarted process has no in-flight children). Bounded LRU per the + * module-singleton convention; unknown ids resolve to themselves, so every + * caller degrades to current behavior when the registry has no entry (e.g. + * the Pi harness, whose subagents run in separate processes). + */ + +import { BoundedSessionMap } from "../../shared/bounded-session-map"; + +const MAX_TRACKED_SESSIONS = 500; +/** Defensive cap on parent-chain walks (cycles cannot be created via + * session.created ordering, but a corrupt registration must not loop). */ +const MAX_PARENT_DEPTH = 5; + +const parentBySession = new BoundedSessionMap(MAX_TRACKED_SESSIONS); + +export function registerSessionParent(childSessionId: string, parentSessionId: string): void { + if (!childSessionId || !parentSessionId) return; + if (childSessionId === parentSessionId) return; + parentBySession.set(childSessionId, parentSessionId); +} + +export function unregisterSessionParent(childSessionId: string): void { + parentBySession.delete(childSessionId); +} + +/** + * Resolve a session id to its root (the user's conversation). Returns the + * input unchanged when no parent is registered — safe to call on main + * sessions and on harnesses that never register parentage. + */ +export function resolveRootSessionId(sessionId: string): string { + let current = sessionId; + for (let depth = 0; depth < MAX_PARENT_DEPTH; depth += 1) { + const parent = parentBySession.peek(current); + if (!parent || parent === current) return current; + current = parent; + } + return current; +} + +export function _resetSessionParentRegistryForTests(): void { + parentBySession.clear(); +} diff --git a/packages/plugin/src/features/magic-context/sidekick/agent.ts b/packages/plugin/src/features/magic-context/sidekick/agent.ts index ee790e509..b9c7477f5 100644 --- a/packages/plugin/src/features/magic-context/sidekick/agent.ts +++ b/packages/plugin/src/features/magic-context/sidekick/agent.ts @@ -6,6 +6,7 @@ import { extractLatestAssistantText } from "../../../shared/assistant-message-ex import { shouldKeepSubagents } from "../../../shared/keep-subagents"; import { log, sessionLog } from "../../../shared/logger"; import { resolveFallbackChain } from "../../../shared/resolve-fallbacks"; +import { registerSessionParent } from "../session-parent-registry"; import { openDatabase } from "../storage"; import { recordChildInvocation } from "../subagent-token-capture"; import { SIDEKICK_SYSTEM_PROMPT, stripThinkingBlocks } from "./core"; @@ -70,6 +71,14 @@ export async function runSidekick(deps: { throw error; } + // Synchronous belt-and-braces alongside the session.created event + // registration: sidekick's ctx_search calls must resolve to the parent + // conversation even if async event delivery races the child's first + // tool call. Cleanup rides session.deleted / LRU eviction. + if (deps.sessionId) { + registerSessionParent(agentSessionId, deps.sessionId); + } + await shared.promptSyncWithModelSuggestionRetry( deps.client, { diff --git a/packages/plugin/src/features/magic-context/storage-db.ts b/packages/plugin/src/features/magic-context/storage-db.ts index 33a652509..c523eaf55 100644 --- a/packages/plugin/src/features/magic-context/storage-db.ts +++ b/packages/plugin/src/features/magic-context/storage-db.ts @@ -37,7 +37,7 @@ export function getSchemaFenceRejection(): { return lastSchemaFenceRejection; } -export const LATEST_SUPPORTED_VERSION = 38; +export const LATEST_SUPPORTED_VERSION = 39; export interface OpenDatabaseOptions { dbPath?: string; @@ -1007,6 +1007,7 @@ CREATE INDEX IF NOT EXISTS idx_dream_queue_pending ON dream_queue(started_at, en ensureColumn(db, "session_meta", "cached_m0_system_hash", "TEXT"); ensureColumn(db, "session_meta", "cached_m0_tool_set_hash", "TEXT"); ensureColumn(db, "session_meta", "cached_m0_model_key", "TEXT"); + ensureColumn(db, "session_meta", "cached_m0_external_recall_hash", "TEXT"); // Pi-only: frozen baseline boundary (end_message_id) captured at // materialization so Pi trims against the snapshot boundary that produced // m[0], not a live-recomputed one a concurrent recomp could have moved. @@ -1393,6 +1394,14 @@ export function openDatabase(dbPathOrOptions?: string | OpenDatabaseOptions): Da mkdirSync(dbDir, { recursive: true }); const db = new Database(dbPath); + // Install busy_timeout IMMEDIATELY after open, before any read. Without + // this, enforceSchemaFence()'s schema_migrations read below can throw + // SQLITE_BUSY when a sibling process holds the WAL writer lock during a + // concurrent cold open (two opencode processes booting at once, or a + // child session spawning while the parent is mid-checkpoint). The + // busy_timeout set later inside initializeDatabase() is too late for + // this first read. See the "database is locked" plugin-load failures. + db.exec("PRAGMA busy_timeout=5000"); if (!enforceSchemaFence(db, dbPath, latestSupportedVersion)) { closeQuietly(db); return null; diff --git a/packages/plugin/src/features/magic-context/storage-meta-session.ts b/packages/plugin/src/features/magic-context/storage-meta-session.ts index 2c3954763..9302849a4 100644 --- a/packages/plugin/src/features/magic-context/storage-meta-session.ts +++ b/packages/plugin/src/features/magic-context/storage-meta-session.ts @@ -40,6 +40,7 @@ const SESSION_META_FALLBACK_SELECTS: Partial< cached_m0_system_hash: "NULL AS cached_m0_system_hash", cached_m0_tool_set_hash: "NULL AS cached_m0_tool_set_hash", cached_m0_model_key: "NULL AS cached_m0_model_key", + cached_m0_external_recall_hash: "NULL AS cached_m0_external_recall_hash", last_observed_model_key: "NULL AS last_observed_model_key", upgrade_reminded_at: "NULL AS upgrade_reminded_at", }; diff --git a/packages/plugin/src/features/magic-context/storage-meta-shared.ts b/packages/plugin/src/features/magic-context/storage-meta-shared.ts index 68e350775..51ccad70f 100644 --- a/packages/plugin/src/features/magic-context/storage-meta-shared.ts +++ b/packages/plugin/src/features/magic-context/storage-meta-shared.ts @@ -43,6 +43,7 @@ export interface SessionMetaRow { cached_m0_system_hash: string | null; cached_m0_tool_set_hash: string | null; cached_m0_model_key: string | null; + cached_m0_external_recall_hash: string | null; last_observed_model_key: string | null; last_usage_context_limit: number | null; prior_boundary_ordinal: number | null; @@ -96,6 +97,7 @@ export const SESSION_META_SELECT_COLUMNS = [ "cached_m0_system_hash", "cached_m0_tool_set_hash", "cached_m0_model_key", + "cached_m0_external_recall_hash", "last_observed_model_key", "last_usage_context_limit", "prior_boundary_ordinal", @@ -148,6 +150,7 @@ export const META_COLUMNS: Record = { cachedM0SystemHash: "cached_m0_system_hash", cachedM0ToolSetHash: "cached_m0_tool_set_hash", cachedM0ModelKey: "cached_m0_model_key", + cachedM0ExternalRecallHash: "cached_m0_external_recall_hash", lastObservedModelKey: "last_observed_model_key", lastUsageContextLimit: "last_usage_context_limit", priorBoundaryOrdinal: "prior_boundary_ordinal", @@ -188,6 +191,7 @@ export const NULL_BIND_META_KEYS = new Set([ "cachedM0MaterializedAt", "cachedM0SessionFactsVersion", "cachedM0UpgradeState", + "cachedM0ExternalRecallHash", "lastObservedModelKey", "upgradeRemindedAt", "piStableIdScheme", @@ -264,6 +268,7 @@ export function isSessionMetaRow(row: unknown): row is SessionMetaRow { isStringOrNull(r.cached_m0_system_hash) && isStringOrNull(r.cached_m0_tool_set_hash) && isStringOrNull(r.cached_m0_model_key) && + isStringOrNull(r.cached_m0_external_recall_hash) && isStringOrNull(r.last_observed_model_key) && isNumberOrNull(r.last_usage_context_limit) && isNumberOrNull(r.prior_boundary_ordinal) && @@ -318,6 +323,7 @@ export function getDefaultSessionMeta(sessionId: string): SessionMeta { cachedM0SystemHash: null, cachedM0ToolSetHash: null, cachedM0ModelKey: null, + cachedM0ExternalRecallHash: null, lastObservedModelKey: null, lastUsageContextLimit: 0, priorBoundaryOrdinal: 1, @@ -433,6 +439,7 @@ export function toSessionMeta(row: SessionMetaRow): SessionMeta { cachedM0SystemHash: stringOrNull(row.cached_m0_system_hash), cachedM0ToolSetHash: stringOrNull(row.cached_m0_tool_set_hash), cachedM0ModelKey: stringOrNull(row.cached_m0_model_key), + cachedM0ExternalRecallHash: stringOrNull(row.cached_m0_external_recall_hash), lastObservedModelKey: stringOrNull(row.last_observed_model_key), lastUsageContextLimit: numOrZero(row.last_usage_context_limit), priorBoundaryOrdinal: Math.max(1, numOrZero(row.prior_boundary_ordinal) || 1), @@ -463,6 +470,7 @@ export interface PersistCachedM0Payload { upgradeState: string | null; systemHash?: string | null; modelKey?: string | null; + externalRecallHash?: string | null; } export function persistCachedM0( @@ -487,7 +495,8 @@ export function persistCachedM0( cached_m0_session_facts_version = ?, cached_m0_upgrade_state = ?, cached_m0_system_hash = ?, - cached_m0_model_key = ? + cached_m0_model_key = ?, + cached_m0_external_recall_hash = ? WHERE session_id = ?`, ).run( Buffer.from(payload.m0Bytes), @@ -505,6 +514,7 @@ export function persistCachedM0( payload.upgradeState, payload.systemHash ?? "", payload.modelKey ?? "", + payload.externalRecallHash ?? "", sessionId, ); } @@ -533,6 +543,7 @@ export function clearCachedM0M1(db: Database, sessionId: string): void { ["cached_m0_system_hash", null], ["cached_m0_tool_set_hash", null], ["cached_m0_model_key", null], + ["cached_m0_external_recall_hash", null], ["cached_m0_last_baseline_end_message_id", null], ["memory_block_cache", ""], ["memory_block_count", 0], diff --git a/packages/plugin/src/features/magic-context/types.ts b/packages/plugin/src/features/magic-context/types.ts index 168253841..a33884e3e 100644 --- a/packages/plugin/src/features/magic-context/types.ts +++ b/packages/plugin/src/features/magic-context/types.ts @@ -90,6 +90,7 @@ export interface SessionMeta { cachedM0SystemHash: string | null; cachedM0ToolSetHash: string | null; cachedM0ModelKey: string | null; + cachedM0ExternalRecallHash: string | null; lastObservedModelKey: string | null; lastUsageContextLimit: number; priorBoundaryOrdinal: number; diff --git a/packages/plugin/src/features/magic-context/user-memory/review-user-memories.ts b/packages/plugin/src/features/magic-context/user-memory/review-user-memories.ts index 95f2a65f9..ab96a5752 100644 --- a/packages/plugin/src/features/magic-context/user-memory/review-user-memories.ts +++ b/packages/plugin/src/features/magic-context/user-memory/review-user-memories.ts @@ -8,6 +8,8 @@ import { log } from "../../../shared/logger"; import type { Database } from "../../../shared/sqlite"; import { renewLease } from "../dreamer/lease"; import { DREAMER_SYSTEM_PROMPT } from "../dreamer/task-prompts"; +import { removeFromExternalBackend, teeToExternalBackend } from "../memory/external-memory"; +import type { ExternalMemoryRemoveItem } from "../memory/external-memory-provider"; import { bumpProjectUserProfileVersion } from "../storage"; import { recordChildInvocation } from "../subagent-token-capture"; import { @@ -236,15 +238,27 @@ If no promotions are warranted, return empty arrays. Always consume reviewed can candidateIds: p.candidate_ids ?? [], })) .filter((p) => p.content.length > 0); + // Coerce LLM-provided ids to integers: a stringified id ("5") would + // pass a truthiness check but miss both the number-keyed snapshot Map + // (silently skipping the external corrective remove — resurrecting + // dismissed memories next session) and any strict-typed DB binding. const updates = (parsed.update_existing ?? []) .map((u) => ({ - memoryId: u.memory_id, + memoryId: Number(u.memory_id), content: u.content?.trim() ?? "", })) - .filter((u) => Boolean(u.memoryId) && u.content.length > 0); - const dismissals = (parsed.dismiss_existing ?? []).filter((d) => Boolean(d.memory_id)); + .filter((u) => Number.isInteger(u.memoryId) && u.memoryId > 0 && u.content.length > 0); + const dismissals = (parsed.dismiss_existing ?? []) + .map((d) => ({ ...d, memory_id: Number(d.memory_id) })) + .filter((d) => Number.isInteger(d.memory_id) && d.memory_id > 0); const consumeCandidateIds = parsed.consume_candidate_ids ?? []; + // Snapshot of pre-mutation content for every stable memory touched by + // this pass — needed to compute the corrective remove items for the + // external store AFTER the transaction (the new content is already + // persisted at that point). + const stableContentById = new Map(stableMemories.map((m) => [m.id, m.content])); + args.db.transaction(() => { for (const promotion of promotions) { insertUserMemory(args.db, promotion.content, promotion.candidateIds); @@ -272,6 +286,59 @@ If no promotions are warranted, return empty arrays. Always consume reviewed can result.dismissed = dismissals.length; result.candidatesConsumed = consumeCandidateIds.length; + // Corrective propagation (W2): dismissed/updated stable user memories + // must leave the external store too, or the profile recall slice will + // resurrect them next session. Updates also re-tee the new content so + // the fresh text lands in the main bank immediately (the next recall + // would otherwise still surface the stale content via hash mismatch). + const removedItems: ExternalMemoryRemoveItem[] = []; + for (const dismissal of dismissals) { + const oldContent = stableContentById.get(dismissal.memory_id); + if (oldContent) { + removedItems.push({ + content: oldContent, + category: "USER_PROFILE", + scope: "user", + }); + } + } + for (const update of updates) { + const oldContent = stableContentById.get(update.memoryId); + if (oldContent && oldContent !== update.content) { + removedItems.push({ + content: oldContent, + category: "USER_PROFILE", + scope: "user", + }); + } + } + if (removedItems.length > 0) { + void removeFromExternalBackend(removedItems); + } + if (updates.length > 0) { + void teeToExternalBackend( + "dreamer", + updates.map((update) => ({ + content: update.content, + category: "USER_PROFILE" as const, + scope: "user" as const, + sourceType: "dreamer" as const, + })), + ); + } + + if (promotions.length > 0) { + void teeToExternalBackend( + "dreamer", + promotions.map((promotion) => ({ + content: promotion.content, + category: "USER_PROFILE" as const, + scope: "user" as const, + sourceType: "dreamer" as const, + })), + ); + } + for (const promotion of promotions) { log(`[dreamer] user-memories: promoted "${promotion.content.slice(0, 60)}..."`); } diff --git a/packages/plugin/src/features/magic-context/workspaces.ts b/packages/plugin/src/features/magic-context/workspaces.ts index 412ad7d8b..cddf9822a 100644 --- a/packages/plugin/src/features/magic-context/workspaces.ts +++ b/packages/plugin/src/features/magic-context/workspaces.ts @@ -1,5 +1,6 @@ import { createHash } from "node:crypto"; import type { Database } from "../../shared/sqlite"; +import { runWriteTransaction } from "../../shared/write-transaction"; import { V2_MEMORY_CATEGORIES } from "./memory/constants"; import { normalizeStoredProjectPath, storedPathBelongsToIdentity } from "./project-identity"; @@ -271,11 +272,6 @@ export function computeWorkspaceEpochFingerprint( return hash.digest("hex"); } -function isInTransaction(db: Database): boolean { - const candidate = db as unknown as { inTransaction?: unknown; isTransaction?: unknown }; - return candidate.inTransaction === true || candidate.isTransaction === true; -} - function workspaceMembersForIdentity(db: Database, identity: string): string[] { if (!tableExists(db, "workspace_members")) return [identity]; const rows = db @@ -313,22 +309,7 @@ export function bumpEpochsForWorkspaceMembers( now = Date.now(), ): void { const run = () => bumpEpochRows(db, workspaceMembersForIdentity(db, identity), now); - if (isInTransaction(db)) { - run(); - return; - } - db.exec("BEGIN IMMEDIATE"); - try { - run(); - db.exec("COMMIT"); - } catch (error) { - try { - db.exec("ROLLBACK"); - } catch { - // ignore rollback failures from an already-closed transaction - } - throw error; - } + runWriteTransaction(db, run); } export function bumpEpochsForWorkspaceMemberSet( @@ -337,20 +318,5 @@ export function bumpEpochsForWorkspaceMemberSet( now = Date.now(), ): void { const run = () => bumpEpochRows(db, identities, now); - if (isInTransaction(db)) { - run(); - return; - } - db.exec("BEGIN IMMEDIATE"); - try { - run(); - db.exec("COMMIT"); - } catch (error) { - try { - db.exec("ROLLBACK"); - } catch { - // ignore rollback failures from an already-closed transaction - } - throw error; - } + runWriteTransaction(db, run); } diff --git a/packages/plugin/src/hooks/magic-context/auto-search-hint.ts b/packages/plugin/src/hooks/magic-context/auto-search-hint.ts index 8876d9eab..8d9caa85a 100644 --- a/packages/plugin/src/hooks/magic-context/auto-search-hint.ts +++ b/packages/plugin/src/hooks/magic-context/auto-search-hint.ts @@ -75,6 +75,10 @@ function renderFragment(result: UnifiedSearchResult, charCap: number): string { const compressed = cavemanCompress(source, "ultra"); return truncate(compressed, charCap); } + case "external": { + const compressed = cavemanCompress(result.content, "ultra"); + return truncate(compressed, charCap); + } } } diff --git a/packages/plugin/src/hooks/magic-context/command-handler.ts b/packages/plugin/src/hooks/magic-context/command-handler.ts index 0f89fa133..ca38b94c7 100644 --- a/packages/plugin/src/hooks/magic-context/command-handler.ts +++ b/packages/plugin/src/hooks/magic-context/command-handler.ts @@ -518,7 +518,7 @@ export function createMagicContextCommandHandler(deps: { } const liveModelKey = deps.getLiveModelKey?.(sessionId); const liveContextLimit = deps.getContextLimit?.(sessionId); - const statusOutput = executeStatus( + const statusOutput = await executeStatus( deps.db, sessionId, deps.protectedTags, diff --git a/packages/plugin/src/hooks/magic-context/compartment-runner-embedding-gate.test.ts b/packages/plugin/src/hooks/magic-context/compartment-runner-embedding-gate.test.ts new file mode 100644 index 000000000..30299bbe1 --- /dev/null +++ b/packages/plugin/src/hooks/magic-context/compartment-runner-embedding-gate.test.ts @@ -0,0 +1,192 @@ +/// + +import { afterEach, describe, expect, it, mock } from "bun:test"; +import { mkdirSync, mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { closeDatabase, openDatabase } from "../../features/magic-context/storage"; +import type { PluginContext } from "../../plugin/types"; +import { Database } from "../../shared/sqlite"; +import { closeQuietly } from "../../shared/sqlite-helpers"; +import { executeContextRecomp } from "./compartment-runner"; + +const tempDirs: string[] = []; +const originalXdgDataHome = process.env.XDG_DATA_HOME; + +afterEach(() => { + closeDatabase(); + process.env.XDG_DATA_HOME = originalXdgDataHome; + + for (const dir of tempDirs) { + try { + rmSync(dir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + // Ignore EBUSY on Windows + } + } + tempDirs.length = 0; + + const dumpDir = join(tmpdir(), "magic-context-historian"); + try { + rmSync(dumpDir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + } catch { + // Ignore + } +}); + +describe("compartment embedding gate (provider, not memory)", () => { + it("runs embedding when memory is off but embedding provider is on", async () => { + useTempDataHome("magic-embedding-gate-mem-off-"); + createOpenCodeDb("ses-mem-off", [ + { id: "m-1", role: "user", text: "eligible one" }, + { id: "m-2", role: "assistant", text: "eligible two" }, + { id: "m-3", role: "user", text: "protected 1" }, + { id: "m-4", role: "user", text: "protected 2" }, + { id: "m-5", role: "user", text: "protected 3" }, + { id: "m-6", role: "user", text: "protected 4" }, + { id: "m-7", role: "user", text: "protected 5" }, + ]); + + const db = openDatabase(); + const client = createRecompClient( + 'Summary', + ); + const ensureProjectRegistered = mock(async () => {}); + + await executeContextRecomp({ + client, + db, + sessionId: "ses-mem-off", + historianChunkTokens: 10_000, + directory: "/tmp", + // Memory feature is OFF but the embedding PROVIDER is ON. + // Embedding is the ctx_search substrate, independent of memory. + memoryEnabled: false, + autoPromote: false, + embeddingEnabled: true, + ensureProjectRegistered, + }); + + // The recomp runner gates project registration + embedding on the + // embedding provider, not the memory feature. With embeddingEnabled=true + // and memoryEnabled=false, project registration must still fire. + expect(ensureProjectRegistered).toHaveBeenCalled(); + }); + + it("skips embedding when the embedding provider is off (even with memory on)", async () => { + useTempDataHome("magic-embedding-gate-emb-off-"); + createOpenCodeDb("ses-emb-off", [ + { id: "m-1", role: "user", text: "eligible one" }, + { id: "m-2", role: "assistant", text: "eligible two" }, + { id: "m-3", role: "user", text: "protected 1" }, + { id: "m-4", role: "user", text: "protected 2" }, + { id: "m-5", role: "user", text: "protected 3" }, + { id: "m-6", role: "user", text: "protected 4" }, + { id: "m-7", role: "user", text: "protected 5" }, + ]); + + const db = openDatabase(); + const client = createRecompClient( + 'Summary', + ); + const ensureProjectRegistered = mock(async () => {}); + + await executeContextRecomp({ + client, + db, + sessionId: "ses-emb-off", + historianChunkTokens: 10_000, + directory: "/tmp", + // Memory is fully on (enabled + auto_promote) but the embedding + // provider is OFF. No project registration, no embedding endpoint hit. + memoryEnabled: true, + autoPromote: true, + embeddingEnabled: false, + ensureProjectRegistered, + }); + + // embeddingEnabled=false is the new gate. Project registration must NOT fire. + expect(ensureProjectRegistered).not.toHaveBeenCalled(); + }); +}); + +function createRecompClient(output: string): PluginContext["client"] { + return { + session: { + get: mock(async () => ({ data: { directory: "/tmp" } })), + create: mock(async () => ({ data: { id: "ses-historian-child" } })), + prompt: mock(async () => ({})), + messages: mock(async () => ({ + data: [ + { + info: { role: "assistant", time: { created: 1 } }, + parts: [{ type: "text", text: output }], + }, + ], + })), + delete: mock(async () => ({})), + }, + } as unknown as PluginContext["client"]; +} + +function useTempDataHome(prefix: string): void { + const dir = mkdtempSync(join(tmpdir(), prefix)); + tempDirs.push(dir); + process.env.XDG_DATA_HOME = dir; +} + +function createOpenCodeDb( + sessionId: string, + messages: Array<{ id: string; role: string; text: string }>, +): void { + const dbPath = join(process.env.XDG_DATA_HOME!, "opencode", "opencode.db"); + mkdirSync(dirname(dbPath), { recursive: true }); + const db = new Database(dbPath); + + try { + db.exec(` + CREATE TABLE IF NOT EXISTS message ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + time_created INTEGER NOT NULL, + time_updated INTEGER NOT NULL, + data TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS part ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + message_id TEXT NOT NULL, + session_id TEXT NOT NULL, + time_created INTEGER NOT NULL, + time_updated INTEGER NOT NULL, + data TEXT NOT NULL + ); + `); + + const insertMessage = db.prepare( + "INSERT INTO message (id, session_id, time_created, time_updated, data) VALUES (?, ?, ?, ?, ?)", + ); + const insertPart = db.prepare( + "INSERT INTO part (message_id, session_id, time_created, time_updated, data) VALUES (?, ?, ?, ?, ?)", + ); + + messages.forEach((message, index) => { + const timestamp = index + 1; + insertMessage.run( + message.id, + sessionId, + timestamp, + timestamp, + JSON.stringify({ id: message.id, role: message.role, sessionID: sessionId }), + ); + insertPart.run( + message.id, + sessionId, + timestamp, + timestamp, + JSON.stringify({ type: "text", text: message.text }), + ); + }); + } finally { + closeQuietly(db); + } +} diff --git a/packages/plugin/src/hooks/magic-context/compartment-runner-incremental.ts b/packages/plugin/src/hooks/magic-context/compartment-runner-incremental.ts index af03c37f4..c334fbbcc 100644 --- a/packages/plugin/src/hooks/magic-context/compartment-runner-incremental.ts +++ b/packages/plugin/src/hooks/magic-context/compartment-runner-incremental.ts @@ -1,3 +1,5 @@ +import { basename } from "node:path"; + import { embedAndStoreCompartmentChunks } from "../../features/magic-context/compartment-embedding"; import { insertCompartmentEvents } from "../../features/magic-context/compartment-events"; import { @@ -45,6 +47,7 @@ import { insertUserMemoryCandidates } from "../../features/magic-context/user-me import { normalizeSDKResponse } from "../../shared"; import { describeError } from "../../shared/error-message"; import { sessionLog } from "../../shared/logger"; +import { runWriteTransaction } from "../../shared/write-transaction"; import { updateCompactionMarkerAfterPublication } from "./compaction-marker-manager"; import { buildCompartmentAgentPrompt } from "./compartment-prompt"; import { queueDropsForCompartmentalizedMessages } from "./compartment-runner-drop-queue"; @@ -552,17 +555,9 @@ export async function runCompartmentAgent(deps: CompartmentRunnerDeps): Promise< rollbackDrainReservation(); return; } - let published = false; - db.exec("BEGIN IMMEDIATE"); - try { + const publishOutcome = runWriteTransaction(db, (): "published" | "lease-lost" => { if (!isCompartmentLeaseHeld(db, sessionId, holderId)) { - db.exec("ROLLBACK"); - rollbackDrainReservation(); - sessionLog( - sessionId, - "historian publish skipped: compartment lease no longer held", - ); - return; + return "lease-lost"; } appendCompartments(db, sessionId, persistedCompartments); // v2 faithful fact lifecycle: facts are NOT a REPLACE-the-whole-list @@ -591,16 +586,12 @@ export async function runCompartmentAgent(deps: CompartmentRunnerDeps): Promise< publishedAt: Date.now(), }); } - db.exec("COMMIT"); - published = true; - } finally { - if (!published) { - try { - db.exec("ROLLBACK"); - } catch { - // Transaction may already be closed by an early rollback. - } - } + return "published"; + }); + if (publishOutcome === "lease-lost") { + rollbackDrainReservation(); + sessionLog(sessionId, "historian publish skipped: compartment lease no longer held"); + return; } // Background publication normally preserves the injection cache until // a materializing pass can rebuild history and apply queued drops @@ -632,13 +623,16 @@ export async function runCompartmentAgent(deps: CompartmentRunnerDeps): Promise< // explicitly disabled the memory feature in config. // Two distinct gates: // - embeddingActive: embeddings + project registration fire whenever the - // memory FEATURE is enabled. They are the substrate for ctx_search + - // future dreamer cross-linking and must NOT depend on auto_promote. - // - promotionActive: writing facts as project memories additionally - // requires auto_promote (a user who disabled auto-promotion still wants - // search/embedding, just not auto-written memories). - const embeddingActive = !!promotionDirectory && deps.memoryEnabled !== false; - const promotionActive = embeddingActive && deps.autoPromote !== false; + // embedding PROVIDER is enabled (config `embedding.provider !== "off"`). + // Embeddings are the substrate for ctx_search + future dreamer cross- + // linking and are independent of the memory store flags. + // - promotionActive: writing facts as project memories requires + // memory.enabled + auto_promote (issue #44). A user with memory off + // gets no memory writes at all; a user with memory on but auto-promote + // off still gets search/embedding, just not auto-written memories. + const embeddingActive = !!promotionDirectory && deps.embeddingEnabled !== false; + const promotionActive = + !!promotionDirectory && deps.memoryEnabled !== false && deps.autoPromote !== false; // Register the project ONCE up front (not inside the promotion block): // embeddings below run even on a discard-last pass that skips promotion, @@ -655,6 +649,7 @@ export async function runCompartmentAgent(deps: CompartmentRunnerDeps): Promise< sessionId, resolveProjectIdentity(promotionDirectory), validatedPass.facts ?? [], + { projectName: basename(promotionDirectory) }, ); } diff --git a/packages/plugin/src/hooks/magic-context/compartment-runner-partial-recomp.ts b/packages/plugin/src/hooks/magic-context/compartment-runner-partial-recomp.ts index 5f4b055b5..7ceefac5d 100644 --- a/packages/plugin/src/hooks/magic-context/compartment-runner-partial-recomp.ts +++ b/packages/plugin/src/hooks/magic-context/compartment-runner-partial-recomp.ts @@ -335,9 +335,9 @@ export async function executePartialRecompInternal( // v2: recompute raw chunk embeddings for the rebuilt compartments. // Partial recomp deletes + reinserts compartments, so their chunk // embeddings must be regenerated or the rebuilt rows vanish from - // ctx_search semantic results. Gated on memory-enabled, distinct from + // ctx_search semantic results. Gated on embeddingEnabled, distinct from // fact promotion (which recomp skips). Fire-and-forget, best-effort. - if (deps.memoryEnabled !== false) { + if (deps.embeddingEnabled !== false) { const projectIdentity = resolveProjectIdentity(sessionDirectory); const liveCompartments = getCompartments(db, sessionId); const chunksToEmbed = liveCompartments.map((c) => ({ diff --git a/packages/plugin/src/hooks/magic-context/compartment-runner-recomp.ts b/packages/plugin/src/hooks/magic-context/compartment-runner-recomp.ts index c2a33bad5..ae8ebf67f 100644 --- a/packages/plugin/src/hooks/magic-context/compartment-runner-recomp.ts +++ b/packages/plugin/src/hooks/magic-context/compartment-runner-recomp.ts @@ -25,6 +25,7 @@ import { getErrorMessage } from "../../shared/error-message"; import { getHarness } from "../../shared/harness"; import { sessionLog } from "../../shared/logger"; import type { Database } from "../../shared/sqlite"; +import { runWriteTransaction } from "../../shared/write-transaction"; import { updateCompactionMarkerAfterPublication } from "./compaction-marker-manager"; import { buildCompartmentAgentPrompt } from "./compartment-prompt"; import { queueDropsForCompartmentalizedMessages } from "./compartment-runner-drop-queue"; @@ -90,19 +91,13 @@ export function promoteRecompStagingWithM0Mutation( facts: Array<{ category: string; content: string }>; } | null { const now = Date.now(); - db.exec("BEGIN IMMEDIATE"); - let finished = false; - try { + return runWriteTransaction(db, () => { if (!isCompartmentLeaseHeld(db, sessionId, holderId)) { - db.exec("ROLLBACK"); - finished = true; return null; } const staging = getRecompStaging(db, sessionId); if (!staging || staging.compartments.length === 0) { - db.exec("ROLLBACK"); - finished = true; return null; } @@ -124,18 +119,8 @@ export function promoteRecompStagingWithM0Mutation( db.prepare("DELETE FROM recomp_facts WHERE session_id = ?").run(sessionId); clearCachedM0M1(db, sessionId); - db.exec("COMMIT"); - finished = true; return { compartments: staging.compartments, facts: staging.facts }; - } finally { - if (!finished) { - try { - db.exec("ROLLBACK"); - } catch { - // Transaction may already be closed by SQLite after an error. - } - } - } + }); } export async function executeContextRecompInternal(deps: CompartmentRunnerDeps): Promise { @@ -273,9 +258,9 @@ export async function executeContextRecompInternal(deps: CompartmentRunnerDeps): // Recomp deletes + reinserts every compartment, so their chunk // embeddings must be regenerated — otherwise the rebuilt rows have no // embeddings and vanish from ctx_search semantic results. Embedding is - // the search substrate (gated on memory-enabled), distinct from fact + // the search substrate (gated on embeddingEnabled), distinct from fact // promotion (which recomp deliberately skips). Fire-and-forget. - if (deps.memoryEnabled !== false) { + if (deps.embeddingEnabled !== false) { const projectIdentity = resolveProjectIdentity(sessionDirectory); // Register the project's embedding provider before embedding; // embedBatchForProject silently no-ops for unregistered projects, @@ -589,9 +574,9 @@ export async function executeContextRecompInternal(deps: CompartmentRunnerDeps): // the NORMAL full-completion path (distinct from promoteAndFinalize, which // handles early-exit/partial cases and already embeds). Without this, a // fully-completed recomp leaves the rebuilt rows without chunk embeddings - // → they vanish from ctx_search semantic results. Gated on memory-enabled, + // → they vanish from ctx_search semantic results. Gated on embeddingEnabled, // distinct from fact promotion (recomp skips). - if (deps.memoryEnabled !== false) { + if (deps.embeddingEnabled !== false) { const projectIdentity = resolveProjectIdentity(sessionDirectory); // Register the embedding provider first; embedBatchForProject silently // no-ops for unregistered projects, leaving no chunk embeddings. diff --git a/packages/plugin/src/hooks/magic-context/compartment-runner-types.ts b/packages/plugin/src/hooks/magic-context/compartment-runner-types.ts index 8f3c8f909..91f429a18 100644 --- a/packages/plugin/src/hooks/magic-context/compartment-runner-types.ts +++ b/packages/plugin/src/hooks/magic-context/compartment-runner-types.ts @@ -86,6 +86,13 @@ export interface CompartmentRunnerDeps { * and must NOT generate or store embeddings. Issue #44. */ memoryEnabled?: boolean; + /** + * Embedding provider on/off (config `embedding.provider !== "off"`). + * Gates compartment P1 embedding + project registration — embeddings are + * the ctx_search substrate, independent of the memory store. Undefined = + * enabled (schema default is the local provider). + */ + embeddingEnabled?: boolean; /** * Automatic-promotion gate (`memory.auto_promote` config). When false (and * memory is otherwise enabled), tools and search still work, but historian diff --git a/packages/plugin/src/hooks/magic-context/event-handler.ts b/packages/plugin/src/hooks/magic-context/event-handler.ts index 6b4f79359..c29b4adca 100644 --- a/packages/plugin/src/hooks/magic-context/event-handler.ts +++ b/packages/plugin/src/hooks/magic-context/event-handler.ts @@ -1,6 +1,10 @@ import type { createCompactionHandler } from "../../features/magic-context/compaction"; import { scheduleClearAndReindex } from "../../features/magic-context/message-index-async"; import { detectOverflow } from "../../features/magic-context/overflow-detection"; +import { + registerSessionParent, + unregisterSessionParent, +} from "../../features/magic-context/session-parent-registry"; import { clearHistorianFailureState, clearPendingCompactionMarkerStateIf, @@ -238,6 +242,16 @@ export function createEventHandler(deps: EventHandlerDeps) { return; } + // Track child→parent linkage for ALL child sessions (sidekick, + // dreamer, user task subagents). ctx_search resolves through this + // so session-scoped reads (message-history boundary, visible + // memory ids, injected external-recall snapshot) target the ROOT + // conversation instead of the child's empty session_meta. In-memory + // only — parentage never spans a restart. + if (info.parentID.length > 0) { + registerSessionParent(info.id, info.parentID); + } + // Flag our own hidden children (historian/dreamer/sidekick/ // memory-migration) by their `magic-context-` title prefix so the // transform + system-prompt hooks can fully exempt them. In-memory @@ -726,6 +740,7 @@ export function createEventHandler(deps: EventHandlerDeps) { clearTransformDecisionSession(sessionId); clearMessageTokensCache(sessionId); invalidateTrueRawTokenCache({ sessionId, reason: "session.deleted" }); + unregisterSessionParent(sessionId); return; } }; diff --git a/packages/plugin/src/hooks/magic-context/execute-status.ts b/packages/plugin/src/hooks/magic-context/execute-status.ts index cfe95ffb0..85cf18ce2 100644 --- a/packages/plugin/src/hooks/magic-context/execute-status.ts +++ b/packages/plugin/src/hooks/magic-context/execute-status.ts @@ -1,5 +1,10 @@ import { DEFAULT_EXECUTE_THRESHOLD_PERCENTAGE } from "../../config/schema/magic-context"; import { getCompartments } from "../../features/magic-context/compartment-storage"; +import { + fetchExternalFailedRetains, + getExternalMemoryStatus, +} from "../../features/magic-context/memory/external-memory"; +import { readExternalRecallSnapshot } from "../../features/magic-context/memory/external-recall-read"; import { parseCacheTtl } from "../../features/magic-context/scheduler"; import { getPendingOps } from "../../features/magic-context/storage"; import { getOrCreateSessionMeta } from "../../features/magic-context/storage-meta"; @@ -31,7 +36,7 @@ function formatExecuteThreshold( return `${thresholdPercentage}%`; } -export function executeStatus( +export async function executeStatus( db: Database, sessionId: string, protectedTags: number, @@ -43,7 +48,7 @@ export function executeStatus( commitClusterTrigger?: { enabled: boolean; min_clusters: number }, executeThresholdTokens?: { default?: number; [modelKey: string]: number | undefined }, contextLimit?: number, -): string { +): Promise { // Single source of truth — resolver tells us both the effective percentage AND // which config source won (tokens vs percentage). Previously /ctx-status // reimplemented the token-match check here and missed progressive base-model @@ -185,6 +190,25 @@ export function executeStatus( } } + const externalStatus = getExternalMemoryStatus(); + if (externalStatus) { + const { state: recallState } = readExternalRecallSnapshot(db, sessionId); + // fetchExternalFailedRetains inherits the 10s fetch timeout and + // circuit breaker from HindsightMemoryBackend.request(); null on + // every failure path so the field is always safe to surface. + const failedRetainCount = await fetchExternalFailedRetains(); + lines.push( + "", + "### External memory", + `- provider: ${externalStatus.provider} (${externalStatus.endpoint ?? "?"})`, + `- circuit: ${externalStatus.circuitState ?? "n/a"}`, + `- session recall: ${recallState ?? "not started"}`, + ...(failedRetainCount !== null + ? [`- failed retains (server): ${failedRetainCount}`] + : []), + ); + } + return lines.join("\n"); } catch (error) { sessionLog(sessionId, "ctx-status failed:", error); diff --git a/packages/plugin/src/hooks/magic-context/hook.ts b/packages/plugin/src/hooks/magic-context/hook.ts index f248ed89b..55ee55b44 100644 --- a/packages/plugin/src/hooks/magic-context/hook.ts +++ b/packages/plugin/src/hooks/magic-context/hook.ts @@ -333,6 +333,7 @@ export function createMagicContextHook(deps: MagicContextDeps) { historianTimeoutMs: deps.config.historian_timeout_ms ?? DEFAULT_HISTORIAN_TIMEOUT_MS, memoryEnabled: deps.config.memory?.enabled ?? true, autoPromote: deps.config.memory?.auto_promote ?? true, + embeddingEnabled: deps.config.embedding?.provider !== "off", fallbackModels: historianFallbackModels, fallbackModelId: (() => { const model = resolveLiveModel(sessionId); @@ -548,6 +549,7 @@ export function createMagicContextHook(deps: MagicContextDeps) { autoPromote: deps.config.memory.auto_promote ?? true, } : undefined, + embeddingEnabled: deps.config.embedding?.provider !== "off", ensureProjectRegistered: ensureProjectRegisteredFromOpenCodeDirectory, getHistorianChunkTokens, historyBudgetPercentage: deps.config.history_budget_percentage, diff --git a/packages/plugin/src/hooks/magic-context/inject-compartments.ts b/packages/plugin/src/hooks/magic-context/inject-compartments.ts index fbb4f37b2..8bcfee50f 100644 --- a/packages/plugin/src/hooks/magic-context/inject-compartments.ts +++ b/packages/plugin/src/hooks/magic-context/inject-compartments.ts @@ -15,6 +15,13 @@ import { MEMORY_CATEGORY_ORDER_SQL, MEMORY_CATEGORY_ORDER_UNKNOWN, } from "../../features/magic-context/memory/constants"; +import { + computeRecallSnapshotHash, + type ExternalRecallSliceItem, + type ExternalRecallSnapshot, + readExternalRecallHash, + readExternalRecallSnapshot, +} from "../../features/magic-context/memory/external-recall-read"; import { getMaxMemoryIdForProjects, getMemoriesByProject, @@ -602,6 +609,10 @@ export interface M0SnapshotMarkers { // read), so readCurrentM0SnapshotMarkers takes them as inputs. systemHash: string; modelKey: string; + /** Hash of the persisted external-recall snapshot baked into m[0] ('' = none). + * NOT a HARD bust trigger (external recall is not a materialization driver) — + * drives the m[1] delta comparison only. */ + externalRecallHash: string; } /** @@ -645,6 +656,7 @@ export interface M0M1State { cachedM0SystemHash: string | null; cachedM0ToolSetHash: string | null; cachedM0ModelKey: string | null; + cachedM0ExternalRecallHash: string | null; snapshotMarkers?: M0SnapshotMarkers | null; } @@ -965,6 +977,7 @@ export function readCurrentM0SnapshotMarkers(args: { upgradeState: getUpgradeState(args.db, args.sessionId), systemHash: hard.systemHash, modelKey: hard.modelKey, + externalRecallHash: readExternalRecallHash(args.db, args.sessionId), }; } @@ -991,6 +1004,7 @@ function snapshotMarkersFromCachedM0(state: M0M1State): M0SnapshotMarkers | null upgradeState: state.cachedM0UpgradeState, systemHash: state.cachedM0SystemHash ?? "", modelKey: state.cachedM0ModelKey ?? "", + externalRecallHash: state.cachedM0ExternalRecallHash ?? "", }; } @@ -1361,16 +1375,113 @@ export function renderMemoryBlockV2( return lines.join("\n"); } -function renderUserProfileBlock(memories: UserMemory[], wrapper = "user-profile"): string { - if (memories.length === 0) return ""; +function renderUserProfileBlock( + memories: UserMemory[], + wrapper = "user-profile", + externalLines: readonly ExternalRecallSliceItem[] = [], +): string { + if (memories.length === 0 && externalLines.length === 0) return ""; const lines = [`<${wrapper}>`]; for (const memory of memories) { lines.push(`- ${escapeXmlContent(memory.content)}`); } + for (const item of externalLines) { + lines.push(`- ${escapeXmlContent(item.content)}`); + } lines.push(``); return lines.join("\n"); } +/** Preamble is the model-facing instruction above every external-memory block. + * Reminds the model that recalled facts are background knowledge, not new + * instructions, and to discard items that aren't directly useful. */ +const EXTERNAL_MEMORY_PREAMBLE = + "Background knowledge from past sessions — prioritize recent information when conflicting; use only what is directly useful, ignore the rest."; + +function renderExternalItem(item: ExternalRecallSliceItem): string { + // Multi-line documents (mental-model briefings) render VERBATIM inside the + // block so the model sees the structured markdown. A + // blank-line separator flanks each multi-line item so single-line list + // items and verbatim documents don't visually merge. + return item.content.includes("\n") + ? escapeXmlContent(item.content) + : `- ${escapeXmlContent(item.content)}`; +} + +/** Body of the block. INTENTIONALLY merges only the + * `project` and `global` slices — the `profile` slice is omitted here by + * design and is rendered separately into the block (see + * `renderUserProfileBlock(..., externalRecall?.profile ?? [])` in `renderM0`). + * Duplicating profile lines into would be redundant + * (the model would see the same recall content twice) and would defeat + * the user-profile budget trim that lives on that block. Note that the + * sibling `renderExternalDeltaLines` (m[1] delta path) DOES include the + * profile slice — there profile lines reconcile into at + * the next HARD fold, per its own comment. */ +function renderExternalLines(snapshot: ExternalRecallSnapshot): string[] { + const items: ExternalRecallSliceItem[] = [...snapshot.project, ...snapshot.global]; + const lines: string[] = []; + for (const item of items) { + if (item.content.includes("\n")) { + if (lines.length > 0) lines.push(""); + lines.push(renderExternalItem(item)); + lines.push(""); + } else { + lines.push(renderExternalItem(item)); + } + } + return lines; +} + +function renderExternalDeltaLines(snapshot: ExternalRecallSnapshot): string[] { + const items: ExternalRecallSliceItem[] = [ + ...snapshot.project, + ...snapshot.global, + ...snapshot.profile, + ]; + const lines: string[] = []; + for (const item of items) { + if (item.content.includes("\n")) { + if (lines.length > 0) lines.push(""); + lines.push(renderExternalItem(item)); + lines.push(""); + } else { + lines.push(renderExternalItem(item)); + } + } + return lines; +} + +/** Sibling block after : project + global recall slices. + * Plain lines, content verbatim — no fake ids (recalled items are not + * ctx_memory-addressable rows). Multi-line mental-model documents render + * verbatim between blank-line separators. */ +export function renderExternalMemoryBlock(snapshot: ExternalRecallSnapshot): string { + const body = renderExternalLines(snapshot); + if (body.length === 0) return ""; + return [ + '', + EXTERNAL_MEMORY_PREAMBLE, + "", + ...body, + "", + ].join("\n"); +} + +/** m[1] delta when recall settles after the last m[0] fold — carries ALL + * slices (profile lines reconcile into at the next HARD fold). */ +export function renderExternalMemoryDelta(snapshot: ExternalRecallSnapshot): string { + const body = renderExternalDeltaLines(snapshot); + if (body.length === 0) return ""; + return [ + '', + EXTERNAL_MEMORY_PREAMBLE, + "", + ...body, + "", + ].join("\n"); +} + /** * v2 decayed session-history rendering delegates entirely to the shared * `decay-render` module (which uses the validated `decay-curve` formula). This @@ -1399,14 +1510,22 @@ export function renderM0(args: { historyBudgetTokens?: number; userProfileBudgetTokens?: number; decayPressureMultiplier?: number; + externalRecall?: ExternalRecallSnapshot | null; }): string { const sections: string[] = []; if (args.projectDocs.length > 0) sections.push(args.projectDocs); + // The external-recall PROFILE slice is merged HERE (into ), + // not into . `renderExternalLines` deliberately emits + // only project + global — see its JSDoc for the rationale. Splitting the + // merge this way lets the user-profile budget trim govern recall-derived + // profile lines without inflating the sibling block. const userProfile = renderUserProfileBlock( trimUserMemoriesToBudget( args.userProfileBaseline, args.userProfileBudgetTokens ?? DEFAULT_USER_PROFILE_BUDGET_TOKENS, ), + "user-profile", + args.externalRecall?.profile ?? [], ); if (userProfile) sections.push(userProfile); @@ -1431,6 +1550,10 @@ export function renderM0(args: { args.memoryRenderOptions, ); if (memoriesBlock) sections.push(memoriesBlock); + if (args.externalRecall) { + const externalBlock = renderExternalMemoryBlock(args.externalRecall); + if (externalBlock) sections.push(externalBlock); + } return sections.join("\n\n").trim(); } @@ -1460,6 +1583,7 @@ function applyMarkersToState( // re-fire the same HARD trigger on the very next pass (double-fold). state.cachedM0SystemHash = markers.systemHash; state.cachedM0ModelKey = markers.modelKey; + state.cachedM0ExternalRecallHash = markers.externalRecallHash; state.snapshotMarkers = markers; } @@ -1493,6 +1617,7 @@ export function materializeM0(options: M0M1RenderOptions): MaterializeM0Result { projectPath, workspaceIdentitySet: options.workspaceIdentitySet, }); + let externalRecall: ExternalRecallSnapshot | null = null; let docs: { renderedBlock: string; canonicalHash: string } = { renderedBlock: "", canonicalHash: "", @@ -1539,6 +1664,13 @@ export function materializeM0(options: M0M1RenderOptions): MaterializeM0Result { : getMemoriesByProject(options.db, projectPath, ["active", "permanent"]) : []; userMemories = safeGetActiveUserMemories(options.db); + // In-transaction read of the persisted external recall snapshot. Overrides + // the value readCurrentM0SnapshotMarkers set so render and marker derive + // from the SAME read (no TOCTOU) — mirrors how projectDocsHash is + // overwritten from readProjectDocsCanonical above. + const recall = readExternalRecallSnapshot(options.db, options.sessionId); + externalRecall = recall.state === "done" ? recall.snapshot : null; + snapshotMarkers.externalRecallHash = computeRecallSnapshotHash(externalRecall); options.db.exec("COMMIT"); } catch (error) { try { @@ -1577,6 +1709,7 @@ export function materializeM0(options: M0M1RenderOptions): MaterializeM0Result { historyBudgetTokens: options.historyBudgetTokens ?? DEFAULT_HISTORY_BUDGET_TOKENS, userProfileBudgetTokens: options.userProfileBudgetTokens, decayPressureMultiplier, + externalRecall, }); let attempts = 0; @@ -1593,6 +1726,7 @@ export function materializeM0(options: M0M1RenderOptions): MaterializeM0Result { historyBudgetTokens: budget, userProfileBudgetTokens: options.userProfileBudgetTokens, decayPressureMultiplier, + externalRecall, }); attempts += 1; } @@ -1648,6 +1782,7 @@ export function materializeM0(options: M0M1RenderOptions): MaterializeM0Result { // so carry the captured values and exclude them from the stale check. systemHash: snapshotMarkers.systemHash, modelKey: snapshotMarkers.modelKey, + externalRecallHash: snapshotMarkers.externalRecallHash, }; // NOTE: maxMemoryId is deliberately EXCLUDED from this stale-check. // Additive memory writes (write/promote) do not invalidate the rendered @@ -1702,6 +1837,7 @@ export function materializeM0(options: M0M1RenderOptions): MaterializeM0Result { upgradeState: snapshotMarkers.upgradeState, systemHash: snapshotMarkers.systemHash, modelKey: snapshotMarkers.modelKey, + externalRecallHash: snapshotMarkers.externalRecallHash, }); // v2 path persists the rendered-memory identity itself. `memory_block_ids` @@ -1838,6 +1974,17 @@ function renderMemoryUpdatesBlock(args: { interface RenderM1Result { text: string; memoryUpdateCount: number; + /** The delta block (late-arrival snapshot) when present, + * "" otherwise. Excluded from the injectM0M1 pressure-refold token math + * so a large recall can NEVER cause an m[0] refold (spec: recall is not + * a bust trigger). */ + externalDeltaText: string; + /** True when this result was freshly rendered from current DB state. False + * when the bytes are a persisted-row replay (sibling-adoption fallback + * or defer-pass replay). The pressure-refold backstop must only fire on + * recomputed bytes — a replayed sibling m[1] that happens to contain a + * large external delta is already settled and must not cause a fold. */ + recomputed: boolean; } function renderM1WithMetadata( @@ -1940,16 +2087,42 @@ function renderM1WithMetadata( if (profileBlock) blocks.push(profileBlock); } + // External recall delta: snapshot settled AFTER the last m[0] fold. We + // intentionally compare the live snapshot hash to markers.externalRecallHash + // (not the DB column) so a sibling that materialized between passes and + // updated the column does NOT leak a stale delta in this soft-refresh. + // Captured separately (not in `blocks`) so injectM0M1 can subtract its + // tokens from the pressure-refold math — recall is NOT a bust trigger. + let externalDeltaText = ""; + const recallRead = readExternalRecallSnapshot(options.db, options.sessionId); + if (recallRead.state === "done" && recallRead.snapshot) { + const currentRecallHash = computeRecallSnapshotHash(recallRead.snapshot); + if (currentRecallHash !== "" && currentRecallHash !== markers.externalRecallHash) { + const delta = renderExternalMemoryDelta(recallRead.snapshot); + if (delta) { + externalDeltaText = delta; + blocks.push(delta); + } + } + } + // v2 faithful facts: session_facts is retired as a render source. Fresh // facts reach the agent as promoted memories via the new-memories block // above (maxMemoryId watermark), not via a delta here. if (blocks.length === 0) { - return { text: M1_EMPTY_PLACEHOLDER, memoryUpdateCount: memoryUpdates.count }; + return { + text: M1_EMPTY_PLACEHOLDER, + memoryUpdateCount: memoryUpdates.count, + externalDeltaText: "", + recomputed: true, + }; } return { text: `\n${blocks.join("\n")}\n`, memoryUpdateCount: memoryUpdates.count, + externalDeltaText, + recomputed: true, }; } @@ -1982,6 +2155,7 @@ interface CachedM0M1Row { cached_m0_upgrade_state: string | null; cached_m0_system_hash: string | null; cached_m0_model_key: string | null; + cached_m0_external_recall_hash: string | null; memory_block_ids: string | null; } @@ -2027,6 +2201,7 @@ function readCachedM0M1Row(db: Database, sessionId: string): CachedM0M1Row | nul cached_m0_upgrade_state, cached_m0_system_hash, cached_m0_model_key, + cached_m0_external_recall_hash, memory_block_ids FROM session_meta WHERE session_id = ?`, @@ -2057,6 +2232,7 @@ function markersFromCachedRow(row: CachedM0M1Row): M0SnapshotMarkers | null { upgradeState: row.cached_m0_upgrade_state, systemHash: row.cached_m0_system_hash ?? "", modelKey: row.cached_m0_model_key ?? "", + externalRecallHash: row.cached_m0_external_recall_hash ?? "", }; } @@ -2078,7 +2254,8 @@ function cachedRowMatchesState(row: CachedM0M1Row, state: M0M1State): boolean { row.cached_m0_session_facts_version === state.cachedM0SessionFactsVersion && (row.cached_m0_upgrade_state ?? null) === (state.cachedM0UpgradeState ?? null) && (row.cached_m0_system_hash ?? "") === (state.cachedM0SystemHash ?? "") && - (row.cached_m0_model_key ?? "") === (state.cachedM0ModelKey ?? "") + (row.cached_m0_model_key ?? "") === (state.cachedM0ModelKey ?? "") && + (row.cached_m0_external_recall_hash ?? "") === (state.cachedM0ExternalRecallHash ?? "") ); } @@ -2102,6 +2279,7 @@ function applyCachedRowToState(state: M0M1State, row: CachedM0M1Row): void { state.cachedM0UpgradeState = markers.upgradeState; state.cachedM0SystemHash = markers.systemHash; state.cachedM0ModelKey = markers.modelKey; + state.cachedM0ExternalRecallHash = markers.externalRecallHash; state.snapshotMarkers = markers; } @@ -2130,7 +2308,18 @@ function softRefreshCachedM1(options: M0M1RenderOptions): RenderM1Result { const sibling = readCachedM0M1Row(options.db, options.sessionId); if (!sibling) throw new RenderM1InvalidMarkersError(options.sessionId); applyCachedRowToState(options.state, sibling); - return { text: replayCachedM1(options.state), memoryUpdateCount: 0 }; + // Replayed sibling bytes — must NOT drive the pressure-refold math + // (the replayed m[1] may already contain a large external delta from + // a prior pass; replaying it should not cause a fold). externalDeltaText + // is "" because we did not re-render and cannot identify the delta + // boundary in the replayed bytes; the recomputed=false flag is what + // actually keeps the backstop off. + return { + text: replayCachedM1(options.state), + memoryUpdateCount: 0, + externalDeltaText: "", + recomputed: false, + }; } const markers = markersFromCachedRow(row); @@ -2265,6 +2454,11 @@ function renderFreshM0NonPersisted(options: M0M1RenderOptions): { ) : []; const userMemories = safeGetActiveUserMemories(options.db); + // External recall read mirrors materializeM0: render and marker must derive + // from the same read. + const recallRead = readExternalRecallSnapshot(options.db, options.sessionId); + const externalRecall = recallRead.state === "done" ? recallRead.snapshot : null; + snapshotMarkers.externalRecallHash = computeRecallSnapshotHash(externalRecall); const memoryBudget = options.memoryInjectionBudgetTokens ?? DEFAULT_MEMORY_BUDGET_TOKENS; const memoryRenderOptions: MemoryRenderOptions = { sourceNameByMemoryId: sourceNamesForMemories({ @@ -2294,6 +2488,7 @@ function renderFreshM0NonPersisted(options: M0M1RenderOptions): { historyBudgetTokens: budget, userProfileBudgetTokens: options.userProfileBudgetTokens, decayPressureMultiplier, + externalRecall, }); let attempts = 0; while (budget > 0 && historySliceTokens(m0Text) > budget * 1.05 && attempts < 3) { @@ -2308,6 +2503,7 @@ function renderFreshM0NonPersisted(options: M0M1RenderOptions): { historyBudgetTokens: budget, userProfileBudgetTokens: options.userProfileBudgetTokens, decayPressureMultiplier, + externalRecall, }); attempts += 1; } @@ -2359,7 +2555,18 @@ export function injectM0M1(options: M0M1RenderOptions): InjectM0M1Result { materialized.snapshotMarkers, materialized.m1Bytes, ); - m1Render = { text: materialized.m1Text, memoryUpdateCount: 0 }; + // The fresh-materialize path's m1Render will not drive a pressure + // refold (rematerialized=true skips the refold block), so the + // externalDeltaText field is logically inert here. Thread it for + // shape consistency; the materialize-with-snapshot case would + // produce externalDeltaText="" anyway (markers.externalRecallHash + // was just stamped to the current hash). + m1Render = { + text: materialized.m1Text, + memoryUpdateCount: 0, + externalDeltaText: "", + recomputed: true, + }; rematerialized = true; } catch (error) { if (!(error instanceof MaterializeContentionError)) throw error; @@ -2412,10 +2619,16 @@ export function injectM0M1(options: M0M1RenderOptions): InjectM0M1Result { let m1Text: string; let memoryUpdateCount = 0; let m1Recomputed = m1Render !== null; + // Tracked across whichever RenderM1Result produced the live m[1] text, so + // the pressure-refold token math can subtract the late-recall delta. + // Replay paths (m1Recomputed=false) skip the refold entirely, so "" is + // safe for those branches. + let externalDeltaText = ""; if (m1Render) { m1Text = m1Render.text; memoryUpdateCount = m1Render.memoryUpdateCount; + externalDeltaText = m1Render.externalDeltaText; } else if (contentionExhausted && freshFallbackRenderedMemoryIds) { const freshM1 = renderM1WithMetadata( { ...options, preRenderedKeyFilesBlock: preRenderKeyFilesBlock(options) }, @@ -2424,6 +2637,7 @@ export function injectM0M1(options: M0M1RenderOptions): InjectM0M1Result { ); m1Text = freshM1.text; memoryUpdateCount = freshM1.memoryUpdateCount; + externalDeltaText = freshM1.externalDeltaText; m1Recomputed = true; } else if (contentionExhausted) { m1Text = replayCachedM1(options.state); @@ -2431,7 +2645,14 @@ export function injectM0M1(options: M0M1RenderOptions): InjectM0M1Result { const refreshed = softRefreshCachedM1(options); m1Text = refreshed.text; memoryUpdateCount = refreshed.memoryUpdateCount; - m1Recomputed = true; + externalDeltaText = refreshed.externalDeltaText; + // Sibling-adoption fallback returns recomputed=false (replayed bytes + // must not drive the pressure backstop). The normal soft-refresh path + // returns recomputed=true (genuinely re-rendered). Replaying defer + // passes' persisted bytes is the same category as the sibling fallback: + // the pressure math is a no-op when m1Recomputed is false, and + // "replayed bytes must not live-read/refold" still holds. + m1Recomputed = refreshed.recomputed; m0Text = decodeM0Bytes(options.state.cachedM0Bytes) ?? M0_EMPTY_BODY; } else { m1Text = replayCachedM1(options.state); @@ -2464,10 +2685,26 @@ export function injectM0M1(options: M0M1RenderOptions): InjectM0M1Result { // ~15% of m[0] tokens". XML-heavy / non-Latin content makes char length // diverge sharply from token count, so the ratio must compare tokens on both // sides. Computed once; this branch is rare (cache-busting + m1Recomputed). + // + // External recall content must NEVER CAUSE a fold (spec: not a bust trigger); + // it rides along when a fold fires for other reasons. Two layers of + // subtraction from m1Tokens: the delta itself (late recall) AND a small + // wrapper overhead (every m[1] carries the wrapper, empty or not — not a + // drift signal). The wrapper tokens are also subtracted from the absolute + // cap budget for symmetry, so a tiny m[0] baseline (where the wrapper + // alone would exceed the cap) does not falsely fire a refold when the + // only m[1] content is the recall delta. const m1HasContent = m1Text !== M1_EMPTY_PLACEHOLDER; const m1Tokens = m1HasContent ? estimateTokens(m1Text) : 0; + const M1_PRESSURE_WRAPPER_TOKENS = 20; + const externalDeltaTokens = externalDeltaText ? estimateTokens(externalDeltaText) : 0; + const m1PressureTokens = Math.max( + 0, + m1Tokens - externalDeltaTokens - M1_PRESSURE_WRAPPER_TOKENS, + ); + const m1AbsoluteContentBudget = Math.max(0, m1AbsoluteBudget - M1_PRESSURE_WRAPPER_TOKENS); const m0Tokens = estimateTokens(m0Text); - const m1OverAbsoluteCap = m1HasContent && m1Tokens > m1AbsoluteBudget; + const m1OverAbsoluteCap = m1HasContent && m1PressureTokens > m1AbsoluteContentBudget; if ( !rematerialized && !contentionExhausted && @@ -2477,7 +2714,7 @@ export function injectM0M1(options: M0M1RenderOptions): InjectM0M1Result { m1OverAbsoluteCap || (m1HasContent && m0Tokens >= M0_DRIFT_RATIO_FLOOR_TOKENS && - m1Tokens > m0Tokens * M1_DRIFT_RATIO)) + m1PressureTokens > m0Tokens * M1_DRIFT_RATIO)) ) { try { const refolded = materializeWithRetry(options); diff --git a/packages/plugin/src/hooks/magic-context/inject-external-recall.test.ts b/packages/plugin/src/hooks/magic-context/inject-external-recall.test.ts new file mode 100644 index 000000000..08e978452 --- /dev/null +++ b/packages/plugin/src/hooks/magic-context/inject-external-recall.test.ts @@ -0,0 +1,471 @@ +/// + +// Render integration for the external-recall snapshot (Tasks 6). +// +// Invariants this test guards: +// 1. A "done" snapshot already persisted before materializeM0 is baked INTO +// m[0] as a sibling block (project + global slices). +// 2. Profile slice lines merge INTO , NOT into the external block. +// 3. No snapshot → no block, marker externalRecallHash is "". +// 4. Late arrival (snapshot lands AFTER m[0] is materialized) routes to the +// m[1] delta on the next cache-busting pass; m[0] does +// NOT rematerialize (mustMaterialize stays false). +// 5. After a HARD fold (next materialize), the delta disappears and the +// snapshot is fully baked into m[0]. +// 6. Recalled items get NO fake markup — plain "- content" lines. + +import { afterEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + appendCompartments, + type CompartmentInput, +} from "../../features/magic-context/compartment-storage"; +import { insertMemory } from "../../features/magic-context/memory/storage-memory"; +import { runMigrations } from "../../features/magic-context/migrations"; +import { getOrCreateSessionMeta } from "../../features/magic-context/storage"; +import { initializeDatabase } from "../../features/magic-context/storage-db"; +import { Database } from "../../shared/sqlite"; +import { + clearInjectionCache, + injectM0M1, + type M0HardSignals, + type M0M1RenderOptions, + type M0M1State, + materializeM0, + mustMaterialize, + renderM1, +} from "./inject-compartments"; + +const SESSION_ID = "ses_ext_recall"; +const PROJECT_PATH = "/tmp/test-ext-recall-project"; +const PROJECT_DIRECTORY = "/tmp/test-ext-recall-project-dir"; + +let db: Database; +const tempDirs: string[] = []; + +function makeDb(): Database { + const d = new Database(":memory:"); + initializeDatabase(d); + // v31 (external recall columns) is required so seedRecallSnapshot can + // UPDATE the recall columns. Without migrations, readExternalRecallSnapshot + // catches the missing-column error and behaves as "never started". + runMigrations(d); + getOrCreateSessionMeta(d, SESSION_ID); + return d; +} + +function makeProjectDir(): string { + const dir = mkdtempSync(join(tmpdir(), "mc-ext-recall-test-")); + tempDirs.push(dir); + return dir; +} + +function compartment(seq: number, title: string, body: string): CompartmentInput { + return { + sequence: seq, + startMessage: seq, + endMessage: seq, + startMessageId: `m${seq}`, + endMessageId: `m${seq}`, + title, + content: body, + p1: body, + }; +} + +const BASE_HARD: M0HardSignals = { + systemHash: "sys-v1", + toolSetHash: "tools-v1", + modelKey: "anthropic/opus", + cacheExpired: false, + lastResponseTime: 0, +}; + +function seedRecallSnapshot( + dbInstance: Database, + sessionId: string, + snapshot: { + project?: Array<{ content: string; category?: string }>; + profile?: Array<{ content: string; category?: string }>; + global?: Array<{ content: string; category?: string }>; + }, +): void { + dbInstance + .prepare( + "UPDATE session_meta SET external_recall_state = ?, external_recall_json = ?, external_recall_at = ? WHERE session_id = ?", + ) + .run("done", JSON.stringify(snapshot), Date.now(), sessionId); +} + +function buildOptions(): M0M1RenderOptions { + return { + db, + sessionId: SESSION_ID, + state: getOrCreateSessionMeta(db, SESSION_ID) as unknown as M0M1State, + projectPath: PROJECT_PATH, + projectDirectory: PROJECT_DIRECTORY, + historyBudgetTokens: 98_000, + isCacheBustingPass: true, + hardSignals: BASE_HARD, + }; +} + +afterEach(() => { + if (db) db.close(); + clearInjectionCache(SESSION_ID); + for (const dir of tempDirs) rmSync(dir, { recursive: true, force: true }); + tempDirs.length = 0; +}); + +describe("external recall in m[0]/m[1]", () => { + test("done snapshot before first materialize bakes into m[0]", () => { + db = makeDb(); + const projectDirectory = makeProjectDir(); + seedRecallSnapshot(db, SESSION_ID, { + project: [{ content: "ext proj fact" }], + profile: [{ content: "ext user pref" }], + global: [{ content: "ext homelab fact" }], + }); + const result = materializeM0({ + ...buildOptions(), + projectDirectory, + }); + // project + global slices in + expect(result.m0Text).toContain(''); + expect(result.m0Text).toContain("- ext proj fact"); + expect(result.m0Text).toContain("- ext homelab fact"); + // profile merges into , NOT into the external block + const userProfileStart = result.m0Text.indexOf(""); + const userProfileEnd = result.m0Text.indexOf(""); + expect(userProfileStart).toBeGreaterThanOrEqual(0); + expect(userProfileEnd).toBeGreaterThan(userProfileStart); + const userProfileSection = result.m0Text.slice(userProfileStart, userProfileEnd); + expect(userProfileSection).toContain("- ext user pref"); + const externalBlockStart = result.m0Text.indexOf(""); + const externalBlockSection = result.m0Text.slice(externalBlockStart, externalBlockEnd); + expect(externalBlockSection).not.toContain("ext user pref"); + // no fake markup for recalled items + expect(result.m0Text).not.toMatch(/]*id="[^"]*"[^>]*>ext (proj|homelab|user) /); + // marker is set + expect(result.snapshotMarkers.externalRecallHash).not.toBe(""); + }); + + test("no snapshot → no external block, marker empty", () => { + db = makeDb(); + const projectDirectory = makeProjectDir(); + const result = materializeM0({ + ...buildOptions(), + projectDirectory, + }); + expect(result.m0Text).not.toContain(" { + db = makeDb(); + const projectDirectory = makeProjectDir(); + // First materialize with no snapshot. + const first = materializeM0({ + ...buildOptions(), + projectDirectory, + }); + expect(first.m0Text).not.toContain(" { + // Regression for the BLOCKING finding: the m[1] drift triggers (ratio + // and absolute cap) were computed from the full m1Text — a large + // late-arrival recall delta could therefore CAUSE a pressure refold + // that would not otherwise fire. Spec invariant: late recall must + // NEVER cause a fold. We exclude the external delta from the pressure + // math, so the backstop only fires for GENUINE non-recall drift. + db = makeDb(); + const projectDirectory = makeProjectDir(); + // Tiny baseline m[0] + small history budget so the absolute cap is + // easy to exceed if measured against the full m1Text. Mirror the + // m0m1-taxonomy.test.ts "pressure backstop" fixture for headroom. + appendCompartments(db, SESSION_ID, [compartment(0, "A", "Ax")]); + const first = injectM0M1({ + ...buildOptions(), + projectDirectory, + historyBudgetTokens: 60, + }); + expect(first.decision.reason).toBe("first_render"); + const baselineM0Bytes = first.m0Bytes; + + // Seed a LARGE late recall snapshot — 30 project items of ~200 chars. + // 30 * 200 = ~6000 chars / ~1500 tokens, which dwarfs the 60-token + // history budget × 0.2 absolute cap (12 tokens). With the bug, this + // would trip the absolute cap and force a refold. + const big = "x".repeat(200); + const bigRecall = { + project: Array.from({ length: 30 }, (_, i) => ({ + content: `${big} ${i}`, + })), + profile: [], + global: [], + }; + seedRecallSnapshot(db, SESSION_ID, bigRecall); + + // Cache-busting pass with no HARD signal — the only trigger that + // COULD refold is the pressure backstop. mustMaterialize returns + // false (recall is not a HARD trigger). + const state = getOrCreateSessionMeta(db, SESSION_ID) as unknown as M0M1State; + const result = injectM0M1({ + ...buildOptions(), + state, + projectDirectory, + historyBudgetTokens: 60, + isCacheBustingPass: true, + hardSignals: BASE_HARD, + }); + + // No refold: m[0] bytes are byte-identical to the pre-recall baseline. + expect(result.m0RematerializedThisPass).toBe(false); + const row = db + .prepare("SELECT cached_m0_bytes FROM session_meta WHERE session_id = ?") + .get(SESSION_ID) as { cached_m0_bytes: Buffer | Uint8Array | null } | null; + const persisted = row?.cached_m0_bytes ? Buffer.from(row.cached_m0_bytes) : null; + expect(persisted?.equals(baselineM0Bytes)).toBe(true); + // m[1] still carries the late delta (the model sees it this pass). + expect(result.m1Text).toContain(" { + // Proves the BLOCKING fix did not disable the backstop for genuine + // drift. Same shape as the m0m1-taxonomy.test.ts "pressure backstop" + // test, exercised here to keep the two adjacent so a future refactor + // of the recall exclusion doesn't accidentally widen it. + db = makeDb(); + const projectDirectory = makeProjectDir(); + appendCompartments(db, SESSION_ID, [compartment(0, "A", "Ax")]); + const first = injectM0M1({ + ...buildOptions(), + projectDirectory, + historyBudgetTokens: 60, + }); + expect(first.decision.reason).toBe("first_render"); + + // Append several compartments → m[1] grows past 20% of the 60-token + // history budget via genuine non-external drift (new compartments). + appendCompartments(db, SESSION_ID, [ + compartment(1, "B", "Bravo delta with enough words to consume tokens"), + compartment(2, "C", "Charlie delta with more words again to consume more tokens"), + compartment(3, "D", "Delta delta even more words here for tokens and tokens"), + ]); + const state = getOrCreateSessionMeta(db, SESSION_ID) as unknown as M0M1State; + const folded = injectM0M1({ + ...buildOptions(), + state, + projectDirectory, + historyBudgetTokens: 60, + isCacheBustingPass: true, + hardSignals: BASE_HARD, + }); + // The absolute-cap backstop folded m[1] into m[0] this pass. + expect(folded.m0RematerializedThisPass).toBe(true); + expect(folded.m1Text).toBe( + "(no new content since last materialization)", + ); + }); + + test("BLOCKING-residual: sibling-replay path does NOT pressure-refold on replayed m[1] bytes (even if those bytes contain a large external delta)", () => { + // Residual of the BLOCKING finding: the softRefresh sibling-adoption + // path (row-mismatch → adopt sibling's cached m[1]) used to return + // m1Recomputed=true unconditionally, so the pressure backstop would + // run on REPLAYED bytes. If the sibling's m[1] happened to contain a + // large late external delta, the backstop could fold it into m[0] — + // i.e. late recall could still trigger a refold via the sibling path. + // + // Setup: + // 1. Materialize m[0] with no recall → DB row is small. + // 2. Manually overwrite DB cached_m1_bytes with a sibling's m[1] + // that contains a LARGE delta (a frozen + // "sibling already settled recall" scenario). + // 3. Mutate the in-memory state so cachedRowMatchesState returns + // false on a marker that is NOT a mustMaterialize trigger + // (cachedM0MaxCompartmentSeq — new compartments are an m[1] delta, + // not a HARD bust signal). This forces softRefreshCachedM1 into + // the sibling-adoption branch without also firing mustMaterialize. + // 4. Call injectM0M1 with isCacheBustingPass=true and a tiny + // historyBudgetTokens (so the absolute cap is easy to exceed if + // the backstop runs on replayed bytes). + db = makeDb(); + const projectDirectory = makeProjectDir(); + appendCompartments(db, SESSION_ID, [compartment(0, "A", "Ax")]); + const baseline = injectM0M1({ + ...buildOptions(), + projectDirectory, + historyBudgetTokens: 60, + }); + expect(baseline.decision.reason).toBe("first_render"); + const baselineM0Bytes = baseline.m0Bytes; + + // Overwrite the DB's cached m[1] with a sibling's m[1] containing a + // large external delta block. The backstop, if it ran on these + // replayed bytes, would trip the absolute cap and force a refold. + const bigRecallBlock = + `\n` + + Array.from({ length: 200 }, (_, i) => `- ${"y".repeat(200)} ${i}`).join("\n") + + `\n`; + const siblingM1 = `\n${bigRecallBlock}\n`; + db.prepare("UPDATE session_meta SET cached_m1_bytes = ? WHERE session_id = ?").run( + Buffer.from(siblingM1, "utf8"), + SESSION_ID, + ); + + // Force a row mismatch on a marker that does not affect + // materialization correctness OR fire mustMaterialize. We pick + // cachedM0MaxCompartmentSeq — it is in cachedRowMatchesState (so + // softRefresh takes the sibling-adoption path) but is deliberately + // NOT a mustMaterialize trigger (new compartments are an m[1] delta, + // not a fold signal — see the comment in mustMaterialize). This + // isolates the test to the pressure-backstop behavior we want to + // guard, without any HARD trigger firing. + const state = getOrCreateSessionMeta(db, SESSION_ID) as unknown as M0M1State; + state.cachedM0MaxCompartmentSeq = 999_999; + + // Cache-busting pass with no HARD signal — only the pressure backstop + // could refold, and only IF the sibling-replay path still sets + // m1Recomputed=true. The fix marks sibling replay recomputed=false. + const result = injectM0M1({ + ...buildOptions(), + state, + projectDirectory, + historyBudgetTokens: 60, + isCacheBustingPass: true, + hardSignals: BASE_HARD, + }); + + // No refold: the backstop must skip replayed sibling bytes. + expect(result.m0RematerializedThisPass).toBe(false); + const row = db + .prepare("SELECT cached_m0_bytes FROM session_meta WHERE session_id = ?") + .get(SESSION_ID) as { cached_m0_bytes: Buffer | Uint8Array | null } | null; + const persisted = row?.cached_m0_bytes ? Buffer.from(row.cached_m0_bytes) : null; + expect(persisted?.equals(baselineM0Bytes)).toBe(true); + // The model still sees the recall this pass (replayed from sibling). + expect(result.m1Text).toContain(" { + db = makeDb(); + const projectDirectory = makeProjectDir(); + seedRecallSnapshot(db, SESSION_ID, { + project: [{ content: "late fact" }], + profile: [], + global: [], + }); + const second = materializeM0({ + ...buildOptions(), + projectDirectory, + }); + // Now baked into m[0] AND no longer in the m[1] delta (the + // markers.externalRecallHash matches the persisted hash). + expect(second.m0Text).toContain("- late fact"); + const m1 = renderM1(buildOptions(), second.snapshotMarkers, second.renderedMemoryIds); + expect(m1).not.toContain("", () => { + db = makeDb(); + const projectDirectory = makeProjectDir(); + // Seed a memory so exists in the render. + insertMemory(db, { + projectPath: PROJECT_PATH, + category: "ARCHITECTURE", + content: "local fact", + importance: 50, + sourceType: "historian", + }); + seedRecallSnapshot(db, SESSION_ID, { + project: [{ content: "ext proj fact" }], + profile: [], + global: [{ content: "ext homelab fact" }], + }); + const result = materializeM0({ + ...buildOptions(), + projectDirectory, + }); + const projectMemoryIdx = result.m0Text.indexOf(""); + const externalIdx = result.m0Text.indexOf(" { + db = makeDb(); + const projectDirectory = makeProjectDir(); + seedRecallSnapshot(db, SESSION_ID, { + project: [{ content: "Doc line 1\nDoc line 2" }, { content: "single fact" }], + profile: [], + global: [], + }); + const result = materializeM0({ + ...buildOptions(), + projectDirectory, + }); + // Preamble appears as the first content line of the block. + expect(result.m0Text).toContain("Background knowledge from past sessions"); + // Multi-line item renders VERBATIM, with blank-line separators around it. + expect(result.m0Text).toContain("Doc line 1\nDoc line 2"); + // Single-line items keep the "- " list prefix. + expect(result.m0Text).toContain("- single fact"); + }); + + test("external block is omitted when all recalled items are empty", () => { + db = makeDb(); + const projectDirectory = makeProjectDir(); + seedRecallSnapshot(db, SESSION_ID, { + project: [], + profile: [], + global: [], + }); + const result = materializeM0({ + ...buildOptions(), + projectDirectory, + }); + expect(result.m0Text).not.toContain(" void; /** @@ -307,6 +311,7 @@ async function runCompartmentPhaseImpl(args: RunCompartmentPhaseArgs): Promise<{ historianTwoPass: args.historianTwoPass, memoryEnabled: args.memoryEnabled, autoPromote: args.autoPromote, + embeddingEnabled: args.embeddingEnabled, onCompartmentStatePublished: args.onCompartmentStatePublished, preserveInjectionCacheUntilConsumed: true, }); @@ -351,6 +356,7 @@ async function runCompartmentPhaseImpl(args: RunCompartmentPhaseArgs): Promise<{ historianTwoPass: args.historianTwoPass, memoryEnabled: args.memoryEnabled, autoPromote: args.autoPromote, + embeddingEnabled: args.embeddingEnabled, onCompartmentStatePublished: args.onCompartmentStatePublished, preserveInjectionCacheUntilConsumed: true, }); diff --git a/packages/plugin/src/hooks/magic-context/transform.ts b/packages/plugin/src/hooks/magic-context/transform.ts index 639b5396e..9f3485844 100644 --- a/packages/plugin/src/hooks/magic-context/transform.ts +++ b/packages/plugin/src/hooks/magic-context/transform.ts @@ -1,4 +1,10 @@ import * as crypto from "node:crypto"; +import { basename } from "node:path"; +import { getExternalRecallConfig } from "../../features/magic-context/memory/external-memory"; +import { + maybeAwaitExternalRecall, + startSessionRecall, +} from "../../features/magic-context/memory/external-recall"; import { resolveProjectIdentity } from "../../features/magic-context/memory/project-identity"; import { scheduleReconciliation } from "../../features/magic-context/message-index-async"; import type { Scheduler } from "../../features/magic-context/scheduler"; @@ -43,6 +49,7 @@ import type { PluginContext } from "../../plugin/types"; import { BoundedSessionMap } from "../../shared/bounded-session-map"; import { getErrorMessage } from "../../shared/error-message"; import { sessionLog } from "../../shared/logger"; +import { removeSystemReminders } from "../../shared/system-directive"; import { applyMidTurnDeferral, detectMidTurnBypassReason } from "./boundary-execution"; import { canConsumeDeferredOnThisPass } from "./cache-busting-signals"; import { replayCavemanCompression } from "./caveman-cleanup"; @@ -73,7 +80,7 @@ import { } from "./protected-tail-boundary"; import { readRawSessionMessages } from "./read-session-chunk"; import { findLastAssistantModelFromOpenCodeDb, isMidTurn } from "./read-session-db"; -import { estimateTokens } from "./read-session-formatting"; +import { estimateTokens, extractTexts, hasMeaningfulUserText } from "./read-session-formatting"; import { extractInMemoryMessageViews } from "./read-session-raw"; import { sendIgnoredMessage } from "./send-session-notification"; import { modelAcceptsEmptyContent } from "./sentinel"; @@ -207,6 +214,25 @@ function findLastAssistantModel( return null; } +/** + * Extract the text of the session's FIRST meaningful user message — the raw + * prompt that opened the conversation. Used to enrich the global external + * recall query (recall.global_from_prompt). Skips synthetic/ignored parts and + * system directives via hasMeaningfulUserText; strips system-reminder blocks + * from the extracted text. Returns undefined when no meaningful user message + * exists yet (e.g. command-only turns). + */ +function extractFirstUserPromptText(messages: MessageLike[]): string | undefined { + for (const message of messages) { + const info = message.info as { role?: string }; + if (info.role !== "user") continue; + if (!hasMeaningfulUserText(message.parts)) continue; + const text = removeSystemReminders(extractTexts(message.parts).join(" ")).trim(); + return text.length > 0 ? text : undefined; + } + return undefined; +} + export interface TransformDeps { tagger: Tagger; scheduler: Scheduler; @@ -261,6 +287,10 @@ export interface TransformDeps { * still write memories explicitly via `ctx_memory write`. Issue #44. */ autoPromote: boolean; }; + /** Embedding provider on/off (config `embedding.provider !== "off"`). + * Gates compartment P1 embedding + project registration at the runner + * call sites — independent of the memory store flags. */ + embeddingEnabled?: boolean; ensureProjectRegistered?: (directory: string, db: ContextDatabase) => Promise; /** * Returns the historian chunk budget. Called at each historian spawn site @@ -898,6 +928,7 @@ export function createTransform(deps: TransformDeps) { // who disable the feature actually see no memories created. memoryEnabled: deps.memoryConfig?.enabled, autoPromote: deps.memoryConfig?.autoPromote, + embeddingEnabled: deps.embeddingEnabled, ensureProjectRegistered: deps.ensureProjectRegistered, // Historian publication invalidates the injection cache AND // changes compartments/facts that render into message[0]. We @@ -1010,6 +1041,42 @@ export function createTransform(deps: TransformDeps) { const projectIdentity = deps.memoryConfig?.enabled ? resolveProjectIdentity(compartmentDirectory || process.cwd()) : undefined; + + // External memory v2: fire the once-per-session recall. Independent of + // memory.enabled (external knowledge is useful with the local store + // off) — identity computed from the directory directly. Internally + // gated on provider/recall.enabled/already-settled; fire-and-forget. + if (fullFeatureMode && compartmentDirectory) { + // Kick project registration so the dedup embedding provider is + // likely registered by recall-settle time; hash-only fallback + // covers the race (spec-accepted). Best-effort: a synchronously + // throwing injected dep must not abort the transform. + if (deps.ensureProjectRegistered) { + try { + void Promise.resolve( + deps.ensureProjectRegistered(compartmentDirectory, db), + ).catch(() => {}); + } catch { + // ignore — registration is best-effort + } + } + // First-prompt enrichment of the global recall slice + // (recall.global_from_prompt). Extracted HERE, pre-injection: the + // m[0]/m[1] prepends are added later this pass and never persisted + // by OpenCode, so the first user message in `messages` is the real + // first prompt — immutable for the session, hence deterministic + // across passes and crash-recovery re-fires. + const firstUserPrompt = getExternalRecallConfig()?.global_from_prompt + ? extractFirstUserPromptText(messages) + : undefined; + startSessionRecall({ + db, + sessionId, + projectIdentity: resolveProjectIdentity(compartmentDirectory), + projectName: basename(compartmentDirectory), + ...(firstUserPrompt ? { firstUserPrompt } : {}), + }); + } // Session-scoped project identity for note-nudge and auto-search, which // must target the SESSION's project — not the launch cwd. `deps.projectPath` // is resolved once at hook init from the launch directory; on @@ -1493,6 +1560,7 @@ export function createTransform(deps: TransformDeps) { // memory.auto_promote. memoryEnabled: deps.memoryConfig?.enabled, autoPromote: deps.memoryConfig?.autoPromote, + embeddingEnabled: deps.embeddingEnabled, ensureProjectRegistered: deps.ensureProjectRegistered, // See startRecoveryRun above for the full rationale — // historian/recomp publication signals history rebuild + @@ -1552,6 +1620,18 @@ export function createTransform(deps: TransformDeps) { : rebuiltHistoryFromInitialPrepare || compartmentPhase.rebuiltHistoryThisPass; const tPostProcess = performance.now(); + // External memory v2 hybrid A-path: when the FIRST m[0] render is + // imminent (no cached baseline — the provider cache is already cold), + // give the in-flight recall up to recall.timeout_ms to land so the + // first materialization bakes it in. Never fires once a baseline + // exists; late recalls ride the m[1] delta instead. + if (fullFeatureMode) { + await maybeAwaitExternalRecall({ + db, + sessionId, + hasCachedM0: sessionMeta.cachedM0Bytes !== null, + }); + } const postTransformResult = await runPostTransformPhase({ sessionId, db, diff --git a/packages/plugin/src/index.ts b/packages/plugin/src/index.ts index b4727c239..15fed7a35 100644 --- a/packages/plugin/src/index.ts +++ b/packages/plugin/src/index.ts @@ -13,6 +13,7 @@ import { loadPluginConfig } from "./config"; import { isDreamerRunnable } from "./config/agent-disable"; import { getMagicContextBuiltinCommands } from "./features/builtin-commands/commands"; import { DREAMER_SYSTEM_PROMPT } from "./features/magic-context/dreamer/task-prompts"; +import { initializeExternalMemory } from "./features/magic-context/memory/external-memory"; import { resolveProjectIdentity } from "./features/magic-context/memory/project-identity"; import { SIDEKICK_SYSTEM_PROMPT } from "./features/magic-context/sidekick/agent"; import { @@ -174,6 +175,7 @@ const plugin: Plugin = async (ctx) => { // Start independent dream schedule timer at plugin level (not inside hooks) // so overnight dreaming works even when the user isn't chatting. if (pluginConfig.enabled) { + initializeExternalMemory(pluginConfig.memory?.external); const dreamerRunnable = isDreamerRunnable(pluginConfig); const timerRegistration = { directory: ctx.directory, @@ -207,7 +209,11 @@ const plugin: Plugin = async (ctx) => { : undefined, ensureRegistered: ensureProjectRegisteredFromOpenCodeDirectory, }; - stopDreamTimerRegistration = await startDreamScheduleTimer(timerRegistration); + try { + stopDreamTimerRegistration = await startDreamScheduleTimer(timerRegistration); + } catch (err) { + log(`[magic-context] dream schedule timer failed to start (non-fatal): ${err}`); + } // Start RPC server for TUI↔server communication (replaces SQLite plugin_messages bus). // `storageDir` is hoisted above so the auto-update checker can also use it. diff --git a/packages/plugin/src/plugin/dream-timer.ts b/packages/plugin/src/plugin/dream-timer.ts index 96707ba72..274535f9e 100644 --- a/packages/plugin/src/plugin/dream-timer.ts +++ b/packages/plugin/src/plugin/dream-timer.ts @@ -59,7 +59,15 @@ let activeTimer: ReturnType | null = null; * deep in embedding registration and throws a confusing TypeError. */ function openTimerDatabaseOrNull(context: string): Database | null { - const db = openDatabase(); + let db: Database | null; + try { + db = openDatabase(); + } catch (error) { + log( + `[dreamer] storage open threw; skipping ${context}: ${error instanceof Error ? error.message : String(error)}`, + ); + return null; + } if (!db) { log( `[dreamer] storage unavailable; skipping ${context} (the cache schema is newer than this binary supports — restart/upgrade OpenCode/Pi/Magic Context to recover)`, diff --git a/packages/plugin/src/plugin/rpc-handlers.test.ts b/packages/plugin/src/plugin/rpc-handlers.test.ts index 09825cece..4f9dd4dce 100644 --- a/packages/plugin/src/plugin/rpc-handlers.test.ts +++ b/packages/plugin/src/plugin/rpc-handlers.test.ts @@ -3,6 +3,11 @@ import { afterEach, describe, expect, test } from "bun:test"; import { replaceAllCompartmentState } from "../features/magic-context/compartment-storage"; import { insertMemory } from "../features/magic-context/memory"; +import { + _resetExternalMemoryForTests, + _setTestExternalBackendFactory, + initializeExternalMemory, +} from "../features/magic-context/memory/external-memory"; import { resolveProjectIdentity } from "../features/magic-context/memory/project-identity"; import { runMigrations } from "../features/magic-context/migrations"; import { initializeDatabase } from "../features/magic-context/storage-db"; @@ -24,6 +29,7 @@ function createTestDb(): Database { afterEach(() => { resetSidebarSnapshotCache(); clearModelsDevCache(); + _resetExternalMemoryForTests(); }); describe("buildSidebarSnapshot — memory tokens fallback (bug #1)", () => { @@ -184,7 +190,7 @@ describe("buildSidebarSnapshot — context limit", () => { }); describe("buildStatusDetail — history token reuse (council audit bg_51106601 #1)", () => { - test("sets historyBlockTokens from compartmentTokens only (facts retired in v2)", () => { + test("sets historyBlockTokens from compartmentTokens only (facts retired in v2)", async () => { const db = createTestDb(); try { const sessionId = "ses-status-history-tokens"; @@ -225,7 +231,7 @@ describe("buildStatusDetail — history token reuse (council audit bg_51106601 # ], ); - const detail = buildStatusDetail(db, sessionId, directory); + const detail = await buildStatusDetail(db, sessionId, directory); // v2: facts are retired as a render source (promoted to memories), so // factTokens is 0 and the history block is compartments only — facts @@ -238,3 +244,105 @@ describe("buildStatusDetail — history token reuse (council audit bg_51106601 # } }); }); + +describe("buildStatusDetail — external memory section", () => { + test("provider off → externalMemory is null", async () => { + const db = createTestDb(); + try { + const sessionId = "ses-status-ext-off"; + db.prepare( + "INSERT INTO session_meta (session_id, last_input_tokens, last_context_percentage) VALUES (?, 0, 0)", + ).run(sessionId); + initializeExternalMemory({ provider: "off" }); + const detail = await buildStatusDetail(db, sessionId, process.cwd()); + expect(detail.externalMemory).toBeNull(); + } finally { + closeQuietly(db); + } + }); + + test("provider on, fake backend exposes fetchFailedRetainCount → detail surfaces the count", async () => { + const db = createTestDb(); + try { + const sessionId = "ses-status-ext-on"; + db.prepare( + "INSERT INTO session_meta (session_id, last_input_tokens, last_context_percentage) VALUES (?, 0, 0)", + ).run(sessionId); + _setTestExternalBackendFactory(() => ({ + backendId: "fake:ext-status", + initialize: async () => true, + retain: async () => 0, + dispose: async () => {}, + _getCircuitState: () => "closed", + fetchFailedRetainCount: async () => 3, + })); + initializeExternalMemory({ + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"], + tags: ["user:test"], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: ["user:test"], + global_from_prompt: false, + search: true, + mental_models: true, + profile_mental_models: ["user-preferences"], + }, + }); + const detail = await buildStatusDetail(db, sessionId, process.cwd()); + expect(detail.externalMemory).not.toBeNull(); + expect(detail.externalMemory?.provider).toBe("hindsight"); + expect(detail.externalMemory?.endpoint).toBe("http://10.0.0.1:8889"); + expect(detail.externalMemory?.circuitState).toBe("closed"); + expect(detail.externalMemory?.failedRetainCount).toBe(3); + } finally { + closeQuietly(db); + } + }); + + test("provider on, backend has no fetchFailedRetainCount hook → failedRetainCount null", async () => { + const db = createTestDb(); + try { + const sessionId = "ses-status-ext-noop"; + db.prepare( + "INSERT INTO session_meta (session_id, last_input_tokens, last_context_percentage) VALUES (?, 0, 0)", + ).run(sessionId); + _setTestExternalBackendFactory(() => ({ + backendId: "fake:ext-noop", + initialize: async () => true, + retain: async () => 0, + dispose: async () => {}, + })); + initializeExternalMemory({ + provider: "hindsight", + endpoint: "http://10.0.0.1:8889", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"], + tags: ["user:test"], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: ["user:test"], + global_from_prompt: false, + search: true, + mental_models: true, + profile_mental_models: ["user-preferences"], + }, + }); + const detail = await buildStatusDetail(db, sessionId, process.cwd()); + expect(detail.externalMemory).not.toBeNull(); + expect(detail.externalMemory?.failedRetainCount).toBeNull(); + } finally { + closeQuietly(db); + } + }); +}); diff --git a/packages/plugin/src/plugin/rpc-handlers.ts b/packages/plugin/src/plugin/rpc-handlers.ts index 9c8812357..c0f627e14 100644 --- a/packages/plugin/src/plugin/rpc-handlers.ts +++ b/packages/plugin/src/plugin/rpc-handlers.ts @@ -3,6 +3,11 @@ * and returns typed responses for TUI consumption. */ import type { MagicContextConfig } from "../config/schema/magic-context"; +import { + fetchExternalFailedRetains, + getExternalMemoryStatus, +} from "../features/magic-context/memory/external-memory"; +import { readExternalRecallSnapshot } from "../features/magic-context/memory/external-recall-read"; import { resolveProjectIdentity } from "../features/magic-context/memory/project-identity"; import { getEmbeddingCoverageStatus } from "../features/magic-context/project-embedding-registry"; import { @@ -495,7 +500,7 @@ export function buildSidebarSnapshot( } } -export function buildStatusDetail( +export async function buildStatusDetail( db: Database, sessionId: string, directory: string, @@ -503,7 +508,7 @@ export function buildStatusDetail( config?: Record, liveSessionState?: LiveSessionState, injectionBudgetTokens?: number, -): StatusDetail { +): Promise { const base = buildSidebarSnapshot( db, sessionId, @@ -535,8 +540,25 @@ export function buildStatusDetail( historyBlockTokens: 0, compressionBudget: null, compressionUsage: null, + externalMemory: null, }; + const externalStatus = getExternalMemoryStatus(); + if (externalStatus) { + const { state: recallState } = readExternalRecallSnapshot(db, sessionId); + // fetchExternalFailedRetains goes through HindsightMemoryBackend's + // request() — inherits the 10s fetch timeout and circuit breaker, so + // a hung backend can't drag the dialog past that cap. Returns null + // on any failure path (offline / endpoint missing / malformed + // envelope), so the field is always safe to surface. + const failedRetainCount = await fetchExternalFailedRetains(); + detail.externalMemory = { + ...externalStatus, + recallState, + failedRetainCount, + }; + } + try { const meta = db .prepare<[string], Record>( @@ -808,6 +830,7 @@ export function registerRpcHandlers( historianTimeoutMs: config.historian_timeout_ms ?? DEFAULT_HISTORIAN_TIMEOUT_MS, memoryEnabled: config.memory?.enabled ?? true, autoPromote: config.memory?.auto_promote ?? true, + embeddingEnabled: config.embedding?.provider !== "off", fallbackModels: resolveFallbackChain(config.historian?.fallback_models), runMigration: config.memory?.enabled !== false && !!config.historian?.model, userMemoriesEnabled: config.dreamer?.user_memories?.enabled === true, diff --git a/packages/plugin/src/shared/models-dev-cache.test.ts b/packages/plugin/src/shared/models-dev-cache.test.ts index f615a304b..0f23f5921 100644 --- a/packages/plugin/src/shared/models-dev-cache.test.ts +++ b/packages/plugin/src/shared/models-dev-cache.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { mkdtempSync, rmSync } from "node:fs"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -263,6 +263,148 @@ describe("models-dev-cache (SDK-only)", () => { expect(getModelsDevCacheState().apiLoaded).toBe(false); }); + describe("cache-first boot", () => { + function persistedFilePath(): string { + return join( + tempDir, + "cortexkit", + "magic-context", + "model-context-limits-opencode.json", + ); + } + + test("persisted cache seeds the in-memory map synchronously, before the API refresh resolves", async () => { + // First run: warm + persist so the persisted file exists for the + // "next boot" we're about to simulate. + await refreshModelLimitsFromApi( + makeClient([{ id: "openai", models: { "gpt-5.5": { limit: { input: 272000 } } } }]), + ); + + // Simulate a restart: in-memory cache gone, persisted file remains. + clearModelsDevCache(); + expect(getModelsDevCacheState().apiLoaded).toBe(false); + + // A deferred fetch — we control when (or if) it resolves. Models + // the boot-time state where OpenCode's provider service isn't + // ready yet: the SDK call hangs, no payload comes back. + let resolveFetch: (value: { data?: { providers?: unknown[] } }) => void = () => {}; + const fetchPromise = new Promise<{ data?: { providers?: unknown[] } }>((resolve) => { + resolveFetch = resolve; + }); + const deferredClient = { config: { providers: () => fetchPromise } }; + + // Kick off the background refresh. Do NOT await — the test asserts + // that the in-memory map is populated synchronously inside the + // function body, before the first `await`, so the very next + // `getSdkContextLimit` lookup returns the persisted value with no + // wait on the API. + const refreshPromise = refreshModelLimitsFromApi(deferredClient, { + retries: 0, + retryDelayMs: 1, + }); + + // CACHE-FIRST: the persisted cache must be loaded SYNCHRONOUSLY + // by refreshModelLimitsFromApi before any await, so the lookup + // resolves immediately from the persisted cache — even though + // the API fetch is still pending (and may never resolve). + expect(getModelsDevCacheState().apiLoaded).toBe(true); + expect(getSdkContextLimit("openai", "gpt-5.5")).toBe(272000); + + // Let the background refresh complete (with an empty payload — + // no overwrite) so the test can finish cleanly. + resolveFetch({ data: { providers: [] } }); + await refreshPromise; + }); + + test("background API refresh success later swaps the in-memory map AND persists the new values", async () => { + // First run: warm + persist with one set of values. + await refreshModelLimitsFromApi( + makeClient([{ id: "openai", models: { "gpt-5.5": { limit: { input: 272000 } } } }]), + ); + + // Simulate a restart: in-memory cache gone, persisted file remains. + clearModelsDevCache(); + + // Deferred fetch returning FRESH data (different limit value). + let resolveFetch: (value: { data?: { providers?: unknown[] } }) => void = () => {}; + const fetchPromise = new Promise<{ data?: { providers?: unknown[] } }>((resolve) => { + resolveFetch = resolve; + }); + const deferredClient = { config: { providers: () => fetchPromise } }; + + // Kick off the background refresh. + const refreshPromise = refreshModelLimitsFromApi(deferredClient, { + retries: 0, + retryDelayMs: 1, + }); + + // CACHE-FIRST: the in-memory map is populated SYNCHRONOUSLY by + // refreshModelLimitsFromApi before the first await, so the + // lookup serves the OLD persisted value (272000) even though the + // API refresh is still in flight. + expect(getModelsDevCacheState().apiLoaded).toBe(true); + expect(getSdkContextLimit("openai", "gpt-5.5")).toBe(272000); + + // Now resolve the fetch with FRESH data. + resolveFetch({ + data: { + providers: [ + { + id: "openai", + models: { "gpt-5.5": { limit: { input: 100000 } } }, + }, + ], + }, + }); + + // Let the refresh function complete. The in-memory map is + // atomically swapped to the API data; the persisted file is + // rewritten with the new values. + await refreshPromise; + + // The in-memory map was swapped to the API data. + expect(getSdkContextLimit("openai", "gpt-5.5")).toBe(100000); + expect(getModelsDevCacheState().apiCount).toBe(1); + + // And the new value was persisted to disk so the NEXT process + // cold-starts with it (not the stale 272k). + const persistedRaw = readFileSync(persistedFilePath(), "utf-8"); + const persisted = JSON.parse(persistedRaw) as Record; + expect(persisted["openai/gpt-5.5"]).toBe(100000); + }); + + test("without a persisted cache, the cache is populated only when the API refresh succeeds (no synchronous preload)", async () => { + // No persisted cache exists (fresh beforeEach, nothing persisted). + expect(getModelsDevCacheState().apiLoaded).toBe(false); + + let calls = 0; + const client = { + config: { + providers: async () => { + calls++; + return { + data: { + providers: [ + { id: "p", models: { m: { limit: { context: 200000 } } } }, + ], + }, + }; + }, + }, + }; + + // With no persisted file, the eager preload inside + // refreshModelLimitsFromApi is a no-op (the file read fails and + // is caught). The existing retry-loop behavior is unchanged: the + // first successful fetch populates the cache. + await refreshModelLimitsFromApi(client, { retries: 0, retryDelayMs: 1 }); + + expect(calls).toBe(1); + expect(getSdkContextLimit("p", "m")).toBe(200000); + expect(getModelsDevCacheState().apiLoaded).toBe(true); + }); + }); + test("repeated refreshes replace cache state without corruption", async () => { const clientA = makeClient([ { diff --git a/packages/plugin/src/shared/models-dev-cache.ts b/packages/plugin/src/shared/models-dev-cache.ts index 6ff270b1e..32060aff6 100644 --- a/packages/plugin/src/shared/models-dev-cache.ts +++ b/packages/plugin/src/shared/models-dev-cache.ts @@ -199,6 +199,22 @@ export async function refreshModelLimitsFromApi( client: OpencodeClientLike, options?: { retries?: number; retryDelayMs?: number }, ): Promise { + // CACHE-FIRST: seed the in-memory map from the persisted last-known-good + // file SYNCHRONOUSLY before any await. This is the only change to the + // boot path: a persisted cache (when present) populates the map the + // moment the caller kicks off the background refresh, so any + // `getSdkContextLimit()` lookup — including the first one, which + // historically fired only after whatever other startup work had cleared + // — returns a real answer with no wait on the API. + // + // `loadPersistedApiCacheOnce()` is idempotent (no-op when the cache is + // already populated or the file was already attempted this process), + // so this is safe to call here and will not overwrite a fresh API + // result on subsequent calls. With no persisted file, the file read + // throws and is caught — the eager preload becomes a no-op and the + // existing retry path is preserved verbatim. + loadPersistedApiCacheOnce(); + const attempts = Math.max(1, (options?.retries ?? 0) + 1); const delayMs = options?.retryDelayMs ?? 1000; for (let attempt = 1; attempt <= attempts; attempt++) { diff --git a/packages/plugin/src/shared/rpc-types.ts b/packages/plugin/src/shared/rpc-types.ts index f81f8396f..3cfa81b6e 100644 --- a/packages/plugin/src/shared/rpc-types.ts +++ b/packages/plugin/src/shared/rpc-types.ts @@ -121,6 +121,22 @@ export interface StatusDetail extends SidebarSnapshot { historyBlockTokens: number; compressionBudget: number | null; compressionUsage: string | null; + /** + * External memory backend status snapshot. Null when the provider is off + * (or never initialized). `recallState` mirrors the session's + * `external_recall_state` so the status dialog can show "pending" / + * "done" / "failed" alongside the backend health. `failedRetainCount` + * comes from the operations endpoint (type=retain&status=failed) and is + * null when the backend is offline, the endpoint is missing, or the + * response is malformed. Populated by the async `buildStatusDetail`. + */ + externalMemory?: { + provider: string; + endpoint?: string; + circuitState?: string; + recallState?: string | null; + failedRetainCount?: number | null; + } | null; } /** Embedding coverage for `/ctx-embed` status (mirrors getEmbeddingCoverageStatus). */ diff --git a/packages/plugin/src/shared/tui-config.ts b/packages/plugin/src/shared/tui-config.ts index b70812e7d..6d7002b95 100644 --- a/packages/plugin/src/shared/tui-config.ts +++ b/packages/plugin/src/shared/tui-config.ts @@ -12,26 +12,17 @@ import { getOpenCodeConfigPaths } from "./opencode-config-dir"; const PLUGIN_NAME = "@cortexkit/opencode-magic-context"; const PLUGIN_ENTRY = `${PLUGIN_NAME}@latest`; -/** - * Detect whether a tui.json plugin entry already references magic-context, in - * any form. Covers: - * - Bare npm name: "@cortexkit/opencode-magic-context" - * - Versioned npm: "@cortexkit/opencode-magic-context@latest" / "@0.15.7" / etc. - * - Local dev directory path (absolute or relative): ".../magic-context" - * or ".../magic-context/packages/plugin" - * - file:// URLs pointing at the same paths - * - Tarball paths ending in opencode-magic-context-*.tgz - * - * Without the path/URL detection, doctor/setup auto-injection adds the npm - * @latest entry on top of an existing dev path, double-loading the plugin. - */ function isMagicContextEntry(entry: string): boolean { if (!entry) return false; if (entry === PLUGIN_NAME) return true; if (entry.startsWith(`${PLUGIN_NAME}@`)) return true; - // Local directory paths: match anywhere in the string so the setup pattern - // (dir-only, dir + /packages/plugin, file:// + either) all qualify. + // Tarball or any path containing the full npm package name. if (entry.includes("opencode-magic-context")) return true; + // Local dev directory paths (absolute, relative, or file:// URLs): + // ".../magic-context" or ".../magic-context/packages/plugin". + // Match "magic-context" as a whole path segment so unrelated entries + // (e.g. "not-magic-contexts") don't false-positive. + if (/(^|\/)magic-context(\/|$)/.test(entry)) return true; return false; } diff --git a/packages/plugin/src/shared/write-transaction.test.ts b/packages/plugin/src/shared/write-transaction.test.ts new file mode 100644 index 000000000..ee3c7f194 --- /dev/null +++ b/packages/plugin/src/shared/write-transaction.test.ts @@ -0,0 +1,150 @@ +import assert from "node:assert/strict"; +import { describe, it } from "node:test"; +import { runWriteTransaction, runWriteTransactionAsync } from "./write-transaction.js"; + +/** + * Minimal fake Database that records exec calls and simulates SQLITE_BUSY on + * the first N attempts. Only the surface area write-transaction.ts uses. + */ +function makeFakeDb(opts: { + inTransaction?: boolean; + isTransaction?: boolean; + busyTimes?: number; +}) { + const calls: string[] = []; + let busyRemaining = opts.busyTimes ?? 0; + const db = { + inTransaction: opts.inTransaction, + isTransaction: opts.isTransaction, + exec(sql: string) { + calls.push(sql); + if (sql === "BEGIN IMMEDIATE" && busyRemaining > 0) { + busyRemaining -= 1; + const err: Error & { code?: string } = new Error("database is locked"); + err.code = "SQLITE_BUSY"; + throw err; + } + }, + _calls: calls, + }; + return db; +} + +describe("runWriteTransaction", () => { + beforeEach(() => { + // node:test doesn't have a global mock restore; we rely on per-test fakes. + }); + + it("issues BEGIN IMMEDIATE / COMMIT around the body", () => { + const db = makeFakeDb({}); + const order: string[] = []; + runWriteTransaction(db as any, () => { + order.push("body"); + return 42; + }); + assert.deepEqual(db._calls, ["BEGIN IMMEDIATE", "COMMIT"]); + assert.deepEqual(order, ["body"]); + }); + + it("returns the body's return value", () => { + const db = makeFakeDb({}); + const result = runWriteTransaction(db as any, () => "ok"); + assert.equal(result, "ok"); + }); + + it("rolls back when the body throws", () => { + const db = makeFakeDb({}); + assert.throws( + () => + runWriteTransaction(db as any, () => { + throw new Error("body failed"); + }), + /body failed/, + ); + assert.deepEqual(db._calls, ["BEGIN IMMEDIATE", "ROLLBACK"]); + }); + + it("retries on transient SQLITE_BUSY and succeeds", () => { + const db = makeFakeDb({ busyTimes: 1 }); + const result = runWriteTransaction(db as any, () => "recovered"); + // First attempt: BEGIN IMMEDIATE throws SQLITE_BUSY (no ROLLBACK — the + // BEGIN itself failed, so there's no transaction to roll back). + // Second attempt: BEGIN IMMEDIATE + COMMIT succeed. + assert.deepEqual(db._calls, ["BEGIN IMMEDIATE", "BEGIN IMMEDIATE", "COMMIT"]); + assert.equal(result, "recovered"); + }); + + it("gives up after WRITE_RETRY_MAX_ATTEMPTS transient failures", () => { + const db = makeFakeDb({ busyTimes: 99 }); + assert.throws(() => runWriteTransaction(db as any, () => "never"), /database is locked/); + // 4 attempts, all on BEGIN IMMEDIATE. + assert.deepEqual(db._calls, [ + "BEGIN IMMEDIATE", + "BEGIN IMMEDIATE", + "BEGIN IMMEDIATE", + "BEGIN IMMEDIATE", + ]); + }); + + it("does NOT issue nested BEGIN when already in a transaction", () => { + const db = makeFakeDb({ inTransaction: true }); + const result = runWriteTransaction(db as any, () => "inline"); + assert.deepEqual(db._calls, []); + assert.equal(result, "inline"); + }); + + it("does NOT issue nested BEGIN when node:sqlite isTransaction flag is set", () => { + const db = makeFakeDb({ isTransaction: true }); + const result = runWriteTransaction(db as any, () => "inline"); + assert.deepEqual(db._calls, []); + assert.equal(result, "inline"); + }); + + it("non-transient errors are not retried", () => { + let attempts = 0; + const db = { + exec() { + attempts += 1; + throw new Error("disk I/O error"); + }, + }; + assert.throws(() => runWriteTransaction(db as any, () => "never"), /disk I\/O error/); + assert.equal(attempts, 1); + }); +}); + +describe("runWriteTransactionAsync", () => { + it("issues BEGIN IMMEDIATE / COMMIT around an async body", async () => { + const db = makeFakeDb({}); + const result = await runWriteTransactionAsync(db as any, async () => { + return "async-ok"; + }); + assert.deepEqual(db._calls, ["BEGIN IMMEDIATE", "COMMIT"]); + assert.equal(result, "async-ok"); + }); + + it("rolls back when the async body throws", async () => { + const db = makeFakeDb({}); + await assert.rejects( + runWriteTransactionAsync(db as any, async () => { + throw new Error("async body failed"); + }), + /async body failed/, + ); + assert.deepEqual(db._calls, ["BEGIN IMMEDIATE", "ROLLBACK"]); + }); + + it("retries on transient SQLITE_BUSY", async () => { + const db = makeFakeDb({ busyTimes: 1 }); + const result = await runWriteTransactionAsync(db as any, async () => "recovered"); + assert.deepEqual(db._calls, ["BEGIN IMMEDIATE", "BEGIN IMMEDIATE", "COMMIT"]); + assert.equal(result, "recovered"); + }); + + it("does NOT issue nested BEGIN when already in a transaction", async () => { + const db = makeFakeDb({ inTransaction: true }); + const result = await runWriteTransactionAsync(db as any, async () => "inline"); + assert.deepEqual(db._calls, []); + assert.equal(result, "inline"); + }); +}); diff --git a/packages/plugin/src/shared/write-transaction.ts b/packages/plugin/src/shared/write-transaction.ts new file mode 100644 index 000000000..eee7b63f9 --- /dev/null +++ b/packages/plugin/src/shared/write-transaction.ts @@ -0,0 +1,202 @@ +/** + * Centralized write-transaction helper for the shared context.db. + * + * Why: the plugin issues 100+ write transactions across 40+ files against ONE + * shared SQLite file, opened by multiple processes (OpenCode + Pi, or two + * OpenCode instances). Each `BEGIN IMMEDIATE` contends for the single WAL + * writer lock. Three problems arose: + * + * 1. SQLITE_BUSY propagation: when a sibling process holds the writer lock + * past busy_timeout, the thrown SQLITE_BUSY propagated up and surfaced + * as "failed to load plugin ... database is locked" or "Hit a transient + * issue comparting history this turn". The plugin would disable itself + * for the run instead of waiting. + * + * 2. Duplicated transaction plumbing: every BEGIN IMMEDIATE site + * reimplemented the same try/commit/finally-rollback pattern, slightly + * differently, sometimes without the rollback path. Bugs leaked in at + * the edges. + * + * 3. Inconsistent busy-retry behavior: some sites retried, some didn't, + * some swallowed, some propagated. The result was unpredictable under + * real multi-process load. + * + * Solution: a single `runWriteTransaction(db, body)` helper that wraps the + * BEGIN IMMEDIATE / COMMIT in a bounded SQLITE_BUSY retry loop, so a + * long-running sibling transaction (large migration, dreamer run, bulk + * Channel-2 delivery) makes us wait-and-retry instead of throwing. This + * centralizes the transaction plumbing and the retry policy in one place. + * + * Cross-process fairness is still handled by SQLite's own WAL writer lock + * plus busy_timeout (set in storage-db.ts initializeDatabase); this helper + * just makes every call site a well-behaved writer that absorbs transient + * BUSY errors instead of propagating them into plugin-disable paths. + * + * Usage (replaces hand-rolled BEGIN IMMEDIATE / COMMIT blocks): + * + * const result = runWriteTransaction(db, () => { + * db.prepare("INSERT ...").run(...); + * return computeResult(db); + * }); + * + * For bodies that must do async work between writes, use the async form: + * + * const result = await runWriteTransactionAsync(db, async () => { + * await someAsyncThing(); + * db.prepare("INSERT ...").run(...); + * }); + * + * Composition: if called inside an existing transaction (db.transaction or + * another runWriteTransaction), the body runs inline WITHOUT issuing a nested + * BEGIN (SQLite doesn't allow nested BEGIN; the outer transaction already + * holds the writer lock). This makes the helper safe to compose. + */ + +import { getErrorMessage } from "./error-message"; +import { log } from "./logger"; +import type { Database } from "./sqlite"; + +/** Bounded retry on transient lock errors. */ +const WRITE_RETRY_MAX_ATTEMPTS = 4; +const WRITE_RETRY_BACKOFF_MS = 100; + +function isTransientBusy(error: unknown): boolean { + if (!error || typeof error !== "object") return false; + const e = error as { code?: unknown; message?: unknown }; + if (typeof e.code === "string") { + if ( + e.code === "SQLITE_BUSY" || + e.code === "SQLITE_LOCKED" || + e.code === "SQLITE_BUSY_SNAPSHOT" || + e.code === "SQLITE_BUSY_RECOVERY" + ) { + return true; + } + } + if (typeof e.message === "string") { + return /database is locked/i.test(e.message) || /sqlite_(busy|locked)/i.test(e.message); + } + return false; +} + +function sleepSync(ms: number): void { + // Synchronous SQLite backends can't await; the durations are tiny (100ms + // cap) and only happen on transient cross-process contention. + const end = Date.now() + ms; + while (Date.now() < end) { + // spin + } +} + +function isInTransaction(db: Database): boolean { + // bun:sqlite and better-sqlite3 expose `inTransaction`; the node:sqlite + // shim in sqlite.ts sets `isTransaction` during savepoint-wrapper + // transactions. Respect either so this helper composes correctly when + // called from inside an existing db.transaction(() => { ... })() block. + const candidate = db as unknown as { inTransaction?: unknown; isTransaction?: unknown }; + return candidate.inTransaction === true || candidate.isTransaction === true; +} + +/** + * Run a synchronous write body inside a BEGIN IMMEDIATE transaction, retried + * on transient SQLITE_BUSY. + * + * - If called inside an existing transaction, the body runs inline WITHOUT + * issuing a nested BEGIN (SQLite doesn't allow nested BEGIN; the outer + * transaction already holds the writer lock). + * - Otherwise: issue BEGIN IMMEDIATE, run the body, COMMIT. On a transient + * SQLITE_BUSY the whole sequence is retried up to WRITE_RETRY_MAX_ATTEMPTS + * times with backoff. + */ +export function runWriteTransaction(db: Database, body: () => T): T { + if (isInTransaction(db)) { + return body(); + } + return runWithBusyRetry(() => { + db.exec("BEGIN IMMEDIATE"); + let committed = false; + try { + const result = body(); + db.exec("COMMIT"); + committed = true; + return result; + } finally { + if (!committed) { + try { + db.exec("ROLLBACK"); + } catch { + // transaction may already be closed by SQLite after an error + } + } + } + }); +} + +/** + * Async form of runWriteTransaction. Use this from any async call site; the + * body may itself be async. Same retry and composition semantics. + */ +export async function runWriteTransactionAsync( + db: Database, + body: () => T | Promise, +): Promise { + if (isInTransaction(db)) { + return body(); + } + return runWithBusyRetryAsync(async () => { + db.exec("BEGIN IMMEDIATE"); + let committed = false; + try { + const result = await body(); + db.exec("COMMIT"); + committed = true; + return result; + } finally { + if (!committed) { + try { + db.exec("ROLLBACK"); + } catch { + // already closed + } + } + } + }); +} + +function runWithBusyRetry(fn: () => T): T { + let lastError: unknown; + for (let attempt = 0; attempt < WRITE_RETRY_MAX_ATTEMPTS; attempt += 1) { + try { + return fn(); + } catch (error) { + lastError = error; + if (!isTransientBusy(error) || attempt === WRITE_RETRY_MAX_ATTEMPTS - 1) { + throw error; + } + log( + `[magic-context] write txn attempt ${attempt + 1}/${WRITE_RETRY_MAX_ATTEMPTS} hit transient lock; retrying in ${WRITE_RETRY_BACKOFF_MS}ms: ${getErrorMessage(error)}`, + ); + sleepSync(WRITE_RETRY_BACKOFF_MS); + } + } + throw lastError; +} + +async function runWithBusyRetryAsync(fn: () => T | Promise): Promise { + let lastError: unknown; + for (let attempt = 0; attempt < WRITE_RETRY_MAX_ATTEMPTS; attempt += 1) { + try { + return await fn(); + } catch (error) { + lastError = error; + if (!isTransientBusy(error) || attempt === WRITE_RETRY_MAX_ATTEMPTS - 1) { + throw error; + } + log( + `[magic-context] write txn attempt ${attempt + 1}/${WRITE_RETRY_MAX_ATTEMPTS} hit transient lock; retrying in ${WRITE_RETRY_BACKOFF_MS}ms: ${getErrorMessage(error)}`, + ); + await new Promise((resolve) => setTimeout(resolve, WRITE_RETRY_BACKOFF_MS)); + } + } + throw lastError; +} diff --git a/packages/plugin/src/tools/ctx-expand/tools.ts b/packages/plugin/src/tools/ctx-expand/tools.ts index 91294c4ca..eac5374a1 100644 --- a/packages/plugin/src/tools/ctx-expand/tools.ts +++ b/packages/plugin/src/tools/ctx-expand/tools.ts @@ -1,5 +1,6 @@ import { type ToolDefinition, tool } from "@opencode-ai/plugin"; import { getLastCompartmentEndMessage } from "../../features/magic-context/compartment-storage"; +import { resolveRootSessionId } from "../../features/magic-context/session-parent-registry"; import type { ContextDatabase } from "../../features/magic-context/storage"; import { readSessionChunk } from "../../hooks/magic-context/read-session-chunk"; import { CTX_EXPAND_DESCRIPTION, CTX_EXPAND_TOKEN_BUDGET } from "./constants"; @@ -40,7 +41,11 @@ function createCtxExpandTool(deps: CtxExpandToolDeps): ToolDefinition { ), }, async execute(args: CtxExpandArgs, toolContext) { - const sessionId = toolContext.sessionID; + // Resolve child sessions (sidekick, task subagents) to the ROOT + // conversation — ctx_search hands out ordinals from the root + // session's message index, so expanding them against the child's + // empty session would always miss. Mirrors ctx_search. + const sessionId = resolveRootSessionId(toolContext.sessionID); // By-ordinal mode: full recovery of a single message from stored history. if (typeof args.message === "number" && args.message >= 1) { diff --git a/packages/plugin/src/tools/ctx-memory/constants.ts b/packages/plugin/src/tools/ctx-memory/constants.ts index 300f47d1b..1ede963b9 100644 --- a/packages/plugin/src/tools/ctx-memory/constants.ts +++ b/packages/plugin/src/tools/ctx-memory/constants.ts @@ -4,7 +4,7 @@ export const CTX_MEMORY_DESCRIPTION = `Durable project knowledge shared across e Your active memories are already visible in (each with its id), and every future session starts with them — write one when you learn something future sessions must know: a project rule, an architectural fact, a hard-won constraint, a config value, or a naming convention. Keep each memory one standalone fact, phrased to make sense without this session's context. Actions: -- write: save a new memory (content + category). +- write: save a new memory (content + category). Optional scope: "project" (default) for this project's store, or "global" for cross-project facts (infrastructure, tooling, environment) stored only in the external long-term memory backend. - update: rewrite one memory whose fact changed (ids: [one], content). - archive: retire wrong or obsolete memories (ids: [one or more], optional reason). - merge: collapse duplicates into one memory (ids: [two or more], content). diff --git a/packages/plugin/src/tools/ctx-memory/tools.test.ts b/packages/plugin/src/tools/ctx-memory/tools.test.ts index f474f8524..23288f7d6 100644 --- a/packages/plugin/src/tools/ctx-memory/tools.test.ts +++ b/packages/plugin/src/tools/ctx-memory/tools.test.ts @@ -9,6 +9,16 @@ import { insertMemory, normalizeStoredProjectPath, } from "../../features/magic-context"; +import { + _resetExternalMemoryForTests, + _setTestExternalBackendFactory, + initializeExternalMemory, +} from "../../features/magic-context/memory/external-memory"; +import type { + ExternalMemoryBackend, + ExternalMemoryRemoveItem, + ExternalMemoryRetainItem, +} from "../../features/magic-context/memory/external-memory-provider"; import { Database } from "../../shared/sqlite"; import { closeQuietly } from "../../shared/sqlite-helpers"; @@ -150,6 +160,56 @@ function getMutationRows(db: Database, projectPath: string, renderedMemoryIds: n ); } +const HINDSIGHT_TEST_CONFIG = { + provider: "hindsight" as const, + endpoint: "http://10.0.0.1:8889", + project_bank: "mc-{name}-{id8}", + main_bank: "main-memory", + retain_sources: ["historian", "agent", "dreamer"] as ("historian" | "agent" | "dreamer")[], + tags: [] as string[], + recall: { + enabled: true, + timeout_ms: 3000, + max_tokens: 2048, + dedup_threshold: 0.85, + global_tags: [] as string[], + global_from_prompt: false, + search: true, + mental_models: false, + profile_mental_models: ["user-preferences"], + }, +}; + +interface ExternalBackendCapture { + retains: ExternalMemoryRetainItem[][]; + removes: ExternalMemoryRemoveItem[][]; +} + +function captureBackend(): ExternalBackendCapture { + const capture: ExternalBackendCapture = { retains: [], removes: [] }; + _setTestExternalBackendFactory( + (): ExternalMemoryBackend => ({ + backendId: "fake:test", + initialize: async () => true, + retain: async (items) => { + capture.retains.push([...items]); + return items.length; + }, + remove: async (items) => { + capture.removes.push([...items]); + return items.length; + }, + dispose: async () => {}, + }), + ); + initializeExternalMemory(HINDSIGHT_TEST_CONFIG); + return capture; +} + +function captureTee(): ExternalMemoryRetainItem[][] { + return captureBackend().retains; +} + afterAll(() => { mock.restore(); }); @@ -170,6 +230,7 @@ describe("createCtxMemoryTools", () => { afterEach(() => { closeQuietly(db); + _resetExternalMemoryForTests(); }); describe("#given write action", () => { @@ -263,6 +324,118 @@ describe("createCtxMemoryTools", () => { expect(memories).toHaveLength(1); expect(memories[0]?.projectPath).toBe("/repo/project"); }); + + it("tees to external backend with project scope", async () => { + const calls = captureTee(); + + const result = await tools.ctx_memory.execute( + { + action: "write", + content: "agent fact", + category: "ARCHITECTURE", + }, + toolContext(), + ); + + expect(result).toContain("Saved memory"); + await Bun.sleep(10); + + expect(calls.length).toBe(1); + expect(calls[0]?.[0]).toMatchObject({ + content: "agent fact", + category: "ARCHITECTURE", + scope: "project", + sourceType: expect.any(String), + }); + }); + + it("scope 'global' tees to the main bank with NO local row, carrying origin provenance", async () => { + const calls = captureTee(); + + const result = await tools.ctx_memory.execute( + { + action: "write", + content: "Homelab reverse proxy lives on 10.1.1.5 (caddy).", + category: "ARCHITECTURE", + scope: "global", + }, + toolContext(), + ); + await Bun.sleep(10); + + expect(result).toContain("Queued global memory"); + expect(result).not.toContain("Saved memory [ID:"); + // No local row — globals live only in the external store. + expect(getMemoriesByProject(db, "/repo/project")).toHaveLength(0); + expect(calls.length).toBe(1); + // Origin provenance rides along (origin-* tags + context at the + // engine layer) while scope stays "global" → main-bank routing. + expect(calls[0]?.[0]).toMatchObject({ + content: "Homelab reverse proxy lives on 10.1.1.5 (caddy).", + category: "ARCHITECTURE", + scope: "global", + projectIdentity: "/repo/project", + }); + }); + + it("scope 'global' errors (and stays local-row-free) when no external backend is configured", async () => { + // No captureBackend() call → provider stays "off". + const result = await tools.ctx_memory.execute( + { + action: "write", + content: "orphan global fact", + category: "ARCHITECTURE", + scope: "global", + }, + toolContext(), + ); + + expect(result).toContain("Error: scope 'global' requires an external memory backend"); + expect(getMemoriesByProject(db, "/repo/project")).toHaveLength(0); + }); + + it("scope 'project' (explicit) behaves exactly like the default", async () => { + const calls = captureTee(); + + const result = await tools.ctx_memory.execute( + { + action: "write", + content: "explicit project fact", + category: "ARCHITECTURE", + scope: "project", + }, + toolContext(), + ); + await Bun.sleep(10); + + expect(result).toContain("Saved memory [ID:"); + expect(getMemoriesByProject(db, "/repo/project")).toHaveLength(1); + expect(calls[0]?.[0]).toMatchObject({ scope: "project" }); + }); + + it("does NOT tee when memory already exists", async () => { + const calls = captureTee(); + + await tools.ctx_memory.execute( + { + action: "write", + content: "dup", + category: "ARCHITECTURE", + }, + toolContext(), + ); + await tools.ctx_memory.execute( + { + action: "write", + content: "dup", + category: "ARCHITECTURE", + }, + toolContext(), + ); + await Bun.sleep(10); + + expect(calls.length).toBe(1); + }); }); describe("#given archive action by a PRIMARY agent", () => { @@ -1143,4 +1316,191 @@ describe("createCtxMemoryTools", () => { expect(result).toContain("Found 1 active memory"); }); }); + + describe("#given corrective propagation to external backend", () => { + function extractMemoryId(result: string): number { + const match = result.match(/\[ID:\s*(\d+)\]/); + if (!match) throw new Error(`could not parse memory id from: ${result}`); + return Number.parseInt(match[1]!, 10); + } + + it("batch archive propagates ONE batched remove covering every archived row", async () => { + const capture = captureBackend(); + const firstId = extractMemoryId( + await tools.ctx_memory.execute( + { action: "write", category: "ARCHITECTURE", content: "doomed fact" }, + toolContext(), + ), + ); + const secondId = extractMemoryId( + await tools.ctx_memory.execute( + { action: "write", category: "ARCHITECTURE", content: "second doomed fact" }, + toolContext(), + ), + ); + + const archiveResult = await tools.ctx_memory.execute( + { action: "archive", ids: [firstId, secondId] }, + toolContext(), + ); + + expect(archiveResult).toContain("Archived memories"); + await Bun.sleep(10); + // ONE batched remove call carrying BOTH pre-mutation rows. + expect(capture.removes.length).toBe(1); + const removed = capture.removes[0] ?? []; + expect(removed.map((item) => item.content).sort()).toEqual([ + "doomed fact", + "second doomed fact", + ]); + expect(removed.every((item) => item.scope === "project")).toBe(true); + }); + + it("archive action propagates remove to external backend", async () => { + const capture = captureBackend(); + const writeResult = await tools.ctx_memory.execute( + { + action: "write", + category: "PROJECT_RULES", + content: "stale fact", + }, + toolContext("ses-dreamer", DREAMER_AGENT), + ); + const id = extractMemoryId(writeResult); + + const archiveResult = await tools.ctx_memory.execute( + { action: "archive", ids: [id], reason: "subsystem removed" }, + toolContext("ses-dreamer", DREAMER_AGENT), + ); + + expect(archiveResult).toContain("Archived memory"); + await Bun.sleep(10); + expect(capture.removes.length).toBe(1); + expect(capture.removes[0]?.[0]?.content).toBe("stale fact"); + expect(capture.removes[0]?.[0]?.category).toBe("PROJECT_RULES"); + }); + + it("update action removes old content and tees new content", async () => { + const capture = captureBackend(); + const writeResult = await tools.ctx_memory.execute( + { + action: "write", + category: "CONFIG_VALUES", + content: "old wording", + }, + toolContext("ses-dreamer", DREAMER_AGENT), + ); + const id = extractMemoryId(writeResult); + // Wait for the write's retain to land before counting subsequent calls. + await Bun.sleep(10); + capture.retains.length = 0; + capture.removes.length = 0; + + const updateResult = await tools.ctx_memory.execute( + { + action: "update", + ids: [id], + content: "new wording", + }, + toolContext("ses-dreamer", DREAMER_AGENT), + ); + + expect(updateResult).toContain("Updated memory"); + await Bun.sleep(10); + // Old content removed from external (the document identity derives + // from the original content hash; without the remove, the new retain + // would create a duplicate document). + expect(capture.removes.length).toBe(1); + expect(capture.removes[0]?.[0]?.content).toBe("old wording"); + // Corrected content teed as a new document. + const teed = capture.retains.flat(); + expect(teed.some((item) => item.content === "new wording")).toBe(true); + }); + + it("verify action sets verification status and upserts verbatim", async () => { + const capture = captureBackend(); + const writeResult = await tools.ctx_memory.execute( + { + action: "write", + category: "PROJECT_RULES", + content: "true fact", + }, + toolContext("ses-dreamer", DREAMER_AGENT), + ); + const id = extractMemoryId(writeResult); + await Bun.sleep(10); + capture.retains.length = 0; + capture.removes.length = 0; + + const verifyResult = await tools.ctx_memory.execute( + { action: "verify", ids: [id] }, + toolContext("ses-dreamer", DREAMER_AGENT), + ); + + expect(verifyResult).toContain("Verified memory"); + // The local row's verification_status flipped. + expect(getMemoryById(db, id)?.verificationStatus).toBe("verified"); + await Bun.sleep(10); + // Verbatim re-retain = same document_id = server-side upsert — the + // single retain should contain the EXACT unchanged content plus + // verifiedAt metadata. No remove is fired (the document is current). + const upserted = capture.retains.flat(); + expect(upserted.length).toBe(1); + expect(upserted[0]?.content).toBe("true fact"); + expect(typeof upserted[0]?.verifiedAt).toBe("number"); + expect(capture.removes.length).toBe(0); + }); + + it("merge does NOT propagate to external backend (canonical rewrite is local-only)", async () => { + const capture = captureBackend(); + const first = insertMemory(db, { + projectPath: "/repo/project", + category: "PROJECT_RULES", + content: "use bun", + }); + const second = insertMemory(db, { + projectPath: "/repo/project", + category: "PROJECT_RULES", + content: "use bun for everything", + }); + await Bun.sleep(10); + capture.retains.length = 0; + capture.removes.length = 0; + + const mergeResult = await tools.ctx_memory.execute( + { + action: "merge", + ids: [first.id, second.id], + content: "use bun for all the things", + }, + toolContext("ses-dreamer", DREAMER_AGENT), + ); + + expect(mergeResult).toContain("Merged memories"); + await Bun.sleep(10); + // Merge is a local canonical rewrite — no external retain, no remove. + // v1 rule: the canonical document was never externally re-teed. + expect(capture.removes.length).toBe(0); + expect(capture.retains.length).toBe(0); + }); + + it("verify rejects non-dreamer agents", async () => { + insertMemory(db, { + projectPath: "/repo/project", + category: "PROJECT_RULES", + content: "primary-only memory", + }); + + // Primary agent tool with default allowedActions = ["write","delete"]. + // "verify" is a dreamer-only action and the action is not in + // allowedActions → rejected with the "not allowed" error. + const result = await tools.ctx_memory.execute( + { action: "verify", id: 1 }, + toolContext(), + ); + + expect(result).toContain("Error"); + expect(result).toContain("not allowed"); + }); + }); }); diff --git a/packages/plugin/src/tools/ctx-memory/tools.ts b/packages/plugin/src/tools/ctx-memory/tools.ts index 2a54a6041..5d2904fd2 100644 --- a/packages/plugin/src/tools/ctx-memory/tools.ts +++ b/packages/plugin/src/tools/ctx-memory/tools.ts @@ -1,9 +1,12 @@ +import { basename } from "node:path"; + import { type ToolDefinition, tool } from "@opencode-ai/plugin"; import { DREAMER_AGENT } from "../../agents/dreamer"; import { SIDEKICK_AGENT } from "../../agents/sidekick"; import { archiveMemory, CATEGORY_PRIORITY, + getExternalMemoryStatus, getMemoriesByProject, getMemoryByHash, getMemoryById, @@ -11,9 +14,13 @@ import { type Memory, type MemoryCategory, mergeMemoryStats, + removeFromExternalBackend, saveEmbedding, supersededMemory, + teeToExternalBackend, updateMemorySeenCount, + updateMemoryVerification, + upsertToExternalBackend, V2_MEMORY_CATEGORIES, } from "../../features/magic-context/memory"; import { @@ -21,6 +28,7 @@ import { getProjectEmbeddingSnapshot, } from "../../features/magic-context/memory/embedding"; import { invalidateMemory } from "../../features/magic-context/memory/embedding-cache"; +import type { ExternalMemoryRemoveItem } from "../../features/magic-context/memory/external-memory-provider"; import { computeNormalizedHash } from "../../features/magic-context/memory/normalize-hash"; import { normalizeStoredProjectPath, @@ -271,6 +279,12 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { .string() .optional() .describe("Why the memory is being archived (optional, recommended)"), + scope: tool.schema + .enum(["project", "global"]) + .optional() + .describe( + 'Write only. "project" (default): this project\'s memory store. "global": a cross-project fact (infrastructure, tooling, environment) stored ONLY in the external long-term memory backend — use when the fact is true regardless of which project you are in. Requires an external backend; recallable from the next session onward.', + ), }, async execute(args: CtxMemoryArgs, toolContext) { // Sidekick consumes untrusted `/ctx-aug` prompt text and is retrieval-only; @@ -282,6 +296,22 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { return `Error: Action '${args.action}' is not allowed in this context.`; } + // Build an ExternalMemoryRemoveItem from a freshly-loaded memory row. + // Document identity in the external backend derives from the original + // content hash + project identity, so a corrective remove needs the + // row AS IT STOOD (not a stale in-memory `memory` from before the + // caller mutated it). Used by the delete/archive/update branches. + const buildRemoveItem = ( + memory: { content: string; category: Memory["category"] }, + projectIdentity: string, + ): ExternalMemoryRemoveItem => ({ + content: memory.content, + category: memory.category as MemoryCategory, + scope: "project", + projectIdentity, + ...(toolContext.directory ? { projectName: basename(toolContext.directory) } : {}), + }); + // Resolve the session's actual project from `toolContext.directory` // each call. OpenCode's top-level `ctx.directory` (the launch dir) // can differ from the session's working directory when the user @@ -362,6 +392,41 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { return `Error: Unknown memory category '${rawCategory}'.`; } + // Global scope: cross-project knowledge goes to the external + // long-term store's main bank ONLY — no local row. The local + // store is project-keyed; parking globals under a pseudo-project + // would corrupt the id-addressable curation model (update/ + // archive/dreamer flows). Server-side document_id idempotency + // (content-hash-derived) replaces the local hash dedup, so a + // re-write of the same fact upserts instead of duplicating. + // projectIdentity/projectName ride along as ORIGIN provenance + // (origin-* tags + extraction context), NOT as routing — the + // item still lands in the main bank with scope:global, but the + // engine links the originating project as an entity so the + // fact surfaces when any project references it by name. + if (args.scope === "global") { + if (!getExternalMemoryStatus()) { + return "Error: scope 'global' requires an external memory backend (memory.external) — none is configured. Use the default project scope instead."; + } + void teeToExternalBackend("agent", [ + { + content, + category, + scope: "global", + projectIdentity: projectPath, + ...(toolContext.directory + ? { projectName: basename(toolContext.directory) } + : {}), + sourceType: + toolContext.agent === DREAMER_AGENT + ? "dreamer" + : getSourceType(deps), + sessionId: toolContext.sessionID, + }, + ]); + return `Queued global memory in ${category} for the long-term store (origin: this project). It has no local ID; it surfaces via the session-start global recall slice and ctx_search source "external" from the next session onward.`; + } + const existingMemory = getMemoryByHash( deps.db, projectPath, @@ -390,9 +455,27 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { content, }); + void teeToExternalBackend("agent", [ + { + content, + category, + scope: "project", + projectIdentity: projectPath, + ...(toolContext.directory + ? { projectName: basename(toolContext.directory) } + : {}), + sourceType: + toolContext.agent === DREAMER_AGENT ? "dreamer" : getSourceType(deps), + sessionId: toolContext.sessionID, + }, + ]); + return `Saved memory [ID: ${memory.id}] in ${category}.`; } + // NOTE: the former `delete` action (an exact alias of archive) was + // removed upstream in v0.23.0; its external corrective-remove hook + // lives on in the `archive` branch below. if (args.action === "list") { const limit = normalizeLimit(args.limit); const category = normalizeCategory(args.category); @@ -460,6 +543,28 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { content, }); + // Corrective propagation: drop the STALE external document (old + // content hash) and tee the corrected fact as a new document. The + // local row's content rewrite already happened in the transaction + // above, so `memory.content` is still the OLD content and + // `content` is the NEW content — both needed for the + // remove-then-tee cascade. + void removeFromExternalBackend([buildRemoveItem(memory, projectIdentity)]); + void teeToExternalBackend("agent", [ + { + content, + category: memory.category as MemoryCategory, + scope: "project", + projectIdentity, + ...(toolContext.directory + ? { projectName: basename(toolContext.directory) } + : {}), + sourceType: + toolContext.agent === DREAMER_AGENT ? "dreamer" : getSourceType(deps), + sessionId: toolContext.sessionID, + }, + ]); + return `Updated memory [ID: ${memory.id}] in ${memory.category}.`; } @@ -650,8 +755,14 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { // Validate the whole batch BEFORE mutating anything so a typo'd // id can't half-archive a batch (all-or-nothing, matching the - // single-transaction write below). - const targets: Array<{ memoryId: number; projectIdentity: string }> = []; + // single-transaction write below). The pre-mutation row rides + // along: the external corrective remove derives its document_id + // from the content AS IT STOOD. + const targets: Array<{ + memoryId: number; + projectIdentity: string; + memory: Memory; + }> = []; for (const memoryId of archiveIds) { const rawProjectPath = projectPathForMemoryId(deps.db, memoryId); const memory = getMemoryById(deps.db, memoryId); @@ -667,6 +778,7 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { targets.push({ memoryId, projectIdentity: targetIdentityForStoredPath(rawProjectPath), + memory, }); } @@ -680,6 +792,12 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { }); } })(); + // Corrective propagation: the facts are gone locally (archived, + // not merely hidden) → drop the external documents. One batched + // call for the whole archive batch, after the local commit. + void removeFromExternalBackend( + targets.map((target) => buildRemoveItem(target.memory, target.projectIdentity)), + ); const idList = targets.map((t) => t.memoryId).join(", "); const plural = targets.length > 1 ? "memories" : "memory"; return args.reason?.trim() @@ -687,6 +805,41 @@ function createCtxMemoryTool(deps: CtxMemoryToolDeps): ToolDefinition { : `Archived ${plural} [ID: ${idList}].`; } + if (args.action === "verify") { + const verifyIds = args.ids; + if (!verifyIds || verifyIds.length !== 1 || !verifyIds.every(Number.isInteger)) { + return "Error: 'ids' must contain exactly one integer memory ID when action is 'verify'."; + } + const verifyId = verifyIds[0]; + const rawProjectPath = projectPathForMemoryId(deps.db, verifyId); + const memory = getMemoryById(deps.db, verifyId); + if (!memory || !rawProjectPath || !memoryBelongsToProject(memory, projectPath)) { + return `Error: Memory with ID ${verifyId} was not found.`; + } + const projectIdentity = projectIdentityForStoredPath(rawProjectPath); + updateMemoryVerification(deps.db, memory.id, "verified"); + // Verbatim re-retain = same document_id = server-side upsert → + // refreshes Hindsight's mentioned_at recency with ZERO duplicate + // risk. verifiedAt lands in metadata.verified_at. + void upsertToExternalBackend([ + { + content: memory.content, + category: memory.category as MemoryCategory, + scope: "project", + projectIdentity, + ...(toolContext.directory + ? { projectName: basename(toolContext.directory) } + : {}), + sourceType: "dreamer", + sessionId: toolContext.sessionID, + verifiedAt: Date.now(), + }, + ]); + // No queueMemoryMutation: verification status is not rendered in + // memory lines, so the cached m[0]/m[1] bytes are unaffected. + return `Verified memory [ID: ${memory.id}].`; + } + return "Error: Unknown action."; }, }); diff --git a/packages/plugin/src/tools/ctx-memory/types.ts b/packages/plugin/src/tools/ctx-memory/types.ts index 801a67cb6..d09cb5c05 100644 --- a/packages/plugin/src/tools/ctx-memory/types.ts +++ b/packages/plugin/src/tools/ctx-memory/types.ts @@ -10,7 +10,11 @@ import type { Database } from "../../shared/sqlite"; // since active memories are already in context) stays dreamer-only. export const CTX_MEMORY_ACTIONS = ["write", "archive", "update", "merge"] as const; -export const CTX_MEMORY_DREAMER_ACTIONS = [...CTX_MEMORY_ACTIONS, "list"] as const; +// `verify` stays dreamer-only: it asserts repo-grounded truth (the dreamer +// greps the actual code before verifying) and refreshes external long-term +// recency — a primary agent confirming its own memory mid-session would be +// circular evidence. +export const CTX_MEMORY_DREAMER_ACTIONS = [...CTX_MEMORY_ACTIONS, "list", "verify"] as const; export type CtxMemoryAction = (typeof CTX_MEMORY_DREAMER_ACTIONS)[number]; @@ -26,6 +30,10 @@ export interface CtxMemoryArgs { ids?: number[]; limit?: number; reason?: string; + /** Write-only. "project" (default) = local store + external tee. + * "global" = cross-project fact stored ONLY in the external long-term + * backend's main bank (requires memory.external configured). */ + scope?: "project" | "global"; } export interface CtxMemoryToolDeps { diff --git a/packages/plugin/src/tools/ctx-search/constants.ts b/packages/plugin/src/tools/ctx-search/constants.ts index a9e559e79..c162f8e6f 100644 --- a/packages/plugin/src/tools/ctx-search/constants.ts +++ b/packages/plugin/src/tools/ctx-search/constants.ts @@ -1,15 +1,17 @@ export const CTX_SEARCH_TOOL_NAME = "ctx_search"; export const CTX_SEARCH_DESCRIPTION = `Your long-term recall for this project — search everything that ever happened here, not just what's currently visible. -Reach for it when something feels familiar but isn't in view: "did we solve this before?", "what did we decide about X?", "when did this break?", "where does Y live?". Results only contain things you CANNOT currently see — memories already shown in and the live conversation tail are filtered out. +Reach for it when something feels familiar but isn't in view: "did we solve this before?", "what did we decide about X?", "when did this break?", "where does Y live?". Results only contain things you CANNOT currently see — memories already shown in , the live conversation tail, and external knowledge already injected via are all filtered out. Sources (omit for a broad search across all): - memory: curated cross-session project knowledge — rules, constraints, conventions. - message: the raw conversation behind your compacted history. Hits include message ordinals — expand the surrounding exchange with ctx_expand(start=N-10, end=N+5). - git_commit: this repository's commit history. +- external: long-term knowledge from past sessions across projects (requires memory.external.recall.search=true; explicit ctx_search calls only). Picking sources: - "when did this change / was this working before" → ["git_commit", "message"] - "did we discuss this earlier" → ["message"] -- "what's our convention / rule for X" → ["memory"]`; +- "what's our convention / rule for X" → ["memory"] +- "is there anything relevant from prior sessions / other projects" → ["external"]`; export const DEFAULT_CTX_SEARCH_LIMIT = 10; diff --git a/packages/plugin/src/tools/ctx-search/tools.test.ts b/packages/plugin/src/tools/ctx-search/tools.test.ts index b86da7d0c..b6b2aab97 100644 --- a/packages/plugin/src/tools/ctx-search/tools.test.ts +++ b/packages/plugin/src/tools/ctx-search/tools.test.ts @@ -1,7 +1,12 @@ import { afterEach, beforeEach, describe, expect, it } from "bun:test"; import { replaceAllCompartments } from "../../features/magic-context/compartment-storage"; import { insertMemory } from "../../features/magic-context/memory"; +import { _resetExternalMemoryForTests } from "../../features/magic-context/memory/external-memory"; import { indexMessagesAfterOrdinal } from "../../features/magic-context/message-index"; +import { + _resetSessionParentRegistryForTests, + registerSessionParent, +} from "../../features/magic-context/session-parent-registry"; import { initializeDatabase } from "../../features/magic-context/storage-db"; import { Database } from "../../shared/sqlite"; import { closeQuietly } from "../../shared/sqlite-helpers"; @@ -21,11 +26,19 @@ describe("createCtxSearchTools", () => { let db: Database; beforeEach(() => { + // The external-memory module keeps a cached backend + test factory at + // module scope. search.test.ts's "external search source" suite + // configures a stub factory; without this reset, ctx_search's explicit + // path (runExternal=true) would route to that stub and surface bogus + // hits in tests that expect an empty result set. + _resetExternalMemoryForTests(); db = createTestDb(); }); afterEach(() => { closeQuietly(db); + _resetExternalMemoryForTests(); + _resetSessionParentRegistryForTests(); }); it("validates required query", async () => { @@ -121,6 +134,54 @@ describe("createCtxSearchTools", () => { expect(result).not.toContain("Expand with ctx_expand(start="); }); + it("child session (sidekick) resolves to the parent's history via the parent registry", async () => { + // Parent session has compacted history + an indexed message; the child + // session has NOTHING (no compartments, no index). Without root + // resolution the boundary is 0 and message search is dead in children. + replaceAllCompartments(db, "ses-parent", [ + { + sequence: 1, + startMessage: 1, + endMessage: 10, + startMessageId: "m1", + endMessageId: "m10", + title: "Compartment", + content: "Summary", + }, + ]); + const indexed = [ + { + ordinal: 5, + id: "m5", + role: "assistant", + parts: [{ type: "text", text: "Alpha migration details are here." }], + }, + ]; + indexMessagesAfterOrdinal(db, "ses-parent", indexed, 0, 5); + const tools = createCtxSearchTools({ + db, + resolveProjectPath: () => "/repo/project", + memoryEnabled: false, + embeddingEnabled: false, + readMessages: () => indexed, + }); + + // Control: unregistered child finds nothing (live-tail exclusion). + const before = await tools.ctx_search.execute( + { query: "alpha migration", sources: ["message"] }, + toolContext("ses-sidekick-child"), + ); + expect(before).toContain("No results found"); + + registerSessionParent("ses-sidekick-child", "ses-parent"); + const after = await tools.ctx_search.execute( + { query: "alpha migration", sources: ["message"] }, + toolContext("ses-sidekick-child"), + ); + expect(after).toContain("[message]"); + expect(after).toContain("ordinal=5"); + }); + it("omits the consolidated expand hint for memory-only results", async () => { insertMemory(db, { projectPath: "/repo/project", diff --git a/packages/plugin/src/tools/ctx-search/tools.ts b/packages/plugin/src/tools/ctx-search/tools.ts index c9520d742..e857a8e6e 100644 --- a/packages/plugin/src/tools/ctx-search/tools.ts +++ b/packages/plugin/src/tools/ctx-search/tools.ts @@ -1,3 +1,5 @@ +import { basename } from "node:path"; + import { type ToolDefinition, tool } from "@opencode-ai/plugin"; import { getLastCompartmentEndMessage } from "../../features/magic-context/compartment-storage"; import { @@ -5,6 +7,7 @@ import { getProjectEmbeddingSnapshot, } from "../../features/magic-context/memory/embedding"; import { type UnifiedSearchResult, unifiedSearch } from "../../features/magic-context/search"; +import { resolveRootSessionId } from "../../features/magic-context/session-parent-registry"; import { getVisibleMemoryIds } from "../../hooks/magic-context/inject-compartments"; import { CTX_SEARCH_DESCRIPTION, @@ -13,7 +16,12 @@ import { } from "./constants"; import type { CtxSearchArgs, CtxSearchSource, CtxSearchToolDeps } from "./types"; -const VALID_SOURCES: ReadonlySet = new Set(["memory", "message", "git_commit"]); +const VALID_SOURCES: ReadonlySet = new Set([ + "memory", + "message", + "git_commit", + "external", +]); function normalizeLimit(limit?: number): number { if (typeof limit !== "number" || !Number.isFinite(limit)) { @@ -81,6 +89,14 @@ function formatResult(result: UnifiedSearchResult, index: number): string { ].join("\n"); } + if (result.source === "external") { + const categoryPart = result.category ? ` category=${result.category}` : ""; + return [ + `[${index}] [external] score=${result.score.toFixed(2)}${categoryPart}`, + result.content, + ].join("\n"); + } + const expandStart = Math.max(1, result.messageOrdinal - 3); const expandEnd = result.messageOrdinal + 3; return [ @@ -91,7 +107,7 @@ function formatResult(result: UnifiedSearchResult, index: number): string { function formatSearchResults(query: string, results: UnifiedSearchResult[]): string { if (results.length === 0) { - return `No results found for "${query}" across memories, git commits, or message history.`; + return `No results found for "${query}" across memories, git commits, message history, or external knowledge.`; } const bodyParts = results.map((result, index) => formatResult(result, index + 1)); @@ -118,10 +134,10 @@ function createCtxSearchTool(deps: CtxSearchToolDeps): ToolDefinition { .optional() .describe("Maximum results to return (default: 10)"), sources: tool.schema - .array(tool.schema.enum(["memory", "message", "git_commit"])) + .array(tool.schema.enum(["memory", "message", "git_commit", "external"])) .optional() .describe( - 'Optional. Restrict to specific sources. Examples: ["git_commit"] for "when did we change X", ["memory"] for naming conventions, ["message"] for "did we discuss this earlier", ["git_commit","message"] for regression hunts. Omit for a broad search across all enabled sources.', + 'Optional. Restrict to specific sources. Examples: ["git_commit"] for "when did we change X", ["memory"] for naming conventions, ["message"] for "did we discuss this earlier", ["git_commit","message"] for regression hunts, ["external"] for long-term knowledge from past sessions. Omit for a broad search across all enabled sources.', ), }, async execute(args: CtxSearchArgs, toolContext) { @@ -130,6 +146,14 @@ function createCtxSearchTool(deps: CtxSearchToolDeps): ToolDefinition { return "Error: 'query' is required."; } + // Child sessions (sidekick, task subagents) search the ROOT + // conversation: their own session has no indexed messages, no + // compartment boundary, and no injection markers, so every + // session-scoped read below would silently no-op — dead message + // search and disabled already-visible filters. Main sessions + // resolve to themselves (registry returns the input unchanged). + const searchSessionId = resolveRootSessionId(toolContext.sessionID); + // Only search message history up to the last compartment boundary — // anything after that (the live tail, including the current turn) is // still in context and already visible to the agent. When NO compartment @@ -138,13 +162,13 @@ function createCtxSearchTool(deps: CtxSearchToolDeps): ToolDefinition { // the live tail and must be excluded. A negative sentinel here would mean // "search everything" and leak the current prompt back to the agent — the // exact opposite of the intent (issue #131). - const lastCompartmentEnd = getLastCompartmentEndMessage(deps.db, toolContext.sessionID); + const lastCompartmentEnd = getLastCompartmentEndMessage(deps.db, searchSessionId); const messageOrdinalCutoff = lastCompartmentEnd >= 0 ? lastCompartmentEnd : 0; // Hard-filter memories already rendered in . // They're visible in message[0], so returning them wastes output // tokens and crowds out high-signal raw-history hits. - const visibleMemoryIds = getVisibleMemoryIds(deps.db, toolContext.sessionID); + const visibleMemoryIds = getVisibleMemoryIds(deps.db, searchSessionId); // Resolve the session's actual project from `toolContext.directory` // each call. OpenCode's top-level `ctx.directory` (the launch dir) @@ -160,36 +184,30 @@ function createCtxSearchTool(deps: CtxSearchToolDeps): ToolDefinition { const gitCommitsEnabled = embeddingSnapshot?.gitCommitEnabled ?? deps.gitCommitsEnabled ?? false; - const results = await unifiedSearch( - deps.db, - toolContext.sessionID, - projectPath, - query, - { - limit: normalizeLimit(args.limit), - memoryEnabled, - embeddingEnabled, - embedQuery: async (text, signal) => { - const result = await embedTextForProject( - projectPath, - text, - signal, - "query", - ); - return result?.vector ?? null; - }, - isEmbeddingRuntimeEnabled: () => embeddingEnabled === true, - readMessages: deps.readMessages, - maxMessageOrdinal: messageOrdinalCutoff, - gitCommitsEnabled, - sources: normalizeSources(args.sources), - visibleMemoryIds, - // Explicit agent search → enable literal-probe multi-query - // recall for symbol/command/path lookups. Auto-search hints - // (the hot path) leave this off to protect their latency. - explicitSearch: true, + const results = await unifiedSearch(deps.db, searchSessionId, projectPath, query, { + limit: normalizeLimit(args.limit), + memoryEnabled, + embeddingEnabled, + embedQuery: async (text, signal) => { + const result = await embedTextForProject(projectPath, text, signal, "query"); + return result?.vector ?? null; }, - ); + isEmbeddingRuntimeEnabled: () => embeddingEnabled === true, + readMessages: deps.readMessages, + maxMessageOrdinal: messageOrdinalCutoff, + gitCommitsEnabled, + sources: normalizeSources(args.sources), + visibleMemoryIds, + // Explicit agent search → enable literal-probe multi-query + // recall for symbol/command/path lookups. Auto-search hints + // (the hot path) leave this off to protect their latency. + explicitSearch: true, + // External bank resolution: basename is the human-readable label the + // engine uses as a bank template parameter, NOT a key. Project + // identity (resolveProjectPath's output) is the key. + // isExternalSearchEnabled() is module-level so no override is needed. + projectName: toolContext.directory ? basename(toolContext.directory) : undefined, + }); return formatSearchResults(query, results); }, diff --git a/packages/plugin/src/tools/ctx-search/types.ts b/packages/plugin/src/tools/ctx-search/types.ts index cd077344e..fbca1db31 100644 --- a/packages/plugin/src/tools/ctx-search/types.ts +++ b/packages/plugin/src/tools/ctx-search/types.ts @@ -2,8 +2,11 @@ import type { Database } from "../../shared/sqlite"; /** Sources the agent can narrow ctx_search to. Facts are intentionally NOT a * source — they're always rendered in in message[0], so - * searching them returns content already visible in context. */ -export type CtxSearchSource = "memory" | "message" | "git_commit"; + * searching them returns content already visible in context. External is a + * long-term knowledge recall channel; it's only fired on explicit + * `ctx_search` calls (not the auto-search hot path) and only when the + * external memory backend is configured. */ +export type CtxSearchSource = "memory" | "message" | "git_commit" | "external"; export interface CtxSearchArgs { query: string; diff --git a/packages/plugin/src/tui/index.tsx b/packages/plugin/src/tui/index.tsx index 3317f637f..3db7a6a2a 100644 --- a/packages/plugin/src/tui/index.tsx +++ b/packages/plugin/src/tui/index.tsx @@ -405,6 +405,45 @@ const StatusDialog = (props: { api: TuiPluginApi; s: StatusDetail }) => { {s().lastDreamerRunAt && ( )} + {/* External memory backend (Hindsight) — only when the + provider is configured. Mirrors the text-mode + executeStatus section: provider/endpoint, circuit + breaker state, this session's recall state, and the + best-effort server-side failed-retain count. */} + {s().externalMemory && (() => { + const em = s().externalMemory! + return ( + + + External Memory + + + {em.endpoint && ( + + )} + + + {em.failedRetainCount != null && ( + 0 ? t().warning : t().textMuted} + /> + )} + + ) + })()}