fix(e2e): set busy_timeout in memory-injection seedMemory - #4
Closed
codeo1io wants to merge 153 commits into
Closed
Conversation
docs.cortexkit.io is now a shared subdomain — sibling CortexKit plugin docs will live alongside under their own base paths. Root redirects to /magic-context/ until a hub page exists. Theme: starlight-theme-obsidian (graph/backlinks off). Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
Body links get the /magic-context base via a remark plugin so authors keep writing site-absolute paths; hero frontmatter links carry the base explicitly. Discord invite corrected to the canonical README one. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
The flat VERBATIM_PROBE_BONUS=0.5 sat 30x above the RRF scale (max list contribution 1/60), so every verbatim hit saturated; divide-by-max then flattened all message scores into a ~0.95-1.0 band, and at the unified layer those ~1.0 scores x MESSAGE_SOURCE_BOOST crowded memories out of the results entirely. Verbatim containment is now worth one rank-0 list appearance (1/RRF_K) of the best matching probe, probe lists are weighted by a smooth document-frequency falloff (a probe matching 2% of the corpus carries a third of a rare probe's signal - the 'AFT' acronym-flood case), and fused results map onto the same linear 0..1 band the single-query path emits so cross-source scales stay comparable. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
The shared DB can be migrated to v34 by a plugin process while the dashboard stays open; caching 'false' permanently would hide workspaces until an app restart. Only the true verdict is cached (schemas never un-migrate); not-ready re-probes on each call. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
The create row and workspace list added their own 20px inline padding on top of .scroll-area's 0 20px, indenting content 40px relative to every other page's content edge. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…dings superseded it Chunk embeddings (per-compartment raw [ordinal] U:/A: vectors) became ctx_search's semantic substrate, leaving p1_embedding (the per-compartment SUMMARY vector) with zero readers — no SELECT of p1_embedding exists in either package. It was still computed on every historian publish and re-computed by four recomp/upgrade blocks, a second embedding call per publish that hammered local endpoints for nothing. Removed embedAndStoreCompartments and all five call sites (incremental publish, full/partial recomp, Pi historian). The p1_embedding column is left inert (no migration mid-cycle); dreamer v2 decides whether to repopulate it for cross-compartment linking or drop it. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…am timer A bounded chunk-embedding batch per 15-min tick is a slow, bursty trickle that repeatedly hammers local embedding endpoints (LM Studio etc.) while never finishing a large historical corpus quickly. New compartments still embed on publish; historical backfill moves to an on-demand command (/ctx-embed-history) so the user controls when the endpoint is hit. The backfill helpers stay exported for that command. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…ddings on demand Replaces the timer-driven trickle (removed in the prior commit) with a user-initiated command that embeds ALL of one session's un-embedded compartments in a single pass, oldest-first. Reuses the recomp progress surface (sidebar + status bar) with a new kind="embed" so OpenCode shows a live N/M bar; Pi reports a single completion message (no progress sidebar). Core: embedSessionCompartmentChunks (shared by both harnesses) runs under the per-project embedding coordinator lease, yields between batches so the multi-core MiniLM burst stays interruptible, and is idempotent/resumable via chunk_hash. Session-scoped candidate loader + count added to compartment-chunk-embedding; the candidate-embed core is factored out of the project sweep so both paths share it. For users with no external provider, the in-process local MiniLM model embeds a large session in ~2 minutes; for external providers this gives explicit control over when the endpoint is hit. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
Three operational risks in the unbounded session drain: - Lease expiry mid-run: the run holds the coordinator lease with no per-run cap and could exceed the 5-min TTL, letting a sibling/passive sweep acquire the 'expired' lease and duplicate work. Added a renewal interval (cleared in finally). - Unbounded provider payload: batchSize capped compartments, not windows, so one large compartment (or big batch) could build an enormous padded tensor / JSON body. Provider calls are now sub-batched by window count (MAX_WINDOWS_PER_EMBED_CALL=16); a compartment is never split across calls. - Abort only between batches: threaded AbortSignal through embedCandidateChunkBatch into embedBatchForProject and re-checked before persist. Also: the no-progress break could falsely return 'done' with embeddable candidates still pending. Now distinguishes no-work skips (empty canonical text / already-current — excluded from re-selection so they can't block the oldest-first cursor) from provider failures, returning a new 'stalled' outcome with a retry hint surfaced in both harness commands. Stale p1 comments in recomp updated. +2 tests (window cap across compartments, provider-failure stall). Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…ilter leak, TOCTOU) Three Oracle audits on the workspaces + dashboard work. Real fixes: - HIGH (cache): mustMaterialize keyed the HARD memory gate only on the CURRENT pass's isWorkspaced, so a cached workspaced m[0] whose session left its workspace fell through to the integer-epoch compare and could keep rendering the stale union. Now compares the workspace fingerprint whenever EITHER cached or current is non-null (matches renderM1 soft- refresh and mustMaterializePi). - HIGH (dashboard): an empty/zero-member workspace filter omitted the IN clause and surfaced ALL memories/stats under that workspace (bulk archive/delete hazard). A requested filter that resolves to zero paths now forces an empty result in both get_memories and get_memory_stats. - MED (dashboard TOCTOU): workspace member fan-out for memory-status changes was resolved BEFORE BEGIN IMMEDIATE; a concurrent add-member could leave a new member's epoch un-bumped. Moved the resolution inside the write transaction (single + bulk paths). - LOW: rename_workspace bumped every member epoch though the workspace name is never rendered (spurious hard fold) — removed. ctx_memory archive now de-dupes ids (both harnesses) so [42,42] queues one row. Accepted-by-design (documented A41-A44): dashboard archive rides the supersede-delta not an epoch bump; foreign-member embeddings aren't backfilled from a member session; m[1] new-memories uses the flat trim; ctx_search RRF band parked pending message-embedding recall. Tests: workspace→single fold regression, empty-workspace zero-result, rename no-fold. Plugin 1997, Pi 441, dashboard 118. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…y on github-copilot stripStructuralNoise/stripClearedReasoning/stripProcessedImages/ dropStaleReduceCalls replace parts with empty-text sentinels assuming OpenCode drops them before the wire. OpenCode only does that for @ai-sdk/anthropic + @ai-sdk/amazon-bedrock — github-copilot (@ai-sdk/github-copilot) forwards the empty-text part as a real content block. When it lands after a tool_use (e.g. a step-finish that natively converts to nothing) it breaks Anthropic's tool_use/tool_result adjacency after copilot's server-side Bedrock re-translation: 400 on a FRESH session. Gate all four empty-sentinel producers on modelAcceptsEmptyContent (anthropic-only). Resolve the provider ONCE (reusing the budget path's map+cold-only DB recovery — no new DB read) and share it across the main transform AND postprocess so cold and hot passes make the same decision (step-start is a message-boundary marker in AI SDK 6.0.168: native vs sentinel diverge on the wire, so a map-only gate would bust a warm anthropic cache on the first hot pass post-restart). The stale-reduce gate covers BOTH frozen-id replay AND persistence so a prior anthropic pass's persisted ids can't re-sentinelize on a later copilot pass. Pi has no makeSentinel wire path (PARITY.md noted). Anthropic behavior byte-unchanged. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…ent drop starvation In an active session the protected tail's newest message changes every turn, so a boundary snapshot captured at trigger time fails validation on the "last ordinal id changed" check by the time the historian runs — even though the eligible head it would compact is untouched. The runner no-op'd forever while the trigger refired each turn, queued drop ops accumulated (observed: 27 consecutive stale no-ops, 179 pending drops, zero reduction). - compartment-runner-incremental: on a stale_snapshot validation result, re-resolve the boundary once from current session state and adopt the fresh snapshot when it still exposes a runnable head. The refreshed snapshot recomputes protectedTailStart/eligibleEnd from live messages, so the protected-tail guarantee is preserved (the head can never include a message now in the live tail). Historian makes real progress, publishes, and queues drops so the accumulated pending ops drain. - startCompartmentAgent: a synchronous runner no-op cleared compartmentInProgress but left the activeRuns registration alive until a microtask, so the same transform pass deferred queued drop ops for a run that already finished. Signal onHistorianRunStarted at the real-run commit point and drop the registration synchronously when the runner no-ops. - Regression tests: stale-tail snapshot re-derives + publishes instead of no-op'ing; synchronous no-op clears the active-run belief in the same pass. Co-authored-by: tracycam <gtracycam@gmail.com> Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…rce attribution Two Pi-only parity gaps from a workspaces council audit: - cachedPiRowMatchesSnapshot compared the session's own projectMemoryEpoch but NOT the workspace fingerprint, unlike OpenCode's cachedRowMatchesState. A FOREIGN workspace member's epoch bump changes the fingerprint without touching this session's epoch, so a sibling-materialized row under different membership could pass the CAS and be adopted with the wrong union baseline. Added the fingerprint compare (both sides already carry it in markers). - Pi ctx_search memory line omitted the source= attribution OpenCode emits, so a foreign member's memory wasn't labeled with its origin project. Verified mustMaterializePi already checks the fingerprint symmetrically (so a membership change is a correct HARD fold, not a soft-refresh race — the council's HIGH #2 is not exploitable). Documented the two post-ship edge cases (v22 rekey fan-out, epoch-0 fingerprint collision) as A45/A46. Pi 441/0. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…, dry-run Four problems from the cortexkit#144 report, all in the setup wizard: - ROOT CAUSE of 'preselected list, none available for me': pi-helpers parseModelListOutput required each token to contain '/', but `pi --list-models` prints a fixed-width TABLE (provider + model in separate columns, no slash). It matched ZERO rows and fell back to a hardcoded STATIC_MODELS list the user didn't have. Rewrote the parser to join the provider/model columns (skips header, keeps forward-compat with a pre-joined first token). Verified: 0 -> 22 real models live. - Removed the per-role recommendation tree (buildModelSelection's hardcoded 'recommended' ids) in BOTH harnesses; replaced with a shared model-picker that shows the user's FULL sorted catalog. - 'type the model name to pick': new selectAutocomplete (clack autocomplete) — a scrollable visible list you also narrow by typing. Used for every model pick. Each role first shows a short explanation of what it does and that it does NOT need a frontier model (smaller/cheaper is fine). - Pi dreamer-after-no bug: chooseModel ran unconditionally even when the user declined the dreamer. Now gated on dreamerEnabled (matches OpenCode). Plus: --dry-run for setup (both harnesses) — runs the full interactive wizard but writes no files / registers no package, printing '[dry-run] would write …'. Verified both flows end-to-end via PTY. Tests: pi-helpers table-parser suite, dreamer-decline regression, updated setup-pi expectations for the no-recommendation-tree behavior. CLI 151/0. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
A content change to ARCHITECTURE.md/STRUCTURE.md was a HARD mustMaterialize trigger, so any on-disk docs change folded m[0] and busted the entire cached prefix. In production this fired every time the dreamer's maintain-docs task rewrote the docs — a real m[0] bust for every active session on that project (confirmed live: cached prefix 599,885B → 28,187B, reason=project_docs_hash). Project docs are slowly-changing reference material living in m[0]; they now follow the 'Deliberately NOT triggers' pattern (like new compartments / additive memories): ride along and fold into m[0] on the NEXT natural hard bust, never force one. Removed projectDocsHash as a DECISION input at 6 sites (both harnesses): mustMaterialize, the m[1] Phase-3 contention stale-check, and the sibling CAS. KEPT computed-at-fold + stored, so a natural fold always reads fresh docs (readProjectDocsCanonical) and persists the hash matching the bytes it rendered. CAS removal is safe because the byte compare runs first — a byte-different m[0] still rejects; only docs-hash-only drift with identical bytes now matches. Also fixes analyze-cache-busts.ts: verdict now compares the cached prefix against the PREVIOUS request's last breakpoint (the old check used the current request's moved-tail breakpoint and stamped normal tail growth as BUST); timestamps include the date (multi-day dump sets were ambiguous). Plugin 2010/0, Pi 445/0. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
Co-authored-by: Ren yiwei <85666259+SSDWGG@users.noreply.github.com> Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
A workspace governs WHICH memory categories its members share to each other. A member session renders its OWN project's memories in all 5 categories, but FOREIGN members' memories only in the workspace's share_categories (new workspaces.share_categories JSON column, default ["CONSTRAINTS"]). [] = foreign share nothing; absent column (pre-v35) = all. Cache-stability core (m[0]/m[1]): - One shared SQL predicate (buildWorkspaceMemorySqlFilter) on ALL union readers — baseline getMemoriesByProjects, m[1] delta readNewMemoriesForM1Union, watermark getMaxMemoryIdForProjects, AND the FTS union — at EVERY call site incl. both OpenCode marker sites (snapshot + Phase-3) and all Pi marker/render sites, so a hidden foreign category can't render in one path while advancing another path's cursor. - ownIdentities = anchor + its v22 aliases (not just [projectIdentity]), so an own-project legacy-alias memory is never wrongly category-filtered. - A share_categories change folds m[0] exactly once: hashed into computeWorkspaceEpochFingerprint (null -> "ALL"), flowing into mustMaterialize + CAS on both harnesses, then byte-stable. - No-workspace sessions are a provable no-op (single-identity fast paths unchanged). resolveWorkspaceShareCategories fails open to null on parse error. Schema: migration v35 (LATEST_SUPPORTED_VERSION 35) — CREATE TABLE IF NOT EXISTS (fresh-safe) + columnExists-guarded ADD COLUMN (upgrade-safe) + NULL backfill + one-tx epoch fan-out over existing workspace members; fresh DDL + ensureColumn carry the default. Pi parity. plugin 2015/0, Pi 446/0. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
Replaces per-action immediate writes (each ran its own epoch fan-out -> one m[0] fold per click) with a staged editor + one batch command. - New apply_workspace_changes Tauri command: one IMMEDIATE tx that builds the FINAL member map in memory (old union adds minus removes), validates display- name uniqueness / membership against THAT final state (so 'remove A, add B reusing A's freed name' passes), and runs CONDITIONAL epoch fan-out over old union new ONLY when membership, member display-names, or share_categories changed — rename-only / no-op Saves bump no epochs. Removes the old add/remove/set-name single-action commands (UI was their only caller). - 5 'Shared categories' checkboxes (default CONSTRAINTS) + share_categories written as a canonical JSON array matching the plugin's V2 category set. - Readiness bumped to v35 + share_categories column check; the card hides entirely pre-column (never writes NULL sharing). Degrades, never migrates. - Staged-editor card uses <Index> (not <For>) for member rows + checkboxes to avoid the prior mid-edit reactivity bug; Save gated on dirty, Discard resets. cargo check + 29 Rust tests + staging unit tests + bun check all green. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
A Pi session overflowed (Codex context_length_exceeded) while MC reported 68%. Root cause: Pi sources pressure from lastContextPercentage persisted at message_end (end of the whole turn), which stays FROZEN during a long multi-step tool-heavy turn while the assembled request balloons past the window — so nothing trips the execute/85%/95% thresholds and the request overflows. OpenCode never hits this: its message.updated fires per-step at step-finish (confirmed against OpenCode source), so its pressure climbs mid-turn and it sheds before overflowing. Fix (Pi-only): floor pressure with ctx.getContextUsage().tokens, which is FORWARD (last assistant usage + estimateTokens of every trailing message, recomputed from the live array each pass) — approximating OpenCode's per-step climb. Shared applyForwardPressureFloor helper applied in BOTH the scheduler pressure block AND maybeFireHistorian (the historian trigger independently re-read the same stale persisted pressure, so it needed the floor too). Keeps only .tokens (forward, input-side); .percent is discarded (counts output on Pi's denominator). Scales the LIMIT (×0.85 FORWARD_PRESSURE_LIMIT_FACTOR) for the forward percentage to compensate estimate-token undercount — does NOT mutate the real limit (history-budget/emergency-ceiling rely on it) and passes the raw forward tokens onward (emergency drop needs the true assembled size). max() so it never lowers; null/missing forward usage = today's behavior. Emergency-recovery bump changed to Math.max(_,95) so it can't cap a higher forward reading. Immune to the NULL token_count tag undercount (uses the live message array, not the tag store). Emergency drops stay cache-stable: same-sample force passes are latched, fresh same-turn growth re-triggers (intended), no-candidate force leaves wire bytes unchanged — all locked by tests. PARITY.md documents the divergence. Pi 455/0, plugin 2015/0 (OpenCode untouched). Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…version-manager shims The Pi config page showed empty model comboboxes for users whose pi (or opencode) binary is installed by a version manager (mise/nvm/fnm/volta/asdf): those write the binary under a per-version dir (e.g. ~/.local/share/mise/installs/node/VER/bin/pi) that the dashboard's hardcoded candidate paths can't enumerate, and the Tauri GUI doesn't inherit the shell PATH so the bare "pi"/"opencode" candidate also fails. get_available_pi_models then returned [] -> empty dropdowns. The table parser and the App.tsx -> getAvailablePiModels -> ConfigEditor wiring were both correct; only binary discovery failed. Fix: - Add version-manager shim dirs (mise/asdf/volta) to the candidate list. - Add a login-shell fallback (run_via_login_shell): when the candidates miss, run the tool through the user's login shell, which resolves it exactly as the user's terminal does. Bounded by an 8s timeout so a slow shell rc can't hang the dropdown; unix-only (gated cfg unix); skipped when SHELL is unset. - Treat an empty parse from a found binary as "keep trying" so a stub on PATH doesn't shadow a working install. Applied to BOTH pi and opencode discovery (same latent bug). Verified E2E that the login-shell command in a GUI-like stripped env resolves the mise pi and returns the model table. cargo check + 46 Rust tests green. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
- Shared-categories checkboxes used raw browser defaults that clashed with the dark theme and aligned poorly with their labels. Reuse the existing custom checkbox styling (.tri-checkbox / .memory-card-checkbox) by extending its selectors to .share-category-label inputs, and add a flex label class for proper checkbox-to-text alignment. - Save changes / Discard are now HIDDEN until there are unsaved edits (were rendered disabled-but-looking-active). This also disambiguates Discard from Delete: Discard only appears next to Save when there are unsaved edits to revert, while Delete (remove the workspace) is always shown — the two no longer sit together at rest. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
A processed-screenshot user message lost its image blocks on a DEFER pass, collapsing the Anthropic cached prefix mid-conversation (verified on the wire: message[48]'s two base64 PNGs present at 18:58:11, gone at the 18:58:31 defer pass → cachedPrefix dropped to that message). Root cause: stripProcessedImages sentinelized images whenever a message's maxTag crossed the live watermark. The empty sentinel is filtered off the Anthropic wire, so that FIRST strip removes the image blocks — a real byte change. Keying it on the live watermark let tail growth advance the boundary across an older image message on a defer pass (which must replay byte-identically), busting the cache. Fix mirrors the adjacent dropStaleReduceCalls frozen-id pattern exactly: - DETECT (cache-busting passes only): find newly-aged processed-image messages, strip them, return their ids. - REPLAY (every pass incl. defer): re-strip only already-frozen ids, byte-identical regardless of how the live array grew. - Frozen id set persisted in session_meta.processed_image_stripped_ids via CAS-merge helpers (sibling-process safe), cleared with the session row. OpenCode-only (Pi intentionally does not strip images). +1 regression test asserting a defer pass never first-strips an aged image; existing defer/replay integration tests updated to the freeze-then-replay contract. plugin 2016/0. Also harden analyze-cache-busts.ts: default to printing ONLY bust rows (+ a bust-count summary), so a real prefix bust is never buried under ordinary tail-growth STABLE rows; --all-rows restores the full table. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…ode) When m[0] is going to HARD-fold this pass (epoch / model / system-hash / ttl-idle / mutation-id / upgrade — whatever mustMaterialize decides), the Anthropic prefix is being re-cached regardless. Previously the drop decision (shouldApplyPendingOps / shouldRunHeuristics, computed at L236/L265) didn't know the pass was about to bust m[0] (discovered later inside injectM0M1), so a hard fold landing on a sub-threshold defer pass busted the whole prefix while reclaiming NOTHING, then a later execute pass busted AGAIN to drain the queued tool drops. Two busts where one would do (wire-confirmed: workspace-delete epoch fold re-billed 611K tokens, then +83s a 2068-op drop pass busted again). Fix: compute the existing mustMaterialize ONCE early (advisory) → m0HardFoldThisPass, and OR it into shouldReadPendingOps / shouldApplyPendingOps / shouldRunHeuristics so a known-bust pass drains the drops into the SAME bust. Council-reviewed (bg_0f5c5832); four corrections folded: - ADVISORY-ONLY: the early call only WIDENS the gates. injectM0M1 keeps its own independent late mustMaterialize recheck — early-false never suppresses it, so a cross-process (dashboard) epoch/mutation bump arriving after the early read still folds. Correctness is never worse than today; cost is one extra mustMaterialize (indexed reads + a cached docs stat). - SEPARATE OR-TERM: never folded into materializationRequested (which drives the lastResponseTime TTL reset + pendingMaterialization cleanup) — folding in would suppress those and oscillate. - Reuses mustMaterialize itself, so the HARD trigger set can't drift (project_docs_hash is correctly NOT a trigger). - Respects the existing compartment-run gate (no new bypass): a hard fold while a historian runs at <85% defers the drain to the next pass — still ≤ today. +2 integration tests: a model-key change on a DEFER scheduler pass drains a queued drop; an unchanged-marker defer pass leaves it queued. plugin 2018/0. Pi parity follows in a separate commit. Design doc: .alfonso/plans/cache-fold-and-image-strip-review.md Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…rity) Pi mirror of 4ce3d73. When Pi's m[0] will HARD-fold this pass (mustMaterializePi: model / system-hash / ttl-idle / project-memory epoch / mutation-id / upgrade), the Anthropic prefix re-caches regardless, so drain the queued tool drops + run heuristics into THAT bust instead of busting again on a later execute pass. Hoisted the piHardSignals block above the exec gates; computes an advisory m0HardFoldThisPass via an early mustMaterializePi (gated on args.injection, its own getCompartments snapshot). injectM0M1Pi keeps its OWN authoritative late mustMaterializePi recheck unchanged — early-false never suppresses it (a cross-process epoch/mutation bump after the early read still folds late). The late injectM0M1Pi reuses the SAME hoisted piHardSignals const, so the advisory and authoritative hard signals are byte-identical. m0HardFoldThisPass is a separate OR-term into shouldRunHeuristics + baseShouldApplyPendingOps only — never folded into the deferred/explicit materialization signals; deferred-drain and compaction-marker ordering untouched. Full parity, no PARITY.md divergence. +2 Pi integration tests (model-key change on a DEFER pass drains a queued drop; unchanged-marker defer leaves it queued). Pi 457/0, plugin 2018/0. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…A + workspaces) Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
…-shared categories The memory-mutation tools (update/archive/merge) gated visibility on workspace MEMBERSHIP only, never on the share-category policy. So an agent in project A could update/archive a memory belonging to foreign member B in a category B does NOT share with the workspace — a memory the agent cannot even SEE in its rendered <project-memory> (the render path filters foreign memories by shareCategories, the tool did not). Tool visibility now mirrors render visibility exactly: own-project memories are mutable in every category; foreign member memories only when shared (shareCategories===null shares all, [] shares none, otherwise only the listed categories) — the same own/foreign split as buildWorkspaceMemorySqlFilter. Both harnesses. +3 regression tests (foreign non-shared category refused, foreign shared allowed, own always allowed). plugin 38/0 in the ctx-memory suite. Co-authored-by: Alfonso [Magic Context] <288211368+alfonso-magic-context@users.noreply.github.com>
Bumps [ws](https://github.com/websockets/ws) to 8.21.0 and updates ancestor dependency [@remotion/cli](https://github.com/remotion-dev/remotion). These dependencies need to be updated together. Updates `ws` from 8.17.1 to 8.21.0 - [Release notes](https://github.com/websockets/ws/releases) - [Commits](websockets/ws@8.17.1...8.21.0) Updates `@remotion/cli` from 4.0.462 to 4.0.481 - [Release notes](https://github.com/remotion-dev/remotion/releases) - [Commits](remotion-dev/remotion@v4.0.462...v4.0.481) --- updated-dependencies: - dependency-name: ws dependency-version: 8.21.0 dependency-type: indirect - dependency-name: "@remotion/cli" dependency-version: 4.0.481 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…r-config-only Engine-agnostic external memory config (provider/endpoint/banks/retain_sources/ tags) + recall sub-block (timeout, per-slice budget, dedup threshold, global_tags, global_from_prompt, search, mental models). Whole block stripped from project-level config (a hostile repo must not redirect a user's memory store). Schema asset regenerated.
Neutral backend interface (retain/recall/remove/mentalModels) with a Hindsight implementation: bank routing (per-project banks + main bank), idempotent content-hash document_ids, circuit breaker, SSRF guard, 422 no-retry, benign-404 read paths, mental-model seeding after first retain. All three durable creation points tee fire-and-forget (historian promotion, ctx_memory write, dreamer user-memory promotion); origin provenance on global-scope items (origin-* tags + extraction-context entity link). Never throws, never blocks a local write.
…er, not memory.enabled Compartment embeddings are the ctx_search substrate, not memory-store data. embedding-on/memory-off users now get search embeddings + project registration; promotion stays gated by memory flags (issue cortexkit#44 preserved).
…safe m[0]/m[1] injection Once-per-session 3-slice recall (project/profile/global; mental-model fast path; optional first-prompt-enriched global query), deduped against local rows (cosine + hash fallback), trimmed, sorted, frozen into session_meta (migration v33 — renumbered from pre-v0.23 v31). Rendered as <external-memory> in m[0] with marker-hash m[1] delta for late arrivals; NOT a mustMaterialize trigger; excluded from the m[1] pressure-refold math; A-path await bounded and first-render-only. Defer passes replay frozen bytes.
…e writes Local memory truth signals propagate to the external store: archive (incl. v0.23.0 batch archive — one batched DELETE per call) removes documents by original-content document_id; update removes the stale doc and tees the corrected fact; NEW dreamer-only verify action records local verification and verbatim-upserts (refreshes engine recency, verifiedAt metadata); user-memory dismiss/update propagate so the profile recall slice cannot resurrect dismissed memories. merge and memory-migration deletes deliberately propagate NOTHING. ctx_memory write gains scope="global": main-bank-only tee (no local row) with origin-project provenance. Removes/upserts bypass retain_sources; the update re-tee respects it.
Provider/endpoint/circuit-state/session-recall-state plus a best-effort failed-retain count from the operations endpoint (benign-404, breaker-safe, null on any failure). executeStatus goes async; both harness call paths await it.
…for read tools session.created parentID registrations (bounded LRU, cycle-safe walk); ctx_search and ctx_expand resolve child sessions (sidekick, task subagents) to the ROOT conversation so message-history search, the visible-memory filter, and the injected-external-recall filter target the session that actually owns that state. Mutating tools stay child-scoped. Sidekick registers synchronously at spawn (event-race belt-and-braces).
Renders the StatusDetail.externalMemory payload that buildStatusDetail already ships over RPC: provider, endpoint, circuit-breaker state (warning color when not closed), this session's recall state, and the server-side failed-retain count. TUI parity with the text-mode /ctx-status section.
isMagicContextEntry only matched the npm package name (opencode-magic-context), so a dev entry like file:///…/magic-context/packages/plugin was invisible to the detector and ensureTuiPluginEntry re-appended @latest on every startup, double-loading the plugin. Match magic-context as a whole path segment too.
When opencode runs inside the user config directory itself
(~/.config/opencode as project root), project-config discovery
resolves to the same magic-context.jsonc as the user config. It was
loaded a second time as untrusted repo config, so the project
security strips fired against the user's own file: {file:} tokens
left literal and memory.external dropped — silently disabling the
external memory backend. Skip the project load when its resolved
path equals the user config path.
…gration fence, search merge
…on test Previous filter admitted ALL stored vectors when queryModelId was falsy or 'off', producing meaningless cosine scores across different embedding spaces. Fix: only populate localVectors when queryModelId is known and not 'off'; otherwise fall back to hash-only dedup. Regression tests: hash-dedup baseline (integration), model-guard unit tests for off/empty queryModelId and known queryModelId with two stored model IDs.
…al parity
- index.ts: call initializeExternalMemory(config.memory?.external) at
startup (mirrors OpenCode). Config flows through the shared
MagicContextConfig schema which already includes memory.external.
- ctx-memory.ts: mirror OpenCode's external wiring:
- write (project scope): tee to external backend after local insert
- write (global scope): external-only write, no local row; gated on
getExternalMemoryStatus()
- update: W2 corrective propagation (remove old + tee new)
- archive: removeFromExternalBackend for all archived memories
- verify (dreamer-only): upsertToExternalBackend to refresh recency
- scope param added to schema (project/global)
- verify added to ALL_ACTIONS and DREAMER_ONLY_ACTIONS
All external calls are fire-and-forget (void), never blocking local
mutations. memoryVisibleToTool gate respected (external calls go after
local mutation, same as OpenCode).
…rch source context-handler.ts: - Fire startSessionRecall on first context pass per session (mirrors OpenCode transform.ts loadedSessions gate). Extracts first user prompt text for global_from_prompt enrichment. - Call maybeAwaitExternalRecall before injectM0M1Pi when injection is enabled (A-path await: waits only when no cached m[0] exists, bounded by recall.timeout_ms). Mirrors OpenCode transform.ts timing. inject-compartments-pi.ts: - Add externalRecallHash to PiM0SnapshotMarkers (NOT a mustMaterializePi trigger — rides m[1] external delta only, invariant preserved). - Add cached_m0_external_recall_hash to CachedPiM0M1Row + SQL SELECT. - readFrozenM0InputsPi: read external recall snapshot in-transaction (TOCTOU-safe, same as projectDocsHash override in OpenCode). - renderM0Pi: accept externalRecallOverride param; merge profile slice into user-profile; render renderExternalMemoryBlock after project-memory. - renderUserProfileBlock: accept externalProfileLines param for profile slice merge (mirrors OpenCode renderUserProfileBlock signature). - materializeM0Pi: pass frozen.externalRecall to renderM0Pi; persist externalRecallHash in persistCachedM0. - renderM1PiWithMetadata: render external memory delta when live hash differs from m[0] baseline hash. - markersFromCachedPiRow: populate externalRecallHash from DB row. - readCurrentMarkersFromCompartments: read live externalRecallHash. ctx-search.ts: already wired (explicitSearch + projectName + external source in schema + formatResult). No changes needed.
Biome import ordering and formatting fixes for: - pi-plugin/src/context-handler.ts (import order + indentation) - pi-plugin/src/index.ts (import order) - pi-plugin/src/inject-compartments-pi.ts (import order + SQL formatting) - pi-plugin/src/tools/ctx-memory.ts (indentation) - plugin/src/.../external-recall.test.ts (import order + arrow fn)
…, STRUCTURE, CONFIGURATION, README) Adds the external-memory-backend feature to root docs: - ARCHITECTURE.md: new 'External-memory flow' subsection under Data Flow (tee-on-write → 3-slice recall + dedup + persist-before-settle → m[1] <external-memory> render → W2 correctives → ctx_search external source); new 'External memory backend (Hindsight)' Key Abstraction; cached_m0_ external_recall_hash marker added to the schema-migrations note (deliberately NOT a mustMaterialize trigger); project-config trust boundary updated to include memory.external in the user-only strip list; unified search entry updated with the external source. - STRUCTURE.md: new external-memory files listed under src/features/ plus a Key File Locations block (engine-agnostic provider, Hindsight impl, recall orchestrator, snapshot read, W2 hooks, project-security strip) and the Pi mirror path (@magic-context/core re-export). - CONFIGURATION.md: full memory.external block (provider, endpoint, api_key, project_bank template, main_bank, retain_sources, tags, recall sub-block) and the memory.external.recall defaults; full failure-semantics + cross-harness note; user-config-only security callout; example updated. - README.md: concise mention in the Recall section + a new callout for long-term memory; external added to the four-source ctx_search description and the agent-tools table.
…, not external-memory block
Three-layer fix for the recurring 'failed to load plugin @cortexkit/opencode-magic-context error="database is locked"' errors that crash opencode startup when multiple processes cold-open context.db concurrently. Root cause: during a concurrent cold open (two opencode processes booting, or a child session spawning while the parent is mid-checkpoint), the WAL writer lock is held by a sibling. The first read in openDatabase() — enforceSchemaFence() -> getPersistedSchemaVersion() — ran BEFORE busy_timeout was installed (it was set later inside initializeDatabase()), so SQLite returned SQLITE_BUSY immediately instead of waiting. Layer 1 (storage-db.ts): Set PRAGMA busy_timeout=5000 immediately after new Database(), before enforceSchemaFence()'s schema_migrations read. This was the actual race window. Layer 2 (dream-timer.ts): openTimerDatabaseOrNull() only handled openDatabase()'s null return (schema-fence path) but NOT its throw path (fatal open error). The throw propagated through startDreamScheduleTimer() -> plugin() -> opencode's plugin loader, which logged 'failed to load plugin' and disabled magic-context for the entire session. Now wrapped in try/catch. Layer 3 (index.ts): Defensive try/catch around the startDreamScheduleTimer() await in the plugin entry, so even if a future code path inside the timer setup throws, plugin() survives and the rest of magic-context (hooks, tools, RPC) still initializes.
…USY retry Replaces 9 hand-rolled BEGIN IMMEDIATE / COMMIT / finally-ROLLBACK blocks with a single shared runWriteTransaction helper that adds a bounded SQLITE_BUSY retry loop (4 attempts, 100ms backoff). This is a better concurrency solution than the previous per-site pattern because: 1. Centralized retry policy: transient SQLITE_BUSY from cross-process WAL writer lock contention (sibling migration, dreamer run, Channel-2 bulk delivery) now waits-and-retries instead of propagating up and disabling Magic Context for the run. Previously each site either swallowed, propagated, or retried inconsistently. 2. Composition-safe: if called inside an existing db.transaction() or another runWriteTransaction, the body runs inline WITHOUT issuing a nested BEGIN (SQLite doesn't allow nested BEGIN; the outer transaction already holds the writer lock). Detected via the bun:sqlite inTransaction flag or the node:sqlite shim's isTransaction flag. 3. One correct transaction plumbing pattern instead of 9 slightly-different copies. Several sites were missing the ROLLBACK path or had subtle ordering bugs at the edges. Sites converted (all sync write paths with hand-rolled BEGIN IMMEDIATE): - dreamer/lease.ts (acquireLease, renewLease, releaseLease) - git-commits/sweep-coordinator.ts (acquireGitSweepLease, renewGitSweepLease, markGitSweepSuccessAndRelease) - message-index.ts (indexMessagesAfterOrdinal — cross-process FTS dedup) - workspaces.ts (bumpEpochsForWorkspaceMembers, bumpEpochsForWorkspaceMemberSet) - compartment-storage.ts (replaceAllCompartmentStateAndBumpDepth, promoteRecompStaging) - key-files/project-key-files.ts (replaceAllKeyFiles) - key-files/identify-key-files.ts (commitKeyFilesUnderLease) - hooks/compartment-runner-recomp.ts (promoteRecompStagingWithM0Mutation) - hooks/compartment-runner-incremental.ts (historian publish path) inject-compartments.ts is intentionally NOT converted: its two BEGIN IMMEDIATE blocks have complex early-return-with-rollback-and-fallback-read patterns (Phase 3 materialization contention retry, soft-refresh cache replay) that don't map cleanly onto the shared helper without significant restructuring. The helper is still a net win across the 9 simpler sites. Verification: - bun run typecheck (tsc --noEmit + scripts) passes - bun run lint (biome check) passes - 12 new unit tests for runWriteTransaction/runWriteTransactionAsync pass (BEGIN/COMMIT, rollback on throw, transient BUSY retry, max-attempts give-up, nested-transaction composition, non-transient error passthrough) - All existing tests in the converted files' test suites pass: dreamer/lease.test.ts (7), git-commits (19), compartment-storage-atomic (8), compartment-storage-v6, compartment-lease, compartment-runner-recomp-fk (4), compartment-runner-partial-recomp, key-files (14), workspaces, storage-db (11), message-index Net: -199 lines / +40 lines across the 9 converted sites, +201 lines for the shared helper + 156 lines for its tests.
…ifest The external-memory-backend branch added the memory.external schema subtree (provider/endpoint/banks/retain_sources/tags + recall sub-block) but did not add a corresponding entry to the dashboard's config-field-coverage manifest. The config-parity guard (config-parity.test.ts) enforces that every schema leaf is either RENDERED by the ConfigEditor form or listed in OMITTED_BY_DESIGN, so CI failed with 16 uncovered leaves. memory.external is USER-config-only and operator-configured (no form widgets exist for it), so the correct classification is OMITTED_BY_DESIGN with the whole subtree covered by the 'memory.external' prefix.
The seedMemory helper opened context.db without setting PRAGMA busy_timeout, so under the live opencode process's concurrent writer (historian checkpoint or dreamer run) the seed INSERT could throw SQLITE_BUSY and fail the 'memory injection > injects <project-memory>' E2E test. This is the same contention pattern fixed in the plugin itself; the test's external write just wasn't opt-in to the retry window. Set busy_timeout=5000 immediately after open, matching the pattern already used by writeContextDb in cache-invariants.test.ts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Triggering CI on the latest commit (1abddf6) to validate the e2e busy_timeout fix. The real review artifact is iceteaSA PR #2.