refactor(migrations): renumber external-recall migration v37→v38 (skill_memory takes v37)

Author: Tehan

ARCHITECTURE.md

@@ -431,7 +431,7 @@ Magic Context runs in three effective modes depending on `ctx_reduce_enabled` an
 
 **Storage:** Use the SQLite database created by `src/features/magic-context/storage-db.ts` under the cortexkit data directory resolved by `src/shared/data-path.ts` (`~/.local/share/cortexkit/magic-context/context.db` on Linux/macOS, XDG-equivalent on Windows). Legacy OpenCode-plugin-folder DBs are migrated forward on first boot. The same DB is shared cross-harness between OpenCode and Pi; session-scoped tables include a `harness` discriminator (`'opencode'` / `'pi'`) while project-scoped tables (memories, git commits) are shared.
 
-**Schema migrations:** `src/features/magic-context/migrations.ts` declares versioned migrations v1–v37 (`LATEST_SUPPORTED_VERSION = 37` in `storage-db.ts` is the schema-fence ceiling and MUST be bumped with every new migration; a unit test — `schema-version-fence.test.ts` — asserts `LATEST_SUPPORTED_VERSION === LATEST_MIGRATION_VERSION` so the two can't drift). Notable: v10 `tool_owner_message_id` (composite tool-tag identity); v11 `todo_synthetic_*` (synthetic-todowrite); v12 orphan `memory_embeddings` cleanup; v13 `pending_compaction_marker_state` (deferred-marker drain); v14 project-scoped key files + version counter; v15 `deferred_execute_state` (boundary execution); v16 context-limit cache sentinels; v17 multi-anchor note-nudge/auto-search JSON storage; v18 `pending_pi_compaction_marker_state`; v19 compartment-state lease table; v20 subagent invocation token accounting; v21 session lifetime work metrics; **v22 the v2.0 cache-architecture foundation (m[0]/m[1] split tables, `project_state` epoch counter, plus per-compartment `p1`–`p4` tier columns, `importance`, `episode_type`, `p1_embedding`, and `legacy` flag); v23 `compartment_events` (historian-extracted causal_incident / trajectory_correction, stored-not-rendered in v2.0); v24 `historian_runs` telemetry (per-run chunk range, compartment/fact/event counts, importance min/max/avg, status + failure reason, FK to `subagent_invocations`); v25 `pi_stable_id_scheme` (Pi stable-id cutover watermark); v26 `memory_mutation_log` + `cached_m1_bytes` (memory supersede-delta — non-additive in-session memory mutations render as an m[1] `<memory-updates>` delta instead of bumping the project epoch, plus the frozen-m[1]-bytes cache column); v27 `tags.entry_fingerprint` (Pi fallback-tag adoption); v28 `git_sweep_coordinator` (lease/cooldown for cross-process git-commit sweeps); v29 `notes.anchor_ordinal` (note→conversation-tail traceback); v30 `cached_m0_system_hash` / `cached_m0_tool_set_hash` / `cached_m0_model_key` (HARD-bust m[0] markers — provider-side cache-eviction detection for the materialization taxonomy; the migration clears the m[0]/m[1] cache once so pre-v30 rows re-materialize cleanly); v31 ctx_reduce-nudge state (`last_nudge_undropped`, `channel2_nudge_state`, `last_emergency_input_sample` + startup heal zeroing legacy sticky/anchor nudge state); v32 protected-tail v3 boundary state + per-tag cached token counts (`tags.token_count` / `input_token_count` / `reasoning_token_count` — computed once on tag insert, summed for sidebar/boundary/nudge math); v33 `compartment_chunk_embeddings` table for cross-session semantic search across compartment windows; v34 `workspaces` / `workspace_members` tables plus `cached_m0_workspace_fingerprint` m[0] marker (with a one-shot m[0]/m[1] cache reset so pre-v34 rows re-materialize cleanly); v35 `workspaces.share_categories` default + epoch refresh for existing members; v36 `session_projects` ownership map + seed for pre-v36 embedded sessions; **v37 the external-memory v2 unified-read foundation: per-session `external_recall_json` / `external_recall_state` / `external_recall_at` columns on `session_meta` (the frozen post-dedup, post-trim snapshot every render replays for byte stability) and the `cached_m0_external_recall_hash` m[0] marker (a carried marker that drives the m[1] `<external-memory>` delta comparison only — deliberately NOT a `mustMaterialize` trigger; the migration is `ensureColumn`-idempotent so a dev DB that ran it under the pre-rename v31 number re-applies harmlessly).** Migration runner uses `schema_migrations` table with version-ordered execution and sibling-startup race protection (duplicate-insert is tolerated).
+**Schema migrations:** `src/features/magic-context/migrations.ts` declares versioned migrations v1–v38 (`LATEST_SUPPORTED_VERSION = 38` in `storage-db.ts` is the schema-fence ceiling and MUST be bumped with every new migration; a unit test — `schema-version-fence.test.ts` — asserts `LATEST_SUPPORTED_VERSION === LATEST_MIGRATION_VERSION` so the two can't drift). Notable: v10 `tool_owner_message_id` (composite tool-tag identity); v11 `todo_synthetic_*` (synthetic-todowrite); v12 orphan `memory_embeddings` cleanup; v13 `pending_compaction_marker_state` (deferred-marker drain); v14 project-scoped key files + version counter; v15 `deferred_execute_state` (boundary execution); v16 context-limit cache sentinels; v17 multi-anchor note-nudge/auto-search JSON storage; v18 `pending_pi_compaction_marker_state`; v19 compartment-state lease table; v20 subagent invocation token accounting; v21 session lifetime work metrics; **v22 the v2.0 cache-architecture foundation (m[0]/m[1] split tables, `project_state` epoch counter, plus per-compartment `p1`–`p4` tier columns, `importance`, `episode_type`, `p1_embedding`, and `legacy` flag); v23 `compartment_events` (historian-extracted causal_incident / trajectory_correction, stored-not-rendered in v2.0); v24 `historian_runs` telemetry (per-run chunk range, compartment/fact/event counts, importance min/max/avg, status + failure reason, FK to `subagent_invocations`); v25 `pi_stable_id_scheme` (Pi stable-id cutover watermark); v26 `memory_mutation_log` + `cached_m1_bytes` (memory supersede-delta — non-additive in-session memory mutations render as an m[1] `<memory-updates>` delta instead of bumping the project epoch, plus the frozen-m[1]-bytes cache column); v27 `tags.entry_fingerprint` (Pi fallback-tag adoption); v28 `git_sweep_coordinator` (lease/cooldown for cross-process git-commit sweeps); v29 `notes.anchor_ordinal` (note→conversation-tail traceback); v30 `cached_m0_system_hash` / `cached_m0_tool_set_hash` / `cached_m0_model_key` (HARD-bust m[0] markers — provider-side cache-eviction detection for the materialization taxonomy; the migration clears the m[0]/m[1] cache once so pre-v30 rows re-materialize cleanly); v31 ctx_reduce-nudge state (`last_nudge_undropped`, `channel2_nudge_state`, `last_emergency_input_sample` + startup heal zeroing legacy sticky/anchor nudge state); v32 protected-tail v3 boundary state + per-tag cached token counts (`tags.token_count` / `input_token_count` / `reasoning_token_count` — computed once on tag insert, summed for sidebar/boundary/nudge math); v33 `compartment_chunk_embeddings` table for cross-session semantic search across compartment windows; v34 `workspaces` / `workspace_members` tables plus `cached_m0_workspace_fingerprint` m[0] marker (with a one-shot m[0]/m[1] cache reset so pre-v34 rows re-materialize cleanly); v35 `workspaces.share_categories` default + epoch refresh for existing members; v36 `session_projects` ownership map + seed for pre-v36 embedded sessions; **v38 the external-memory v2 unified-read foundation: per-session `external_recall_json` / `external_recall_state` / `external_recall_at` columns on `session_meta` (the frozen post-dedup, post-trim snapshot every render replays for byte stability) and the `cached_m0_external_recall_hash` m[0] marker (a carried marker that drives the m[1] `<external-memory>` delta comparison only — deliberately NOT a `mustMaterialize` trigger; the migration is `ensureColumn`-idempotent so a dev DB that ran it under the pre-rename v31 number re-applies harmlessly).** Migration runner uses `schema_migrations` table with version-ordered execution and sibling-startup race protection (duplicate-insert is tolerated).
 
 **Harness-aware behavior:** `src/shared/harness.ts` exposes `setHarness()`/`getHarness()` for the runtime to identify itself; production INSERTs into session-scoped tables tag rows with the current harness. Pi-specific session-resolution paths are skipped on OpenCode and vice versa.
 

STRUCTURE.md

@@ -124,7 +124,7 @@
 - `src/features/magic-context/memory/external-recall-read.ts`: `readExternalRecallSnapshot` (sanitizeSlice validates per-item to fail safe on corrupted rows), `readExternalRecallHash`, `computeRecallSnapshotHash` (deterministic 16-char SHA-256 prefix of the JSON-serialized snapshot).
 - `src/features/magic-context/storage-db.ts`: Create durable storage; run versioned migrations; resolve runtime SQLite backend.
 - `src/features/magic-context/storage-meta-persisted.ts`: Read and write per-session persisted scalars and JSON blobs.
-- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v37 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep). v37 adds `external_recall_json` / `external_recall_state` / `external_recall_at` and the `cached_m0_external_recall_hash` m[0] marker.
+- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v38 (`LATEST_SUPPORTED_VERSION` in `storage-db.ts` must track the highest; `schema-version-fence.test.ts` asserts they stay in lockstep). v38 adds `external_recall_json` / `external_recall_state` / `external_recall_at` and the `cached_m0_external_recall_hash` m[0] marker.
 - `src/features/magic-context/message-index.ts`: FTS-backed raw-message index for `ctx_search`.
 - `src/features/magic-context/search.ts`: Unified retrieval over memories, raw messages, git commits, and (explicit-only) the external long-term memory backend (`SearchSource += "external"`; `searchExternal` runs project + main bank in parallel, content-hash-dedups, drops hits already in the session's `<external-memory>` block, respects a 5s `AbortSignal`).
 - `src/tools/ctx-memory/tools.ts`: W2 correctives (Pi mirror: `packages/pi-plugin/src/tools/ctx-memory.ts`) — `write` tees to external, `update` remove-then-tee cascade, `archive` batched remove, `merge` no-op, `verify` (dreamer-only) verbatim re-upsert. Also adds `scope: "global"` which routes ONLY to the external main bank with origin provenance and no local row.

packages/plugin/src/features/magic-context/migrations-v34.test.ts

@@ -78,13 +78,13 @@ describe("migration v34/v35 — workspaces", () => {
             expect(
                 db.prepare("SELECT share_categories FROM workspaces WHERE name = 'fresh'").get(),
             ).toEqual({ share_categories: '["CONSTRAINTS"]' });
-            expect(LATEST_SUPPORTED_VERSION).toBe(37);
-            expect(LATEST_MIGRATION_VERSION).toBe(37);
+            expect(LATEST_SUPPORTED_VERSION).toBe(38);
+            expect(LATEST_MIGRATION_VERSION).toBe(38);
             expect(
                 db
                     .prepare("SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1")
                     .get(),
-            ).toEqual({ version: 37 });
+            ).toEqual({ version: 38 });
         } finally {
             closeQuietly(db);
         }
@@ -215,7 +215,7 @@ describe("migration v34/v35 — workspaces", () => {
                 db
                     .prepare("SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1")
                     .get(),
-            ).toEqual({ version: 37 });
+            ).toEqual({ version: 38 });
         } finally {
             closeQuietly(db);
         }

packages/plugin/src/features/magic-context/migrations-v36.test.ts

@@ -29,13 +29,13 @@ describe("migration v36 — session project ownership", () => {
                 expect.arrayContaining(["session_id", "harness", "project_path", "updated_at"]),
             );
             expect(indexNames(db)).toContain("idx_session_projects_project");
-            expect(LATEST_SUPPORTED_VERSION).toBe(37);
-            expect(LATEST_MIGRATION_VERSION).toBe(37);
+            expect(LATEST_SUPPORTED_VERSION).toBe(38);
+            expect(LATEST_MIGRATION_VERSION).toBe(38);
             expect(
                 db
                     .prepare("SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1")
                     .get(),
-            ).toEqual({ version: 37 });
+            ).toEqual({ version: 38 });
         } finally {
             closeQuietly(db);
         }
@@ -58,7 +58,7 @@ describe("migration v36 — session project ownership", () => {
                 db
                     .prepare("SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1")
                     .get(),
-            ).toEqual({ version: 37 });
+            ).toEqual({ version: 38 });
         } finally {
             closeQuietly(db);
         }

…res/magic-context/migrations-v37.test.ts …res/magic-context/migrations-v38.test.tspackages/plugin/src/features/magic-context/migrations-v37.test.ts renamed to packages/plugin/src/features/magic-context/migrations-v38.test.ts packages/plugin/src/features/magic-context/migrations-v37.test.ts renamed to packages/plugin/src/features/magic-context/migrations-v38.test.ts

@@ -10,7 +10,7 @@ function columnNames(db: Database, table: string): string[] {
     );
 }
 
-describe("migration v37 — external recall snapshot + m[0] marker", () => {
+describe("migration v38 — external recall snapshot + m[0] marker", () => {
     test("adds external recall columns to session_meta on a fresh DB, idempotently", () => {
         const db = new Database(":memory:");
         try {
@@ -40,14 +40,14 @@ describe("migration v37 — external recall snapshot + m[0] marker", () => {
             runMigrations(db);
 
             db.prepare("INSERT INTO session_meta (session_id, harness) VALUES (?, ?)").run(
-                "ses_v37",
+                "ses_v38",
                 "test-harness",
             );
             const row = db
                 .prepare(
                     "SELECT external_recall_json, external_recall_state, external_recall_at, cached_m0_external_recall_hash FROM session_meta WHERE session_id = ?",
                 )
-                .get("ses_v37") as Record<string, unknown>;
+                .get("ses_v38") as Record<string, unknown>;
 
             expect(row.external_recall_json).toBeNull();
             expect(row.external_recall_state).toBeNull();

packages/plugin/src/features/magic-context/migrations.ts

@@ -1439,9 +1439,10 @@ const MIGRATIONS: Migration[] = [
     {
         // Was v31 on the pre-v0.23 external-memory-backend branch; renumbered
         // to v33 pre-v0.24-rebase, then to v37 when upstream v0.24 shipped its
-        // own v33/34/35/36. The body is ensureColumn-idempotent, so a dev DB
-        // that already ran it under an old number re-applies harmlessly.
-        version: 37,
+        // own v33/34/35/36, then to v38 when skill_memory took v37.
+        // The body is ensureColumn-idempotent, so a dev DB that already ran it
+        // under an old number re-applies harmlessly.
+        version: 38,
         description: "External memory v2: session recall snapshot + m[0] recall marker",
         up: (db: Database) => {
             // session_meta existence guard — see v30's comment (partial test fixtures).

packages/plugin/src/features/magic-context/storage-db.ts

@@ -36,7 +36,7 @@ export function getSchemaFenceRejection(): {
     return lastSchemaFenceRejection;
 }
 
-export const LATEST_SUPPORTED_VERSION = 37;
+export const LATEST_SUPPORTED_VERSION = 38;
 
 export interface OpenDatabaseOptions {
     dbPath?: string;