feat(mcp-server): persist async job registry across restarts (#237) (#251)

Co-authored-by: Roman Melnikov <roman@neftedollar.com>

Author: Neftedollar

bun.lock

@@ -47,15 +47,16 @@ decides how to reuse them and what the MCP tool surface looks like.
   progress/elicitation as today. In job mode, elicitation surfaces via
   `get_workflow_status.currentTask` — no out-of-band prompt because the
   originating client may be gone.
-- **Job registry is in-process.** Matches `@ageflow/server` today. An
-  interface hook (`RunStore`) is sketched for future persistent backends.
+- **Job registry defaults to in-process, with optional persistence.**
+  By default jobs use an in-memory store. With `--job-db <path>` the
+  registry persists snapshots to SQLite and hydrates known jobs on startup.
 
 ## Non-goals
 
 - **No distributed jobs.** `jobId` is valid only on the server that created
   it. Horizontal scale requires sticky routing — out of scope.
-- **No persistence across restarts.** Jobs live in memory; restart drops
-  them. Durable jobs → future work (see "Future: RunStore").
+- **No distributed persistence / replication.** Durable snapshots are local
+  to a single server instance (for example SQLite on local disk).
 - **No job prioritization / queueing.** Single-run `BUSY` lock preserved
   (§5). First caller wins; second caller gets `BUSY`.
 - **No new HITL mechanism.** Existing `hitl-bridge`; only **surfacing**
@@ -405,7 +406,8 @@ input type is structurally identical to the sync tool's input type
 Restated for emphasis, since issue #18 is deliberately narrow:
 
 - **No distributed jobs.** Jobs are single-instance only.
-- **No persistence across server restart.** In-memory `RunRegistry` only.
+- **No distributed persistence across server restart.** Restart recovery is
+  supported only when a durable local `RunStore` backend is configured.
 - **No job prioritization / queueing.** Single `BUSY` lock — same policy
   as sync mode.
 - **No `list_jobs` / `wait_for_job` bulk APIs.** v2 if ever requested.
@@ -430,8 +432,8 @@ Restated for emphasis, since issue #18 is deliberately narrow:
 
 ## Open follow-ups / future work
 
-- **`RunStore` persistence.** File-backed or SQLite-backed
-  `RunRegistry` for jobs that survive server restart.
+- **Additional durable backends.** Add Redis/Postgres-grade `RunStore`
+  adapters where restart recovery must survive host replacement.
 - **Webhook / push notifications.** Optional "call me at URL X when job
   finishes" so polling isn't required for clients that can accept
   callbacks.

docs/superpowers/specs/2026-04-16-mcp-async-design.md

@@ -44,8 +44,9 @@ transports are all re-expressed on top of it.
 
 - **No bundled HTTP server.** We do not ship Express / Fastify middleware.
   Framework integrations live in userland or future packages.
-- **No persistence.** Runs live in process memory. Restarting the server
-  drops in-flight runs. Durable runs are a v0.2+ feature.
+- **In-memory by default; persistence is pluggable.** Without a `RunStore`,
+  runs live in process memory. With a durable `RunStore` backend, run
+  snapshots can survive restart.
 - **No distributed execution.** A `runId` is only valid on the instance that
   created it. Horizontal scale requires sticky sessions or external state,
   out of scope here.
@@ -146,7 +147,8 @@ behavior unchanged). So CLI keeps prompting on TTY exactly like today.
 
 ### Run registry
 
-`@ageflow/server` owns a `RunRegistry` — an in-memory `Map<string, RunHandle>`.
+`@ageflow/server` owns a `RunRegistry` (active handles) and can mirror
+run snapshots through a pluggable `RunStore` backend.
 
 ```ts
 interface RunHandle {

docs/superpowers/specs/2026-04-16-server-execution-design.md

@@ -17,6 +17,7 @@
   },
   "devDependencies": {
     "@ageflow/mcp-server": "workspace:*",
+    "@ageflow/server-sqlite": "workspace:*",
     "@modelcontextprotocol/sdk": "^1.0.0",
     "@types/node": "^22.0.0",
     "vitest": "^2.1.0",

examples/mcp-server/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ageflow/cli",
-  "version": "0.5.5",
+  "version": "0.6.0",
   "description": "CLI for ageflow \u2014 agentwf run / validate / dry-run / init",
   "homepage": "https://github.com/Neftedollar/ageflow/tree/master/packages/cli",
   "type": "module",
@@ -26,7 +26,9 @@
     "@ageflow/executor": "^0.7.0",
     "@ageflow/learning": "^0.5.0",
     "@ageflow/learning-sqlite": "^0.4.1",
-    "@ageflow/mcp-server": "^0.5.0",
+    "@ageflow/mcp-server": "^0.7.0",
+    "@ageflow/server": "^0.6.0",
+    "@ageflow/server-sqlite": "^0.2.0",
     "chalk": "^5.3.0",
     "ora": "^8.0.0",
     "boxen": "^8.0.0",

packages/cli/package.json

@@ -216,6 +216,16 @@ describe("parseMcpServeArgs: async mode flags (#18)", () => {
     expect(parsed.jobCheckpointTtlMs).toBe(900_000);
   });
 
+  it("parses --job-db <path>", () => {
+    const parsed = parseMcpServeArgs([
+      "wf.ts",
+      "--async",
+      "--job-db",
+      "/tmp/jobs.sqlite",
+    ]);
+    expect(parsed.jobDb).toBe("/tmp/jobs.sqlite");
+  });
+
   it("rejects --job-ttl with no value", () => {
     expect(() => parseMcpServeArgs(["wf.ts", "--async", "--job-ttl"])).toThrow(
       /requires/,
@@ -239,4 +249,10 @@ describe("parseMcpServeArgs: async mode flags (#18)", () => {
       parseMcpServeArgs(["wf.ts", "--checkpoint-ttl", "1000"]),
     ).toThrow(/requires --async/);
   });
+
+  it("rejects --job-db without --async", () => {
+    expect(() =>
+      parseMcpServeArgs(["wf.ts", "--job-db", "/tmp/jobs.sqlite"]),
+    ).toThrow(/requires --async/);
+  });
 });

packages/cli/src/__tests__/mcp-serve.test.ts

@@ -48,6 +48,8 @@ export interface McpServeArgs {
   readonly jobTtlMs?: number;
   /** Override default 1-hour checkpoint TTL in ms (--checkpoint-ttl <ms>). */
   readonly jobCheckpointTtlMs?: number;
+  /** Persist async job registry to a SQLite database (--job-db <path>). */
+  readonly jobDb?: string;
   /** Use Streamable HTTP transport instead of stdio (--http). */
   readonly http?: boolean;
   /** HTTP port (--port <n>, required with --http). */
@@ -90,6 +92,7 @@ export function parseMcpServeArgs(argv: readonly string[]): McpServeArgs {
   let asyncMode: boolean | undefined = undefined;
   let jobTtlMs: number | undefined = undefined;
   let jobCheckpointTtlMs: number | undefined = undefined;
+  let jobDb: string | undefined = undefined;
   let httpMode: boolean | undefined = undefined;
   let httpPort: number | undefined = undefined;
   let httpHost: string | undefined = undefined;
@@ -215,6 +218,15 @@ export function parseMcpServeArgs(argv: readonly string[]): McpServeArgs {
         break;
       }
 
+      case "--job-db": {
+        const val = args[++i];
+        if (val === undefined || val.startsWith("-")) {
+          throw new Error("--job-db requires a path argument");
+        }
+        jobDb = val;
+        break;
+      }
+
       case "--http":
         httpMode = true;
         break;
@@ -259,9 +271,11 @@ export function parseMcpServeArgs(argv: readonly string[]): McpServeArgs {
 
   if (
     asyncMode !== true &&
-    (jobTtlMs !== undefined || jobCheckpointTtlMs !== undefined)
+    (jobTtlMs !== undefined ||
+      jobCheckpointTtlMs !== undefined ||
+      jobDb !== undefined)
   ) {
-    throw new Error("--job-ttl / --checkpoint-ttl requires --async");
+    throw new Error("--job-ttl / --checkpoint-ttl / --job-db requires --async");
   }
 
   if (httpPort !== undefined && httpMode !== true) {
@@ -302,6 +316,7 @@ export function parseMcpServeArgs(argv: readonly string[]): McpServeArgs {
     ...(asyncMode !== undefined ? { async: asyncMode } : {}),
     ...(jobTtlMs !== undefined ? { jobTtlMs } : {}),
     ...(jobCheckpointTtlMs !== undefined ? { jobCheckpointTtlMs } : {}),
+    ...(jobDb !== undefined ? { jobDb } : {}),
     ...(httpMode !== undefined ? { http: httpMode } : {}),
     ...(httpPort !== undefined ? { port: httpPort } : {}),
     ...(httpHost !== undefined ? { httpHost } : {}),
@@ -381,6 +396,7 @@ async function runMcpServe(rawArgv: string[]): Promise<void> {
     ...(parsed.jobCheckpointTtlMs !== undefined
       ? { jobCheckpointTtlMs: parsed.jobCheckpointTtlMs }
       : {}),
+    ...(parsed.jobDb !== undefined ? { jobDbPath: parsed.jobDb } : {}),
   });
 
   if (parsed.http === true) {
@@ -464,6 +480,7 @@ export function registerMcpCommand(program: Command): void {
         "  --async                enable async job mode (5 extra tools)\n" +
         "  --job-ttl <ms>         job TTL in ms (default: 1800000, requires --async)\n" +
         "  --checkpoint-ttl <ms>  checkpoint TTL in ms (default: 3600000, requires --async)\n" +
+        "  --job-db <path>        persist async job registry to SQLite (requires --async)\n" +
         "  --http                 use Streamable HTTP transport instead of stdio\n" +
         "  --port <n>             HTTP port (required with --http)\n" +
         "  --host <addr>          HTTP bind address (default: 127.0.0.1, requires --http)\n" +

packages/cli/src/commands/mcp-serve.ts

@@ -23,6 +23,7 @@ const PUBLISH_ORDER = [
   "@ageflow/runner-anthropic",
   "@ageflow/testing",
   "@ageflow/server",
+  "@ageflow/server-sqlite",
   "@ageflow/mcp-server",
   "@ageflow/learning",
   "@ageflow/learning-sqlite",

packages/dev-workflow/pipelines/release.ts

@@ -89,8 +89,9 @@ agentwf mcp serve ./workflow.ts --async --checkpoint-ttl 7200000
 
 ### Known limitations
 
-- **No persistence** — the job registry is in-memory only. Restarting the
-  server loses all job state.
+- **Durability is opt-in** — by default the job registry is in-memory.
+  Use `--job-db <path>` to persist async job snapshots to SQLite and
+  recover known jobs after restart.
 - **Single-instance** — the registry is not shared across processes. Running
   multiple server processes will have independent job stores.
 - **Single BUSY lock** — only one `start_*` call can be in-flight at a time.

packages/mcp-server/README.md

@@ -1,6 +1,6 @@
 {
   "name": "@ageflow/mcp-server",
-  "version": "0.5.1",
+  "version": "0.7.0",
   "description": "Expose ageflow workflows as MCP tools (stdio transport, progress streaming, HITL via elicitation).",
   "homepage": "https://github.com/Neftedollar/ageflow/tree/master/packages/mcp-server",
   "type": "module",
@@ -18,12 +18,14 @@
   "dependencies": {
     "@ageflow/core": "^0.6.0",
     "@ageflow/executor": "^0.7.0",
-    "@ageflow/server": "^0.4.4",
+    "@ageflow/server": "^0.6.0",
+    "@ageflow/server-sqlite": "^0.2.0",
     "@modelcontextprotocol/sdk": "^1.0.0",
     "zod-to-json-schema": "^3.23.0"
   },
   "devDependencies": {
     "@ageflow/testing": "workspace:*",
+    "@types/bun": "^1.3.12",
     "@types/node": "^22.0.0",
     "vitest": "^2.1.0",
     "zod": "^3.23.0"