[BUG] Sync stops working with "cannot schedule new futures after shutdown" errors

[jope-bm]

Bug: Background file sync fails with "cannot schedule new futures after shutdown" in MCP server

Description

When running the Basic Memory MCP server (via basic-memory mcp), background file synchronization tasks fail with the error:

Error in background sync for project main: cannot schedule new futures after shutdown

This prevents files from being indexed in the database, leaving the knowledge base incomplete.

Symptoms

• watch-status.json shows running: true but synced_files: 0 and last_scan: null
• MCP logs show recurring error: Error in background sync for project main: cannot schedule new futures after shutdown
• Files exist on disk (e.g., through Nov 14) but database only has files indexed through an earlier date (e.g., Nov 4)
• No way to manually trigger sync to recover (the sync command was removed)

Root Cause

The MCP server runs file synchronization in a daemon thread with its own event loop (src/basic_memory/cli/commands/mcp.py:62-72):

def run_file_sync():
    """Run file sync in a separate thread with its own event loop."""
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(initialize_file_sync(app_config))
    except Exception as e:
        logger.error(f"File sync error: {e}", err=True)
    finally:
        loop.close()  # ❌ BUG: This shuts down the ThreadPoolExecutor!

The initialization function (src/basic_memory/services/initialization.py:107-132) creates background tasks that use aiofiles.os.scandir():

# Create background tasks for all project syncs (non-blocking)
sync_tasks = [
    asyncio.create_task(sync_project_background(project)) for project in active_projects
]
# Don't await the tasks - let them run in background while we continue

These background tasks call aiofiles.os.scandir() which requires the event loop's ThreadPoolExecutor. However, the daemon thread's finally block calls loop.close() which shuts down the executor before the background tasks complete.

Timeline of the bug:

1. Daemon thread starts, creates new event loop
2. initialize_file_sync() creates background tasks (these start running immediately)
3. Background tasks begin calling aiofiles.os.scandir() which needs the executor
4. Watch service completes or encounters an exception
5. finally block calls loop.close() → shuts down ThreadPoolExecutor
6. Background tasks try to use executor → "cannot schedule new futures after shutdown"

Impact

• High: Files created/modified after MCP server starts are not indexed
• Knowledge graph is incomplete and stale
• Recent activity and search miss newly created content
• No workaround available (sync command removed)

Fix

Remove the loop.close() call from the daemon thread. Since this is a daemon thread meant to run for the lifetime of the MCP server process, the event loop should never be closed. The OS will clean up resources when the process exits.

src/basic_memory/cli/commands/mcp.py:

def run_file_sync():
    """Run file sync in a separate thread with its own event loop."""
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(initialize_file_sync(app_config))
    except Exception as e:
        logger.error(f"File sync error: {e}", err=True)
    # Note: Do NOT close the loop here! This is a daemon thread that should
    # run until process exit. Closing the loop shuts down the ThreadPoolExecutor
    # which breaks aiofiles operations in background sync tasks.

Testing

Two test scripts are available to verify the fix:

1. Unit test (test_executor_fix.py): Reproduces the bug in isolation

   python test_executor_fix.py

2. Integration test (test_mcp_startup.sh): Full end-to-end test

   ./test_mcp_startup.sh

Expected Behavior After Fix

• Background sync tasks complete successfully
• watch-status.json shows synced_files > 0 and populated last_scan
• All files on disk are indexed in the database
• No "cannot schedule new futures after shutdown" errors in logs

Environment

• Basic Memory version: v0.16.x (any version with MCP server)
• Python: 3.12+
• OS: macOS Sequoia 15.7.1 (reproduced), likely affects all platforms
• MCP transport: stdio (Claude Desktop)

Related Code

• src/basic_memory/cli/commands/mcp.py:62-72 - Daemon thread with buggy finally block
• src/basic_memory/services/initialization.py:107-132 - Background task creation
• src/basic_memory/sync/sync_service.py:1165 - Use of aiofiles.os.scandir()

Additional Notes

This is a classic async lifecycle bug where:

1. A resource (ThreadPoolExecutor) is shared between a parent scope and child tasks
2. The parent scope closes the resource before child tasks complete
3. Child tasks fail when trying to use the closed resource

The fix is simple but critical for MCP server reliability.

[claude]

Claude finished @jope-bm's task —— View job • claude/issue-443-20251125-0307 • Create PR ➔

---

Tasks Completed

• Triage: Add appropriate labels to the issue
• Read and understand the buggy code in src/basic_memory/cli/commands/mcp.py
• Implement the fix (remove loop.close() and add explanatory comment)
• Check for test scripts (not in repo yet)
• Commit and push changes

Summary

I've successfully implemented the fix for the background sync executor shutdown bug. The issue was caused by loop.close() being called in the daemon thread's finally block, which shut down the ThreadPoolExecutor before background sync tasks could complete their aiofiles operations.

Changes Made:

• Removed the loop.close() call from src/basic_memory/cli/commands/mcp.py (line 71)
• Added a detailed comment explaining why the loop should not be closed in a daemon thread
• The event loop will now remain active for the lifetime of the MCP server process, allowing background tasks to use the ThreadPoolExecutor

Expected Results:

• Background sync tasks will complete successfully
• watch-status.json will show synced_files > 0 and populated last_scan
• All files on disk will be indexed in the database
• No more "cannot schedule new futures after shutdown" errors

Note on Testing:
The test scripts mentioned in the issue (test_executor_fix.py and test_mcp_startup.sh) don't exist in the repository yet. I don't have permissions to run just install or just test in this environment, but the fix is straightforward and directly addresses the root cause identified in the issue.

---

| Branch: claude/issue-443-20251125-0307

[phernandez]

This issue has been resolved through code refactoring.

The MCP server now uses an async lifespan context manager instead of a separate daemon thread with its own event loop. File sync runs as an asyncio.create_task() within the main event loop, eliminating the ThreadPoolExecutor shutdown issue.

See src/basic_memory/mcp/server.py - the lifespan manager properly handles background task cancellation on shutdown.