feat: add trigger file watcher for instant re-index#332
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an opt-in filesystem trigger to request an immediate re-index, reducing the latency window inherent in the existing 5-minute background polling sync.
Changes:
- Add
fs.watch()on~/.context/to detect.sync-triggermodifications and trigger re-index. - Debounce trigger events (2s) to collapse rapid touches into a single sync.
- Keep existing periodic sync interval as a fallback while adding trigger-based sync.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| fs.watch(contextDir, (event, filename) => { | ||
| if (filename === triggerFile) { | ||
| if (debounceTimer) clearTimeout(debounceTimer); | ||
| debounceTimer = setTimeout(() => { | ||
| console.log('[SYNC] 🔔 Trigger file detected, starting instant re-index...'); | ||
| this.handleSyncIndex(); | ||
| }, 2000); | ||
| } | ||
| }); |
There was a problem hiding this comment.
fs.watch() can emit an asynchronous error event (e.g., if ~/.context is deleted or the filesystem stops watching). Because the returned FSWatcher isn’t captured and no error listener is attached, an emitted error can crash the process (unhandled EventEmitter error). Capture the watcher returned by fs.watch(...) and attach an on('error', ...) handler (and consider closing/recreating the watcher on failure).
| fs.watch(contextDir, (event, filename) => { | ||
| if (filename === triggerFile) { | ||
| if (debounceTimer) clearTimeout(debounceTimer); | ||
| debounceTimer = setTimeout(() => { | ||
| console.log('[SYNC] 🔔 Trigger file detected, starting instant re-index...'); | ||
| this.handleSyncIndex(); | ||
| }, 2000); | ||
| } |
There was a problem hiding this comment.
filename from fs.watch is not guaranteed to be a string (it can be undefined or a Buffer depending on platform/options). Comparing it directly to '.sync-trigger' can cause the trigger to be missed. Normalize/guard the value (e.g., coerce Buffer to string and handle undefined), or pass an explicit encoding option to fs.watch.
| fs.watch(contextDir, (event, filename) => { | |
| if (filename === triggerFile) { | |
| if (debounceTimer) clearTimeout(debounceTimer); | |
| debounceTimer = setTimeout(() => { | |
| console.log('[SYNC] 🔔 Trigger file detected, starting instant re-index...'); | |
| this.handleSyncIndex(); | |
| }, 2000); | |
| } | |
| fs.watch(contextDir, { encoding: 'utf8' }, (event, filename) => { | |
| const watchedFilename = | |
| typeof filename === 'string' | |
| ? filename | |
| : Buffer.isBuffer(filename) | |
| ? filename.toString('utf8') | |
| : undefined; | |
| if (watchedFilename !== triggerFile) { | |
| return; | |
| } | |
| if (debounceTimer) clearTimeout(debounceTimer); | |
| debounceTimer = setTimeout(() => { | |
| console.log('[SYNC] 🔔 Trigger file detected, starting instant re-index...'); | |
| this.handleSyncIndex(); | |
| }, 2000); |
| if (debounceTimer) clearTimeout(debounceTimer); | ||
| debounceTimer = setTimeout(() => { | ||
| console.log('[SYNC] 🔔 Trigger file detected, starting instant re-index...'); | ||
| this.handleSyncIndex(); |
There was a problem hiding this comment.
handleSyncIndex() is async; calling it without await/.catch() inside the debounce callback can create an unhandled rejection if handleSyncIndex() throws before its internal try/catch runs (or if future refactors add awaits before the try). Consider invoking it as a fire-and-forget promise with explicit .catch(...) logging, consistent with the initial sync handler.
| this.handleSyncIndex(); | |
| void this.handleSyncIndex().catch((error) => { | |
| const errorMessage = error instanceof Error ? error.message : String(error); | |
| if (errorMessage.includes('Failed to query collection')) { | |
| console.log('[SYNC-DEBUG] Collection not yet established during trigger sync, this is expected for new cluster users. Will retry on next sync cycle.'); | |
| } else { | |
| console.error('[SYNC-DEBUG] Triggered sync failed with unexpected error:', error); | |
| // Do not re-throw here: this callback runs via setTimeout with no caller to propagate to. | |
| } | |
| }); |
| // Set up trigger file watcher for instant re-index (e.g., from Claude Code hooks) | ||
| this.setupTriggerWatcher(); | ||
| } |
There was a problem hiding this comment.
startBackgroundSync() unconditionally calls setupTriggerWatcher(). If startBackgroundSync() is ever invoked more than once (tests, hot-reload, re-init), this will register multiple watchers and debounce timers. Consider storing the watcher/timer as class fields and guarding against double-initialization (and optionally expose a stopBackgroundSync() to close the watcher).
| console.log(`[SYNC-DEBUG] Trigger watcher active on ${contextDir}/${triggerFile}`); | ||
| } catch (error) { | ||
| console.warn(`[SYNC-DEBUG] Could not set up trigger watcher: ${error}`); |
There was a problem hiding this comment.
The watcher “active on …” path is built with a hardcoded / separator; on Windows this will be misleading. Prefer path.join(contextDir, triggerFile) for logging. Also, console.warn(... ${error}) will typically stringify to [object Object] (and lose stack traces, especially with the custom console wrapper in index.ts); log error.message/error.stack explicitly (or pass structured fields).
| console.log(`[SYNC-DEBUG] Trigger watcher active on ${contextDir}/${triggerFile}`); | |
| } catch (error) { | |
| console.warn(`[SYNC-DEBUG] Could not set up trigger watcher: ${error}`); | |
| console.log(`[SYNC-DEBUG] Trigger watcher active on ${path.join(contextDir, triggerFile)}`); | |
| } catch (error) { | |
| if (error instanceof Error) { | |
| console.warn('[SYNC-DEBUG] Could not set up trigger watcher:', error.message); | |
| if (error.stack) { | |
| console.warn(error.stack); | |
| } | |
| } else { | |
| console.warn('[SYNC-DEBUG] Could not set up trigger watcher:', String(error)); | |
| } |
12e9eaf to
ca3da7c
Compare
|
Addressed Copilot review feedback in ca3da7c:
|
|
I tested this on top of the current master by merging the PR branch locally, then running: That path passes. One behavior concern before merging: the trigger path is global at Could you clarify whether that is the intended tradeoff, or add a guard so only one process performs a triggered sync at a time? A shared lock around triggered sync, a per-process trigger path, or documentation that this is still a per-process trigger would make the behavior clearer. Also, since this adds |
|
Thanks for the work on the trigger watcher approach. v0.1.10 now includes a global cross-process background sync lock. That should cover the main concern discussed above: when multiple local MCP server processes are running, only one process should perform sync work at a time and the others skip that cycle while the lock is held. This may also be useful if this PR continues, since a global trigger can otherwise wake every running process at once. Please test against the latest version and let us know whether the remaining trigger-watcher behavior still needs a separate guard or documentation changes. |
Watch ~/.context/.sync-trigger for changes to enable instant re-indexing. Claude Code PostToolUse hooks can touch this file after Write/Edit operations. Debounced at 2s to batch rapid edits. Runs alongside existing 5-minute polling as fallback.
- Capture FSWatcher and attach error listener (prevent unhandled crash)
- Pass {encoding: 'utf8'} so filename is string across platforms
- Coerce Buffer/undefined filename safely
- Wrap handleSyncIndex in .catch() inside setTimeout (no unhandled rejection)
- Guard against double-init via class field; add stopTriggerWatcher()
- Use path.join for log path; log error.message/stack instead of [object Object]
Refs zilliztech#285 (txhno's CPU-churn report on multi-instance MCP sessions) and zilliztech#314 (txhno's opt-in background-sync fix). Their analysis showed that periodic background sync should be opt-in to avoid duplicated work when N workspaces each spawn their own MCP. Our trigger watcher is the natural complement: instead of polling every 5 min, the user (or a Claude Code PostToolUse hook) touches ~/.context/.sync-trigger to request an immediate re-index. To make the two play together, lift setupTriggerWatcher() to the start of startBackgroundSync() so the watcher still runs when polling is gated off (the recommended config for multi-instance setups in zilliztech#285). Add CLAUDE_CONTEXT_TRIGGER_WATCHER (default true) so users can opt out of the watcher entirely if they don't want any filesystem watching. Credit: @txhno for zilliztech#285 + zilliztech#314.
Address review feedback from @zc277584121 on zilliztech#332: - Document `CLAUDE_CONTEXT_TRIGGER_WATCHER` env in MCP README and the `--help` output (config.ts). - Note that triggered sync goes through the same global cross-process lock added in v0.1.10, so multi-instance setups stay coordinated when the trigger file is touched. - Add Claude Code PostToolUse hook example.
ba1c308 to
589f121
Compare
|
Thanks @zc277584121 — rebased onto
Branch state:
|
Summary
Add a file-system watcher on
~/.context/.sync-triggerso external tools (e.g., Claude Code'sPostToolUsehooks) can request an immediate re-index instead of waiting up to 5 minutes for the existing background poll.Relation to #285 / #314
Filed concurrently with @jmmaloney4's #285 (multi-instance MCP CPU churn) and @txhno's #314 (opt-in background sync, which fixes #285). Their work addresses the polling side; this PR addresses the on-demand side. They are complementary — together they let users pick how aggressive sync should be:
CLAUDE_CONTEXT_BACKGROUND_SYNC(#314)CLAUDE_CONTEXT_TRIGGER_WATCHER(this PR)truetrue(default)falsetrue(default)truefalsefalsefalseindex_codebaseonlyThis PR is structured to be merge-order independent:
setupTriggerWatcher()is now invoked at the top ofstartBackgroundSync(), before the polling block, so #314 can land in either order without conflict and without accidentally skipping the watcher.Motivation
The MCP server currently re-indexes every 5 minutes via a
setInterval. That's fine for passive polling, but creates a frustrating gap when the user (or an agent) makes a code change and immediately runs a search — the search hits stale embeddings.A trigger file is the simplest possible coordination primitive:
fs.watchis built in.touch ~/.context/.sync-triggerto request a re-index.Concrete use case (Claude Code
settings.json):After this, every Edit/Write the agent makes triggers a debounced re-index — no manual
index_codebasecall, no 5-minute wait.Changes
packages/mcp/src/sync.ts:path,os,envManager.setupTriggerWatcher()andisTriggerWatcherEnabled()methods.setupTriggerWatcher()is invoked at the start ofstartBackgroundSync(), independent of the polling block. This makes the watcher coexist cleanly with feat(mcp): make background sync opt-in #314's opt-in polling gate (the watcher still runs even when periodic sync is disabled).fs.watch()on~/.context/(not the file directly — files don't exist until first touch and watching a non-existent path throws).fs.mkdirSync(contextDir, { recursive: true })ensures dir exists before watching (snapshot manager already creates it, but being defensive avoids race on cold start).fs.watch(contextDir, { encoding: 'utf8' }, ...)— explicit encoding sofilenameis a string across platforms; null/non-string filenames guarded.FSWatcherwithon('error', ...)listener so async watcher errors (dir deletion, fs unmount) don't crash the process..sync-trigger.save → save → save) collapse to a single re-index.handleSyncIndex()invocation inside the debounce is wrapped in.catch(...)to avoid unhandled promise rejection from thesetTimeoutcallback.triggerWatcher/triggerDebounceTimerfields guard against double-init (hot reload, repeated test setup).stopTriggerWatcher()for graceful shutdown / tests.Configuration
CLAUDE_CONTEXT_TRIGGER_WATCHERtruefs.watchon a single directory) but can be disabled on read-only or sandboxed filesystems.Behavior
If the watcher fails to start (filesystem doesn't support
fs.watch, e.g., some CIFS mounts), a warning is logged witherror.message+ stack and operation continues — graceful degradation.Test plan
pnpm buildpasses[SYNC-DEBUG] Trigger watcher activeappears in logstouch ~/.context/.sync-trigger→[SYNC] 🔔 Trigger file detectedwithin 2sCLAUDE_CONTEXT_TRIGGER_WATCHER=false→ log shows watcher disabled, no FS watch active~/.context/while server runs → watcher emits error, server logs warning, polling (if enabled) continuesfs.watchsupport → warning logged, polling (if enabled) still worksCLAUDE_CONTEXT_BACKGROUND_SYNC=false→ polling skipped, watcher still activeNotes for reviewers
fs.watchsemantics differ across platforms (recursive on macOS, not on Linux), but we only watch a single directory non-recursively, so cross-platform behavior is consistent.SYNC_TRIGGER_FILEenv var in a follow-up if needed, but a hardcoded sentinel is simpler for hook authors.setupTriggerWatcher()was deliberately moved above the polling block in response to that work to keep the three PRs orthogonal.