You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
✨ feat(app): add filesystem notification support and checkpoint recovery to follow-imports
- Add --watch-mode auto|notify|poll flag with fsnotify integration
- Store boundary hash in checkpoint sidecar for file replacement detection
- Report requested/active watch mode, fallback count, and fallback reason
- Reset checkpoint when file is replaced without shrinking
- Update memory rules to be more selective about note/handoff saves
- Move fsnotify from indirect to direct dependency
Copy file name to clipboardExpand all lines: AGENTS.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,8 +9,9 @@
9
9
## Memory Rules
10
10
11
11
- At the start of a fresh session in this repository, call `memory_bootstrap_session`.
12
-
- Save a memory note when work produces a lasting decision, bugfix insight, reusable discovery, or durable implementation constraint.
13
-
- Save a handoff before pausing, switching tasks, or ending the session.
12
+
- Save a memory note only when work produces a lasting decision, bugfix insight, reusable discovery, or durable implementation constraint that is likely to matter beyond the current task checkpoint.
13
+
- Save a handoff only before pausing, switching tasks, ending the session, or when the user explicitly asks for a checkpoint or resume record.
14
+
- Do not save both in the same turn by default. Write both only when one artifact captures reusable long-term knowledge and the other captures task-specific continuation state.
Imports newline-delimited watcher or relay note events into durable imported notes plus audit records, with optional partial-success handling plus retry-oriented failure exports.
Follows a watcher or relay JSONL file incrementally, checkpoints the last consumed offset, and reuses the same imported-note workflow for newly appended complete lines.
Follows a watcher or relay JSONL file incrementally, prefers filesystem notifications with polling fallback by default, checkpoints the last consumed offset, and reports the requested/active watch mode plus fallback state alongside imported-note results.
83
83
-`codex-mem migrate`
84
84
Opens the configured SQLite database and applies embedded migrations.
Copy file name to clipboardExpand all lines: docs/go/maintainer/development-tracker.md
+18Lines changed: 18 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -343,6 +343,24 @@ Current blockers:
343
343
- In progress: none.
344
344
- Blockers: none.
345
345
- Next step: decide whether polling-based follow mode is sufficient for watcher/relay integration for now, or whether a later slice should add native filesystem notifications, rotation metadata, or multi-input fan-in.
346
+
### 2026-03-16 Session Update
347
+
348
+
- Completed: Strengthened `follow-imports` checkpoint recovery so it no longer relies only on `size < offset` truncation detection. The sidecar now stores a hash of the last consumed boundary bytes plus file metadata, and follow mode resets to offset `0` when the current file no longer matches that checkpoint, including same-size replacement cases. App coverage now verifies checkpoint hashes, normal restart recovery, truncation resets, and replacement without shrink.
349
+
- In progress: none.
350
+
- Blockers: none.
351
+
- Next step: decide whether the current boundary-hash approach is enough for production watcher/relay rotation patterns, or whether a later slice should add stronger file identity metadata or native filesystem notifications.
352
+
### 2026-03-16 Session Update
353
+
354
+
- Completed: Added filesystem notification support to `follow-imports` with a new `--watch-mode auto|notify|poll` operator switch. Auto mode now prefers `fsnotify` on the input directory and still keeps the polling timer as a safety net, notify mode fails hard on watcher setup/runtime errors, and poll mode preserves the previous timer-only behavior. App coverage now includes watch-mode parsing and event filtering for input-file notifications.
355
+
- In progress: none.
356
+
- Blockers: none.
357
+
- Next step: decide whether `follow-imports` needs richer observability for watcher fallbacks and dropped-event recovery, or whether the current auto/notify/poll split is enough for operators.
358
+
### 2026-03-16 Session Update
359
+
360
+
- Completed: Added follow-mode observability for watcher state. `follow-imports` reports now include the requested watch mode, the currently active mode, fallback count, and last fallback reason, so operators can tell when auto mode has degraded to polling. Text output and JSON output both expose those fields, and app coverage verifies runtime-state injection plus report formatting.
361
+
- In progress: none.
362
+
- Blockers: none.
363
+
- Next step: decide whether the next long-lived import slice should expose these watch-state transitions as structured metrics/events, or move on to multi-input fan-in support.
Copy file name to clipboardExpand all lines: docs/go/operator/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ Start here:
7
7
-[Client Examples](./client-examples.md)
8
8
Real MCP client registration examples for local stdio and remote HTTP.
9
9
-[Import Ingestion](./import-ingestion.md)
10
-
JSONL batch ingestion through `ingest-imports` plus checkpointed follow-mode ingestion through `follow-imports`.
10
+
JSONL batch ingestion through `ingest-imports` plus checkpointed follow-mode ingestion through `follow-imports`, including notify-vs-poll operator guidance.
Copy file name to clipboardExpand all lines: docs/go/operator/import-ingestion.md
+17-5Lines changed: 17 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,7 +23,7 @@ Do not use this for:
23
23
## Command Shape
24
24
25
25
Use `ingest-imports` when you already have a bounded batch to replay.
26
-
Use `follow-imports` when another process keeps appending to the same JSONL file and you want `codex-mem` to checkpoint progress between polling passes.
26
+
Use `follow-imports` when another process keeps appending to the same JSONL file and you want `codex-mem` to checkpoint progress between notification or polling passes.
Run in notify-first mode and let polling stay as a safety fallback:
71
+
72
+
```powershell
73
+
codex-mem.exe follow-imports --source watcher_import --input .\events.jsonl --watch-mode auto --poll-interval 5s
68
74
```
69
75
70
76
Useful flags:
@@ -95,7 +101,9 @@ Useful flags:
95
101
-`--state-file <path>`
96
102
`follow-imports` only. Optional. Stores the consumed byte offset checkpoint. Defaults to `<input>.offset.json`.
97
103
-`--poll-interval <duration>`
98
-
`follow-imports` only. Optional. Controls how often the input file is polled for appended complete lines. Defaults to `5s`.
104
+
`follow-imports` only. Optional. Controls how often the input file is polled for appended complete lines and how often notify mode performs a safety poll. Defaults to `5s`.
105
+
-`--watch-mode auto|notify|poll`
106
+
`follow-imports` only. Optional. `auto` prefers filesystem notifications and falls back to polling on watcher setup/runtime issues. `notify` requires filesystem notifications and fails if they cannot be used. `poll` disables notifications and uses polling only. Defaults to `auto`.
99
107
-`--once`
100
108
`follow-imports` only. Optional. Runs one poll/ingest pass and exits instead of staying in the polling loop.
101
109
@@ -172,17 +180,21 @@ JSON mode returns the same summary plus per-line results, including the created
172
180
When a line fails in `--continue-on-error` mode, that result entry includes a structured `error` payload instead.
173
181
If `--failed-output` is set, the report also includes the resolved output path and how many failed lines were written there.
174
182
If `--failed-manifest` is set, the report also includes the manifest path and how many failures were captured there.
175
-
`follow-imports` reports the input path, checkpoint file, consumed offset, pending trailing bytes, truncation detection, and the nested batch report for whatever newly appended complete lines were imported during that poll.
183
+
`follow-imports` reports the input path, checkpoint file, requested watch mode, active watch mode, fallback count, last fallback reason, consumed offset, pending trailing bytes, whether the checkpoint was reset, the reset reason, truncation detection, and the nested batch report for whatever newly appended complete lines were imported during that poll.
176
184
177
185
## Operational Notes
178
186
179
187
-`ingest-imports` starts one fresh session for the whole batch after resolving scope.
180
188
-`follow-imports` starts one fresh session per consumed polling batch, not one session for the lifetime of the process.
189
+
- In `auto` mode, `follow-imports` prefers filesystem notifications for lower latency and keeps the poll timer as a safety net in case a platform drops an event.
190
+
- In `notify` mode, watcher setup or runtime failures stop the command instead of silently switching to polling.
191
+
- The follow-mode report now exposes both the requested watch mode and the currently active mode, so operators can tell when `auto` has fallen back to polling and how many fallbacks have happened in the current process.
181
192
- Each event uses the same imported-note workflow as `memory_save_imported_note`.
182
193
- Existing explicit memory wins over weaker imported duplicates in the same project.
183
194
- The default implementation is fail-fast: the first invalid line stops the batch and returns an error.
184
195
-`--continue-on-error` preserves successful lines, reports per-line failures, and still exits with an error if nothing in the batch imports successfully.
185
196
-`--failed-output` writes the original failed JSONL lines without wrapping them, so operators can edit that file and replay it through the same command later.
186
197
-`--failed-manifest` writes a structured JSON sidecar with line numbers, error codes, error messages, raw failed lines, and failed-output line numbers when available.
187
198
-`follow-imports` only consumes complete newline-terminated lines. A partially written trailing line is left in place until a later poll sees its terminating newline.
188
-
- If the followed input file is truncated or rotated to a smaller size, `follow-imports` resets its checkpoint to byte offset `0` and continues from the start of the new file contents.
199
+
- The `follow-imports` checkpoint sidecar stores both the consumed byte offset and a hash of the last consumed boundary bytes so replacement or rotation can be detected even when the new file does not shrink first.
200
+
- If the followed input file is truncated, rotated, or replaced with different bytes before the saved offset, `follow-imports` resets its checkpoint to byte offset `0` and continues from the start of the new file contents.
0 commit comments