Skip to content

Commit caa09da

Browse files
committed
✨ feat(app): add filesystem notification support and checkpoint recovery to follow-imports
- Add --watch-mode auto|notify|poll flag with fsnotify integration - Store boundary hash in checkpoint sidecar for file replacement detection - Report requested/active watch mode, fallback count, and fallback reason - Reset checkpoint when file is replaced without shrinking - Update memory rules to be more selective about note/handoff saves - Move fsnotify from indirect to direct dependency
1 parent 504c1f3 commit caa09da

13 files changed

Lines changed: 670 additions & 72 deletions

AGENTS.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,9 @@
99
## Memory Rules
1010

1111
- At the start of a fresh session in this repository, call `memory_bootstrap_session`.
12-
- Save a memory note when work produces a lasting decision, bugfix insight, reusable discovery, or durable implementation constraint.
13-
- Save a handoff before pausing, switching tasks, or ending the session.
12+
- Save a memory note only when work produces a lasting decision, bugfix insight, reusable discovery, or durable implementation constraint that is likely to matter beyond the current task checkpoint.
13+
- Save a handoff only before pausing, switching tasks, ending the session, or when the user explicitly asks for a checkpoint or resume record.
14+
- Do not save both in the same turn by default. Write both only when one artifact captures reusable long-term knowledge and the other captures task-specific continuation state.
1415

1516
## Related Project Policy
1617

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,8 +78,8 @@ They are not MCP tools and are not the normal end-user interaction path.
7878
Prints the same diagnostics in machine-readable JSON for automation or CI checks.
7979
- `codex-mem ingest-imports --source watcher_import [--input events.jsonl] [--json] [--continue-on-error] [--failed-output failed.jsonl] [--failed-manifest failed.json]`
8080
Imports newline-delimited watcher or relay note events into durable imported notes plus audit records, with optional partial-success handling plus retry-oriented failure exports.
81-
- `codex-mem follow-imports --source watcher_import --input events.jsonl [--state-file events.offset.json] [--poll-interval 5s] [--once] [--json]`
82-
Follows a watcher or relay JSONL file incrementally, checkpoints the last consumed offset, and reuses the same imported-note workflow for newly appended complete lines.
81+
- `codex-mem follow-imports --source watcher_import --input events.jsonl [--state-file events.offset.json] [--watch-mode auto|notify|poll] [--poll-interval 5s] [--once] [--json]`
82+
Follows a watcher or relay JSONL file incrementally, prefers filesystem notifications with polling fallback by default, checkpoints the last consumed offset, and reports the requested/active watch mode plus fallback state alongside imported-note results.
8383
- `codex-mem migrate`
8484
Opens the configured SQLite database and applies embedded migrations.
8585
- `codex-mem serve`

docs/go/maintainer/development-tracker.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -343,6 +343,24 @@ Current blockers:
343343
- In progress: none.
344344
- Blockers: none.
345345
- Next step: decide whether polling-based follow mode is sufficient for watcher/relay integration for now, or whether a later slice should add native filesystem notifications, rotation metadata, or multi-input fan-in.
346+
### 2026-03-16 Session Update
347+
348+
- Completed: Strengthened `follow-imports` checkpoint recovery so it no longer relies only on `size < offset` truncation detection. The sidecar now stores a hash of the last consumed boundary bytes plus file metadata, and follow mode resets to offset `0` when the current file no longer matches that checkpoint, including same-size replacement cases. App coverage now verifies checkpoint hashes, normal restart recovery, truncation resets, and replacement without shrink.
349+
- In progress: none.
350+
- Blockers: none.
351+
- Next step: decide whether the current boundary-hash approach is enough for production watcher/relay rotation patterns, or whether a later slice should add stronger file identity metadata or native filesystem notifications.
352+
### 2026-03-16 Session Update
353+
354+
- Completed: Added filesystem notification support to `follow-imports` with a new `--watch-mode auto|notify|poll` operator switch. Auto mode now prefers `fsnotify` on the input directory and still keeps the polling timer as a safety net, notify mode fails hard on watcher setup/runtime errors, and poll mode preserves the previous timer-only behavior. App coverage now includes watch-mode parsing and event filtering for input-file notifications.
355+
- In progress: none.
356+
- Blockers: none.
357+
- Next step: decide whether `follow-imports` needs richer observability for watcher fallbacks and dropped-event recovery, or whether the current auto/notify/poll split is enough for operators.
358+
### 2026-03-16 Session Update
359+
360+
- Completed: Added follow-mode observability for watcher state. `follow-imports` reports now include the requested watch mode, the currently active mode, fallback count, and last fallback reason, so operators can tell when auto mode has degraded to polling. Text output and JSON output both expose those fields, and app coverage verifies runtime-state injection plus report formatting.
361+
- In progress: none.
362+
- Blockers: none.
363+
- Next step: decide whether the next long-lived import slice should expose these watch-state transitions as structured metrics/events, or move on to multi-input fan-in support.
346364

347365
## Recommended Next Step
348366

docs/go/operator/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ Start here:
77
- [Client Examples](./client-examples.md)
88
Real MCP client registration examples for local stdio and remote HTTP.
99
- [Import Ingestion](./import-ingestion.md)
10-
JSONL batch ingestion through `ingest-imports` plus checkpointed follow-mode ingestion through `follow-imports`.
10+
JSONL batch ingestion through `ingest-imports` plus checkpointed follow-mode ingestion through `follow-imports`, including notify-vs-poll operator guidance.
1111
- [Release Readiness](./release-readiness.md)
1212
Packaging, readiness, and release checklist.
1313
- [Troubleshooting](./troubleshooting.md)

docs/go/operator/import-ingestion.md

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Do not use this for:
2323
## Command Shape
2424

2525
Use `ingest-imports` when you already have a bounded batch to replay.
26-
Use `follow-imports` when another process keeps appending to the same JSONL file and you want `codex-mem` to checkpoint progress between polling passes.
26+
Use `follow-imports` when another process keeps appending to the same JSONL file and you want `codex-mem` to checkpoint progress between notification or polling passes.
2727

2828
Minimal stdin example:
2929

@@ -64,7 +64,13 @@ codex-mem.exe follow-imports --source watcher_import --input .\events.jsonl --on
6464
Run as a long-lived poller with an explicit checkpoint file:
6565

6666
```powershell
67-
codex-mem.exe follow-imports --source relay_import --input .\relay-events.jsonl --state-file .\relay-events.offset.json --poll-interval 10s
67+
codex-mem.exe follow-imports --source relay_import --input .\relay-events.jsonl --state-file .\relay-events.offset.json --watch-mode poll --poll-interval 10s
68+
```
69+
70+
Run in notify-first mode and let polling stay as a safety fallback:
71+
72+
```powershell
73+
codex-mem.exe follow-imports --source watcher_import --input .\events.jsonl --watch-mode auto --poll-interval 5s
6874
```
6975

7076
Useful flags:
@@ -95,7 +101,9 @@ Useful flags:
95101
- `--state-file <path>`
96102
`follow-imports` only. Optional. Stores the consumed byte offset checkpoint. Defaults to `<input>.offset.json`.
97103
- `--poll-interval <duration>`
98-
`follow-imports` only. Optional. Controls how often the input file is polled for appended complete lines. Defaults to `5s`.
104+
`follow-imports` only. Optional. Controls how often the input file is polled for appended complete lines and how often notify mode performs a safety poll. Defaults to `5s`.
105+
- `--watch-mode auto|notify|poll`
106+
`follow-imports` only. Optional. `auto` prefers filesystem notifications and falls back to polling on watcher setup/runtime issues. `notify` requires filesystem notifications and fails if they cannot be used. `poll` disables notifications and uses polling only. Defaults to `auto`.
99107
- `--once`
100108
`follow-imports` only. Optional. Runs one poll/ingest pass and exits instead of staying in the polling loop.
101109

@@ -172,17 +180,21 @@ JSON mode returns the same summary plus per-line results, including the created
172180
When a line fails in `--continue-on-error` mode, that result entry includes a structured `error` payload instead.
173181
If `--failed-output` is set, the report also includes the resolved output path and how many failed lines were written there.
174182
If `--failed-manifest` is set, the report also includes the manifest path and how many failures were captured there.
175-
`follow-imports` reports the input path, checkpoint file, consumed offset, pending trailing bytes, truncation detection, and the nested batch report for whatever newly appended complete lines were imported during that poll.
183+
`follow-imports` reports the input path, checkpoint file, requested watch mode, active watch mode, fallback count, last fallback reason, consumed offset, pending trailing bytes, whether the checkpoint was reset, the reset reason, truncation detection, and the nested batch report for whatever newly appended complete lines were imported during that poll.
176184

177185
## Operational Notes
178186

179187
- `ingest-imports` starts one fresh session for the whole batch after resolving scope.
180188
- `follow-imports` starts one fresh session per consumed polling batch, not one session for the lifetime of the process.
189+
- In `auto` mode, `follow-imports` prefers filesystem notifications for lower latency and keeps the poll timer as a safety net in case a platform drops an event.
190+
- In `notify` mode, watcher setup or runtime failures stop the command instead of silently switching to polling.
191+
- The follow-mode report now exposes both the requested watch mode and the currently active mode, so operators can tell when `auto` has fallen back to polling and how many fallbacks have happened in the current process.
181192
- Each event uses the same imported-note workflow as `memory_save_imported_note`.
182193
- Existing explicit memory wins over weaker imported duplicates in the same project.
183194
- The default implementation is fail-fast: the first invalid line stops the batch and returns an error.
184195
- `--continue-on-error` preserves successful lines, reports per-line failures, and still exits with an error if nothing in the batch imports successfully.
185196
- `--failed-output` writes the original failed JSONL lines without wrapping them, so operators can edit that file and replay it through the same command later.
186197
- `--failed-manifest` writes a structured JSON sidecar with line numbers, error codes, error messages, raw failed lines, and failed-output line numbers when available.
187198
- `follow-imports` only consumes complete newline-terminated lines. A partially written trailing line is left in place until a later poll sees its terminating newline.
188-
- If the followed input file is truncated or rotated to a smaller size, `follow-imports` resets its checkpoint to byte offset `0` and continues from the start of the new file contents.
199+
- The `follow-imports` checkpoint sidecar stores both the consumed byte offset and a hash of the last consumed boundary bytes so replacement or rotation can be detected even when the new file does not shrink first.
200+
- If the followed input file is truncated, rotated, or replaced with different bytes before the saved offset, `follow-imports` resets its checkpoint to byte offset `0` and continues from the start of the new file contents.

docs/go/operator/release-readiness.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,8 @@ In a clean repository:
133133

134134
1. Run `memory_install_agents` in safe mode.
135135
2. Start a session with `memory_bootstrap_session`.
136-
3. Save one note with `memory_save_note`.
137-
4. Save one handoff with `memory_save_handoff`.
136+
3. Save one note with `memory_save_note` in a step where you are testing durable note writes.
137+
4. Save one handoff with `memory_save_handoff` in a separate step where you are testing continuation writes.
138138
5. Start a later bootstrap and confirm continuity is recovered.
139139

140140
### 5. Config Smoke Check
@@ -158,8 +158,8 @@ For a quick end-to-end demo:
158158
2. Start `codex-mem serve`
159159
3. Call `memory_resolve_scope`
160160
4. Call `memory_start_session`
161-
5. Call `memory_save_note`
162-
6. Call `memory_save_handoff`
161+
5. Call `memory_save_note` to verify durable note writes
162+
6. Call `memory_save_handoff` to verify continuation writes
163163
7. Call `memory_search`
164164
8. Call `memory_get_recent`
165165

docs/go/user/how-memory-works.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -511,10 +511,15 @@ A 项目的支付流程依赖 B 项目的订单接口和字段约定。后续凡
511511
- 想保存“以后很多次都可能会用到的经验或结论” -> `memory_save_note`
512512
- 想保存“这次任务下次如何继续” -> `memory_save_handoff`
513513

514-
很多时候两者可以一起保存:
514+
默认不要把两者一起保存。
515515

516-
1. 先把值得长期保留的结论存成 note
517-
2. 再把当前任务进度和 next steps 存成 handoff
516+
只有同时满足下面两件事时,才建议一起保存:
517+
518+
1. 有一条值得长期复用的结论,应该存成 note
519+
2. 还有一份这次任务的续做状态,应该存成 handoff
520+
521+
如果只是想保存“以后会反复用到的经验或结论”,通常只写 `memory_save_note` 就够了。
522+
如果只是想保存“这次任务下次如何继续”,通常只写 `memory_save_handoff` 就够了。
518523

519524
## FAQ: `workspace``project` 有什么区别
520525

docs/go/user/prompt-examples.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,18 @@
109109

110110
- `memory_save_handoff`
111111

112+
### 默认不要双写
113+
114+
如果你不希望同一轮同时写长期 note 和 handoff,可以在提示词里明确说只保存一种:
115+
116+
```text
117+
这条信息只需要保存成长期记忆,不要另外写 handoff。
118+
```
119+
120+
```text
121+
这次只保存任务交接记录,不要另外写长期 note。
122+
```
123+
112124
### 查历史信息
113125

114126
#### 提示词:搜索以前的记忆
@@ -298,6 +310,26 @@ Likely tools:
298310

299311
- `memory_save_handoff`
300312

313+
#### Prompt: save only a note, not a handoff
314+
315+
```text
316+
This should be saved only as a durable note for future reuse. Do not also write a handoff.
317+
```
318+
319+
Likely tools:
320+
321+
- `memory_save_note`
322+
323+
#### Prompt: save only a handoff, not a note
324+
325+
```text
326+
This should be saved only as a handoff for task continuation. Do not also write a durable note.
327+
```
328+
329+
Likely tools:
330+
331+
- `memory_save_handoff`
332+
301333
### Retrieval
302334

303335
#### Prompt: search prior memory

go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ module codex-mem
33
go 1.26.0
44

55
require (
6+
github.com/fsnotify/fsnotify v1.9.0
67
github.com/modelcontextprotocol/go-sdk v1.4.1
78
github.com/spf13/viper v1.21.0
89
gopkg.in/natefinch/lumberjack.v2 v2.2.1
@@ -11,7 +12,6 @@ require (
1112

1213
require (
1314
github.com/dustin/go-humanize v1.0.1 // indirect
14-
github.com/fsnotify/fsnotify v1.9.0 // indirect
1515
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
1616
github.com/google/jsonschema-go v0.4.2 // indirect
1717
github.com/google/uuid v1.6.0 // indirect

0 commit comments

Comments
 (0)