✨ feat(app): add partial-success mode and failure exports to ingest-imports

- Add --continue-on-error for partial-success batch processing
- Add --failed-output for exporting failed lines for retry
- Add --failed-manifest for structured JSON retry manifest
- Extract App.IngestImports for reusable embedded integration
- Update CLI and operator documentation

Author: cocosip

README.md

@@ -10,7 +10,7 @@ It stores structured notes and handoffs in SQLite, restores continuity across re
 - `serve-http` runs a native MCP HTTP server for remote or private deployment.
 - `doctor` reports config, database readiness, migration status, provenance coverage, and MCP tool availability.
 - AGENTS template installation is implemented for global and project workflows.
-- one-shot watcher/relay batch ingestion is available through `ingest-imports`.
+- watcher/relay import ingestion is available as one-shot batches through `ingest-imports` and as a checkpointed long-lived adapter through `follow-imports`.
 
 Normative product docs live in [docs/spec/README.md](docs/spec/README.md).
 Go implementation docs now live under [docs/go/README.md](docs/go/README.md), grouped into user, operator, and maintainer directories.
@@ -24,7 +24,7 @@ Use the docs by audience:
 - [Operator docs](docs/go/operator/README.md)
   Client registration, deployment/readiness, packaging, and troubleshooting.
 - [Import ingestion guide](docs/go/operator/import-ingestion.md)
-  JSONL batch ingestion for watcher and relay artifacts through `ingest-imports`.
+  JSONL batch and checkpointed follow-mode ingestion for watcher and relay artifacts.
 - [Maintainer docs](docs/go/maintainer/README.md)
   Source-tree MCP integration, implementation planning, and development tracking.
 
@@ -76,8 +76,10 @@ They are not MCP tools and are not the normal end-user interaction path.
   Prints effective config plus runtime readiness and audit diagnostics.
 - `codex-mem doctor --json`
   Prints the same diagnostics in machine-readable JSON for automation or CI checks.
-- `codex-mem ingest-imports --source watcher_import [--input events.jsonl] [--json]`
-  Imports newline-delimited watcher or relay note events into durable imported notes plus audit records.
+- `codex-mem ingest-imports --source watcher_import [--input events.jsonl] [--json] [--continue-on-error] [--failed-output failed.jsonl] [--failed-manifest failed.json]`
+  Imports newline-delimited watcher or relay note events into durable imported notes plus audit records, with optional partial-success handling plus retry-oriented failure exports.
+- `codex-mem follow-imports --source watcher_import --input events.jsonl [--state-file events.offset.json] [--poll-interval 5s] [--once] [--json]`
+  Follows a watcher or relay JSONL file incrementally, checkpoints the last consumed offset, and reuses the same imported-note workflow for newly appended complete lines.
 - `codex-mem migrate`
   Opens the configured SQLite database and applies embedded migrations.
 - `codex-mem serve`

docs/go/maintainer/development-tracker.md

@@ -313,6 +313,36 @@ Current blockers:
 - In progress: none.
 - Blockers: none.
 - Next step: decide whether `ingest-imports` should remain the main watcher/relay bridge for now or whether a more direct long-lived integration path is worth adding later.
+### 2026-03-16 Session Update
+
+- Completed: Added `ingest-imports --continue-on-error` so watcher/relay batches can keep importing valid lines while collecting per-line decode/write failures in the text/JSON report. Default behavior remains fail-fast for compatibility, but partial-success mode now reports `status`, attempted/failed counts, and structured line errors; app coverage verifies partial success and the all-failed path.
+- In progress: none.
+- Blockers: none.
+- Next step: decide whether partial-success mode is enough for watcher/relay operators for now, or whether they also need richer retry/export behavior for failed lines.
+### 2026-03-16 Session Update
+
+- Completed: Added `ingest-imports --failed-output <path>` for `--continue-on-error` batches so failed raw JSONL lines can be exported unchanged for later replay. The CLI report now includes the resolved failed-output path plus written count, and app coverage verifies both partial-success export and all-failed export behavior.
+- In progress: none.
+- Blockers: none.
+- Next step: decide whether failed-line export is enough for operators or whether the next slice should add a richer retry manifest with error metadata alongside the raw replay file.
+### 2026-03-16 Session Update
+
+- Completed: Added `ingest-imports --failed-manifest <path>` so `--continue-on-error` batches can emit a JSON retry manifest with line numbers, error payloads, raw failed lines, and failed-output line numbers. The main report now surfaces the manifest path/count, and app coverage verifies manifest validation plus partial/all-failed export behavior.
+- In progress: none.
+- Blockers: none.
+- Next step: decide whether the operator path is now sufficient, or whether the next import slice should focus on a more direct watcher/relay integration instead of more CLI/reporting polish.
+### 2026-03-16 Session Update
+
+- Completed: Extracted the import batch workflow into a reusable app-level entrypoint `(*App).IngestImports(...)` so future in-process watcher/relay integrations can reuse the same scope resolution, session creation, imported-note materialization, and failure-export behavior without shelling out to the CLI. The CLI command now delegates to that method, and app coverage verifies the embedded path directly.
+- In progress: none.
+- Blockers: none.
+- Next step: decide whether to build an actual in-tree watcher/relay adapter on top of `App.IngestImports`, or stop here and treat the reusable app method as sufficient integration scaffolding for now.
+### 2026-03-16 Session Update
+
+- Completed: Added `follow-imports` as a checkpointed long-lived adapter on top of `App.IngestImports(...)`. The new command polls a JSONL file for newly appended complete lines, persists byte-offset state in a sidecar checkpoint file, resets cleanly on truncation, and derives per-batch failed-output / failed-manifest paths so operator retry artifacts are not overwritten.
+- In progress: none.
+- Blockers: none.
+- Next step: decide whether polling-based follow mode is sufficient for watcher/relay integration for now, or whether a later slice should add native filesystem notifications, rotation metadata, or multi-input fan-in.
 
 ## Recommended Next Step
 

docs/go/maintainer/implementation-plan.md

@@ -120,6 +120,7 @@ Responsibilities:
 - initialize services
 - register MCP tools
 - start server mode
+- expose reusable app-level workflows such as import ingestion for future in-process integrations
 
 ### `internal/config`
 

docs/go/operator/README.md

@@ -7,7 +7,7 @@ Start here:
 - [Client Examples](./client-examples.md)
   Real MCP client registration examples for local stdio and remote HTTP.
 - [Import Ingestion](./import-ingestion.md)
-  JSONL batch ingestion for watcher or relay artifacts through `ingest-imports`.
+  JSONL batch ingestion through `ingest-imports` plus checkpointed follow-mode ingestion through `follow-imports`.
 - [Release Readiness](./release-readiness.md)
   Packaging, readiness, and release checklist.
 - [Troubleshooting](./troubleshooting.md)

docs/go/operator/import-ingestion.md

@@ -2,7 +2,7 @@
 
 ## Purpose
 
-This document explains how operators can use `codex-mem ingest-imports` to turn watcher or relay batches into durable imported notes plus import audit records.
+This document explains how operators can use `codex-mem ingest-imports` for one-shot batches and `codex-mem follow-imports` for long-lived incremental consumption of watcher or relay JSONL feeds.
 
 Audience:
 
@@ -12,6 +12,7 @@ Audience:
 Use this when:
 
 - you need a one-shot batch bridge into the imported-note workflow
+- you need a checkpointed long-lived bridge for a growing JSONL file
 - your upstream process can emit newline-delimited JSON events
 
 Do not use this for:
@@ -21,6 +22,9 @@ Do not use this for:
 
 ## Command Shape
 
+Use `ingest-imports` when you already have a bounded batch to replay.
+Use `follow-imports` when another process keeps appending to the same JSONL file and you want `codex-mem` to checkpoint progress between polling passes.
+
 Minimal stdin example:
 
 ```powershell
@@ -33,12 +37,43 @@ Read from a file and print JSON:
 codex-mem.exe ingest-imports --source relay_import --input .\relay-events.jsonl --json
 ```
 
+Continue past bad lines and keep successful imports:
+
+```powershell
+codex-mem.exe ingest-imports --source watcher_import --input .\events.jsonl --continue-on-error --json
+```
+
+Export failed lines for retry after the batch finishes:
+
+```powershell
+codex-mem.exe ingest-imports --source watcher_import --input .\events.jsonl --continue-on-error --failed-output .\failed-events.jsonl --json
+```
+
+Export a machine-readable retry manifest alongside the raw failed lines:
+
+```powershell
+codex-mem.exe ingest-imports --source watcher_import --input .\events.jsonl --continue-on-error --failed-output .\failed-events.jsonl --failed-manifest .\failed-events.json --json
+```
+
+Follow a growing JSONL file once and checkpoint the consumed offset:
+
+```powershell
+codex-mem.exe follow-imports --source watcher_import --input .\events.jsonl --once --json
+```
+
+Run as a long-lived poller with an explicit checkpoint file:
+
+```powershell
+codex-mem.exe follow-imports --source relay_import --input .\relay-events.jsonl --state-file .\relay-events.offset.json --poll-interval 10s
+```
+
 Useful flags:
 
 - `--source watcher_import|relay_import`
-  Required. Declares the provenance source for every event in the batch.
+  Required. Declares the provenance source for every event in the input stream.
 - `--input <path>`
-  Optional. Reads JSONL from a file instead of stdin.
+  Optional for `ingest-imports`. Reads JSONL from a file instead of stdin.
+  Required for `follow-imports`.
 - `--cwd <path>`
   Optional. Resolves scope from a specific workspace root.
 - `--branch-name <name>`
@@ -49,6 +84,20 @@ Useful flags:
   Optional. Overrides the default ingestion session task summary.
 - `--json`
   Optional. Prints a structured report instead of line-oriented text output.
+- `--continue-on-error`
+  `ingest-imports` only. Keeps scanning after per-line decode or import failures and returns a partial-success report when at least one event succeeds.
+- `--failed-output <path>`
+  Optional. For `ingest-imports`, requires `--continue-on-error` and writes the original failed input lines to a JSONL file for manual fix-up or replay.
+  For `follow-imports`, each polling batch derives a range-suffixed file from the provided base path so earlier failures are not overwritten.
+- `--failed-manifest <path>`
+  Optional. For `ingest-imports`, requires `--continue-on-error` and writes a JSON manifest with per-line error metadata and raw failed input.
+  For `follow-imports`, each polling batch derives a range-suffixed manifest path from the provided base path.
+- `--state-file <path>`
+  `follow-imports` only. Optional. Stores the consumed byte offset checkpoint. Defaults to `<input>.offset.json`.
+- `--poll-interval <duration>`
+  `follow-imports` only. Optional. Controls how often the input file is polled for appended complete lines. Defaults to `5s`.
+- `--once`
+  `follow-imports` only. Optional. Runs one poll/ingest pass and exits instead of staying in the polling loop.
 
 ## Event Schema
 
@@ -103,11 +152,15 @@ Text mode prints a compact summary such as:
 
 ```text
 ingest imports ok
+status=ok
 source=watcher_import
 input=stdin
 session_id=sess_20260316_001
 resolved_by=repo_remote
+continue_on_error=false
+attempted=2
 processed=2
+failed=0
 materialized=1
 suppressed=1
 note_deduplicated=0
@@ -116,10 +169,20 @@ warnings=1
 ```
 
 JSON mode returns the same summary plus per-line results, including the created or reused `note_id` and `import_id`.
+When a line fails in `--continue-on-error` mode, that result entry includes a structured `error` payload instead.
+If `--failed-output` is set, the report also includes the resolved output path and how many failed lines were written there.
+If `--failed-manifest` is set, the report also includes the manifest path and how many failures were captured there.
+`follow-imports` reports the input path, checkpoint file, consumed offset, pending trailing bytes, truncation detection, and the nested batch report for whatever newly appended complete lines were imported during that poll.
 
 ## Operational Notes
 
 - `ingest-imports` starts one fresh session for the whole batch after resolving scope.
+- `follow-imports` starts one fresh session per consumed polling batch, not one session for the lifetime of the process.
 - Each event uses the same imported-note workflow as `memory_save_imported_note`.
 - Existing explicit memory wins over weaker imported duplicates in the same project.
-- The current implementation is fail-fast: the first invalid line stops the batch and returns an error.
+- The default implementation is fail-fast: the first invalid line stops the batch and returns an error.
+- `--continue-on-error` preserves successful lines, reports per-line failures, and still exits with an error if nothing in the batch imports successfully.
+- `--failed-output` writes the original failed JSONL lines without wrapping them, so operators can edit that file and replay it through the same command later.
+- `--failed-manifest` writes a structured JSON sidecar with line numbers, error codes, error messages, raw failed lines, and failed-output line numbers when available.
+- `follow-imports` only consumes complete newline-terminated lines. A partially written trailing line is left in place until a later poll sees its terminating newline.
+- If the followed input file is truncated or rotated to a smaller size, `follow-imports` resets its checkpoint to byte offset `0` and continues from the start of the new file contents.