Skip to content

Commit 4817554

Browse files
committed
✨ feat(import-follow): add --target-profile presets for cleanup and audit hygiene commands
- Add target-profile flag with values: all, artifacts, state, retry, health - artifacts profile covers checkpoint plus retry targets without follow-health - all profile enables every target class for full hygiene sweeps - state, retry, and health profiles enable single target classes - Add parser normalization and profile-to-boolean expansion helpers - Add unit tests for profile parsing and end-to-end cleanup/audit runs - Add example fixtures for target-profile-all and target-profile-retry - Update operator docs with preset examples and usage guidance
1 parent 9acaff4 commit 4817554

9 files changed

Lines changed: 657 additions & 2 deletions

README.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,10 @@ They are not MCP tools and are not the normal end-user interaction path.
8080
Imports newline-delimited watcher or relay note events into durable imported notes plus audit records, with optional partial-success handling plus retry-oriented failure exports.
8181
- `codex-mem follow-imports --source watcher_import --input events-a.jsonl [--input events-b.jsonl ...] [--state-file events-a.offset.json --state-file events-b.offset.json ...] [--watch-mode auto|notify|poll] [--poll-interval 5s] [--once] [--json]`
8282
Follows one or more watcher or relay JSONL files incrementally, prefers filesystem notifications with polling fallback by default, keeps one checkpoint per input, automatically retries watcher recovery in `auto` mode, and reports command-level watch state plus poll-catchup/recovery events and warnings alongside per-input imported-note results.
83+
- `codex-mem cleanup-follow-imports [--target-profile all|artifacts|state|retry|health] [...]`
84+
Removes selected follow-imports checkpoint, retry-artifact, and stale-health artifacts. `--target-profile` can enable common cleanup target sets before you add path, age, or dry-run filters.
85+
- `codex-mem audit-follow-imports [--target-profile all|artifacts|state|retry|health] [...]`
86+
Reports the same hygiene targets without deleting them. `--target-profile` can enable common audit target sets before you add path, age, or fail-if-matched controls.
8387
- `codex-mem migrate`
8488
Opens the configured SQLite database and applies embedded migrations.
8589
- `codex-mem serve`

docs/go/maintainer/development-tracker.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,27 @@ Immediate next tasks:
151151

152152
### 2026-03-17 Session Update
153153

154+
- Completed: Added a fifth follow-import hygiene target preset: `--target-profile artifacts`. It expands to checkpoint-sidecar plus retry-artifact targets without touching follow-health snapshots, which fills the gap between `all` and the narrower `state` / `retry` profiles. Parser and end-to-end CLI tests now verify that `artifacts` selects state plus retry work while leaving follow-health out of both cleanup and audit reports, and the operator docs/README now advertise the new preset.
155+
- In progress: none.
156+
- Blockers: none.
157+
- Next step: decide whether the current target-profile catalog is now complete enough, or whether future operator feedback should drive any further preset additions.
158+
159+
### 2026-03-17 Session Update
160+
161+
- Completed: Added checked-in sample outputs that explicitly exercise the new follow-import hygiene target presets. The repository now includes a `cleanup-follow-imports` text fixture for `--target-profile all` plus an `audit-follow-imports` JSON fixture for `--target-profile retry`, and the example-list tests plus operator docs now advertise those preset-focused fixtures alongside the older age/pattern examples.
162+
- In progress: none.
163+
- Blockers: none.
164+
- Next step: decide whether the preset catalog is complete enough now, or whether operators would benefit more from another profile such as "artifacts-only" that excludes follow-health.
165+
166+
### 2026-03-17 Session Update
167+
168+
- Completed: Added `--target-profile` presets to `cleanup-follow-imports` and `audit-follow-imports` with initial `all`, `state`, `retry`, and `health` values. The profile layer expands to the existing prune/check booleans instead of introducing a second hygiene engine, so explicit `--prune-*` and `--check-*` flags still work and can be combined with a profile. App and parser coverage now verifies target-profile parsing plus end-to-end cleanup/audit runs, and the import-ingestion/README docs now show the new shorthand for common hygiene slices.
169+
- In progress: none.
170+
- Blockers: none.
171+
- Next step: decide whether the current target-profile catalog is enough, or whether follow/import operators would benefit more from checked-in sample outputs that exercise the new presets explicitly.
172+
173+
### 2026-03-17 Session Update
174+
154175
- Completed: Fixed Linux CI drift in unit tests and pinned the workflow lint toolchain to the same `golangci-lint` version used locally. The follow-import and readiness example-list tests now build expected fixture paths with `filepath.Join(...)` instead of hard-coding Windows separators, the duplicate follow-input test now uses a platform-neutral relative path, and `.github/workflows/build-release.yml` now pins `golangci-lint` to `v2.10.1` with the same `./...` target as the local command.
155176
- In progress: none.
156177
- Blockers: none.

docs/go/operator/import-ingestion.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,12 @@ Use a named retention preset instead of spelling out an age threshold:
105105
codex-mem.exe cleanup-follow-imports --failed-output .\failed-events.jsonl --prune-failed-output --retention-profile daily --dry-run
106106
```
107107

108+
Use a named target preset when the command should cover a common hygiene slice:
109+
110+
```powershell
111+
codex-mem.exe cleanup-follow-imports --target-profile all --input .\events.jsonl --failed-output .\failed-events.jsonl --failed-manifest .\failed-events.json --dry-run
112+
```
113+
108114
Audit whether cleanup work is pending and fail the command if anything in the selected target set matches:
109115

110116
```powershell
@@ -117,6 +123,18 @@ Run the same hygiene audit as a dedicated read-only report command:
117123
codex-mem.exe audit-follow-imports --input .\events.jsonl --check-state --failed-output .\failed-events.jsonl --check-failed-output --failed-manifest .\failed-events.json --check-failed-manifest --check-follow-health --retention-profile daily --fail-if-matched
118124
```
119125

126+
Focus that audit on retry exports only:
127+
128+
```powershell
129+
codex-mem.exe audit-follow-imports --target-profile retry --failed-output .\failed-events.jsonl --failed-manifest .\failed-events.json --retention-profile daily --fail-if-matched
130+
```
131+
132+
Cover checkpoint plus retry artifacts together while leaving follow-health untouched:
133+
134+
```powershell
135+
codex-mem.exe cleanup-follow-imports --target-profile artifacts --input .\events.jsonl --failed-output .\failed-events.jsonl --failed-manifest .\failed-events.json --dry-run
136+
```
137+
120138
Useful flags:
121139

122140
- `--source watcher_import|relay_import`
@@ -181,6 +199,8 @@ Useful flags:
181199
`cleanup-follow-imports` and `audit-follow-imports` only. Optional. Repeats or accepts comma-separated glob patterns that act as a whitelist for checkpoint and retry-artifact candidate paths. Patterns are matched against both the absolute path and the basename.
182200
- `--exclude <glob>`
183201
`cleanup-follow-imports` and `audit-follow-imports` only. Optional. Repeats or accepts comma-separated glob patterns that remove checkpoint and retry-artifact candidates from the matched set after includes are considered. Excludes win over includes.
202+
- `--target-profile all|artifacts|state|retry|health`
203+
`cleanup-follow-imports` and `audit-follow-imports` only. Optional. Expands to a documented default target set instead of making you spell out the target flags every time. `all` enables every target class, `artifacts` enables checkpoint plus retry-artifact targets while leaving follow-health alone, `state` enables only checkpoint sidecars, `retry` enables only failed-output plus failed-manifest artifacts, and `health` enables only follow-health inspection/pruning. Explicit `--prune-*` or `--check-*` flags still add extra targets on top.
184204
- `--retention-profile stale|daily|reset`
185205
`cleanup-follow-imports` and `audit-follow-imports` only. Optional. Expands to a documented default age threshold instead of making you spell out `--older-than` every time. `stale` means `1h`, `daily` means `24h`, and `reset` means `0s`. An explicit `--older-than` still overrides the profile.
186206

@@ -266,8 +286,10 @@ Checked-in sample outputs for common cleanup flows live under [../../../internal
266286

267287
- [cleanup-follow-imports-daily-dry-run.txt](../../../internal/app/testdata/cleanup-follow-imports-daily-dry-run.txt)
268288
- [cleanup-follow-imports-filtered-cleanup.json](../../../internal/app/testdata/cleanup-follow-imports-filtered-cleanup.json)
289+
- [cleanup-follow-imports-target-profile-all.txt](../../../internal/app/testdata/cleanup-follow-imports-target-profile-all.txt)
269290
- [audit-follow-imports-daily-audit.txt](../../../internal/app/testdata/audit-follow-imports-daily-audit.txt)
270291
- [audit-follow-imports-filtered-audit.json](../../../internal/app/testdata/audit-follow-imports-filtered-audit.json)
292+
- [audit-follow-imports-target-profile-retry.json](../../../internal/app/testdata/audit-follow-imports-target-profile-retry.json)
271293

272294
If a deliberate output change makes those fixtures drift, refresh the cleanup fixtures from the repository root through the test-only maintainer helper:
273295

@@ -326,6 +348,7 @@ go test ./internal/app -run "Test(Audit|Cleanup)FollowImportsExampleOutputsStayI
326348
- Add `--older-than <duration>` when you want cleanup or audit to ignore very recent checkpoint or retry files. This age gate applies to checkpoint sidecars plus range-suffixed retry artifacts, not to the stale-health decision, which still uses the existing follow-health staleness policy.
327349
- Add `--include` when the cleanup or audit target should stay inside one file family, input label, or path prefix. This is especially useful for multi-input runs where you only want to inspect or clean artifacts related to one input at a time.
328350
- Add `--exclude` for the final guardrail when a broad include or input set still catches more than you want. Exclude patterns override includes, so they are the safer place to carve out known paths or suffixes.
351+
- Use `--target-profile all` when you want the old "turn on every target flag" behavior without spelling out each flag, `--target-profile artifacts` when the command should cover checkpoint and retry files but intentionally leave follow-health snapshots alone, `--target-profile retry` when scheduled audits should look only at range-suffixed retry exports, `--target-profile state` when checkpoint hygiene is the only concern, and `--target-profile health` when you only need the last-known follow-health snapshot.
329352
- Use `--retention-profile stale` when you want a quick ad-hoc cleanup or audit pass that ignores artifacts newer than one hour, `--retention-profile daily` for a broader once-per-day sweep, and `--retention-profile reset` when you want the selected target set matched immediately. If one preset is close but not quite right, keep the profile for readability and add an explicit `--older-than` override.
330353
- `cleanup-follow-imports --prune-state` derives the same default checkpoint path that `follow-imports` would use for each `--input`, so you can clean old sidecars without repeating every `.offset.json` name manually.
331354
- `cleanup-follow-imports --prune-failed-output` and `--prune-failed-manifest` remove only batch-scoped retry artifacts whose names end in the byte-range suffix that `follow-imports` generates (for example `failed.events-a.0-42.jsonl`). The base path you pass stays untouched, which avoids deleting a placeholder or unrelated manually curated file with the unsuffixed name.

internal/app/follow_import_example_fixtures_test.go

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,50 @@ func cleanupFollowImportsExampleFixtures() []cleanupFollowImportsExampleFixture
217217
},
218218
},
219219
},
220+
{
221+
Name: "target-profile-all-text",
222+
RelativePath: "cleanup-follow-imports-target-profile-all.txt",
223+
JSON: false,
224+
Report: cleanupFollowImportsReport{
225+
DryRun: false,
226+
FailIfMatched: true,
227+
MatchFound: true,
228+
TargetProfile: followImportsTargetProfileAll,
229+
RetentionProfile: cleanupFollowImportsRetentionProfileReset,
230+
OlderThanSeconds: 0,
231+
Status: "ok",
232+
StateFiles: cleanupFollowImportsPathSummary{
233+
Requested: 1,
234+
Matched: 1,
235+
Removed: 1,
236+
Missing: 0,
237+
MatchedPaths: []string{`D:\Ops\follow\events.offset.json`},
238+
RemovedPaths: []string{`D:\Ops\follow\events.offset.json`},
239+
},
240+
FailedOutputs: cleanupFollowImportsPatternSummary{
241+
Bases: 1,
242+
Matched: 2,
243+
Removed: 2,
244+
BasePaths: []string{`D:\Ops\follow\failed\failed.jsonl`},
245+
MatchedPaths: []string{`D:\Ops\follow\failed\failed.0-42.jsonl`, `D:\Ops\follow\failed\failed.43-84.jsonl`},
246+
RemovedPaths: []string{`D:\Ops\follow\failed\failed.0-42.jsonl`, `D:\Ops\follow\failed\failed.43-84.jsonl`},
247+
},
248+
FailedManifests: cleanupFollowImportsPatternSummary{
249+
Bases: 1,
250+
Matched: 2,
251+
Removed: 2,
252+
BasePaths: []string{`D:\Ops\follow\failed\failed.json`},
253+
MatchedPaths: []string{`D:\Ops\follow\failed\failed.0-42.json`, `D:\Ops\follow\failed\failed.43-84.json`},
254+
RemovedPaths: []string{`D:\Ops\follow\failed\failed.0-42.json`, `D:\Ops\follow\failed\failed.43-84.json`},
255+
},
256+
FollowHealth: cleanupFollowImportsFollowHealthView{
257+
File: `D:\Ops\follow\logs\follow-imports.health.json`,
258+
WouldPrune: false,
259+
Pruned: true,
260+
PruneReason: "stale",
261+
},
262+
},
263+
},
220264
}
221265
}
222266

@@ -347,6 +391,40 @@ func auditFollowImportsExampleFixtures() []auditFollowImportsExampleFixture {
347391
},
348392
},
349393
},
394+
{
395+
Name: "target-profile-retry-json",
396+
RelativePath: "audit-follow-imports-target-profile-retry.json",
397+
JSON: true,
398+
Report: auditFollowImportsReport{
399+
FailIfMatched: true,
400+
MatchFound: true,
401+
TargetProfile: followImportsTargetProfileRetry,
402+
RetentionProfile: cleanupFollowImportsRetentionProfileDaily,
403+
OlderThanSeconds: 86400,
404+
Status: "ok",
405+
StateFiles: auditFollowImportsPathSummary{},
406+
FailedOutputs: auditFollowImportsPatternSummary{
407+
Bases: 1,
408+
Matched: 1,
409+
SkippedByAge: 1,
410+
BasePaths: []string{`D:\Ops\follow\failed\failed.jsonl`},
411+
MatchedPaths: []string{`D:\Ops\follow\failed\failed.0-42.jsonl`},
412+
SkippedByAgePaths: []string{`D:\Ops\follow\failed\failed.43-84.jsonl`},
413+
},
414+
FailedManifests: auditFollowImportsPatternSummary{
415+
Bases: 1,
416+
Matched: 1,
417+
SkippedByAge: 1,
418+
BasePaths: []string{`D:\Ops\follow\failed\failed.json`},
419+
MatchedPaths: []string{`D:\Ops\follow\failed\failed.0-42.json`},
420+
SkippedByAgePaths: []string{`D:\Ops\follow\failed\failed.43-84.json`},
421+
},
422+
FollowHealth: auditFollowImportsHealthView{
423+
File: `D:\Ops\follow\logs\follow-imports.health.json`,
424+
Present: false,
425+
},
426+
},
427+
},
350428
}
351429
}
352430

0 commit comments

Comments
 (0)