Skip to content

Commit 23bdbee

Browse files
committed
✨ feat(import-follow): add summary-only hygiene reports
- add --summary-only to cleanup-follow-imports and audit-follow-imports - suppress per-path detail output while preserving counts and follow-health status - cover compact reporting in tests and operator documentation
1 parent 4817554 commit 23bdbee

6 files changed

Lines changed: 208 additions & 4 deletions

File tree

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,9 +81,9 @@ They are not MCP tools and are not the normal end-user interaction path.
8181
- `codex-mem follow-imports --source watcher_import --input events-a.jsonl [--input events-b.jsonl ...] [--state-file events-a.offset.json --state-file events-b.offset.json ...] [--watch-mode auto|notify|poll] [--poll-interval 5s] [--once] [--json]`
8282
Follows one or more watcher or relay JSONL files incrementally, prefers filesystem notifications with polling fallback by default, keeps one checkpoint per input, automatically retries watcher recovery in `auto` mode, and reports command-level watch state plus poll-catchup/recovery events and warnings alongside per-input imported-note results.
8383
- `codex-mem cleanup-follow-imports [--target-profile all|artifacts|state|retry|health] [...]`
84-
Removes selected follow-imports checkpoint, retry-artifact, and stale-health artifacts. `--target-profile` can enable common cleanup target sets before you add path, age, or dry-run filters.
84+
Removes selected follow-imports checkpoint, retry-artifact, and stale-health artifacts. `--target-profile` can enable common cleanup target sets before you add path, age, dry-run, or `--summary-only` report filters.
8585
- `codex-mem audit-follow-imports [--target-profile all|artifacts|state|retry|health] [...]`
86-
Reports the same hygiene targets without deleting them. `--target-profile` can enable common audit target sets before you add path, age, or fail-if-matched controls.
86+
Reports the same hygiene targets without deleting them. `--target-profile` can enable common audit target sets before you add path, age, fail-if-matched, or `--summary-only` controls.
8787
- `codex-mem migrate`
8888
Opens the configured SQLite database and applies embedded migrations.
8989
- `codex-mem serve`

docs/go/maintainer/development-tracker.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,13 @@ Immediate next tasks:
151151

152152
### 2026-03-17 Session Update
153153

154+
- Completed: Added `--summary-only` to `cleanup-follow-imports` and `audit-follow-imports` for automation-friendly compact reports. The hygiene commands still compute the same target counts, fail-if-matched behavior, and follow-health status, but they now omit detailed checkpoint and retry-artifact path lists from text and JSON output when summary-only mode is requested. Parser coverage plus end-to-end run tests now verify both flag parsing and the suppression of per-path details, and the operator docs/README now show when to use the compact mode for scheduled hygiene runs.
155+
- In progress: none.
156+
- Blockers: none.
157+
- Next step: decide whether follow/import operators would benefit more from another report-shaping control such as capped path samples, or whether the next slice should move back to a new ingestion/follow capability.
158+
159+
### 2026-03-17 Session Update
160+
154161
- Completed: Added a fifth follow-import hygiene target preset: `--target-profile artifacts`. It expands to checkpoint-sidecar plus retry-artifact targets without touching follow-health snapshots, which fills the gap between `all` and the narrower `state` / `retry` profiles. Parser and end-to-end CLI tests now verify that `artifacts` selects state plus retry work while leaving follow-health out of both cleanup and audit reports, and the operator docs/README now advertise the new preset.
155162
- In progress: none.
156163
- Blockers: none.

docs/go/operator/import-ingestion.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,12 @@ Focus that audit on retry exports only:
129129
codex-mem.exe audit-follow-imports --target-profile retry --failed-output .\failed-events.jsonl --failed-manifest .\failed-events.json --retention-profile daily --fail-if-matched
130130
```
131131

132+
Keep scheduled hygiene logs compact while preserving the matched counts:
133+
134+
```powershell
135+
codex-mem.exe audit-follow-imports --target-profile artifacts --input .\events.jsonl --failed-output .\failed-events.jsonl --failed-manifest .\failed-events.json --summary-only --json
136+
```
137+
132138
Cover checkpoint plus retry artifacts together while leaving follow-health untouched:
133139

134140
```powershell
@@ -195,6 +201,8 @@ Useful flags:
195201
`cleanup-follow-imports` only. Optional. Computes the same cleanup candidates and reports what would be removed, but leaves every file in place.
196202
- `--fail-if-matched`
197203
`cleanup-follow-imports` and `audit-follow-imports` only. Optional. Returns a non-zero exit after writing the report when the selected target set matched at least one checkpoint sidecar, retry artifact, or stale follow-health snapshot. This is especially useful for CI or scheduled hygiene audits.
204+
- `--summary-only`
205+
`cleanup-follow-imports` and `audit-follow-imports` only. Optional. Keeps the same aggregate counts, target metadata, and follow-health status, but omits enumerated checkpoint and retry-artifact path lists from the text or JSON report. This is useful for scheduled automation where path-by-path detail would create noisy logs.
198206
- `--include <glob>`
199207
`cleanup-follow-imports` and `audit-follow-imports` only. Optional. Repeats or accepts comma-separated glob patterns that act as a whitelist for checkpoint and retry-artifact candidate paths. Patterns are matched against both the absolute path and the basename.
200208
- `--exclude <glob>`
@@ -279,8 +287,9 @@ If `--failed-output` is set, the report also includes the resolved output path a
279287
If `--failed-manifest` is set, the report also includes the manifest path and how many failures were captured there.
280288
Single-input `follow-imports` reports the input path, checkpoint file, requested watch mode, active watch mode, fallback count, transition count, cumulative poll-catchup count and bytes, any warning summaries, any structured watch events since the previous emitted report, consumed offset, pending trailing bytes, whether the checkpoint was reset, the reset reason, truncation detection, and the nested batch report for whatever newly appended complete lines were imported during that poll.
281289
Multi-input `follow-imports` returns one aggregate report with command-level watch state, cumulative poll-catchup counters, warning summaries, per-process watch events, total consumed and pending bytes, and one nested per-input report for each followed file.
282-
`cleanup-follow-imports` reports whether the run was a dry-run, whether `--fail-if-matched` was active, whether the selected target set matched anything, the named retention profile when one is active, the configured age gate in seconds, any include/exclude patterns in effect, how many checkpoint sidecars and derived retry artifacts matched cleanup versus were actually removed, which files were skipped because they were filtered out by pattern or were too new, which explicit state files were already missing, and whether it pruned or would prune the stale follow-health sidecar.
283-
`audit-follow-imports` reports the same target-selection metadata and matched-versus-skipped counts as a read-only hygiene pass, plus whether the follow-health snapshot is present, when it was last updated, whether it is stale, and any warning summaries carried by that snapshot.
290+
`cleanup-follow-imports` reports whether the run was a dry-run, whether `--fail-if-matched` was active, whether the selected target set matched anything, whether `--summary-only` was active, the named retention profile when one is active, the configured age gate in seconds, any include/exclude patterns in effect, how many checkpoint sidecars and derived retry artifacts matched cleanup versus were actually removed, which files were skipped because they were filtered out by pattern or were too new, which explicit state files were already missing, and whether it pruned or would prune the stale follow-health sidecar.
291+
`audit-follow-imports` reports the same target-selection metadata and matched-versus-skipped counts as a read-only hygiene pass, plus whether `--summary-only` was active, whether the follow-health snapshot is present, when it was last updated, whether it is stale, and any warning summaries carried by that snapshot.
292+
When `--summary-only` is set, the aggregate counts stay the same but the detailed checkpoint and retry-artifact path lists are omitted from both text and JSON output.
284293

285294
Checked-in sample outputs for common cleanup flows live under [../../../internal/app/testdata](../../../internal/app/testdata/):
286295

@@ -345,6 +354,7 @@ go test ./internal/app -run "Test(Audit|Cleanup)FollowImportsExampleOutputsStayI
345354
- If you want the same target selection and age/pattern filtering logic without any deletion, use `codex-mem.exe audit-follow-imports`. It is the cleaner fit for scheduled reports, pre-cleanup review, and automation that should fail on pending hygiene work before anything is removed.
346355
- Add `--dry-run` first when you are not fully sure about the target set. The report shows the same matched file list and stale-health outcome it would use for a real cleanup pass, but without deleting anything.
347356
- Add `--fail-if-matched` when the command should act as a hygiene gate instead of only as an informational report. On `audit-follow-imports` the command stays read-only; on `cleanup-follow-imports --dry-run` it behaves the same way while still showing what a future deletion pass would remove.
357+
- Add `--summary-only` when scheduled automation or CI should preserve the counts and status metadata but avoid printing long per-path lists into logs or inbox output.
348358
- Add `--older-than <duration>` when you want cleanup or audit to ignore very recent checkpoint or retry files. This age gate applies to checkpoint sidecars plus range-suffixed retry artifacts, not to the stale-health decision, which still uses the existing follow-health staleness policy.
349359
- Add `--include` when the cleanup or audit target should stay inside one file family, input label, or path prefix. This is especially useful for multi-input runs where you only want to inspect or clean artifacts related to one input at a time.
350360
- Add `--exclude` for the final guardrail when a broad include or input set still catches more than you want. Exclude patterns override includes, so they are the safer place to carve out known paths or suffixes.

internal/app/import_follow.go

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ type followImportsHygieneOptions struct {
6767
CWD string
6868
JSON bool
6969
FailIfMatched bool
70+
SummaryOnly bool
7071
OlderThan time.Duration
7172
olderThanExplicit bool
7273
}
@@ -221,6 +222,7 @@ type cleanupFollowImportsReport struct {
221222
DryRun bool `json:"dry_run"`
222223
FailIfMatched bool `json:"fail_if_matched"`
223224
MatchFound bool `json:"match_found"`
225+
SummaryOnly bool `json:"summary_only,omitempty"`
224226
TargetProfile string `json:"target_profile,omitempty"`
225227
RetentionProfile string `json:"retention_profile,omitempty"`
226228
OlderThanSeconds int64 `json:"older_than_seconds,omitempty"`
@@ -270,6 +272,7 @@ type cleanupFollowImportsFollowHealthView struct {
270272
type auditFollowImportsReport struct {
271273
FailIfMatched bool `json:"fail_if_matched"`
272274
MatchFound bool `json:"match_found"`
275+
SummaryOnly bool `json:"summary_only,omitempty"`
273276
TargetProfile string `json:"target_profile,omitempty"`
274277
RetentionProfile string `json:"retention_profile,omitempty"`
275278
OlderThanSeconds int64 `json:"older_than_seconds,omitempty"`
@@ -810,6 +813,9 @@ func parseFollowImportsHygieneBooleanFlag(arg string, options *followImportsHygi
810813
case "--fail-if-matched":
811814
options.FailIfMatched = true
812815
return true
816+
case "--summary-only":
817+
options.SummaryOnly = true
818+
return true
813819
default:
814820
return false
815821
}
@@ -1853,6 +1859,7 @@ func cleanupFollowImportsAt(cfg config.Config, options cleanupFollowImportsOptio
18531859
report := cleanupFollowImportsReport{
18541860
DryRun: options.DryRun,
18551861
FailIfMatched: options.FailIfMatched,
1862+
SummaryOnly: options.SummaryOnly,
18561863
TargetProfile: options.TargetProfile,
18571864
RetentionProfile: options.RetentionProfile,
18581865
OlderThanSeconds: int64(options.OlderThan / time.Second),
@@ -1889,6 +1896,9 @@ func cleanupFollowImportsAt(cfg config.Config, options cleanupFollowImportsOptio
18891896
}
18901897
}
18911898
report.MatchFound = cleanupFollowImportsHasMatches(report)
1899+
if options.SummaryOnly {
1900+
report = cleanupFollowImportsSummaryOnlyReport(report)
1901+
}
18921902

18931903
return report, nil
18941904
}
@@ -1921,6 +1931,7 @@ func auditFollowImportsAt(cfg config.Config, options auditFollowImportsOptions,
19211931
scanOptions := options.cleanupDryRunOptions()
19221932
report := auditFollowImportsReport{
19231933
FailIfMatched: options.FailIfMatched,
1934+
SummaryOnly: options.SummaryOnly,
19241935
TargetProfile: options.TargetProfile,
19251936
RetentionProfile: options.RetentionProfile,
19261937
OlderThanSeconds: int64(options.OlderThan / time.Second),
@@ -1961,9 +1972,47 @@ func auditFollowImportsAt(cfg config.Config, options auditFollowImportsOptions,
19611972
}
19621973

19631974
report.MatchFound = auditFollowImportsHasMatches(report)
1975+
if options.SummaryOnly {
1976+
report = auditFollowImportsSummaryOnlyReport(report)
1977+
}
19641978
return report, nil
19651979
}
19661980

1981+
func cleanupFollowImportsSummaryOnlyReport(report cleanupFollowImportsReport) cleanupFollowImportsReport {
1982+
report.StateFiles.MatchedPaths = nil
1983+
report.StateFiles.RemovedPaths = nil
1984+
report.StateFiles.MissingPaths = nil
1985+
report.StateFiles.SkippedByPatternPaths = nil
1986+
report.StateFiles.SkippedByAgePaths = nil
1987+
report.FailedOutputs.BasePaths = nil
1988+
report.FailedOutputs.MatchedPaths = nil
1989+
report.FailedOutputs.RemovedPaths = nil
1990+
report.FailedOutputs.SkippedByPatternPaths = nil
1991+
report.FailedOutputs.SkippedByAgePaths = nil
1992+
report.FailedManifests.BasePaths = nil
1993+
report.FailedManifests.MatchedPaths = nil
1994+
report.FailedManifests.RemovedPaths = nil
1995+
report.FailedManifests.SkippedByPatternPaths = nil
1996+
report.FailedManifests.SkippedByAgePaths = nil
1997+
return report
1998+
}
1999+
2000+
func auditFollowImportsSummaryOnlyReport(report auditFollowImportsReport) auditFollowImportsReport {
2001+
report.StateFiles.MatchedPaths = nil
2002+
report.StateFiles.MissingPaths = nil
2003+
report.StateFiles.SkippedByPatternPaths = nil
2004+
report.StateFiles.SkippedByAgePaths = nil
2005+
report.FailedOutputs.BasePaths = nil
2006+
report.FailedOutputs.MatchedPaths = nil
2007+
report.FailedOutputs.SkippedByPatternPaths = nil
2008+
report.FailedOutputs.SkippedByAgePaths = nil
2009+
report.FailedManifests.BasePaths = nil
2010+
report.FailedManifests.MatchedPaths = nil
2011+
report.FailedManifests.SkippedByPatternPaths = nil
2012+
report.FailedManifests.SkippedByAgePaths = nil
2013+
return report
2014+
}
2015+
19672016
func newAuditFollowImportsPathSummary(summary cleanupFollowImportsPathSummary) auditFollowImportsPathSummary {
19682017
return auditFollowImportsPathSummary{
19692018
Requested: summary.Requested,
@@ -2531,6 +2580,9 @@ func formatCleanupFollowImportsReport(report cleanupFollowImportsReport) string
25312580
fmt.Sprintf("follow_health_pruned=%t", report.FollowHealth.Pruned),
25322581
fmt.Sprintf("follow_health_prune_reason=%s", fallbackString(report.FollowHealth.PruneReason)),
25332582
}
2583+
if report.SummaryOnly {
2584+
lines = append(lines, "summary_only=true")
2585+
}
25342586
if report.TargetProfile != "" {
25352587
lines = append(lines, fmt.Sprintf("target_profile=%s", report.TargetProfile))
25362588
}
@@ -2630,6 +2682,9 @@ func formatAuditFollowImportsReport(report auditFollowImportsReport) string {
26302682
fmt.Sprintf("follow_health_watch_poll_catchup_bytes=%d", report.FollowHealth.WatchPollCatchupBytes),
26312683
fmt.Sprintf("follow_health_warnings=%d", len(report.FollowHealth.Warnings)),
26322684
}
2685+
if report.SummaryOnly {
2686+
lines = append(lines, "summary_only=true")
2687+
}
26332688
if report.TargetProfile != "" {
26342689
lines = append(lines, fmt.Sprintf("target_profile=%s", report.TargetProfile))
26352690
}

internal/app/import_follow_test.go

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ func TestParseCleanupFollowImportsOptions(t *testing.T) {
143143
"--older-than", "2h",
144144
"--dry-run",
145145
"--fail-if-matched",
146+
"--summary-only",
146147
"--prune-state",
147148
"--prune-failed-output",
148149
"--prune-failed-manifest",
@@ -164,6 +165,9 @@ func TestParseCleanupFollowImportsOptions(t *testing.T) {
164165
if !options.FailIfMatched {
165166
t.Fatal("expected fail-if-matched option")
166167
}
168+
if !options.SummaryOnly {
169+
t.Fatal("expected summary-only option")
170+
}
167171
if got, want := options.OlderThan, 2*time.Hour; got != want {
168172
t.Fatalf("older-than mismatch: got %s want %s", got, want)
169173
}
@@ -339,6 +343,7 @@ func TestParseAuditFollowImportsOptions(t *testing.T) {
339343
"--cwd", "D:/Code/go/codex-mem",
340344
"--older-than", "2h",
341345
"--fail-if-matched",
346+
"--summary-only",
342347
"--check-state",
343348
"--check-failed-output",
344349
"--check-failed-manifest",
@@ -357,6 +362,9 @@ func TestParseAuditFollowImportsOptions(t *testing.T) {
357362
if !options.FailIfMatched {
358363
t.Fatal("expected fail-if-matched option")
359364
}
365+
if !options.SummaryOnly {
366+
t.Fatal("expected summary-only option")
367+
}
360368
if got, want := options.OlderThan, 2*time.Hour; got != want {
361369
t.Fatalf("older-than mismatch: got %s want %s", got, want)
362370
}

0 commit comments

Comments
 (0)