Skip to content

Commit 268f5cd

Browse files
committed
✨ feat(imports): add aggregate summary blocks to multi-input follow-imports reports
- Add batch_summary rollup for attempted, processed, failed, materialized, suppressed, dedupe, and warning-by-code counts - Add retry_summary block totaling failed-output and failed-manifest activity with aggregate retry artifact paths - Add batch_error_summary block counting inputs with follow-level batch_error payloads and error code distribution - Add state_summary block rolling up truncation and checkpoint-reset visibility plus reset-reason counts - Add pending_summary block counting inputs with trailing pending bytes and highlighting largest pending backlog - Add watch_summary block rolling up watch-event kind counts plus mode-transition pairs - Emit idle aggregate reports when pending_summary or state_summary are present to surface backlog and reset events - Update operator docs and JSON fixtures to describe all six summary blocks
1 parent 9faa17f commit 268f5cd

7 files changed

Lines changed: 810 additions & 60 deletions

File tree

docs/go/maintainer/development-tracker.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# codex-mem Go Development Tracker
22

3-
Last updated: 2026-03-18
3+
Last updated: 2026-03-19
44
Status: active
55

66
## Purpose
@@ -149,6 +149,13 @@ Immediate next tasks:
149149

150150
## Decisions Log
151151

152+
### 2026-03-19 Session Update
153+
154+
- Completed: Extended multi-input `follow-imports` aggregate reports with a top-level `batch_summary` rollup so operators can see aggregate attempted, processed, failed, materialized, suppressed, dedupe, suppression-reason, and warning-by-code counts without manually summing each nested input batch. Added a companion `retry_summary` block that totals failed-output and failed-manifest activity across the consumed inputs in the same pass and now also surfaces aggregate retry artifact paths, added a compact `batch_error_summary` block so the aggregate report also shows how many inputs surfaced follow-level `batch_error` payloads and which error codes appeared, added a `state_summary` block that rolls up truncation and checkpoint-reset visibility plus reset-reason counts across inputs, added a `pending_summary` block that counts inputs with trailing pending bytes and highlights the largest pending backlog, and then added a compact `watch_summary` block that rolls up watch-event kind counts plus mode-transition pairs for the current emitted watch event batch. Idle aggregate reports with pending-summary-only or state-summary-only changes now still emit so backlog and reset/truncation events are not silently hidden. JSON fixtures, format coverage, and operator docs now describe all six summary blocks alongside the existing per-input reports.
155+
- In progress: none.
156+
- Blockers: none.
157+
- Next step: decide whether this aggregate reporting slice is now complete, or whether the better next maintenance step is to consolidate the aggregate-report fixture/test setup so future shape changes touch fewer duplicated literals.
158+
152159
### 2026-03-18 Session Update
153160

154161
- Completed: Tightened command-example manifest parsing to reject duplicate fields inside one entry instead of silently letting a later value overwrite an earlier one. Parser tests now cover duplicate-key rejection alongside missing metadata and malformed tag lists, which makes hand-edited catalog drift easier to catch before it reaches the embedded binary.

docs/go/operator/import-ingestion.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,7 @@ When a line fails in `--continue-on-error` mode, that result entry includes a st
309309
If `--failed-output` is set, the report also includes the resolved output path and how many failed lines were written there.
310310
If `--failed-manifest` is set, the report also includes the manifest path and how many failures were captured there.
311311
Single-input `follow-imports` reports the input path, checkpoint file, requested watch mode, active watch mode, fallback count, transition count, cumulative poll-catchup count and bytes, any warning summaries, any structured watch events since the previous emitted report, consumed offset, pending trailing bytes, whether the checkpoint was reset, the reset reason, truncation detection, and the nested batch report for whatever newly appended complete lines were imported during that poll.
312-
Multi-input `follow-imports` returns one aggregate report with command-level watch state, cumulative poll-catchup counters, warning summaries, per-process watch events, total consumed and pending bytes, and one nested per-input report for each followed file.
312+
Multi-input `follow-imports` returns one aggregate report with command-level watch state, cumulative poll-catchup counters, warning summaries, per-process watch events, a compact `watch_summary` block that rolls up watch-event kinds plus mode-transition pairs seen in the current emitted event batch, total consumed and pending bytes, a `pending_summary` block that counts how many inputs still have trailing pending bytes and identifies the largest pending backlog, a `state_summary` block that rolls up how many inputs were truncated or had their checkpoint reset plus reset-reason counts, a `batch_summary` rollup of the nested import counters across every consumed input in that pass, including aggregate suppression buckets plus `warning_count` and per-code `warning_codes`, a compact `batch_error_summary` block that counts how many inputs surfaced follow-level `batch_error` payloads and how those error codes were distributed, a `retry_summary` block that totals failed-output and failed-manifest activity across the consumed inputs in that pass and also lists the aggregate retry artifact paths, and one nested per-input report for each followed file.
313313
`cleanup-follow-imports` reports whether the run was a dry-run, whether `--fail-if-matched` was active, whether the selected target set matched anything, whether `--summary-only` was active, the named retention profile when one is active, the configured age gate in seconds, any include/exclude patterns in effect, how many checkpoint sidecars and derived retry artifacts matched cleanup versus were actually removed, which files were skipped because they were filtered out by pattern or were too new, which explicit state files were already missing, and whether it pruned or would prune the stale follow-health sidecar.
314314
`audit-follow-imports` reports the same target-selection metadata and matched-versus-skipped counts as a read-only hygiene pass, plus whether `--summary-only` was active, whether the follow-health snapshot is present, when it was last updated, whether it is stale, and any warning summaries carried by that snapshot.
315315
When `--summary-only` is set, the aggregate counts stay the same but the detailed checkpoint and retry-artifact path lists are omitted from both text and JSON output.

internal/app/import_command_example_fixtures_test.go

Lines changed: 79 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -170,29 +170,81 @@ func followImportsCommandExampleFixtures() []followImportsCommandFixture {
170170
RelativePath: "follow-imports-audit-only-multi.json",
171171
JSON: true,
172172
Report: followImportsAggregateReport{
173-
Status: "ok",
173+
Status: "partial",
174174
Source: "relay_import",
175175
InputCount: 2,
176176
AuditOnly: true,
177177
ConsumedInputs: 1,
178178
IdleInputs: 1,
179+
PartialInputs: 1,
179180
RequestedWatchMode: "auto",
180181
ActiveWatchMode: "notify",
181182
WatchFallbacks: 1,
182183
WatchTransitions: 3,
183184
LastFallbackReason: "watcher_unavailable",
184185
WatchEventCount: 1,
185186
WatchEvents: []followImportsEvent{exampleEvent},
186-
WatchPollCatchups: 1,
187-
WatchCatchupBytes: 42,
187+
WatchSummary: &followImportsWatchSummary{
188+
EventKinds: map[string]int{
189+
"watch_fallback": 1,
190+
},
191+
ModeTransitions: map[string]int{
192+
"notify_to_poll": 1,
193+
},
194+
},
195+
WatchPollCatchups: 1,
196+
WatchCatchupBytes: 42,
188197
Warnings: []common.Warning{
189198
{Code: common.WarnFollowImportsPollCatchup, Message: "notify mode repeatedly relied on poll catchup; treat watcher health as degraded"},
190199
},
191200
TotalConsumedBytes: 42,
192201
TotalPendingBytes: 7,
202+
PendingSummary: &followImportsPendingSummary{
203+
InputsWithPending: 1,
204+
MaxPendingBytes: 7,
205+
MaxPendingInput: `D:\Ops\follow\events-b.jsonl`,
206+
},
207+
StateSummary: &followImportsStateSummary{
208+
TruncatedInputs: 1,
209+
CheckpointResetInputs: 1,
210+
ResetReasons: map[string]int{
211+
"truncated": 1,
212+
},
213+
},
214+
BatchSummary: &followImportsBatchSummary{
215+
Attempted: 3,
216+
Processed: 2,
217+
Failed: 1,
218+
Materialized: 0,
219+
Suppressed: 1,
220+
SuppressionReasons: map[string]int{"import_policy": 1},
221+
WarningCount: 2,
222+
WarningCodes: map[string]int{
223+
common.WarnDedupeApplied: 1,
224+
common.WarnImportSuppressed: 1,
225+
},
226+
WouldMaterialize: 0,
227+
LinkedExistingNote: 1,
228+
NoteDeduplicated: 1,
229+
ImportDeduplicated: 1,
230+
},
231+
BatchErrorSummary: &followImportsBatchErrorSummary{
232+
Count: 1,
233+
Codes: map[string]int{
234+
common.ErrWriteFailed: 1,
235+
},
236+
},
237+
RetrySummary: &followImportsRetrySummary{
238+
FailedOutputWritten: 1,
239+
FailedManifestCount: 1,
240+
InputsWithFailedOutput: 1,
241+
InputsWithFailedManifest: 1,
242+
FailedOutputPaths: []string{`D:\Ops\follow\failed\failed.events-a.0-42.jsonl`},
243+
FailedManifestPaths: []string{`D:\Ops\follow\failed\failed.events-a.0-42.json`},
244+
},
193245
Inputs: []followImportsReport{
194246
{
195-
Status: "ok",
247+
Status: "partial",
196248
Source: "relay_import",
197249
Input: `D:\Ops\follow\events-a.jsonl`,
198250
StateFile: `D:\Ops\follow\events-a.offset.json`,
@@ -202,23 +254,36 @@ func followImportsCommandExampleFixtures() []followImportsCommandFixture {
202254
Offset: 42,
203255
ConsumedBytes: 42,
204256
PendingBytes: 0,
257+
BatchError: &common.ErrorPayload{
258+
Code: common.ErrWriteFailed,
259+
Message: "follow-imports batch failed",
260+
},
205261
Batch: &ingestImportsReport{
206-
Status: "ok",
262+
Status: "partial",
207263
Source: "relay_import",
208264
Input: `D:\Ops\follow\events-a.jsonl`,
209265
Scope: exampleScope,
210266
Session: session.Session{ID: "sess_20260318_013600", Scope: exampleScope.Ref(), Status: session.StatusActive, Task: "audit imported notes (relay_import)", BranchName: "master", StartedAt: time.Date(2026, 3, 18, 1, 36, 0, 0, time.UTC)},
211267
AuditOnly: true,
212-
Attempted: 2,
268+
ContinueOnError: true,
269+
Attempted: 3,
213270
Processed: 2,
214-
Failed: 0,
271+
Failed: 1,
215272
Materialized: 0,
216273
Suppressed: 1,
217274
SuppressionReasons: map[string]int{"import_policy": 1},
218-
WouldMaterialize: 0,
219-
LinkedExistingNote: 1,
220-
NoteDeduplicated: 1,
221-
ImportDeduplicated: 1,
275+
Warnings: []common.Warning{
276+
{Code: common.WarnDedupeApplied, Message: "matched an existing imported note and reused it"},
277+
{Code: common.WarnImportSuppressed, Message: "matched an existing import record and skipped duplicate import"},
278+
},
279+
WouldMaterialize: 0,
280+
LinkedExistingNote: 1,
281+
NoteDeduplicated: 1,
282+
ImportDeduplicated: 1,
283+
FailedOutput: `D:\Ops\follow\failed\failed.events-a.0-42.jsonl`,
284+
FailedOutputWritten: 1,
285+
FailedManifest: `D:\Ops\follow\failed\failed.events-a.0-42.json`,
286+
FailedManifestCount: 1,
222287
},
223288
},
224289
{
@@ -229,6 +294,9 @@ func followImportsCommandExampleFixtures() []followImportsCommandFixture {
229294
AuditOnly: true,
230295
RequestedWatchMode: "auto",
231296
ActiveWatchMode: "notify",
297+
Truncated: true,
298+
CheckpointReset: true,
299+
ResetReason: "truncated",
232300
Offset: 0,
233301
ConsumedBytes: 0,
234302
PendingBytes: 7,

0 commit comments

Comments
 (0)