feat: storage maintenance — janitor, cleanup CLI, retention config - #92
Merged
Conversation
New internal/maintenance package: Sweep (one pass) and Start (periodic janitor goroutine) covering session retention (reuses Store.Cleanup), audit-log retention (previously zero retention), telegram/schedule log rotation at 50MB, telegram plans retention, media sweep including per-chat subdirs (the per-turn sweep skipped them), and skill skip-list GC. New 'maintenance' config section (enabled by default, 60m interval, 30d sessions / 14d audit / 50MB logs / 30d plans / 90d skips) with ODEK_MAINTENANCE_* env plumbing; rejected from project-level odek.json like other deletion-relevant settings.
…path cap - odek cleanup runs one maintenance sweep with a human report; --dry-run previews candidates (sessions/audit/plans/skips/logs) without deleting - the janitor goroutine now runs in the telegram bot, odek serve, and the schedule daemon - session files can no longer grow past the 32 MiB load cap: the save path trims oldest message groups (system kept, tool groups intact) using exact per-group sizing - one sizing pass per trim instead of a full re-marshal per dropped message, which was O(n^2) on long transcripts and hung on oversized fixtures
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
odek | c6d0bf2 | Commit Preview URL Branch Preview URL |
Jul 22 2026, 05:35 PM |
…ction TestSave_TrimsOversizedSession timed out at 10m in CI: the 40 MiB fixture forced redact.RedactSecrets (20+ regexes) over the whole transcript on save, and saveLocked re-redacts the ENTIRE history on every save - O(history) per write, O(n^2) over a session's life. - MaxSessionFileBytes is now a var (comment: production treats it as fixed) so cap-trimming tests use a 64 KiB cap with tiny fixtures instead of multi-MiB transcripts; the oversized trim tests drop from ~25s to <0.1s and the session package from ~42s to ~1.5s. - Real production win behind the same failure: sessions are append-only, so saveLocked now redacts only messages at or beyond a persisted RedactBoundary instead of the full transcript on every save. Old files default to 0 (= redact all once); the boundary is set to the surviving message count before the final marshal so it persists. - New TestSave_IncrementalRedaction pins the boundary behavior; small-cap fixtures keep identical trim semantics coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the approved storage-maintenance design: every byte odek keeps is now either useful or time-bounded, and the unbounded growth categories for long-lived agents (telegram bot,
serve) are gone. Three commits.The problem (from the audit)
Store.Cleanupexisted but was only wired to the manual CLI. Worse, session files could grow past the 32 MiB load cap on the write path and become unloadable.~/.odek/sessions/audit/): zero retention — grew per turn forever.chat<id>/subdirs; plans had no limits; skill skip-list grew forever.session_ttl_hoursdocumented as expiry was actually cache-only (doc fixed).What's in it
Core (
0a835e1) — newinternal/maintenancepackage:Sweep(one pass) +Start(periodic goroutine). Covers session retention (reusesStore.Cleanupincl. index/vector scrubbing), audit-log retention,telegram.log/schedule.logrotation atlog_max_mb(one.1generation), plans retention, media sweep including per-chat subdirs, and skill skip-list GC. Newmaintenanceconfig section (enabled by default: 60m interval, 30d sessions, 14d audit, 50MB logs, 30d plans, 90d skips) withODEK_MAINTENANCE_*env plumbing — operator-only: rejected from project-level./odek.jsonlike other deletion-relevant settings.CLI + wiring (
5aa099c) —odek cleanup [--dry-run]runs one sweep with a human report; dry-run previews candidates without deleting. The janitor runs in the telegram bot,odek serve, and the schedule daemon. Session write-path cap: saves now trim oldest message groups (system kept, tool groups intact) so files can't outgrow the load cap — using exact per-group sizing, one pass per trim. (The naive drop-one-group-per-re-marshal version was O(n²) and hung on oversized fixtures; caught in cross-review, fixed with a regression test.)Docs (
84df555) — newdocs/MAINTENANCE.mdoperator guide (what's cleaned vs. never touched, config reference, example output),docs/CLI.md,docs/CONFIG.md(maintenance section + thesession_ttl_hourshonesty fix),docs/SCHEDULES.md,AGENTS.md.Coverage (99%+ of reachable introduced code)
cmd/odek/cleanup.go: 100%, every functioninternal/configmaintenance helpers: 100%internal/maintenance: 98.2% — the documented remainder is provably dead defensive code (WalkDircan't error when the callback always returns nil;Closeon a just-opened file can't fail)internal/sessiontrim paths: 92% — remainder isjson.Marshalerror returns that can't fire on concrete types (documented in test comments)Verification
go test ./...— all green;-racegreen on maintenance/config/sessiongolangci-lint run ./...— 0 issuesWhat's never touched (by design)
Memory atoms/episodes/facts (already bounded by their own caps/TTLs), skill files, schedules, trust anchors, and — of course —
maintenanceonly ever deletes by age/size, never by content.