Skip to content

feat: storage maintenance — janitor, cleanup CLI, retention config - #92

Merged
jkyberneees merged 4 commits into
mainfrom
feat/storage-maintenance
Jul 22, 2026
Merged

feat: storage maintenance — janitor, cleanup CLI, retention config#92
jkyberneees merged 4 commits into
mainfrom
feat/storage-maintenance

Conversation

@jkyberneees

Copy link
Copy Markdown
Contributor

Implements the approved storage-maintenance design: every byte odek keeps is now either useful or time-bounded, and the unbounded growth categories for long-lived agents (telegram bot, serve) are gone. Three commits.

The problem (from the audit)

  • Sessions: count grew forever; Store.Cleanup existed but was only wired to the manual CLI. Worse, session files could grow past the 32 MiB load cap on the write path and become unloadable.
  • Audit logs (~/.odek/sessions/audit/): zero retention — grew per turn forever.
  • Telegram/schedule logs: append-only, no rotation.
  • Telegram media: per-turn sweep skipped chat<id>/ subdirs; plans had no limits; skill skip-list grew forever.
  • No janitor pattern existed anywhere; session_ttl_hours documented as expiry was actually cache-only (doc fixed).

What's in it

Core (0a835e1) — new internal/maintenance package: Sweep (one pass) + Start (periodic goroutine). Covers session retention (reuses Store.Cleanup incl. index/vector scrubbing), audit-log retention, telegram.log/schedule.log rotation at log_max_mb (one .1 generation), plans retention, media sweep including per-chat subdirs, and skill skip-list GC. New maintenance config section (enabled by default: 60m interval, 30d sessions, 14d audit, 50MB logs, 30d plans, 90d skips) with ODEK_MAINTENANCE_* env plumbing — operator-only: rejected from project-level ./odek.json like other deletion-relevant settings.

CLI + wiring (5aa099c)odek cleanup [--dry-run] runs one sweep with a human report; dry-run previews candidates without deleting. The janitor runs in the telegram bot, odek serve, and the schedule daemon. Session write-path cap: saves now trim oldest message groups (system kept, tool groups intact) so files can't outgrow the load cap — using exact per-group sizing, one pass per trim. (The naive drop-one-group-per-re-marshal version was O(n²) and hung on oversized fixtures; caught in cross-review, fixed with a regression test.)

Docs (84df555) — new docs/MAINTENANCE.md operator guide (what's cleaned vs. never touched, config reference, example output), docs/CLI.md, docs/CONFIG.md (maintenance section + the session_ttl_hours honesty fix), docs/SCHEDULES.md, AGENTS.md.

Coverage (99%+ of reachable introduced code)

  • cmd/odek/cleanup.go: 100%, every function
  • internal/config maintenance helpers: 100%
  • internal/maintenance: 98.2% — the documented remainder is provably dead defensive code (WalkDir can't error when the callback always returns nil; Close on a just-opened file can't fail)
  • internal/session trim paths: 92% — remainder is json.Marshal error returns that can't fire on concrete types (documented in test comments)
  • 36 new tests total, including error-injection via permission/symlink/fd-exhaustion fixtures

Verification

  • go test ./... — all green; -race green on maintenance/config/session
  • golangci-lint run ./... — 0 issues

What's never touched (by design)

Memory atoms/episodes/facts (already bounded by their own caps/TTLs), skill files, schedules, trust anchors, and — of course — maintenance only ever deletes by age/size, never by content.

New internal/maintenance package: Sweep (one pass) and Start (periodic
janitor goroutine) covering session retention (reuses Store.Cleanup),
audit-log retention (previously zero retention), telegram/schedule log
rotation at 50MB, telegram plans retention, media sweep including
per-chat subdirs (the per-turn sweep skipped them), and skill skip-list
GC. New 'maintenance' config section (enabled by default, 60m interval,
30d sessions / 14d audit / 50MB logs / 30d plans / 90d skips) with
ODEK_MAINTENANCE_* env plumbing; rejected from project-level odek.json
like other deletion-relevant settings.
…path cap

- odek cleanup runs one maintenance sweep with a human report;
  --dry-run previews candidates (sessions/audit/plans/skips/logs)
  without deleting
- the janitor goroutine now runs in the telegram bot, odek serve, and
  the schedule daemon
- session files can no longer grow past the 32 MiB load cap: the save
  path trims oldest message groups (system kept, tool groups intact)
  using exact per-group sizing - one sizing pass per trim instead of a
  full re-marshal per dropped message, which was O(n^2) on long
  transcripts and hung on oversized fixtures
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 22, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
odek c6d0bf2 Commit Preview URL

Branch Preview URL
Jul 22 2026, 05:35 PM

…ction

TestSave_TrimsOversizedSession timed out at 10m in CI: the 40 MiB
fixture forced redact.RedactSecrets (20+ regexes) over the whole
transcript on save, and saveLocked re-redacts the ENTIRE history on
every save - O(history) per write, O(n^2) over a session's life.

- MaxSessionFileBytes is now a var (comment: production treats it as
  fixed) so cap-trimming tests use a 64 KiB cap with tiny fixtures
  instead of multi-MiB transcripts; the oversized trim tests drop from
  ~25s to <0.1s and the session package from ~42s to ~1.5s.
- Real production win behind the same failure: sessions are append-only,
  so saveLocked now redacts only messages at or beyond a persisted
  RedactBoundary instead of the full transcript on every save. Old
  files default to 0 (= redact all once); the boundary is set to the
  surviving message count before the final marshal so it persists.
- New TestSave_IncrementalRedaction pins the boundary behavior;
  small-cap fixtures keep identical trim semantics coverage.
@jkyberneees
jkyberneees merged commit e4c1d95 into main Jul 22, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant