Skip to content

Commit e4c1d95

Browse files
authored
feat: storage maintenance — janitor, cleanup CLI, retention config (#92)
* feat(maintenance): storage maintenance package + operator-only config New internal/maintenance package: Sweep (one pass) and Start (periodic janitor goroutine) covering session retention (reuses Store.Cleanup), audit-log retention (previously zero retention), telegram/schedule log rotation at 50MB, telegram plans retention, media sweep including per-chat subdirs (the per-turn sweep skipped them), and skill skip-list GC. New 'maintenance' config section (enabled by default, 60m interval, 30d sessions / 14d audit / 50MB logs / 30d plans / 90d skips) with ODEK_MAINTENANCE_* env plumbing; rejected from project-level odek.json like other deletion-relevant settings. * feat(cli): 'odek cleanup [--dry-run]', janitor wiring, session write-path cap - odek cleanup runs one maintenance sweep with a human report; --dry-run previews candidates (sessions/audit/plans/skips/logs) without deleting - the janitor goroutine now runs in the telegram bot, odek serve, and the schedule daemon - session files can no longer grow past the 32 MiB load cap: the save path trims oldest message groups (system kept, tool groups intact) using exact per-group sizing - one sizing pass per trim instead of a full re-marshal per dropped message, which was O(n^2) on long transcripts and hung on oversized fixtures * docs: storage maintenance guide, CLI/CONFIG updates, AGENTS layout * fix(session): unbreak CI - small-cap trim fixtures + incremental redaction TestSave_TrimsOversizedSession timed out at 10m in CI: the 40 MiB fixture forced redact.RedactSecrets (20+ regexes) over the whole transcript on save, and saveLocked re-redacts the ENTIRE history on every save - O(history) per write, O(n^2) over a session's life. - MaxSessionFileBytes is now a var (comment: production treats it as fixed) so cap-trimming tests use a 64 KiB cap with tiny fixtures instead of multi-MiB transcripts; the oversized trim tests drop from ~25s to <0.1s and the session package from ~42s to ~1.5s. - Real production win behind the same failure: sessions are append-only, so saveLocked now redacts only messages at or beyond a persisted RedactBoundary instead of the full transcript on every save. Old files default to 0 (= redact all once); the boundary is set to the surviving message count before the final marshal so it persists. - New TestSave_IncrementalRedaction pins the boundary behavior; small-cap fixtures keep identical trim semantics coverage.
1 parent af875d8 commit e4c1d95

21 files changed

Lines changed: 2957 additions & 35 deletions

AGENTS.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ cmd/odek/
4848
skill_promote.go `odek skill promote` — clear NeedsReview on a tainted skill
4949
schedule.go `odek schedule` command and scheduler wiring
5050
memory_cmd.go `odek memory` command
51+
cleanup.go `odek cleanup [--dry-run]` one-shot storage sweep + janitor wiring (telegram/serve/schedule daemon)
5152
parallel.go Parallelism helpers
5253
toolctx.go Tool-call context plumbing
5354
security_report_validation_test.go Regression bar for every documented mitigation
@@ -60,6 +61,7 @@ internal/
6061
auth/ Interactive approval system
6162
memory/ MemoryManager (facts, buffer, episodes, merge, scan). EpisodeProvenance — tainted episodes never auto-replayed.
6263
session/ Session store (CRUD, trim, cleanup, compact JSON). AuditStore + divergence heuristic.
64+
maintenance/ Storage-maintenance janitor (session/audit/plan retention, log rotation, media sweep, skill skip-list GC). Config: `maintenance` section (operator-only).
6365
skills/ Skill system (types, loader, triggers, self-improve, curator, import, cache). SkillProvenance gate.
6466
config/ Config file loading, env vars, secrets.env, priority merge
6567
telegram/ Telegram bot: bot.go, poller.go, handler.go, commands.go, session.go, health.go, plan.go, media_path.go
@@ -91,6 +93,7 @@ ReAct cycle: observe → think → act → repeat.
9193
- **Interaction modes** — engaging (narrated), enhance (persistent), verbose (raw), off.
9294
- Max 300 iterations by default.
9395
- **Post-response async processing** — skill learning, episode extraction, and per-turn extended-memory extraction run in background goroutines tracked by `MemoryManager.RunBackground`; `Agent.Close` drains them via `WaitForBackground` (bounded, ~15s) so the work survives CLI exit without hanging `odek run`.
96+
- **Storage maintenance janitor**`maintenance.Start` (internal/maintenance) runs a periodic sweep of `~/.odek` (expired sessions/audit/plans, oversized-log rotation, media sweep, skill skip-list GC) inside `odek telegram`, `odek serve`, and `odek schedule daemon` when `maintenance.enabled` is set; `odek cleanup [--dry-run]` runs the same sweep on demand. Session files are also trimmed at write time (oldest groups first, system message kept) when they would exceed `MaxSessionFileBytes`, so a session can never grow past the load cap. Operator-only config — project `./odek.json` cannot set it. See docs/MAINTENANCE.md.
9497
- **Artifact-aware file search**`search_files` and `multi_grep` skip build/artifact directories (`node_modules`, `vendor`, `.git`, `__pycache__`, `.venv`, etc.) automatically, reducing noise and speeding scans.
9598
- **Semantic session search** — the `session_search` tool uses go-vector RandomProjections + k-NN for semantic similarity search through session content, with a two-tier pipeline: vector index (fast, ~1ms) → deepSearch fallback (exhaustive, slower).
9699
- **Security-first defaults** — the latest hardening closes the high/medium/low findings tracked in `sec_findings.md`: default `non_interactive` is `deny`, project-level `odek.json` cannot redirect backends or hijack delivery, project-level sandbox knobs require explicit operator approval, `~/.odek` trust anchors are protected, WebSocket upgrades require a per-instance CSRF token, and all untrusted content is wrapped before reaching the model. See Security Architecture below for the full list.

cmd/odek/cleanup.go

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
package main
2+
3+
import (
4+
"context"
5+
"encoding/json"
6+
"fmt"
7+
"io/fs"
8+
"os"
9+
"path/filepath"
10+
"strings"
11+
"time"
12+
13+
"github.com/BackendStack21/odek/internal/config"
14+
"github.com/BackendStack21/odek/internal/maintenance"
15+
"github.com/BackendStack21/odek/internal/session"
16+
)
17+
18+
// cleanupCmd implements `odek cleanup [--dry-run]`: a one-shot, operator-
19+
// invoked storage sweep over ~/.odek (expired sessions, audit records, plans,
20+
// skill skip entries, oversized logs). It runs the same maintenance.Sweep the
21+
// background janitor uses in long-lived processes (telegram, serve, schedule
22+
// daemon). Like `odek session cleanup`, this deletes data without a
23+
// confirmation prompt — it is a local, operator-run command.
24+
func cleanupCmd(args []string) error {
25+
dryRun := false
26+
for _, a := range args {
27+
switch a {
28+
case "--dry-run":
29+
dryRun = true
30+
case "--help", "-h":
31+
fmt.Println(`Usage: odek cleanup [--dry-run]
32+
33+
Remove expired odek storage from ~/.odek per the [maintenance] config
34+
section: old sessions, audit records, plans, and skill skip entries, and
35+
rotate oversized logs.
36+
37+
--dry-run Show what would be removed without removing anything.`)
38+
return nil
39+
default:
40+
return fmt.Errorf("unknown flag %q for cleanup", a)
41+
}
42+
}
43+
44+
resolved := config.LoadConfig(config.CLIFlags{})
45+
cfg := maintenanceConfig(resolved)
46+
home := expandHome("~/.odek")
47+
48+
if dryRun {
49+
printCleanupDryRun(home, cfg)
50+
return nil
51+
}
52+
53+
report, err := maintenance.Sweep(context.Background(), home, cfg)
54+
if err != nil {
55+
return fmt.Errorf("cleanup: %w", err)
56+
}
57+
printCleanupReport(report)
58+
return nil
59+
}
60+
61+
// maintenanceConfig returns the resolved maintenance section as a
62+
// maintenance.Config. resolved.Maintenance already carries the fully
63+
// defaulted type; this helper is the single mapping point if the resolved
64+
// shape ever diverges.
65+
func maintenanceConfig(resolved config.ResolvedConfig) maintenance.Config {
66+
return resolved.Maintenance
67+
}
68+
69+
// startStorageMaintenance starts the background storage janitor when the
70+
// resolved maintenance config enables it. Long-lived processes (telegram bot,
71+
// web UI server, schedule daemon) call this at startup; the janitor stops
72+
// when ctx is cancelled.
73+
func startStorageMaintenance(ctx context.Context, resolved config.ResolvedConfig) {
74+
cfg := maintenanceConfig(resolved)
75+
if !cfg.Enabled {
76+
return
77+
}
78+
maintenance.Start(ctx, expandHome("~/.odek"), cfg)
79+
fmt.Fprintf(os.Stderr, "odek: storage maintenance enabled (interval %dm)\n", cfg.IntervalMinutes)
80+
}
81+
82+
// printCleanupReport prints the human-readable result of a sweep, or a quiet
83+
// success line when there was nothing to do.
84+
func printCleanupReport(r maintenance.Report) {
85+
if r.SessionsRemoved == 0 && r.AuditRemoved == 0 && r.PlansRemoved == 0 &&
86+
r.SkipsRemoved == 0 && r.MediaFreedBytes == 0 && len(r.LogsRotated) == 0 {
87+
fmt.Println("Storage is clean — nothing to remove.")
88+
return
89+
}
90+
fmt.Println("Cleanup complete:")
91+
fmt.Printf(" sessions removed: %d\n", r.SessionsRemoved)
92+
fmt.Printf(" audit records removed: %d\n", r.AuditRemoved)
93+
fmt.Printf(" plans removed: %d\n", r.PlansRemoved)
94+
fmt.Printf(" skip entries removed: %d\n", r.SkipsRemoved)
95+
fmt.Printf(" media freed: %s\n", humanBytes(r.MediaFreedBytes))
96+
for _, p := range r.LogsRotated {
97+
fmt.Printf(" log rotated: %s\n", p)
98+
}
99+
}
100+
101+
// humanBytes formats a byte count for human consumption (e.g. "12.3 MB").
102+
func humanBytes(n int64) string {
103+
const unit = 1024
104+
if n < unit {
105+
return fmt.Sprintf("%d B", n)
106+
}
107+
div, exp := int64(unit), 0
108+
for v := n / unit; v >= unit; v /= unit {
109+
div *= unit
110+
exp++
111+
}
112+
return fmt.Sprintf("%.1f %cB", float64(n)/float64(div), "KMGTPE"[exp])
113+
}
114+
115+
// ── Dry run ────────────────────────────────────────────────────────────
116+
//
117+
// maintenance.Sweep has no dry-run mode, so the CLI builds the same candidate
118+
// list locally for display only. Media cleanup is not previewed — its
119+
// retention policy lives inside the maintenance package.
120+
121+
// cleanupCandidates lists what a sweep WOULD remove, per category.
122+
type cleanupCandidates struct {
123+
sessions []string
124+
audit []string
125+
plans []string
126+
skips int
127+
logs []string
128+
}
129+
130+
// collectCleanupCandidates enumerates expired files under home without
131+
// removing anything.
132+
func collectCleanupCandidates(home string, cfg maintenance.Config) cleanupCandidates {
133+
now := time.Now()
134+
var c cleanupCandidates
135+
136+
if cfg.SessionsMaxAgeDays > 0 {
137+
c.sessions = sessionCandidates(home, now.AddDate(0, 0, -cfg.SessionsMaxAgeDays))
138+
}
139+
if cfg.AuditMaxAgeDays > 0 {
140+
c.audit = filesOlderThan(filepath.Join(home, "sessions", "audit"), now.AddDate(0, 0, -cfg.AuditMaxAgeDays), false)
141+
}
142+
if cfg.PlansMaxAgeDays > 0 {
143+
// Plans may be nested per chat (plans/chat<id>/), so walk recursively.
144+
c.plans = filesOlderThan(filepath.Join(home, "plans"), now.AddDate(0, 0, -cfg.PlansMaxAgeDays), true)
145+
}
146+
if cfg.SkillsSkipMaxAgeDays > 0 {
147+
c.skips = staleSkipEntries(filepath.Join(home, "skills", ".skipped.json"), now.AddDate(0, 0, -cfg.SkillsSkipMaxAgeDays))
148+
}
149+
if cfg.LogMaxMB > 0 {
150+
for _, name := range []string{"schedule.log", "telegram.log"} {
151+
p := filepath.Join(home, name)
152+
if info, err := os.Stat(p); err == nil && info.Size() > cfg.LogMaxMB*1024*1024 {
153+
c.logs = append(c.logs, p)
154+
}
155+
}
156+
}
157+
return c
158+
}
159+
160+
// sessionCandidates lists session files whose UpdatedAt is before cutoff,
161+
// using the session store's own listing so the dry-run preview matches what
162+
// Store.Cleanup (and therefore the sweep) would delete. Unreadable sessions
163+
// are skipped, mirroring Cleanup.
164+
func sessionCandidates(home string, cutoff time.Time) []string {
165+
store, err := session.NewStoreWithDir(filepath.Join(home, "sessions"))
166+
if err != nil {
167+
return nil
168+
}
169+
sessions, err := store.List(0)
170+
if err != nil {
171+
return nil
172+
}
173+
var out []string
174+
for _, s := range sessions {
175+
if s.UpdatedAt.Before(cutoff) {
176+
out = append(out, store.Path(s.ID))
177+
}
178+
}
179+
return out
180+
}
181+
182+
// filesOlderThan returns the regular files under dir whose modification time
183+
// is before cutoff. Session index/metadata files are excluded. Missing
184+
// directories yield an empty list.
185+
func filesOlderThan(dir string, cutoff time.Time, recursive bool) []string {
186+
var out []string
187+
walk := func(path string, d fs.DirEntry, err error) error {
188+
if err != nil {
189+
return nil // unreadable entries are not candidates
190+
}
191+
if d.IsDir() {
192+
if path != dir && !recursive {
193+
return fs.SkipDir
194+
}
195+
return nil
196+
}
197+
if d.Type()&fs.ModeSymlink != 0 {
198+
return nil // never follow or report symlinks
199+
}
200+
name := d.Name()
201+
if name == "index.json" || !strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".md") {
202+
return nil
203+
}
204+
info, err := d.Info()
205+
if err != nil {
206+
return nil
207+
}
208+
if info.ModTime().Before(cutoff) {
209+
out = append(out, path)
210+
}
211+
return nil
212+
}
213+
_ = filepath.WalkDir(dir, walk) // missing dir → no candidates
214+
return out
215+
}
216+
217+
// staleSkipEntries counts entries in the skills .skipped.json file whose
218+
// skipped_at timestamp is before cutoff. An unreadable or malformed file
219+
// counts as zero — the sweep itself decides what to do with it.
220+
func staleSkipEntries(path string, cutoff time.Time) int {
221+
data, err := os.ReadFile(path)
222+
if err != nil {
223+
return 0
224+
}
225+
var file struct {
226+
Skipped map[string]struct {
227+
SkippedAt time.Time `json:"skipped_at"`
228+
} `json:"skipped"`
229+
}
230+
if err := json.Unmarshal(data, &file); err != nil {
231+
return 0
232+
}
233+
n := 0
234+
for _, e := range file.Skipped {
235+
if e.SkippedAt.Before(cutoff) {
236+
n++
237+
}
238+
}
239+
return n
240+
}
241+
242+
// printCleanupDryRun reports the candidate list without removing anything.
243+
func printCleanupDryRun(home string, cfg maintenance.Config) {
244+
c := collectCleanupCandidates(home, cfg)
245+
if len(c.sessions) == 0 && len(c.audit) == 0 && len(c.plans) == 0 && c.skips == 0 && len(c.logs) == 0 {
246+
fmt.Println("Dry run: storage is clean — nothing would be removed.")
247+
return
248+
}
249+
fmt.Println("Dry run — nothing removed. Would remove:")
250+
fmt.Printf(" sessions: %d\n", len(c.sessions))
251+
fmt.Printf(" audit records: %d\n", len(c.audit))
252+
fmt.Printf(" plans: %d\n", len(c.plans))
253+
fmt.Printf(" skip entries: %d\n", c.skips)
254+
for _, p := range c.logs {
255+
fmt.Printf(" log rotated: %s\n", p)
256+
}
257+
}

0 commit comments

Comments
 (0)