Skip to content

Disk-full owner email (shard-initiated event, #76)#174

Open
ClaydeCode wants to merge 6 commits into
mainfrom
feature/clayde/disk-full-email
Open

Disk-full owner email (shard-initiated event, #76)#174
ClaydeCode wants to merge 6 commits into
mainfrom
feature/clayde/disk-full-email

Conversation

@ClaydeCode

Copy link
Copy Markdown
Contributor

Closes #76.

What

Shard-initiated disk-full owner email. When disk usage crosses a configurable threshold (default 90%), the shard emails its owner once via the controller relay (POST api/email_relay, shard-signed, recipient locked to owner server-side).

Per the owner's scope-narrowing comment on #76, this covers only the disk-full event for now: sent once, re-armed only when usage drops back below the threshold, English + German templates prepared but English sent only.

Depends on the controller relay from #232 / freeshard-controller#258 (already merged).

How

  • event_notifications.disk_full config: enabled (default true), threshold_percent (default 90.0). Env override: FREESHARD_EVENT_NOTIFICATIONS__DISK_FULL__THRESHOLD_PERCENT.
  • service/disk_full_notification.pycheck_disk_full(usage) computes used-%, compares >= threshold, and dedupes via a persistent kv_store flag (disk_full_notification_sent): sends once on the rising edge, clears the flag on the falling edge. Sent once per crossing; survives restart.
  • Runs as its own PeriodicTask (30s), reading the disk snapshot maintained by the existing disk monitor — deliberately not inside update_disk_space, so a slow/hung relay call can never stall disk monitoring (matches the telemetry-task pattern).
  • English + German templates kept in one dict; ACTIVE_LANGUAGE = "en" selects the sent language.

Recommended reading order

  1. shard_core/settings.py — new config models
  2. config.toml / tests/config.toml — defaults (tests disable it globally; the suite re-enables per-test)
  3. shard_core/service/disk_full_notification.py — the feature
  4. shard_core/app_factory.py — background-task wiring
  5. tests/test_disk_full_notification.py — trigger/dedupe/reset/opt-out/boundary coverage
  6. agents.md — docs

Verification

uv run ruff check clean; pytest tests/test_disk_full_notification.py tests/test_disk_space.py → 15 passed. Boundary test proven to have teeth (>=> breaks test_triggers_at_exact_threshold).

Review panel

Three adversarial reviewers ran (adversarial correctness, test adversary, DevEx). Specialists skipped as out-of-lens: security (no new endpoint/secret/input parsing — outbound signed call, recipient server-locked), DB/migration (existing kv_store, no schema change), API-contract (no FastAPI route / OpenAPI change), UX (not Sundial).

Blocking findings — resolved:

  • Disk-monitor stall (adversarial). Original code awaited the timeout-less relay call inside the sequential 30s disk PeriodicTask; a black-holed controller connection would freeze current_disk_usage and on_disk_usage_update for all consumers. Fixed in b753f9e: moved the check to its own PeriodicTask; also release the streamed response.
  • Exact-threshold boundary untested (test adversary). >= vs > at exactly 90% — the core "at 90%" criterion — was unpinned. Fixed in ad346e2: added test_triggers_at_exact_threshold + test_does_not_trigger_just_below_threshold.

Advisory — addressed: pinned literal English copy + dedupe-notice line (W1), added test_both_templates_are_well_formed guarding the unsent DE template (M2), added test_run_check_uses_current_disk_snapshot for the trigger wiring (M3), dropped the rationale comment and aligned the kv-store constant name with its value, clearer test helper names.

Advisory — noted, not changed (rationale):

  • Default enabled = true means a self-hosted/unpaired shard over 90% retries the relay every 30s and logs an error each cycle. The feature is meant on for the hosted fleet; the errors are bounded to an active disk-pressure condition and tolerated like other periodic-task failures. Left on by design; opt-out is the config flag.
  • "Sent once" leans on controller-side rate-limit/dedup if a 201 is lost in transit. The flag is set only after a confirmed send — the safe direction (prefer a rare resend over a silent no-send); the controller enforces its own per-shard rate limit.
  • One kv_store SELECT per 30s tick while below threshold. Trivial single indexed read; an in-memory mirror would add a restart-correctness footgun for no real gain.

🤖 Generated with Claude Code

ClaydeCode and others added 6 commits July 20, 2026 21:04
Introduce event_notifications.disk_full settings (enabled, threshold_percent,
default 90%) as the config axis for shard-initiated event emails. Disabled in
tests/config.toml so unrelated disk tests stay hermetic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add disk_full_notification service that sends an email to the shard owner via
the controller email relay (POST api/email_relay) when disk usage rises above
the configured threshold. Deduplicated with a persistent kv_store flag so the
alert fires once and only re-arms after usage drops back below the threshold —
the flag survives restarts to avoid re-notifying on every boot. DE and EN
templates are defined; only EN is sent for now. Called from update_disk_space
so it reuses the existing 30s monitoring tick (checked async, not via the sync
on_disk_usage_update signal).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cover: send above threshold, silence below, dedupe (send once), re-arm after
dropping below, opt-out when disabled, no-send on relay failure/error status
(flag stays unset so it retries), and ignore unmeasured (total=0) disk.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A hung controller relay call could otherwise stall the sequential 30s disk
PeriodicTask, freezing disk-usage monitoring for all consumers during a
disk-pressure event. Move the check to its own PeriodicTask (matching the
telemetry pattern) so the relay call can never block disk monitoring. Also
release the streamed relay response and align the kv-store constant name with
its value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pin the exact-90% boundary (guards >= vs >), the run_check entry point that
reads the live disk snapshot, literal English copy, and that both language
templates format cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard-initiierte Event-Emails (Disk voll, Inaktivität, ...)

1 participant