Disk-full owner email (shard-initiated event, #76)#174
Open
ClaydeCode wants to merge 6 commits into
Open
Conversation
Introduce event_notifications.disk_full settings (enabled, threshold_percent, default 90%) as the config axis for shard-initiated event emails. Disabled in tests/config.toml so unrelated disk tests stay hermetic. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add disk_full_notification service that sends an email to the shard owner via the controller email relay (POST api/email_relay) when disk usage rises above the configured threshold. Deduplicated with a persistent kv_store flag so the alert fires once and only re-arms after usage drops back below the threshold — the flag survives restarts to avoid re-notifying on every boot. DE and EN templates are defined; only EN is sent for now. Called from update_disk_space so it reuses the existing 30s monitoring tick (checked async, not via the sync on_disk_usage_update signal). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cover: send above threshold, silence below, dedupe (send once), re-arm after dropping below, opt-out when disabled, no-send on relay failure/error status (flag stays unset so it retries), and ignore unmeasured (total=0) disk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A hung controller relay call could otherwise stall the sequential 30s disk PeriodicTask, freezing disk-usage monitoring for all consumers during a disk-pressure event. Move the check to its own PeriodicTask (matching the telemetry pattern) so the relay call can never block disk monitoring. Also release the streamed relay response and align the kv-store constant name with its value. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pin the exact-90% boundary (guards >= vs >), the run_check entry point that reads the live disk snapshot, literal English copy, and that both language templates format cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #76.
What
Shard-initiated disk-full owner email. When disk usage crosses a configurable threshold (default 90%), the shard emails its owner once via the controller relay (
POST api/email_relay, shard-signed, recipient locked to owner server-side).Per the owner's scope-narrowing comment on #76, this covers only the disk-full event for now: sent once, re-armed only when usage drops back below the threshold, English + German templates prepared but English sent only.
Depends on the controller relay from #232 / freeshard-controller#258 (already merged).
How
event_notifications.disk_fullconfig:enabled(defaulttrue),threshold_percent(default90.0). Env override:FREESHARD_EVENT_NOTIFICATIONS__DISK_FULL__THRESHOLD_PERCENT.service/disk_full_notification.py—check_disk_full(usage)computes used-%, compares>= threshold, and dedupes via a persistentkv_storeflag (disk_full_notification_sent): sends once on the rising edge, clears the flag on the falling edge. Sent once per crossing; survives restart.PeriodicTask(30s), reading the disk snapshot maintained by the existing disk monitor — deliberately not insideupdate_disk_space, so a slow/hung relay call can never stall disk monitoring (matches the telemetry-task pattern).ACTIVE_LANGUAGE = "en"selects the sent language.Recommended reading order
shard_core/settings.py— new config modelsconfig.toml/tests/config.toml— defaults (tests disable it globally; the suite re-enables per-test)shard_core/service/disk_full_notification.py— the featureshard_core/app_factory.py— background-task wiringtests/test_disk_full_notification.py— trigger/dedupe/reset/opt-out/boundary coverageagents.md— docsVerification
uv run ruff checkclean;pytest tests/test_disk_full_notification.py tests/test_disk_space.py→ 15 passed. Boundary test proven to have teeth (>=→>breakstest_triggers_at_exact_threshold).Review panel
Three adversarial reviewers ran (adversarial correctness, test adversary, DevEx). Specialists skipped as out-of-lens: security (no new endpoint/secret/input parsing — outbound signed call, recipient server-locked), DB/migration (existing
kv_store, no schema change), API-contract (no FastAPI route / OpenAPI change), UX (not Sundial).Blocking findings — resolved:
PeriodicTask; a black-holed controller connection would freezecurrent_disk_usageandon_disk_usage_updatefor all consumers. Fixed inb753f9e: moved the check to its ownPeriodicTask; also release the streamed response.>=vs>at exactly 90% — the core "at 90%" criterion — was unpinned. Fixed inad346e2: addedtest_triggers_at_exact_threshold+test_does_not_trigger_just_below_threshold.Advisory — addressed: pinned literal English copy + dedupe-notice line (W1), added
test_both_templates_are_well_formedguarding the unsent DE template (M2), addedtest_run_check_uses_current_disk_snapshotfor the trigger wiring (M3), dropped the rationale comment and aligned the kv-store constant name with its value, clearer test helper names.Advisory — noted, not changed (rationale):
enabled = truemeans a self-hosted/unpaired shard over 90% retries the relay every 30s and logs an error each cycle. The feature is meant on for the hosted fleet; the errors are bounded to an active disk-pressure condition and tolerated like other periodic-task failures. Left on by design; opt-out is the config flag.kv_storeSELECT per 30s tick while below threshold. Trivial single indexed read; an in-memory mirror would add a restart-correctness footgun for no real gain.🤖 Generated with Claude Code