feat(state_monitor): disk-safety monitors (recovery drives, H&S staleness, futile-scrub alert)#80
Open
pbitzer wants to merge 1 commit into
Open
Conversation
…eness, futile-scrub alert Follow-on to the auto-scrub reliability fix (#78), closing the residual risks its assurance report flagged. Three additions, all in the plugin + config (no brokkr/sindri): - check_recovery_drives: alert when the roomiest /media/pi/DATA?? drive is below recovery_low_gb (default 25). Closes the WILL-RECUR gap where the mj-side recovery targets were unmonitored (DATA55 is already 100%). Local, telemetry-independent, alert-only; guards max([]) when all paths error. - check_hs_staleness: alert when no numeric bytes_remaining (H&S) has arrived for hs_stale_s (default 900). Catches the AGS-silent/reboot-loop case that the value trigger goes blind on and that check_ping misses (ping hits the AGS kernel, which comes up each ~60s boot; H&S needs the AGS app, which doesn't). A prior reboot loop ran ~5.5 days unnoticed for exactly this gap. - Futile-scrub-loop alert in check_sensor_drive: escalate when the auto-scrub re-fires futile_scrub_alert_after times (default 3) without clearing low space. A symptom alert — no SSH/df/log-parsing, sidesteps the policy question that made full futile-detection unshippable. Scope decided by two rounds of adversarial review: the SSH df probe and full futile-scrub auto-detection were cut (blind during the real failure, new failure classes) and are tracked as follow-ups. Design + review trail in docs/residual-risk-design-v{1,2}.md. 52/52 state_monitor tests pass (18 new, TDD). No brokkr/sindri changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #78 (base =
feature/scrub-trigger-fix; retarget to0.4.xonce #78 merges). Closes the residual risks #78's assurance report flagged (§5), scoped by two rounds of adversarial review. mjolnir-hamma only — no brokkr/sindri.What
Three additions to
plugins/state_monitor.py(+ 3 config keys):check_recovery_drives— alert when the roomiest/media/pi/DATA??drive is belowrecovery_low_gb(25). Closes the WILL-RECUR gap: the mj-side recovery targets had no free-space monitoring (check_pi_spacewatches/only;check_drivewatches presence only), and DATA55 is already 100%. Local, telemetry-independent, alert-only (a full DATA drive isn't fixed by scrubbing). Per-drive-100% is normal rotation → only the roomiest triggers.check_hs_staleness— alert when no numericbytes_remaining(H&S) has arrived forhs_stale_s(900 s). Catches the AGS-silent / reboot-loop case where the value trigger goes blind. Verified not redundant withcheck_ping: ping hits the AGS kernel (up each ~60 s boot →bad_pingresets belowping_max), while H&S needs the AGS app (stays down). This is the gap that let a prior reboot loop run ~5.5 days unnoticed. Zero network dependency.Futile-scrub-loop alert (in
check_sensor_drive) — escalate when the auto-scrub re-firesfutile_scrub_alert_aftertimes (3) without clearing low space. A symptom alert (no SSH/df/log-parsing) — sidesteps the policy question that made full futile-detection unshippable.Scope (two review rounds)
Round 1 cut a dark-shipped SSH
dfprobe and a fragile full futile-detector. Round 2 verified the cuts are safe (the SSH probe would be blind during the confirmed VL805-wedge failure and couldn't remediate a wedged USB anyway) and added the lightweight futile alert above. Deferred (tracked): directdf /ags/dataover SSH; full futile auto-detection; per-write-target DATA check. Full trail:docs/residual-risk-design-v{1,2}.md.Tests
52/52 state_monitor tests pass (18 new, TDD — watched fail then pass), incl. all-paths-OSError→None (no
max([])crash), lazy-init staleness clock,None-is-not-numeric guard, boundary re-arm, futile-loop escalation + reset. Full suite: 464 passed (3 pre-existingtest_hamma_noisefailures unrelated). Not deployed anywhere.🤖 Generated with Claude Code