fix(state_monitor): level-triggered, numeric-robust, observable auto-scrub

[pbitzer]

Why

mj05's AGS drive /ags/data filled to 0 bytes (Jul 2026). The data was real (~20,446 legit 22 MB triggers; brokkr had already copied ~20,400 to MJ — only 10 were missing). The automated scrub safety net did not keep the AGS side clear; the last effective drain was a manual scrub.

The auto-scrub trigger (StateMonitor.check_sensor_drive) had three code-visible weaknesses (independent of the — unprovable, logs rotated — question of whether it fired on the key day):

• one-shot falling-edge: fires at most once per high→low crossing, never retries while the drive stays low;
• NA/nan/non-numeric fragility: a None/string value raises TypeError that run_checks swallows, silently disabling the check;
• silent failure: scrub output → /dev/null, exit status ignored — a scrub that runs and frees nothing is invisible.

What

• check_sensor_drive is now level-triggered (fires on any below-threshold sample), cooldown-gated (scrub_cooldown_s, default 1800s) so it retries without spamming, and numeric-robust (skips NA/nan/non-numeric; 0 still fires).
• Alerts escalate: on descent-entry and on each cooldown-gated respawn (not one-message-then-silence).
• _spawn_scrub returns success; cooldown is stamped only on a successful launch; scrub output is appended to scrub_log (config) instead of /dev/null.
• Config: adds scrub_cooldown_s + scrub_log to [steps.state_monitor].

Tests

• 34/34 state_monitor tests pass. TDD: 6 new-contract tests were watched to fail on the old code, then pass. Full suite 446 passed (3 pre-existing test_hamma_noise failures are an installed-hamma version issue, unrelated).

Review trail (in this PR)

• docs/scrub-trigger-fix-analysis.md — root cause + 3-way adversarial red-team + debugger adjudication (my first nan theory was refuted and the report re-scoped honestly).
• docs/scrub-fix-assurance-report.md — effectiveness + peer review. Honest bottom line: this fix alone is not a guarantee of no recurrence.

Known residual (tracked follow-ups, NOT closed here)

1. Telemetry blind spot — trigger still depends on AGS-pushed bytes_remaining; needs a direct df /ags/data probe.
2. /media/pi/DATA?? free space is unmonitored (DATA55 is also full) — needs a local disk_usage check.
3. Futile-scrub detection (freed-bytes verification).

Not deployed to any unit yet (deploy via normal git pull + brokkr restart after merge).

🤖 Generated with Claude Code