Add migration deadman switch and Australia migration docs#46
Open
pbitzer wants to merge 10 commits into
Open
Conversation
Migration plan for mj41/42/43 from wwan_install to feature/daily-compression. Includes dead man's switch script (systemd timer-based rollback) for safety during remote cellular network changes on sensors accessible only via SSH tunnels. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deadman script fixes: - Validate minutes argument (prevent 0-second timer on typo) - Check systemctl enable/start exit codes, verify timer is running - Detect in-progress rollback in defuse() with warning - Signal trap in arm() cleans up partial state on Ctrl-C/SSH drop - Force git checkout (-f) in rollback for dirty working trees - Use runuser instead of sudo in rollback (avoids PAM issues) - Use --absolute-names in tar to suppress warnings - Capture tar exit code explicitly in rollback - Trap SIGTERM in rollback after file restore (point of no return) - Remove unreliable $0 from user-facing messages Plan fixes: - Reorder git fetch before stopping wwan-check timer (shrink no-recovery window) - Add ip route check after install.sh (verify wwan0 default route) - Use ; instead of && in verification checklist (one failure doesn't abort remaining checks) - Use precise grep patterns (^DNS=, ^APN\s*=) - Add 30-eth1.network to verification checklist - Add fallback defuse instructions for post-reboot scenario - Fix heredoc-over-SSH for power_delim (use printf instead) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive hardening of migration-deadman.sh based on iterative expert review (bash, systemd, signal handling, adversarial testing): - Fix git checkout: remove `--` separator that treated branch as pathspec - Move trap '' TERM INT HUP to first line of rollback (prevent mid-restore kill) - Add ConditionPathExists + backup dir deletion to break infinite reboot loops - Add flock concurrency control to prevent arm/defuse races - Add timeout on systemctl stop in defuse (TOCTOU race with timer firing) - Add post-stop re-check of service state to prevent false "DEFUSED" message - Add timeout 30 on systemd-networkd restart in rollback (prevent hang) - Add timeout 60 on git checkout in rollback (prevent hang on hooks/locks) - Add tar integrity verification before extraction - Add sync before reboot, reboot fallback cascade (reboot -f, sysrq) - Add input validation (positive int, max 1440 min), cleanup on all error paths - Fix detached HEAD handling (guard against literal "HEAD" branch name) - Use sudo -H -u pi per project rules (not runuser) - Use systemctl show ActiveState for oneshot service detection (not is-active) - Add logger calls for syslog audit trail Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes 2 P0, 4 P1, 4 P2, and 2 P3 findings from independent review:
P0: Defuse re-check now matches both activating AND deactivating states
(timeout kills systemctl client, not the rollback process)
P0: Read-only filesystem detection with rollback attempt counter (max 3)
to break infinite reboot loops when SD card goes read-only
P1: Add SIGPIPE to rollback trap (dead syslog socket can't kill us)
P1: Constrain minutes regex to ^[0-9]{1,4}$ (prevent bash arithmetic overflow)
P1: Add sync immediately after tar extraction (not just before reboot)
P1: Add --unlink-first to tar extract (handle file-type changes)
P2: Add daemon-reload after unit deletion in rollback
P2: Persistent rollback log at /var/log/deadman-rollback.log
P2: Verify chmod +x and unit file writes before enabling timer
P3: Stop timer before service in defuse (close re-trigger window)
P3: Eliminate double tar read on degraded SD cards
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Round 7 review findings:
- Verify rollback script heredoc was fully written (grep sentinel)
to catch truncation from full disk — without this, a truncated
rollback script gets armed but never completes on fire
- Reject leading zeros in minutes arg (^[1-9][0-9]{0,3}$) to avoid
bash octal interpretation errors (e.g. "09" is invalid octal)
- Sanitize attempt counter from disk before arithmetic to handle
corrupted file contents gracefully
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Test both /var/lib and /etc writability (selective sector wear can make one RO while other is writable) - Atomic attempt counter write via tmp+mv (prevents corruption on power loss mid-write that would reset counter and enable infinite loop) - Delete rollback script before rm -rf backup dir (ConditionPathExists target removed first, so partial rm -rf still breaks the loop) - Add exit 1 after reboot cascade failure (systemd marks service failed) - Document --unlink-first tradeoff (file-type fix vs power-loss risk) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- arm() now checks if rollback service is running before proceeding (flock only guards this script, not the systemd-launched rollback) - arm() checks for unit file existence as secondary guard against early-boot false negatives from systemctl is-active - logger wrapped in timeout 5 to prevent hangs when journald is stuck - RO filesystem now attempts remount rw before giving up; if truly read-only, sleeps 5min before reboot to throttle the loop instead of falling through to cleanup code that silently fails - Trap handlers now pass correct exit codes per signal Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address 5 findings from Round 10 (4 independent reviewers): - P1: Add --kill-after=10 to all timeout invocations in rollback heredoc (trap '' TERM sets SIG_IGN, inherited across fork+exec per POSIX; SIGTERM ignored by children, SIGKILL cannot be) - P1: Wrap logger in cleanup_partial_arm with timeout 5 - P2: Wrap all bare logger calls in arm/defuse with timeout 5 - P2: Add timeout to systemctl daemon-reload in rollback - P2: Clean up orphaned wwan-check files not in backup tar after rollback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- P1: Guard orphan cleanup with tar readability check — if backup tar is corrupt, skip file deletion entirely (conservative: leaving extra files is safer than deleting needed wwan-check mechanism) - P3: Add || true to defuse() daemon-reload for consistency Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implementation plans and design specs are working documents, not permanent repo artifacts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/migration-deadman.sh— Safety net for remote sensor migrations: arms a systemd timer that auto-restores from backup and reboots if not defused within a configurable timeoutdocs/superpowers/specs/— Migration design specdocs/superpowers/plans/— Migration execution planUsed in production
Successfully used for mj41, mj42, mj43 migration (2026-03-27/30). Armed, verified, and defused on all three sensors with zero rollbacks needed.
Test plan
🤖 Generated with Claude Code