Skip to content

Automate manual post-install steps + install hardening (HAM-120)#79

Open
pbitzer wants to merge 7 commits into
0.4.xfrom
feature/ham120-install-steps
Open

Automate manual post-install steps + install hardening (HAM-120)#79
pbitzer wants to merge 7 commits into
0.4.xfrom
feature/ham120-install-steps

Conversation

@pbitzer

@pbitzer pbitzer commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Automates the recurring manual post-install steps that keep getting missed on sensor bring-up (Epic HAM-120), plus a set of install-hardening fixes surfaced by a Jira + sensor-log audit. Every new install step is best-effort / non-fatal — a failure logs a warning and never aborts the install.

Base: 0.4.x. 7 commits, +671 lines. 106 unit/dry-run tests passing.

What's in it

Post-install automation (new Phase 7, --skip-postinstall to opt out):

brokkr install:

  • HAM-84 — pin gpiozero<2.0 on Python ≤3.7 (2.0 breaks relay.py on Buster); auto-heals on newer Python
  • HAM-158 — install notifiers editable-as-pi so on-sensor git pull deploys and brokkr survives reboot (same safe pattern as the sindri HAM-163 plan)

hardware phase:

  • Package + enable a boot-time stale-mountpoint cleanup oneshot (files/hamma-cleanup-stale-mountpoints.service) that rmdirs empty /media/pi/DATA?? orphans before the drive is mounted, so a dirty unmount can't push it to a suffixed path brokkr's DATA?? glob misses (sensor-log feat: on-demand sensor noise check script #52; previously hand-installed per unit). rmdir is empty-only → safe no-op once mounted.

verify_deployment.sh:

  • New check_post_install_config (timezone/HAM-71, gpiozero import, .googlechat, datasync, pi-ownership) and a --full check_hamma_repo_access deploy-key check (bounded with BatchMode+ConnectTimeout).

install.sh:

  • REPO_BRANCH 0.3.x0.4.x so the mjolnir-hamma self-update actually runs on 0.4.x units (it only pulls when already on that branch, so the stale value silently skipped updates).

Jira sub-tickets

Closeable on merge: HAM-71, HAM-84, HAM-118, HAM-158.
Partially addressed (install hook only; per-sensor backfill remains): HAM-80.
Untracked audit items implemented here (will file under HAM-120): legacy-unit cleanup, ownership normalization, deploy-key verify, udisks oneshot.

Testing

  • pytest tests/unified106 passing (new tests/unified/test_postinstall.py: dry-run manifest assertions + a sandboxed set -e non-fatal harness that is mutation-verified to fail if a guard is dropped).
  • Full no-skip install.sh --dry-run → exit 0; no new shellcheck findings in any changed line.
  • Read-only E2E on mj02: all new verify_deployment checks pass live; the oneshot unit passes systemd-analyze verify (rc=0) and is a correct no-op on that healthy unit; rmdir-only safety proven (removes empty, keeps non-empty/mounted).
  • Two independent review rounds (incl. a Raspberry-Pi/systemd expert) — all confirmed findings addressed; the Pi expert's key catch (making the oneshot's real Before=multi-user.target / brokkr ordering explicit rather than relying on the non-load-bearing Before=udisks2.service) is in.

Known follow-ups (not blocking)

  • Docker/systemd integration suite (tests/integration/test-with-systemd.sh) not run here (no Docker on the dev host) — worth one real-Buster run before fleet rollout.
  • Boot-time E2E of the oneshot (verifying it prevents the suffixed mountpoint on a unit with the hard-reset pattern, e.g. mj51) needs a reboot — deferred to avoid disrupting a live sensor.

🤖 Generated with Claude Code

pbitzer and others added 7 commits July 9, 2026 16:31
Bring-up repeatedly landed without documented-but-unautomated steps, causing
silent runtime degradation caught only months later. Wire the small ones into
unified_install as best-effort steps and add matching verify_deployment checks:

- HAM-84:  pin gpiozero<2.0 on Python 3.7 (2.0 breaks relay.py on Buster)
- HAM-118: fetch .googlechat key so state_monitor notifications work
- HAM-80:  provision the local datasync user (key install still manual)
- HAM-71:  add the timezone verify check (install step already in bootstrap)

Post-install steps run in a new Phase 7 (--skip-postinstall to opt out) and
never abort the install on failure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review found two unguarded commands (mkdir&&chmod, chown) in the
"best-effort" datasync step: under set -e a failure there aborted the whole
install at finalize — the opposite of the documented contract. Guard both,
and add a sandboxed regression test that proves the Phase 7 steps survive a
failing privileged command (fails against the pre-fix code).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- fetch_notification_key: capture scp stderr so the warning shows the real
  cause (auth vs DNS vs unreachable) instead of swallowing it
- align the .googlechat host to www.hamma.dev (matches README + verify script)
- harden the skip test to assert the skip branch ran, add a positive control

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repo self-update only pulls when the unit is already on REPO_BRANCH, so
the stale 0.3.x value silently skipped updates on 0.4.x-deployed units. Bump
to the current version branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fold in recurring manual fixes surfaced by a Jira + sensor-log audit:

- notifiers installed editable-as-pi so on-sensor git pulls deploy and brokkr
  survives reboot (HAM-158; same safe pattern as sindri HAM-163)
- cleanup_legacy_services: drop retired pre-default units (autossh-hamma,
  brokkr-hamma) that compete with the -default units (sensor-log #43/#9)
- normalize_pi_ownership: best-effort chown of pi paths + /etc/systemd/system
  mode, fixing the recurring root-owned-from-sudo breakage (sensor-log #78/#11/#33)
- verify_deployment: pi-ownership checks + a --full hamma deploy-key check

All new install steps are best-effort/non-fatal (regression-tested under set -e).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A dirty unmount leaves an empty /media/pi/DATA?? dir; on next boot udisks
mounts the drive at a suffixed path (DATA071) that brokkr's DATA?? glob misses,
so science data silently falls back to the SD card. Ship + enable a boot-time
oneshot (ordered Before=udisks2.service) that rmdir's the empty orphans first.
rmdir is empty-only, so it's a safe no-op once drives are mounted. Previously
hand-installed per-sensor (sensor-log #52); this makes it fleet-standard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review-team findings (none blocking) addressed:
- oneshot: make the load-bearing ordering explicit (Before=brokkr-hamma-default
  /multi-user.target) and fix the comment — Before=udisks2.service alone is not
  the real guarantee (udisks is D-Bus-activated; brokkr is the actual mounter)
- verify: git ls-remote deploy-key check now uses BatchMode+ConnectTimeout so a
  stalled cellular link can't hang a --full verify
- fetch_notification_key: chmod 600 the .googlechat webhook token
- gpiozero probe: fall back to 99 so an empty result can't silently pin
- tests: fix the set -e non-fatal harness so it actually exercises the guards
  (sudo-flag-stripping shim; LEGACY_SYSTEMD_DIR / PI_OWN_PATHS sandbox overrides)
  — mutation-verified the cleanup test now fails if a guard is dropped
- tests: assert the oneshot's [Install]/WantedBy + DATA?? glob; refresh docstring

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@pbitzer
pbitzer requested a review from jcburchfield July 12, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant