app-health-probe.md

Per-app HTTP health-probe + app-unresponsive detector

• Status: done
• Date: 2026-06-03
• Specs touched: none — realizes the already-locked 2026-06-02 decision (DECISIONS.md "app-unresponsive un-deferred: opt-in health_probe, probed through Caddy", APP_MANIFEST.md # B, HEALTH.md # Detector catalog locus C, THREAT_MODEL.md # B2). No divergence.

Closes issue #54. The 2026-06-02 spec drop un-deferred app-unresponsive and pinned the health_probe manifest field, the locus-C detector, and the probe-through-Caddy security call — but no code implemented them: container-restart-loop was the only app-liveness signal, and a container can be running and accepting TCP yet not answering HTTP coherently. This adds the optional health_probe field and the app-unresponsive detector that consumes it. It is the "up but not responding" half of app liveness.

What was done

• internal/manifest — new optional HealthProbe field (health_probe). Custom UnmarshalYAML accepts both the shorthand string (health_probe: /healthz → {path: /healthz}) and the full mapping (path / healthy_status / start_period); start_period is a Go duration string on the wire. A matching MarshalYAML emits the mapping form (start_period back to a duration string) so a parsed manifest survives the marshal→parse round-trip that writeInstanceDir / loadInstanceManifest does on the per-instance manifest.yml. Validation rejects an empty/relative path, a negative start_period, and out-of-range healthy_status codes, and normalizes the start_period default (60s) in place. Absent ⇒ nil ⇒ the app is never probed.
• internal/health — registered the app-unresponsive definition (Category version, warning, Tier 2, blocks nothing), mirroring container-restart-loop. Brain-owned (no ReportCategory, so a host-report poll never clears it). Debounce left false on purpose: the detector calls Raise/Clear directly (not via ApplyFindings), so the flag would be inert — the 2-bad/1-good debounce lives in the detector, like container-restart-loop's window does.
• internal/lifecycle — added ManagedContainers(ctx) ([]ManagedContainer, …) to the Docker seam: one docker inspect over managed containers returning per-container {instance_id, service, running, StartedAt}, for the steady-running + start-period gate. Added Manager.InstanceManifest(id) — a thin exported wrapper over the on-disk loadInstanceManifest so cmd/brain doesn't duplicate the instance-dir layout.
• cmd/brain — new appProbeDetector (locus C). On each tick it lists instances, loads each manifest, and for every instance that declares a health_probe, isn't already flagged container-restart-loop, has its main_service container running, and is past the start-period grace, it GETs the probe path through Caddy — a request to Caddy's listen address with Host: <route host>, never dialing the container (THREAT_MODEL.md # B2). Healthy = status in healthy_status (default any < 500); a 5xx / timeout / connection failure (Caddy's 502 for a dead upstream) is unhealthy. Reconciles the per-instance app-unresponsive issue: raise on the 2nd consecutive failed probe (or immediately while already active), clear on one good probe or loss of eligibility (stopped / now crash-looping / uninstalled). Audit + notify wiring mirrors container-restart-loop.
• Shared poll goroutine — per the issue ("reuse the timer, do not add a parallel goroutine"), the restart-loop and probe detectors now run from one goroutine (appRuntimeHealthLoop) on the existing 60s health-poll cadence, restart-loop first so the probe can defer to a freshly-raised container-restart-loop and not double-banner a crash-looping app. This replaced restartLoopDetector.run (its sole caller), which was removed as the orphan of this change.

How it maps to the specs

• APP_MANIFEST.md # B — health_probe is opt-in and malmo-executed (not Docker HEALTHCHECK), shorthand-expanding, default healthy < 500, with a start_period warm-up grace. Door-2 synthetic manifests omit it (they build the struct directly with HealthProbe nil).
• HEALTH.md # Detector catalog (locus C) — measurement is "HTTP GET through Caddy (Host: <slug>, path = health_probe.path); status vs healthy_status", cadence 60s, raise on 2 consecutive bad after the grace, clear on 1 good. Anti-flap guards (start-period grace; probe only steady-running) are implemented exactly.
• THREAT_MODEL.md # B2 / DECISIONS.md 2026-06-02 — probing through Caddy (not dialing the container) keeps the control plane off every app-reachable Docker network. The detector only ever talks to Caddy's listener.
• CLAUDE.md # Go discipline — consumer-side narrow interfaces in cmd/brain (managedContainerReader, instanceLister, instanceManifestLoader); log/slog only; transitions audited per issue (health.issue.*).

Known gaps & deviations

• No real Docker/Caddy end-to-end. The api harness builds with life=nil and ManagedContainers shells out to Docker, so the unit tests assert at hermetic seams: a stub http.RoundTripper keyed on the request Host drives healthy/unhealthy/connection-failure probes, and fakes supply the instance list, manifests, and container states. The "routes through Caddy by Host" test pins that the probe hits the Caddy listener base URL with Host: <route host> and the manifest path — but no test exercises a real Caddy round-trip. The docker inspect format string in ManagedContainers is not unit-tested (no Docker in the dev box); it mirrors the verified RestartCounts shape.
• Bell notification is a no-op until allowlisted. Like container-restart-loop, app-unresponsive (warning, per-app) isn't in internal/notify's healthRules allowlist, so emitHealthNotifications dispatches and notify drops it. It still raises the dashboard health issue + audit record. Wiring a bell rule is a separate notify change, deliberately out of scope.
• Probe dial address. The probe targets the Caddy listen address EnsureServer was given (MALMO_CADDY_LISTEN, default :80 → http://127.0.0.1:80), overridable via MALMO_CADDY_PROBE_URL for a containerized brain reaching a Caddy container by service name. v1/.local is plain HTTP, so the HTTP listener + Host header is the path; HTTPS/SNI in secure-URL mode is deferred (default client follows redirects, untested here).
• Open sub-decisions (recorded per the issue): probe timeout = 5s (matches the brain's other host-call timeouts, well under the 60s tick; tune at first soak); start-period default = 60s; probes are sequential per tick (fine for v1 catalog sizes).

Tests

go test ./cmd/brain/ ./internal/manifest/ ./internal/health/ ./internal/lifecycle/ green, plus the full non-PAM suite (go test over ./cmd/brain/... ./internal/... minus the PAM-cgo target). go vet + gofmt clean on the touched packages. Manifest tests cover shorthand→object expansion, full-form parse, bad path/status/duration rejection, and the marshal→parse round-trip. Detector tests cover: 2-bad-sample debounce raise, 1-good-sample clear, default <500 (404 doesn't raise), narrowed healthy_status, start-period suppression then raise, opt-out (no probe → never probed), crash-looper deferral to container-restart-loop, connection-failure-is-unhealthy, not-running skip, clear-on-uninstall, Host-routing wiring, and a probeHealthy classification table. (go vet ./... and make test-nopam both trip on a pre-existing root-owned dev/test-qemu/mkosi.tools/boot/loader permission error unrelated to this change; ran the package list directly instead. The cmd/host-agent PAM-cgo build needs libpam0g-dev, absent in this dev box — pre-existing, unrelated.)

What's next

• Allowlist app-unresponsive for the bell if product wants it surfaced beyond the health banner (a internal/notify healthRules entry; same pending step as container-restart-loop / version-mismatch).
• Real Caddy round-trip coverage once a lane can stand up Caddy + a managed app (the medium QEMU lane) — assert a dead upstream surfaces as 502 → app-unresponsive.
• health_probe in the Door-2 form is explicitly out of v1 (#57's "Edit as YAML" escape hatch authors it by hand); no form control planned.