fix(worker): give the worker a real health endpoint and a mode-aware HEALTHCHECK

[localai-bot]

Problem

The image bakes in one HEALTHCHECK that curls http://localhost:8080/readyz, but the same image also runs local-ai worker, which serves HTTP on the gRPC base port minus one and never binds 8080. Every worker container is therefore permanently unhealthy — 43 consecutive failures observed on a production node.

That is worse than having no healthcheck. A genuinely broken worker and a perfectly good one both report unhealthy, so the signal carries no information, depends_on: service_healthy and restart-on-unhealthy misbehave, and operators learn to ignore (unhealthy) on every worker.

Fixes #10987.

What was already there — and why fixing only the Dockerfile was not enough

The worker does already serve /readyz on port 50050 via the file-transfer server, started unconditionally in worker mode (core/services/worker/worker.go:157), not gated on file staging.

But the handler was a constant 200 "ok". It proved only that the listener was bound — which is precisely the failure mode this issue is about, one layer in: a process that is up and serving nothing still answers healthy. It also starts before the NATS connect and kept answering 200 after NATS dropped.

So there were two independent bugs. Fixing only the Dockerfile would have made every worker report healthy forever instead of unhealthy forever — still zero information.

Changes

Worker readiness is now real. /readyz returns 200 only when the worker is registered and its NATS connection is live; 503 otherwise.

• NATS is the load-bearing signal: all of a worker's actual work — backend install/start/stop, inference dispatch, staging notifications — arrives over it. A worker with a dead link is up and useless.
• Registration is already implied, since the file-transfer server only starts after registration succeeds.
• This reports something the controller cannot see. The registry's status/last_heartbeat is fed by an HTTP heartbeat to the frontend — a different network path. A worker can keep heartbeating happily while NATS is dead and still look healthy in the registry. The local probe closes exactly that gap.
• "Backends present" was rejected as a signal: a worker legitimately has zero backends until the router pushes one.
• /healthz stays a constant 200. Liveness must not follow readiness, or a NATS blip becomes a cluster-wide restart storm.

The HEALTHCHECK is now mode-aware. scripts/build/healthcheck.sh detects the mode from the running process's argv via /proc and derives the port from the same env vars that configure the bind address — chosen over a static endpoint because the port is both mode- and config-dependent.

This also fixes an unreported sibling bug: a frontend moved off 8080 with LOCALAI_ADDRESS was permanently unhealthy for the same reason. Modes with no HTTP surface (agent-worker, one-shot commands) report healthy rather than false-unhealthy. HEALTHCHECK_ENDPOINT is retained as an explicit override, so the existing compose workaround keeps working — both overrides in docker-compose.distributed.yaml are now unnecessary and removed.

--start-period added: --start-period=60m --interval=1m --timeout=10s --retries=3 (was --interval=1m --timeout=10m --retries=10).

Since #10949 a frontend's startup preload materializes HF artifacts before the HTTP server binds — 31 GB observed on a live cluster — so a healthy replica can legitimately fail probes for a long time. Today it is starting; a small timing change would flip it to unhealthy. --start-period is Docker's purpose-built knob: failures inside it do not burn retries, and it ends early on the first success, so a generous 60m costs a fast-starting container nothing. --timeout drops from 10m to 10s — it is a per-probe deadline, and a localhost curl that has not answered in 10s is itself the fault. Interacts correctly with #10989's preload-aware 503: curl -f fails fast and the start period absorbs it.

Testing

• New Ginkgo specs for the readiness gate, NATSReadiness, and the /readyz / /healthz split. Red first: serves /readyz 503 once the probe reports not-ready → Expected <int>: 200 to equal <int>: 503.
• New scripts/build/healthcheck_test.sh (15 cases, picked up by the existing make test-build-scripts), covering mode/port derivation for both modes, the HEALTHCHECK_ENDPOINT override, exit-code normalisation, and /proc detection under both normal and init: true container shapes. Red first with 6 failures including worker defaults to the file-transfer port: expected 'http://localhost:50050/readyz', got 'http://localhost:8080/readyz' — the literal bug.

Two regressions those tests caught before merge: run is kong default:"withargs", so local-ai gemma-4 whisper is the frontend with model args — the first implementation classified gemma-4 as an unknown mode and silently skipped the probe for the most common invocation in the docs. And a sort -t/ -k3 over /proc sorted lexicographically, putting PID 64 before PID 7 and picking the wrong process under init: true.

go build ./core/... ./pkg/..., go test on both touched packages, make lint (0 issues), and shellcheck all clean. Shell suite stable over 3 runs.

Not yet verified in a real container. The /proc detection path was tested against fixture trees via a LOCALAI_HEALTHCHECK_PROC seam, since this environment has no Docker or unshare. Worth a smoke test on a real worker image before merge.

---

🤖 Generated with Claude Code