feat(admin): /admin/healthz/leader for load-balancer leader probes (#689)

## Summary

- Add `/admin/healthz/leader` so an upstream load balancer (Caddy,
HAProxy, ...) can route admin traffic only to the current Raft leader,
skipping the follower→leader AdminForward proxy hop on writes.
- Mirrors the `/healthz/leader` endpoints already shipped on the S3
(`adapter/s3.go:serveS3LeaderHealthz`) and DynamoDB
(`adapter/dynamodb.go:serveDynamoLeaderHealthz`) adapters — same
semantics, same body shape, same method allowlist — so a multi-protocol
load balancer probing any of the three sees identical responses.
- New `LeaderProbe` interface in `internal/admin`;
`NewRouterWithLeaderProbe` constructor (`NewRouter` delegates with `nil`
probe, preserving existing callers); `ServerDeps.LeaderProbe`;
`newAdminLeaderProbe` in `main_admin.go` consults the local Raft group
runtimes.

## Behaviour matrix

| Path | Probe state | Response |

|----------------------------|--------------------|----------------------------|
| `/admin/healthz` | (n/a) | 200 `ok` |
| `/admin/healthz/leader` | wired, leader | 200 `ok` |
| `/admin/healthz/leader` | wired, follower | 503 `not leader` |
| `/admin/healthz/leader` | not wired | 404 `not_found` (JSON) |
| `/admin/healthz/leader` POST | (any) | 405 `method_not_allowed` |

## Self-review (CLAUDE.md 5 lenses)

1. **Data loss** — None. Read-only handler; no Raft proposals.
2. **Concurrency** — `VerifyLeader` is bounded at 2 s, called only on
nodes that already claim leadership via `Status()`. No new shared state.
3. **Performance** — Cheap path: followers short-circuit via `Status()`
(no Raft round-trip). Leader path: one ReadIndex per probe per group;
load balancers typically probe every few seconds, so the overhead is
bounded and well below the existing background HLC lease + ReadIndex
traffic.
4. **Data consistency** — `VerifyLeader` is the same ReadIndex
round-trip the lease-read path uses; a stale-leader follower
mid-silent-leadership-change cannot return 200.
5. **Test coverage** — six router-level tests: 200-when-leader,
503-when-follower, HEAD on both branches, POST rejection, nil-probe →
404, classify ordering (path is not swallowed by the SPA fallback).

## Test plan

- [x] `go test -race ./internal/admin/...` — passes (1.3s)
- [x] `go test -race -run TestStartAdminServer .` — passes
- [x] `golangci-lint run ./...` — clean
- [ ] CI: full suite under `-race`

@claude review

Author: bootjp

internal/admin/router.go

@@ -26,18 +26,49 @@ import (
 // versions (`/admin/api`, `/admin/api/v2`, ...) return a JSON 404
 // instead of being silently answered with the SPA HTML.
 const (
-	pathPrefixAdmin   = "/admin"
-	pathAPIRoot       = "/admin/api"
-	pathPrefixAPI     = pathAPIRoot + "/"
-	pathAPIv1Root     = "/admin/api/v1"
-	pathPrefixAPIv1   = pathAPIv1Root + "/"
-	pathHealthz       = "/admin/healthz"
+	pathPrefixAdmin = "/admin"
+	pathAPIRoot     = "/admin/api"
+	pathPrefixAPI   = pathAPIRoot + "/"
+	pathAPIv1Root   = "/admin/api/v1"
+	pathPrefixAPIv1 = pathAPIv1Root + "/"
+	pathHealthz     = "/admin/healthz"
+	// pathHealthzLeader is a sibling of pathHealthz for load balancers
+	// that want to route admin traffic only to the current Raft leader
+	// (so admin writes do not bounce through the AdminForward proxy on
+	// followers). Mirrors the /healthz/leader endpoints already shipped
+	// on the S3 and DynamoDB adapters: returns 200 only when this node
+	// is the verified leader of the default Raft group; 503 otherwise.
+	// classify() routes this path before the SPA catch-all so it cannot
+	// be silently answered with index.html.
+	pathHealthzLeader = "/admin/healthz/leader"
 	pathAssetsRoot    = "/admin/assets"
 	pathPrefixAssets  = pathAssetsRoot + "/"
 	pathRootAssetsDir = "assets"
 	pathIndexHTML     = "index.html"
 )
 
+// LeaderProbe is the cheap healthz contract used by /admin/healthz/leader.
+// Implementations should return true only when this node is the verified
+// Raft leader — a stale-leader follower returning true during a silent
+// leadership change defeats the load balancer's purpose. Production wires
+// this to the default group's coordinator IsLeader + VerifyLeader pair;
+// tests use a stub returning a fixed bool.
+//
+// A nil LeaderProbe makes /admin/healthz/leader unavailable — the router
+// answers it with the standard JSON 404, distinguishing "not enabled" from
+// the operational "503 not leader" state. Mirrors the S3/DynamoDB
+// /healthz/leader contract.
+type LeaderProbe interface {
+	IsVerifiedLeader() bool
+}
+
+// LeaderProbeFunc is a convenience adapter for wiring a plain function
+// without defining an interface implementation. Mirrors ClusterInfoFunc.
+type LeaderProbeFunc func() bool
+
+// IsVerifiedLeader implements LeaderProbe.
+func (f LeaderProbeFunc) IsVerifiedLeader() bool { return f() }
+
 // APIHandler is the bridge between the router and all JSON API endpoints.
 // Everything under /admin/api/v1/ resolves through it; individual endpoint
 // routing is the handler's responsibility (see apiMux below).
@@ -51,6 +82,7 @@ type APIHandler http.Handler
 type Router struct {
 	api      http.Handler
 	static   fs.FS
+	leader   LeaderProbe
 	notFound http.Handler
 }
 
@@ -62,10 +94,24 @@ type Router struct {
 //     SPA catch-all (which always serves index.html). A nil static FS
 //     causes 404s for asset and SPA routes, which is the expected state
 //     while the SPA has not been built yet.
+//
+// /admin/healthz/leader is wired via NewRouterWithLeaderProbe; this
+// constructor leaves it unrouted (404) for callers that do not need the
+// leader healthz endpoint.
 func NewRouter(api http.Handler, static fs.FS) *Router {
+	return NewRouterWithLeaderProbe(api, static, nil)
+}
+
+// NewRouterWithLeaderProbe is the long-form constructor used by
+// production wiring (see ServerDeps.LeaderProbe). The probe drives
+// /admin/healthz/leader: 200 when probe.IsVerifiedLeader() is true, 503
+// otherwise. A nil probe behaves identically to NewRouter — the path
+// returns the standard JSON 404.
+func NewRouterWithLeaderProbe(api http.Handler, static fs.FS, leader LeaderProbe) *Router {
 	return &Router{
 		api:      api,
 		static:   static,
+		leader:   leader,
 		notFound: http.HandlerFunc(writeJSONNotFound),
 	}
 }
@@ -90,6 +136,7 @@ const (
 	routeAPIv1 routeKind = iota
 	routeAPIOther
 	routeHealthz
+	routeHealthzLeader
 	routeAssetsRoot
 	routeAsset
 	routeSPA
@@ -103,6 +150,14 @@ func (rt *Router) classify(p string) routeKind {
 	if k, ok := classifyAssets(p); ok {
 		return k
 	}
+	// Both /admin/healthz and /admin/healthz/leader are exact-match
+	// equality checks — relative ordering between them does not affect
+	// correctness, but BOTH must run before the SPA prefix branch
+	// below; otherwise the catch-all "anything under /admin/" resolves
+	// these paths to index.html.
+	if p == pathHealthzLeader {
+		return routeHealthzLeader
+	}
 	if p == pathHealthz {
 		return routeHealthz
 	}
@@ -146,6 +201,11 @@ func (rt *Router) dispatch(k routeKind) http.Handler {
 		return rt.api
 	case routeHealthz:
 		return http.HandlerFunc(rt.serveHealth)
+	case routeHealthzLeader:
+		if rt.leader == nil {
+			return rt.notFound
+		}
+		return http.HandlerFunc(rt.serveLeaderHealth)
 	case routeAsset:
 		return http.HandlerFunc(rt.serveAsset)
 	case routeSPA:
@@ -156,9 +216,27 @@ func (rt *Router) dispatch(k routeKind) http.Handler {
 	return rt.notFound
 }
 
+// allowGetHead is the canonical Allow value for read-only handlers
+// (healthz / SPA / static assets). RFC 7231 §6.5.5 requires every 405
+// to advertise the supported methods; load balancers and synthetic-
+// monitor tools key off this header to discover the right verbs
+// without scraping the body. Mirrors the value the S3 and DynamoDB
+// /healthz/leader handlers already set
+// (adapter/s3.go:2404, adapter/dynamodb.go:399).
+const allowGetHead = "GET, HEAD"
+
+// writeMethodNotAllowed centralises the read-only 405 response shape
+// for router-served paths. The Allow header is set BEFORE
+// writeJSONError because writeJSONError calls w.WriteHeader, after
+// which header mutations would silently no-op.
+func writeMethodNotAllowed(w http.ResponseWriter) {
+	w.Header().Set("Allow", allowGetHead)
+	writeJSONError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only GET or HEAD supported")
+}
+
 func (rt *Router) serveHealth(w http.ResponseWriter, r *http.Request) {
 	if r.Method != http.MethodGet && r.Method != http.MethodHead {
-		writeJSONError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only GET or HEAD supported")
+		writeMethodNotAllowed(w)
 		return
 	}
 	w.Header().Set("Content-Type", "text/plain; charset=utf-8")
@@ -169,6 +247,33 @@ func (rt *Router) serveHealth(w http.ResponseWriter, r *http.Request) {
 	}
 }
 
+// serveLeaderHealth answers /admin/healthz/leader. 200 + "ok" only when
+// the LeaderProbe reports a verified leader; 503 + "not leader"
+// otherwise. The body shape and method-allowlist mirror the S3 / Dynamo
+// /healthz/leader endpoints (adapter/s3.go:serveS3LeaderHealthz,
+// adapter/dynamodb.go:serveDynamoLeaderHealthz) so an upstream load
+// balancer that probes any of the three sees identical semantics.
+//
+// Precondition: rt.leader is non-nil. dispatch() short-circuits the
+// nil case to rt.notFound before this handler is ever invoked, so a
+// belt-and-braces nil check inside this body would be dead code.
+func (rt *Router) serveLeaderHealth(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodGet && r.Method != http.MethodHead {
+		writeMethodNotAllowed(w)
+		return
+	}
+	status, body := http.StatusOK, "ok\n"
+	if !rt.leader.IsVerifiedLeader() {
+		status, body = http.StatusServiceUnavailable, "not leader\n"
+	}
+	w.Header().Set("Content-Type", "text/plain; charset=utf-8")
+	w.Header().Set("Cache-Control", "no-store")
+	w.WriteHeader(status)
+	if r.Method == http.MethodGet {
+		_, _ = w.Write([]byte(body))
+	}
+}
+
 func (rt *Router) serveAsset(w http.ResponseWriter, r *http.Request) {
 	// Static assets are read-only resources; rejecting non-GET/HEAD
 	// here keeps the response uniform with /admin/healthz and the
@@ -177,7 +282,7 @@ func (rt *Router) serveAsset(w http.ResponseWriter, r *http.Request) {
 	// the file body for a write request) or surface as a confusing
 	// 404 — neither matches the API contract for assets.
 	if r.Method != http.MethodGet && r.Method != http.MethodHead {
-		writeJSONError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only GET or HEAD supported")
+		writeMethodNotAllowed(w)
 		return
 	}
 	if rt.static == nil {
@@ -254,7 +359,7 @@ func (rt *Router) serveSPA(w http.ResponseWriter, r *http.Request) {
 	// /admin/something returned a JSON 404 with a nil static and a
 	// JSON 405 with a populated static — same URL, different answer.
 	if r.Method != http.MethodGet && r.Method != http.MethodHead {
-		writeJSONError(w, http.StatusMethodNotAllowed, "method_not_allowed", "only GET or HEAD supported")
+		writeMethodNotAllowed(w)
 		return
 	}
 	if rt.static == nil {

internal/admin/router_test.go

@@ -65,6 +65,135 @@ func TestRouter_HealthzRejectsPost(t *testing.T) {
 	require.Equal(t, http.StatusMethodNotAllowed, rec.Code)
 }
 
+// TestRouter_HealthzLeader_ReturnsOKWhenLeader locks down the
+// happy path on /admin/healthz/leader: a verified leader probe
+// produces 200 + "ok\n" + text/plain. Mirrors the S3 / Dynamo
+// /healthz/leader contract so a multi-protocol load balancer
+// sees identical semantics.
+func TestRouter_HealthzLeader_ReturnsOKWhenLeader(t *testing.T) {
+	r := NewRouterWithLeaderProbe(nil, nil, LeaderProbeFunc(func() bool { return true }))
+	req := httptest.NewRequest(http.MethodGet, "/admin/healthz/leader", nil)
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+
+	require.Equal(t, http.StatusOK, rec.Code)
+	require.Contains(t, rec.Header().Get("Content-Type"), "text/plain")
+	require.Equal(t, "ok\n", rec.Body.String())
+	require.Equal(t, "no-store", rec.Header().Get("Cache-Control"))
+}
+
+// TestRouter_HealthzLeader_Returns503WhenNotLeader locks down the
+// non-leader contract: 503 Service Unavailable + "not leader\n".
+// A load balancer probing this endpoint takes the node out of
+// rotation when it loses leadership; the body string is informative
+// for operators reading curl output.
+func TestRouter_HealthzLeader_Returns503WhenNotLeader(t *testing.T) {
+	r := NewRouterWithLeaderProbe(nil, nil, LeaderProbeFunc(func() bool { return false }))
+	req := httptest.NewRequest(http.MethodGet, "/admin/healthz/leader", nil)
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+
+	require.Equal(t, http.StatusServiceUnavailable, rec.Code)
+	require.Contains(t, rec.Header().Get("Content-Type"), "text/plain")
+	require.Equal(t, "not leader\n", rec.Body.String())
+}
+
+// TestRouter_HealthzLeader_HeadOmitsBody mirrors the existing
+// healthz HEAD test. The status code must still indicate the
+// leader state; only the body is suppressed.
+func TestRouter_HealthzLeader_HeadOmitsBody(t *testing.T) {
+	rLeader := NewRouterWithLeaderProbe(nil, nil, LeaderProbeFunc(func() bool { return true }))
+	req := httptest.NewRequest(http.MethodHead, "/admin/healthz/leader", nil)
+	rec := httptest.NewRecorder()
+	rLeader.ServeHTTP(rec, req)
+	require.Equal(t, http.StatusOK, rec.Code)
+	require.Equal(t, "", rec.Body.String())
+
+	rFollower := NewRouterWithLeaderProbe(nil, nil, LeaderProbeFunc(func() bool { return false }))
+	req = httptest.NewRequest(http.MethodHead, "/admin/healthz/leader", nil)
+	rec = httptest.NewRecorder()
+	rFollower.ServeHTTP(rec, req)
+	require.Equal(t, http.StatusServiceUnavailable, rec.Code)
+	require.Equal(t, "", rec.Body.String())
+}
+
+// TestRouter_HealthzLeader_RejectsPost guards the method allowlist:
+// only GET / HEAD are accepted. Mirrors TestRouter_HealthzRejectsPost.
+// Also asserts the Allow: GET, HEAD header required by RFC 7231
+// §6.5.5 — load balancers and synthetic-monitor tools key off this
+// header to discover supported verbs.
+func TestRouter_HealthzLeader_RejectsPost(t *testing.T) {
+	r := NewRouterWithLeaderProbe(nil, nil, LeaderProbeFunc(func() bool { return true }))
+	req := httptest.NewRequest(http.MethodPost, "/admin/healthz/leader", strings.NewReader(""))
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+	require.Equal(t, http.StatusMethodNotAllowed, rec.Code)
+	require.Equal(t, "GET, HEAD", rec.Header().Get("Allow"))
+}
+
+// TestRouter_405_AllowHeader sweeps every router-served path that
+// rejects non-GET/HEAD verbs and asserts each emits the
+// RFC-required Allow header. Locks down the writeMethodNotAllowed
+// invariant — a future handler that bypasses the helper would
+// regress this test rather than silently shipping a non-compliant
+// 405.
+func TestRouter_405_AllowHeader(t *testing.T) {
+	t.Parallel()
+	cases := []struct {
+		name string
+		path string
+	}{
+		{"healthz", "/admin/healthz"},
+		{"healthz_leader", "/admin/healthz/leader"},
+		{"asset", "/admin/assets/app.js"},
+		{"spa", "/admin/somewhere"},
+	}
+	r := NewRouterWithLeaderProbe(nil, newTestStatic(), LeaderProbeFunc(func() bool { return true }))
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			req := httptest.NewRequest(http.MethodPost, tc.path, strings.NewReader(""))
+			rec := httptest.NewRecorder()
+			r.ServeHTTP(rec, req)
+			require.Equal(t, http.StatusMethodNotAllowed, rec.Code)
+			require.Equal(t, "GET, HEAD", rec.Header().Get("Allow"),
+				"path %s missing RFC 7231 Allow header on 405", tc.path)
+		})
+	}
+}
+
+// TestRouter_HealthzLeader_NilProbeReturns404 locks down the
+// "feature off" pattern: when the LeaderProbe is not wired, the
+// router answers /admin/healthz/leader with the standard JSON 404
+// — distinct from the operational 503 (which means "wired, not
+// leader"). Single-node dev runs and admin builds without runtime
+// access fall here.
+func TestRouter_HealthzLeader_NilProbeReturns404(t *testing.T) {
+	r := NewRouter(nil, newTestStatic()) // NewRouter passes nil probe to NewRouterWithLeaderProbe
+	req := httptest.NewRequest(http.MethodGet, "/admin/healthz/leader", nil)
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+	require.Equal(t, http.StatusNotFound, rec.Code)
+	require.Contains(t, rec.Header().Get("Content-Type"), "application/json")
+	require.Contains(t, rec.Body.String(), "not_found")
+}
+
+// TestRouter_HealthzLeader_NotSwallowedBySPA locks down the
+// classify ordering: /admin/healthz/leader must reach the leader
+// handler (200/503) — not the SPA fallback that would otherwise
+// answer with index.html. A regression here would cause a load
+// balancer probing the path to see HTML 200 forever and never
+// detect a leadership change.
+func TestRouter_HealthzLeader_NotSwallowedBySPA(t *testing.T) {
+	probe := LeaderProbeFunc(func() bool { return false })
+	r := NewRouterWithLeaderProbe(nil, newTestStatic(), probe)
+	req := httptest.NewRequest(http.MethodGet, "/admin/healthz/leader", nil)
+	rec := httptest.NewRecorder()
+	r.ServeHTTP(rec, req)
+	require.Equal(t, http.StatusServiceUnavailable, rec.Code)
+	require.NotContains(t, rec.Body.String(), "<html")
+}
+
 func TestRouter_StaticAssetServed(t *testing.T) {
 	r := NewRouter(nil, newTestStatic())
 	req := httptest.NewRequest(http.MethodGet, "/admin/assets/app.js", nil)

internal/admin/server.go

@@ -80,6 +80,15 @@ type ServerDeps struct {
 	// /admin/assets/* and the SPA fallback in that case.
 	StaticFS fs.FS
 
+	// LeaderProbe drives /admin/healthz/leader: 200 when probe.
+	// IsVerifiedLeader() is true, 503 otherwise. A nil probe keeps the
+	// path unrouted (returns the standard JSON 404), matching the
+	// "feature off" pattern Tables / Buckets / Queues already use.
+	// Production wires this to the default group's IsLeader +
+	// VerifyLeader pair so a stale-leader follower in the middle of a
+	// silent leadership change cannot return 200.
+	LeaderProbe LeaderProbe
+
 	// AuthOpts configures cookie attributes and rate limiting. Zero
 	// values pick production-appropriate defaults.
 	AuthOpts AuthServiceOpts
@@ -129,7 +138,7 @@ func NewServer(deps ServerDeps) (*Server, error) {
 	keyviz := NewKeyVizHandler(deps.KeyViz).WithLogger(logger).WithFanout(deps.KeyVizFanout)
 	sqs := buildSqsHandlerForDeps(deps, logger)
 	mux := buildAPIMux(auth, deps.Verifier, cluster, dynamo, s3, keyviz, sqs, logger)
-	router := NewRouter(mux, deps.StaticFS)
+	router := NewRouterWithLeaderProbe(mux, deps.StaticFS, deps.LeaderProbe)
 	return &Server{deps: deps, router: router, auth: auth, mux: mux}, nil
 }