fix(admin): forward session cookie on KeyViz fan-out so peers do not 401

PR #686 shipped fan-out with the design assumption that peers
would accept anonymous calls on a private network (#685 3). But
the receiving side runs the same SessionAuth middleware as the
browser-facing path, so anonymous fan-out calls are rejected with
401 missing-session-cookie -- the cluster heatmap collapsed to
"1 of N nodes responded" in any production-like deploy.

Fix: forward the inbound user's cookies on every peer call so the
peer's SessionAuth middleware sees a valid principal. Cluster nodes
already share --adminSessionSigningKey for HA, so a cookie minted on
node A is verifiable on node B without any new infrastructure.

Behaviour delta:

- KeyVizFanout.Run gains a cookies []*http.Cookie parameter; nil
  preserves the legacy unauthenticated behaviour for tests / future
  non-browser callers.
- The handler passes r.Cookies() so a SPA poll forwards admin_session
  and admin_csrf transparently.
- New TestKeyVizFanoutRunForwardsCookies pins that both cookies
  reach the peer verbatim.

Design doc 3 rewritten: the "anonymous on a private network" claim
was wrong (the earlier draft assumed the peer side would also be
anonymous, which is not how the admin path is built). Phase 2-C MVP
now documents cookie-forwarding plus the operator-trust assumption
that --keyvizFanoutNodes points at trusted hosts only. Phase 2-C+
will replace cookie forwarding with a dedicated inter-node token so
browser-session expiry and inter-node call validity are decoupled.

Tested against a 6-node cluster: with this fix the heatmap shows
all 6 nodes (was 1 of 6 with anonymous calls).

Author: bootjp

docs/design/2026_04_27_proposed_keyviz_cluster_fanout.md

@@ -132,22 +132,35 @@ elastickv \
   network); the HTTP client uses `http://`. A follow-up will
   introduce `--keyvizFanoutTLS` once the rest of the admin path
   has TLS too.
-- **Auth (Phase 2-C MVP)**: the fan-out path is anonymous — the
-  aggregator issues unauthenticated GETs to peers. This is only
-  acceptable on a fully-private intra-cluster network, which is
-  the contract `--keyvizFanoutNodes` documents. **Do NOT enable
-  fan-out across an untrusted network until Phase 2-C+ ships
-  proper auth.** Per-call timeout: 2 s default, override via
-  `--keyvizFanoutTimeout`.
-- **Auth (Phase 2-C+)**: a follow-up extends the fan-out path with
-  a short-lived signed token derived from the existing admin
-  session-signing key (`ELASTICKV_ADMIN_SESSION_SIGNING_KEY`).
-  Pre-shared inter-node token, NOT a replay of the browser's
-  session cookie — re-using the cookie would couple browser session
-  TTL to inter-node call validity, and a compromised browser
-  session would gain peer-call authority. The two paths are
-  intentionally distinct. The earlier draft of this section
-  conflated them; this paragraph supersedes it.
+- **Auth (Phase 2-C MVP)**: the aggregator forwards the inbound
+  user's `admin_session` cookie (and any other cookies the request
+  carries) on every peer call. The peer's `SessionAuth` middleware
+  verifies the cookie against its own `--adminSessionSigningKey`;
+  cluster nodes already share that key for HA, so a cookie minted
+  on node A is verifiable on node B without any new infrastructure.
+  - The earlier draft of this paragraph said "anonymous on a
+    private network" — that was wrong: the receiving side enforces
+    session auth, so anonymous calls are rejected with 401 and the
+    cluster heatmap collapses to "1 of N nodes responded". The
+    cookie-forwarding scheme above is what actually works.
+  - **Trust model**: forwarding a session cookie to peers is safe
+    when (a) every peer is operator-configured and trusted, and
+    (b) the network is private (cookies are HttpOnly but ride
+    plaintext HTTP for now). `--keyvizFanoutNodes` is the
+    operator's explicit trust list. **Do NOT point
+    `--keyvizFanoutNodes` at an untrusted host** — the user's
+    admin session would be replayed there.
+  - Per-call timeout: 2 s default, override via
+    `--keyvizFanoutTimeout`.
+- **Auth (Phase 2-C+)**: a follow-up replaces cookie forwarding
+  with a short-lived **inter-node** token derived from
+  `ELASTICKV_ADMIN_SESSION_SIGNING_KEY`. Pre-shared, NOT a replay
+  of the browser cookie — re-using the cookie couples browser
+  session TTL to inter-node call validity, and a compromised
+  browser session gains peer-call authority. The follow-up
+  decouples the two paths so the inter-node call survives a
+  browser-session expiry and a compromised cookie doesn't extend
+  laterally.
 
 ## 4. Merge rules
 

internal/admin/keyviz_fanout.go

@@ -205,9 +205,17 @@ type peerResult struct {
 // cluster (peers empty) Run returns local with a Fanout block that
 // reports Expected=1, Responded=1.
 //
+// cookies are attached to every peer request so the receiving node's
+// SessionAuth middleware sees a valid admin session. Production
+// passes the inbound request's cookies; nil disables cookie
+// forwarding (peers will 401 unless they have their own bypass).
+// All cluster nodes must share the same --adminSessionSigningKey for
+// the cookie minted by node A to be verifiable on node B; the
+// existing HA setup already requires this.
+//
 // Run never returns an error: peer-level failures surface in the
 // FanoutResult; aggregation is best-effort.
-func (f *KeyVizFanout) Run(ctx context.Context, params keyVizParams, local KeyVizMatrix) KeyVizMatrix {
+func (f *KeyVizFanout) Run(ctx context.Context, params keyVizParams, local KeyVizMatrix, cookies []*http.Cookie) KeyVizMatrix {
 	if f == nil || len(f.peers) == 0 {
 		merged := local
 		merged.Fanout = &FanoutResult{
@@ -218,7 +226,7 @@ func (f *KeyVizFanout) Run(ctx context.Context, params keyVizParams, local KeyVi
 		return merged
 	}
 
-	results := f.fetchPeersParallel(ctx, params)
+	results := f.fetchPeersParallel(ctx, params, cookies)
 
 	matrices := []KeyVizMatrix{local}
 	statuses := []FanoutNodeStatus{{Node: f.selfName(), OK: true}}
@@ -260,7 +268,7 @@ func countOK(statuses []FanoutNodeStatus) int {
 	return n
 }
 
-func (f *KeyVizFanout) fetchPeersParallel(ctx context.Context, params keyVizParams) []peerResult {
+func (f *KeyVizFanout) fetchPeersParallel(ctx context.Context, params keyVizParams, cookies []*http.Cookie) []peerResult {
 	// Cap per-peer wall time so a single slow node cannot hold the
 	// SPA poll open beyond the configured timeout. The parent
 	// context is preserved as the cancellation root so an early
@@ -274,15 +282,15 @@ func (f *KeyVizFanout) fetchPeersParallel(ctx context.Context, params keyVizPara
 		wg.Add(1)
 		go func(i int, peer string) {
 			defer wg.Done()
-			matrix, err := f.fetchPeer(callCtx, peer, params)
+			matrix, err := f.fetchPeer(callCtx, peer, params, cookies)
 			results[i] = peerResult{node: peer, matrix: matrix, err: err}
 		}(i, peer)
 	}
 	wg.Wait()
 	return results
 }
 
-func (f *KeyVizFanout) fetchPeer(ctx context.Context, peer string, params keyVizParams) (*KeyVizMatrix, error) {
+func (f *KeyVizFanout) fetchPeer(ctx context.Context, peer string, params keyVizParams, cookies []*http.Cookie) (*KeyVizMatrix, error) {
 	target, err := buildKeyVizPeerURL(peer, params)
 	if err != nil {
 		return nil, pkgerrors.Wrap(err, "build peer url")
@@ -292,6 +300,18 @@ func (f *KeyVizFanout) fetchPeer(ctx context.Context, peer string, params keyViz
 		return nil, pkgerrors.Wrap(err, "new request")
 	}
 	req.Header.Set("Accept", "application/json")
+	// Forward the inbound user's admin session cookie so the peer's
+	// SessionAuth middleware sees a valid principal. Without this
+	// the peer rejects every fan-out request with 401 — the missing
+	// piece in the Phase 2-C MVP design (#685 §3 said "anonymous on
+	// a private network" but the receiving side still enforces
+	// session auth, so anonymous calls don't actually go through).
+	// Forwarding the cookie works because all admin nodes share
+	// --adminSessionSigningKey for HA; a cookie minted by node A is
+	// verifiable on node B.
+	for _, c := range cookies {
+		req.AddCookie(c)
+	}
 	resp, err := f.client.Do(req)
 	if err != nil {
 		return nil, pkgerrors.Wrap(err, "peer request")

internal/admin/keyviz_fanout_test.go

@@ -6,6 +6,7 @@ import (
 	"net/http"
 	"net/http/httptest"
 	"strings"
+	"sync"
 	"testing"
 	"time"
 
@@ -152,6 +153,50 @@ func TestMergeKeyVizMatricesDistinctRowsPreserveOrder(t *testing.T) {
 	require.Equal(t, "route:9", merged.Rows[2].BucketID)
 }
 
+// TestKeyVizFanoutRunForwardsCookies pins the auth bug fix that
+// brought all peers up from 401 to 200: the inbound user's session
+// cookies are forwarded on every peer request so the receiving
+// node's SessionAuth middleware sees a valid principal. Without
+// this, peers reject every fan-out call with 401 missing-session-
+// cookie and the cluster heatmap collapses to "1 of N nodes
+// responded".
+func TestKeyVizFanoutRunForwardsCookies(t *testing.T) {
+	t.Parallel()
+	var seenCookies [][]*http.Cookie
+	var seenMu sync.Mutex
+	peer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if !strings.HasSuffix(r.URL.Path, "/admin/api/v1/keyviz/matrix") {
+			http.NotFound(w, r)
+			return
+		}
+		seenMu.Lock()
+		seenCookies = append(seenCookies, r.Cookies())
+		seenMu.Unlock()
+		w.Header().Set("Content-Type", "application/json")
+		w.WriteHeader(http.StatusOK)
+		_ = json.NewEncoder(w).Encode(KeyVizMatrix{Series: keyVizSeriesReads})
+	}))
+	defer peer.Close()
+
+	f := NewKeyVizFanout("self:8080", []string{peer.URL}).WithHTTPClient(peer.Client())
+	cookies := []*http.Cookie{
+		{Name: "admin_session", Value: "session-token-abc"},
+		{Name: "admin_csrf", Value: "csrf-token-def"},
+	}
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads}, cookies)
+	require.True(t, merged.Fanout.Nodes[1].OK)
+	seenMu.Lock()
+	defer seenMu.Unlock()
+	require.Len(t, seenCookies, 1)
+	got := seenCookies[0]
+	names := make(map[string]string, len(got))
+	for _, c := range got {
+		names[c.Name] = c.Value
+	}
+	require.Equal(t, "session-token-abc", names["admin_session"], "session cookie must be forwarded verbatim")
+	require.Equal(t, "csrf-token-def", names["admin_csrf"], "all inbound cookies forwarded; CSRF is innocuous on a GET")
+}
+
 // TestKeyVizFanoutRunSinglePeerOK exercises the end-to-end happy
 // path: one peer responds with a parseable matrix; the aggregator
 // merges it with the local view and reports both nodes ok.
@@ -176,7 +221,7 @@ func TestKeyVizFanoutRunSinglePeerOK(t *testing.T) {
 		},
 	}
 
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, local)
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, local, nil)
 	require.Equal(t, []uint64{10}, merged.Rows[0].Values, "reads must sum across local + peer")
 	require.NotNil(t, merged.Fanout)
 	require.Equal(t, 2, merged.Fanout.Expected)
@@ -209,7 +254,7 @@ func TestKeyVizFanoutRunPeerHTTPError(t *testing.T) {
 		},
 	}
 
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, local)
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, local, nil)
 	require.Equal(t, []uint64{3}, merged.Rows[0].Values, "5xx peer must not perturb local counts")
 	require.NotNil(t, merged.Fanout)
 	require.Equal(t, 2, merged.Fanout.Expected)
@@ -238,7 +283,7 @@ func TestKeyVizFanoutRunPeerTimeout(t *testing.T) {
 		WithTimeout(50 * time.Millisecond)
 
 	start := time.Now()
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads})
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads}, nil)
 	require.Less(t, time.Since(start), 1*time.Second, "fan-out must not wait beyond its per-peer timeout")
 	require.NotNil(t, merged.Fanout)
 	require.Equal(t, 1, merged.Fanout.Responded)
@@ -258,7 +303,7 @@ func TestKeyVizFanoutRunNoPeers(t *testing.T) {
 			{BucketID: "route:1", Values: []uint64{99}},
 		},
 	}
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesWrites, rows: 1024}, local)
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesWrites, rows: 1024}, local, nil)
 	require.Equal(t, []uint64{99}, merged.Rows[0].Values)
 	require.Equal(t, 1, merged.Fanout.Expected)
 	require.Equal(t, 1, merged.Fanout.Responded)
@@ -322,7 +367,7 @@ func TestKeyVizFanoutRunPeerOrder(t *testing.T) {
 	defer second.Close()
 
 	f := NewKeyVizFanout("self:8080", []string{first.URL, second.URL}).WithHTTPClient(first.Client())
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, matrix)
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, matrix, nil)
 	require.Equal(t, []FanoutNodeStatus{
 		{Node: "self:8080", OK: true},
 		{Node: first.URL, OK: true},
@@ -369,7 +414,7 @@ func TestKeyVizFanoutRunPeerExceedsBodyLimit(t *testing.T) {
 		WithResponseBodyLimit(testCap)
 
 	start := time.Now()
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads})
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads}, nil)
 	require.Less(t, time.Since(start), 5*time.Second, "decode must respect the size cap and complete promptly")
 	require.NotNil(t, merged.Fanout)
 	require.Equal(t, 2, merged.Fanout.Expected)
@@ -416,7 +461,7 @@ func TestKeyVizFanoutRunPeerNearCapSucceedsWithWarning(t *testing.T) {
 		WithHTTPClient(peer.Client()).
 		WithResponseBodyLimit(testCap)
 
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads})
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads}, nil)
 	require.NotNil(t, merged.Fanout)
 	require.Len(t, merged.Fanout.Nodes, 2)
 	require.True(t, merged.Fanout.Nodes[0].OK, "self always reports ok")
@@ -458,7 +503,7 @@ func TestKeyVizFanoutRunPeerOverlargeBody(t *testing.T) {
 	f := NewKeyVizFanout("self:8080", []string{peer.URL}).WithHTTPClient(peer.Client())
 
 	start := time.Now()
-	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads})
+	merged := f.Run(context.Background(), keyVizParams{series: keyVizSeriesReads, rows: 1024}, KeyVizMatrix{Series: keyVizSeriesReads}, nil)
 	require.Less(t, time.Since(start), 5*time.Second, "decode must respect the size cap and complete promptly")
 	require.NotNil(t, merged.Fanout)
 	require.True(t, merged.Fanout.Nodes[1].OK, "in-cap response must succeed; cap is 64 MiB and the synthetic body is well under that")

internal/admin/keyviz_handler.go

@@ -178,7 +178,12 @@ func (h *KeyVizHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	matrix := pivotKeyVizColumns(cols, params.series, params.rows)
 	matrix.GeneratedAt = h.now()
 	if h.fanout != nil {
-		matrix = h.fanout.Run(r.Context(), params, matrix)
+		// Forward the inbound user's cookies so the peer's SessionAuth
+		// middleware sees a valid principal. Cluster nodes share
+		// --adminSessionSigningKey for HA, so a cookie minted on this
+		// node verifies on any peer. nil cookies make peers 401,
+		// which surfaces as ok=false in the per-node status.
+		matrix = h.fanout.Run(r.Context(), params, matrix, r.Cookies())
 	}
 	w.Header().Set("Content-Type", "application/json; charset=utf-8")
 	w.Header().Set("Cache-Control", "no-store")