docs(authentication): document cluster auth replication (Phase A, v26.06.1+)

Arc Enterprise v26.06.1 routes API token writes through the Raft FSM
so tokens propagate cluster-wide (a token created on any node is
valid on every node; revocation propagates cluster-wide within ~50
ms). This docs update reflects that.

**docs/configuration/authentication.md** — new "Cluster auth
replication (Enterprise)" section under "Bootstrap & Recovery":
- What replicates (tokens yes; RBAC tables in v26.07.1; SSO/audit
  intentionally not).
- Eventual-consistency semantics (~50 ms convergence, customer SDKs
  already retry on transient 401).
- Leader-only bootstrap banner: every cluster node calls
  EnsureInitialToken on boot, only the Raft leader's proposal lands,
  followers silently no-op. Operators previously had to scrape every
  pod for the banner; now there's exactly one.
- Plaintext-secrecy invariant: only bcrypt hash + prefix go through
  Raft; snapshot dumps don't contain plaintext.
- Six new Prometheus counters (arc_cluster_auth_apply_* +
  arc_cluster_auth_rejected_total) with operator-facing semantics
  (which divergence patterns mean what).
- No-auto-migration policy: pre-26.06.1 tokens remain valid only on
  the node that issued them; operators re-issue via the API after
  upgrade.
- Divergence-detection error log + remediation: the v26.06.1
  materialiser refuses to overwrite a pre-existing AUTOINCREMENT row
  that collides with a new cluster-stamped ID, surfacing the upgrade
  hazard rather than silently diverging. Includes the sqlite3 cleanup
  command.
- Required configuration: ARC_CLUSTER_SHARED_SECRET (32+ chars, same
  on every node).

**docs/installation/kubernetes.md** — updated the "Get Your Admin
Token" section with an info callout for v26.06.1+ explaining the
leader-only banner behaviour. The existing `kubectl logs -l app=arc
| grep -i "admin"` selector still works — it just returns the
single banner from whichever pod won the election.

Author: Ignacio Van Droogenbroeck

docs/configuration/authentication.md

@@ -105,6 +105,133 @@ After recovering access:
 If Arc restarts with `ARC_AUTH_FORCE_BOOTSTRAP=true` and the `arc-recovery` token already exists, it is a no-op. You still hold the token value you provided.
 :::
 
+## Cluster auth replication (Enterprise)
+
+:::info Available since v26.06.1
+Cluster-wide token replication is available in Arc Enterprise v26.06.1 and later. OSS / standalone deployments are unaffected — tokens stay in the local SQLite as they always have.
+:::
+
+Before v26.06.1, every Arc Enterprise cluster node carried its own SQLite auth DB. A token created via `POST /api/v1/auth/tokens` on the writer was **not** valid on the reader — the reader's local SQLite never saw the row. Operators worked around this by pre-seeding `ARC_AUTH_BOOTSTRAP_TOKEN` with the same value on every node, but API-created tokens and revocations did not propagate. A revocation on the writer left the same token still valid on every reader for the lifetime of those reader processes.
+
+v26.06.1 routes auth **writes** through the cluster's Raft consensus. Auth **reads** still hit the local cache — there's no Raft round trip on every API call.
+
+### What replicates
+
+| Operation | Replicates cluster-wide? |
+|-----------|---|
+| Create token | Yes |
+| Update token (rename, change permissions, change expiry) | Yes |
+| Revoke token | Yes |
+| Delete token | Yes |
+| Rotate token (new value, same metadata) | Yes |
+| `EnsureInitialToken` (first-run bootstrap) | Yes — only the Raft leader's proposal lands; other nodes get an "already exists" no-op |
+| RBAC tables (organizations, teams, roles, measurement permissions, token memberships) | **No — planned for v26.07.1** |
+| Audit log entries | **No — intentionally per-node** (high-volume append-only, no consensus needed) |
+| SSO / OIDC / LDAP | **No — Phase B, separate roadmap item** |
+
+### Convergence semantics
+
+Eventual consistency, typically under 50 ms via Raft apply on local loopback or LAN. Customer SDKs already retry on transient 401, so the brief window between leader commit and follower materialise is invisible in normal usage.
+
+A read-after-write barrier is **not** included in v26.06.1. If a real customer report surfaces the window, a follow-up will add a per-query `LastApplied()` barrier on the query path.
+
+### Bootstrap banner now prints on the leader only
+
+Before v26.06.1, every cluster node printed its own randomly-generated admin token at first start, so a 4-node boot produced 4 banners and 4 different admin tokens (each valid only on its own node).
+
+From v26.06.1, every node still calls `EnsureInitialToken` on boot with its own random plaintext, but only the Raft leader's proposal lands cluster-wide. Followers receive the FSM's `"token name already exists"` rejection from the leader and return an empty plaintext to the caller, so **no banner is emitted on losers**. The losing node's local SQLite still gets the winner's bcrypt hash + prefix via the FSM materialise callback — every node converges on the same admin token.
+
+If you watch a 3-writer + 1-reader cluster boot, expect:
+- 1 `Admin API token:` banner on the Raft leader's stderr
+- 3 `INFO Deferring initial token bootstrap until cluster Raft proposer is wired (Phase A)` lines during startup (one per node)
+- 1 `INFO Cluster auth state replication enabled — token writes now propagate via Raft` line per node after Raft elects a leader
+
+### Security posture
+
+Plaintext token values are **never** written to the Raft log. The proposer generates the token, hashes it with bcrypt locally, and only the hash + prefix go into the replicated payload. The plaintext is returned to the API caller out-of-band before any Raft work begins. Snapshot dumps don't contain plaintext either — verified by a snapshot-grep test in the test suite.
+
+Applier-side validation runs on every node before a token command lands in the FSM. Empty name, missing bcrypt hash, missing prefix, malformed permission string, or zero `created_at` all cause the entry to be rejected on every node. The cluster-wide rejection counter (`arc_cluster_auth_rejected_total`) increments and the rejection is logged at `Error`.
+
+### Prometheus counters
+
+Per node, on the `/metrics` endpoint:
+
+```
+arc_cluster_auth_apply_create_total
+arc_cluster_auth_apply_update_total
+arc_cluster_auth_apply_revoke_total
+arc_cluster_auth_apply_delete_total
+arc_cluster_auth_apply_rotate_total
+arc_cluster_auth_rejected_total
+```
+
+In a healthy cluster every node sees the same monotonic count for each `apply_*` counter — they all apply the same Raft log. Divergence across nodes is the load-bearing signal that one of them is missing applies (network partition, FSM stall).
+
+`arc_cluster_auth_rejected_total` is the **security alerting signal** — non-zero growth means somebody is proposing tokens that fail applier-side validation (malformed payload, fuzz attempt, or a buggy client). Alert on growth, not on absolute value.
+
+### Pre-existing tokens DO NOT auto-migrate
+
+Tokens created on a pre-v26.06.1 node by the local-only API path remain valid **only on that node** after upgrade. The expected migration path is:
+
+1. Upgrade every cluster node to v26.06.1.
+2. Re-issue API tokens via `POST /api/v1/auth/tokens` after restart. New tokens are cluster-wide automatically.
+3. Revoke the old per-node tokens via `POST /api/v1/auth/tokens/:id/revoke` (the revoke also propagates cluster-wide).
+
+Bootstrap tokens set via `ARC_AUTH_BOOTSTRAP_TOKEN` are unaffected if the same value was used on every node (which the pre-v26.06.1 workaround required) — the bytes match, so all nodes effectively share the same admin token already.
+
+### Divergence detection
+
+If a pre-v26.06.1 AUTOINCREMENT row in your local `auth.db` happens to share an ID with a new cluster-replicated token (Raft log indices land in the same `INTEGER` space), the cluster apply on that node will detect the collision and **refuse to overwrite** the pre-existing row. The cluster's in-memory FSM map remains authoritative; the local SQLite cache stays divergent until the operator resolves it.
+
+You'll see an `Error`-level log line on the affected node:
+
+```
+ApplyCreateToken: id <N> already exists locally with different token (cluster<->local divergence; see upgrade notes for pre-26.06.1 tokens)
+```
+
+And the `arc_cluster_auth_rejected_total` counter increments on that node only.
+
+**Remediation**: drop the diverging rows from the local `auth.db`, or drop the whole local auth DB and let the FSM repopulate from the cluster's snapshot. Stop Arc on the affected node, run:
+
+```sql
+sqlite3 /app/data/arc.db "DELETE FROM api_tokens WHERE id = <N>"
+```
+
+Then restart Arc — it'll re-apply the cluster's authoritative state on the affected ID range.
+
+Identical hash + name is treated as idempotent log replay (no-op, no error), so a normal cluster restart never surfaces this.
+
+### `arcx`-style upgrade path
+
+Operators of pre-v26.06.1 Enterprise clusters with many AUTOINCREMENT tokens may prefer to **drain the local auth DB** before re-joining:
+
+1. Note down the names of any service tokens currently in active use.
+2. Stop Arc on the node.
+3. Move `auth.db`, `auth.db-shm`, `auth.db-wal` aside (don't delete — keep as backup).
+4. Restart Arc with `ARC_AUTH_BOOTSTRAP_TOKEN` matching the cluster's admin token.
+5. Re-issue the service tokens cluster-wide via the API on any leader-eligible node.
+
+The cluster's FSM is the source of truth, so this drain-and-rejoin is non-destructive — the only state at risk is per-node tokens that weren't intended to be cluster-wide, and the release notes already document that they don't carry over.
+
+### Required configuration
+
+In addition to standard cluster mode (`cluster.enabled = true`, `cluster.raft_data_dir`, etc.), token replication requires:
+
+```toml
+[cluster]
+shared_secret = "..."  # min 32 chars; same value on every node
+```
+
+The shared secret authenticates leader-forward HMAC for non-leader nodes proposing token writes. Without it, follower nodes refuse to forward auth proposals and the cluster falls back to OSS-mode bootstrap on every node (you'll see 4 banners again).
+
+```bash
+export ARC_CLUSTER_SHARED_SECRET="$(openssl rand -hex 32)"
+```
+
+:::caution Same value on every node
+If nodes have different secrets, follower-to-leader forward-apply fails HMAC validation and token writes that originate on a non-leader silently fail. There is no graceful fallback — operators must ensure the secret is identical across the cluster (e.g. via Kubernetes Secrets or environment-variable injection from a single source).
+:::
+
 ## Token Management
 
 All token management endpoints require **admin** authentication.

docs/installation/kubernetes.md

@@ -53,6 +53,10 @@ You should see:
 Copy this token immediately - you won't see it again!
 :::
 
+:::info Arc Enterprise cluster mode (v26.06.1+)
+In a multi-pod Arc Enterprise cluster, **only one pod prints the banner** — the Raft leader that wins the bootstrap election. Other pods log `INFO Deferring initial token bootstrap until cluster Raft proposer is wired` during startup and then `INFO Cluster auth state replication enabled — token writes now propagate via Raft` once the leader is elected. The non-leader pods silently no-op the bootstrap (they get an "already exists" response from the leader's FSM) and converge on the leader's token via Raft. The `kubectl logs -l app=arc | grep -i "admin"` command above still works — it just returns the single banner from whichever pod won the election. See [Cluster auth replication](/docs/configuration/authentication#cluster-auth-replication-enterprise) for the full semantics, including token-propagation behaviour and the divergence-detection error log.
+:::
+
 ## Installation Methods
 
 <Tabs>