You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(authentication): document cascade-on-delete soft cap (v26.06.1+)
Adds the "Cascade-on-delete soft cap" subsection covering the new
cluster.rbac.max_cascade_descendants config (default 50000), the
ARC_CLUSTER_RBAC_MAX_CASCADE_DESCENDANTS env var, 409 Conflict
behaviour, error body shape, operator workaround, and the new
arc_cluster_rbac_cascade_rejected_total counter. Updates the
"Required configuration" section so it no longer claims "no new
env vars."
The motivation prose explicitly disclaims that long applies cost
the leader its lease: hashicorp/raft runs runFSM async of
heartbeats, so the real failure mode is client-visible
proposeTimeout blowout + apply-pipeline backpressure, not lost
leadership.
Also gitignores .claude/ (per-machine project instructions).
If nodes have different secrets, follower-to-leader forward-apply fails HMAC validation and token writes that originate on a non-leader silently fail. There is no graceful fallback — operators must ensure the secret is identical across the cluster (e.g. via Kubernetes Secrets or environment-variable injection from a single source).
233
233
:::
234
234
235
+
## RBAC replication (Enterprise)
236
+
237
+
:::info Available since v26.06.1
238
+
Phase A.1 of Cluster Auth Convergence — every RBAC write (organizations, teams, roles, measurement permissions, token memberships) propagates cluster-wide via the same Raft FSM seam used for tokens. Lands in the same v26.06.1 release as Phase A token replication.
239
+
:::
240
+
241
+
Before v26.06.1, RBAC writes hit only the local node's SQLite — same shape as the token gap Phase A closes. An organization created on the writer was invisible to RBAC checks on every reader; a role grant on one node didn't grant the corresponding permission anywhere else. v26.06.1 routes all 13 RBAC writes through Raft so the cluster converges on a single RBAC state.
Deleting an organization removes every descendant — teams, roles, measurement permissions, and token memberships — under a single Raft log entry. Same for `DeleteTeam` (cascades to roles + measurement permissions + memberships) and `DeleteRole` (cascades to measurement permissions). Concurrent writes targeting a being-cascaded entity are serialised and see the post-cascade state.
264
+
265
+
When a token is hard-deleted via `DELETE /api/v1/auth/tokens/:id` its memberships are also removed cluster-wide — mirroring the SQLite FK cascade `rbac_token_memberships.token_id REFERENCES api_tokens(id) ON DELETE CASCADE` at the FSM layer so the in-memory state stays consistent across nodes.
266
+
267
+
### Cascade-on-delete soft cap
268
+
269
+
:::info Available since v26.06.1
270
+
Phase A.2 Item 2 — a configurable cap on the number of descendants `DeleteOrganization` / `DeleteTeam` will cascade through in cluster mode.
271
+
:::
272
+
273
+
The FSM cascade-on-delete runs under `f.mu.Lock()` on the single-threaded Raft apply goroutine. hashicorp/raft runs `runFSM` async of heartbeats, so a long apply does **not** directly cost the leader its lease — but for a pathologically large tenant (~100k+ descendants under one organization), the cascade can hold the apply goroutine long enough to blow past the proposer-side 5 s `proposeTimeout`. The originating client sees an opaque timeout while the apply still completes in the background; meanwhile later commands queue behind the slow apply and risk failing their own timeout budgets. Operators see unclear "propose timeout" diagnostics on a delete that "should have worked," instead of a clear "you tried to delete too much at once."
274
+
275
+
v26.06.1 ships a configurable proposer-side cap. Before proposing `CommandDeleteOrganization` or `CommandDeleteTeam`, the proposer counts the descendants in local SQLite (`teams + roles + measurement_permissions + token_memberships`). If the total exceeds the cap, the API returns **HTTP 409 Conflict** without spending a Raft log entry on a cascade that would block the apply path.
Operator workaround when 409 lands: `DELETE` the affected children first (roles → teams → re-attempt the org delete), or raise the cap if your tenant size justifies it. `DeleteRole`'s cascade is 1-level (only measurement_permissions) and is not capped — it can't plausibly blow up the apply path.
296
+
297
+
The pre-check costs four small `COUNT(*)` queries against indexed columns — sub-millisecond at realistic cap values, well under the 5-second Raft proposal timeout.
298
+
299
+
### Pre-existing RBAC rows: auto-seed for orgs, manual re-issue for the rest
300
+
301
+
On the first leader boot after upgrading to v26.06.1, Arc runs a one-time **upgrade seed** that walks the leader's local `rbac_organizations` table and proposes a `CommandCreateOrganization` for every pre-existing row. The cluster's in-memory FSM learns the org under a fresh log-index ID; the leader's local SQLite keeps the row under its pre-v26.06.1 AUTOINCREMENT ID; both IDs map to the same logical org because the `UNIQUE(name)` constraint is enforced cluster-wide. Followers see the new org via Raft replication and store it under the cluster ID.
302
+
303
+
**Teams, roles, measurement permissions, and token memberships are NOT auto-seeded.** They reference parent entities by surrogate ID, and the pre-v26.06.1 local AUTOINCREMENT IDs don't generally match the cluster's log-index-stamped IDs after the org seed runs. The seed logs a `Warn` at startup listing the unseeded counts per table:
304
+
305
+
```
306
+
WARN child RBAC rows present in local SQLite are NOT auto-seeded (FK-ID rebase ambiguity);
307
+
re-issue them via the API post-upgrade for cluster-wide replication
Re-issue each affected team, role, measurement permission, and token membership via the API after upgrade. Existing local rows stay readable on the leader (the FK chain in local SQLite is intact) — they just won't replicate to followers until re-created.
312
+
313
+
The seed runs only on the Raft leader, gated by `WaitForLeader(30s)`. It is idempotent — re-running on the same leader is a no-op (each proposal is rejected as `"organization name already exists"` and the seed counts it as skipped rather than retrying).
`arc_cluster_rbac_rejected_total` is a **single counter aggregating applier-side validation failures across all 13 RBAC command types** — empty names, missing parent IDs, malformed permission strings, UNIQUE collisions. Same security-alerting semantics as `arc_cluster_auth_rejected_total`: non-zero growth means somebody is proposing malformed RBAC commands; alert on growth.
338
+
339
+
`arc_cluster_rbac_cascade_rejected_total` counts proposer-side cascade-cap refusals (see [Cascade-on-delete soft cap](#cascade-on-delete-soft-cap) above). Non-zero growth means operators are issuing cascades larger than `cluster.rbac.max_cascade_descendants`. Alert if you'd rather raise the cap than have operators retry after manual cleanup.
340
+
341
+
In a healthy cluster every node sees the same monotonic count for each `apply_*` counter. Per-node divergence indicates a node missing applies (network partition, FSM stall).
342
+
343
+
### Required configuration
344
+
345
+
RBAC replication itself requires no new env vars — it is gated by the same `cluster.enabled = true` + Enterprise license + `cluster.shared_secret` that gate Phase A token replication.
346
+
347
+
One **optional** knob is documented above: `cluster.rbac.max_cascade_descendants` (env `ARC_CLUSTER_RBAC_MAX_CASCADE_DESCENDANTS`, default `50000`) caps cluster-mode `DeleteOrganization` / `DeleteTeam` cascades. See [Cascade-on-delete soft cap](#cascade-on-delete-soft-cap).
348
+
235
349
## Token Management
236
350
237
351
All token management endpoints require **admin** authentication.
0 commit comments