Skip to content

Commit 2d62e31

Browse files
author
Ignacio Van Droogenbroeck
committed
docs(authentication): document cascade-on-delete soft cap (v26.06.1+)
Adds the "Cascade-on-delete soft cap" subsection covering the new cluster.rbac.max_cascade_descendants config (default 50000), the ARC_CLUSTER_RBAC_MAX_CASCADE_DESCENDANTS env var, 409 Conflict behaviour, error body shape, operator workaround, and the new arc_cluster_rbac_cascade_rejected_total counter. Updates the "Required configuration" section so it no longer claims "no new env vars." The motivation prose explicitly disclaims that long applies cost the leader its lease: hashicorp/raft runs runFSM async of heartbeats, so the real failure mode is client-visible proposeTimeout blowout + apply-pipeline backpressure, not lost leadership. Also gitignores .claude/ (per-machine project instructions).
1 parent 85bb6a4 commit 2d62e31

2 files changed

Lines changed: 118 additions & 1 deletion

File tree

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,3 +90,6 @@ lerna-debug.log*
9090
*.bak
9191
*.backup
9292
*~
93+
94+
# Claude Code project instructions (your-machine-only)
95+
.claude/

docs/configuration/authentication.md

Lines changed: 115 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ v26.06.1 routes auth **writes** through the cluster's Raft consensus. Auth **rea
125125
| Delete token | Yes |
126126
| Rotate token (new value, same metadata) | Yes |
127127
| `EnsureInitialToken` (first-run bootstrap) | Yes — only the Raft leader's proposal lands; other nodes get an "already exists" no-op |
128-
| RBAC tables (organizations, teams, roles, measurement permissions, token memberships) | **Noplanned for v26.07.1** |
128+
| RBAC tables (organizations, teams, roles, measurement permissions, token memberships) | Yeslanded in v26.06.1 alongside Phase A. See [RBAC replication](#rbac-replication-enterprise) below. |
129129
| Audit log entries | **No — intentionally per-node** (high-volume append-only, no consensus needed) |
130130
| SSO / OIDC / LDAP | **No — Phase B, separate roadmap item** |
131131

@@ -232,6 +232,120 @@ export ARC_CLUSTER_SHARED_SECRET="$(openssl rand -hex 32)"
232232
If nodes have different secrets, follower-to-leader forward-apply fails HMAC validation and token writes that originate on a non-leader silently fail. There is no graceful fallback — operators must ensure the secret is identical across the cluster (e.g. via Kubernetes Secrets or environment-variable injection from a single source).
233233
:::
234234

235+
## RBAC replication (Enterprise)
236+
237+
:::info Available since v26.06.1
238+
Phase A.1 of Cluster Auth Convergence — every RBAC write (organizations, teams, roles, measurement permissions, token memberships) propagates cluster-wide via the same Raft FSM seam used for tokens. Lands in the same v26.06.1 release as Phase A token replication.
239+
:::
240+
241+
Before v26.06.1, RBAC writes hit only the local node's SQLite — same shape as the token gap Phase A closes. An organization created on the writer was invisible to RBAC checks on every reader; a role grant on one node didn't grant the corresponding permission anywhere else. v26.06.1 routes all 13 RBAC writes through Raft so the cluster converges on a single RBAC state.
242+
243+
### What replicates
244+
245+
| RBAC operation | Replicates cluster-wide? |
246+
|----------------|---|
247+
| `POST /api/v1/rbac/organizations` (create) | Yes |
248+
| `PATCH /api/v1/rbac/organizations/:id` (update) | Yes |
249+
| `DELETE /api/v1/rbac/organizations/:id` (delete + cascade to teams, roles, measurement permissions, memberships) | Yes |
250+
| `POST /api/v1/rbac/organizations/:org_id/teams` (create) | Yes |
251+
| `PATCH /api/v1/rbac/teams/:id` (update) | Yes |
252+
| `DELETE /api/v1/rbac/teams/:id` (delete + cascade to roles, measurement permissions, memberships) | Yes |
253+
| `POST /api/v1/rbac/teams/:team_id/roles` (create) | Yes |
254+
| `PATCH /api/v1/rbac/roles/:id` (update) | Yes |
255+
| `DELETE /api/v1/rbac/roles/:id` (delete + cascade to measurement permissions) | Yes |
256+
| `POST /api/v1/rbac/roles/:role_id/measurements` (create) | Yes |
257+
| `DELETE /api/v1/rbac/measurement-permissions/:id` (leaf delete) | Yes |
258+
| `POST /api/v1/auth/tokens/:id/teams` (add membership) | Yes |
259+
| `DELETE /api/v1/auth/tokens/:id/teams/:team_id` (remove membership) | Yes |
260+
261+
### Cascade-on-delete
262+
263+
Deleting an organization removes every descendant — teams, roles, measurement permissions, and token memberships — under a single Raft log entry. Same for `DeleteTeam` (cascades to roles + measurement permissions + memberships) and `DeleteRole` (cascades to measurement permissions). Concurrent writes targeting a being-cascaded entity are serialised and see the post-cascade state.
264+
265+
When a token is hard-deleted via `DELETE /api/v1/auth/tokens/:id` its memberships are also removed cluster-wide — mirroring the SQLite FK cascade `rbac_token_memberships.token_id REFERENCES api_tokens(id) ON DELETE CASCADE` at the FSM layer so the in-memory state stays consistent across nodes.
266+
267+
### Cascade-on-delete soft cap
268+
269+
:::info Available since v26.06.1
270+
Phase A.2 Item 2 — a configurable cap on the number of descendants `DeleteOrganization` / `DeleteTeam` will cascade through in cluster mode.
271+
:::
272+
273+
The FSM cascade-on-delete runs under `f.mu.Lock()` on the single-threaded Raft apply goroutine. hashicorp/raft runs `runFSM` async of heartbeats, so a long apply does **not** directly cost the leader its lease — but for a pathologically large tenant (~100k+ descendants under one organization), the cascade can hold the apply goroutine long enough to blow past the proposer-side 5 s `proposeTimeout`. The originating client sees an opaque timeout while the apply still completes in the background; meanwhile later commands queue behind the slow apply and risk failing their own timeout budgets. Operators see unclear "propose timeout" diagnostics on a delete that "should have worked," instead of a clear "you tried to delete too much at once."
274+
275+
v26.06.1 ships a configurable proposer-side cap. Before proposing `CommandDeleteOrganization` or `CommandDeleteTeam`, the proposer counts the descendants in local SQLite (`teams + roles + measurement_permissions + token_memberships`). If the total exceeds the cap, the API returns **HTTP 409 Conflict** without spending a Raft log entry on a cascade that would block the apply path.
276+
277+
| Setting | Value |
278+
|---|---|
279+
| Config key (TOML) | `cluster.rbac.max_cascade_descendants` |
280+
| Env var | `ARC_CLUSTER_RBAC_MAX_CASCADE_DESCENDANTS` |
281+
| Default | `50000` |
282+
| Disable | `0` (no cap; escape hatch for operators who know their workload) |
283+
| HTTP code on rejection | `409 Conflict` |
284+
| Metric | `arc_cluster_rbac_cascade_rejected_total` |
285+
286+
The error body includes the actual descendant count, the configured cap, and the operator workaround:
287+
288+
```json
289+
{
290+
"success": false,
291+
"error": "cascade exceeds configured limit: 73214 descendants under organization 42 (max 50000); delete child entities (teams, roles, measurement_permissions, token_memberships) first"
292+
}
293+
```
294+
295+
Operator workaround when 409 lands: `DELETE` the affected children first (roles → teams → re-attempt the org delete), or raise the cap if your tenant size justifies it. `DeleteRole`'s cascade is 1-level (only measurement_permissions) and is not capped — it can't plausibly blow up the apply path.
296+
297+
The pre-check costs four small `COUNT(*)` queries against indexed columns — sub-millisecond at realistic cap values, well under the 5-second Raft proposal timeout.
298+
299+
### Pre-existing RBAC rows: auto-seed for orgs, manual re-issue for the rest
300+
301+
On the first leader boot after upgrading to v26.06.1, Arc runs a one-time **upgrade seed** that walks the leader's local `rbac_organizations` table and proposes a `CommandCreateOrganization` for every pre-existing row. The cluster's in-memory FSM learns the org under a fresh log-index ID; the leader's local SQLite keeps the row under its pre-v26.06.1 AUTOINCREMENT ID; both IDs map to the same logical org because the `UNIQUE(name)` constraint is enforced cluster-wide. Followers see the new org via Raft replication and store it under the cluster ID.
302+
303+
**Teams, roles, measurement permissions, and token memberships are NOT auto-seeded.** They reference parent entities by surrogate ID, and the pre-v26.06.1 local AUTOINCREMENT IDs don't generally match the cluster's log-index-stamped IDs after the org seed runs. The seed logs a `Warn` at startup listing the unseeded counts per table:
304+
305+
```
306+
WARN child RBAC rows present in local SQLite are NOT auto-seeded (FK-ID rebase ambiguity);
307+
re-issue them via the API post-upgrade for cluster-wide replication
308+
teams_local=3 roles_local=7 measurement_permissions_local=12 token_memberships_local=4
309+
```
310+
311+
Re-issue each affected team, role, measurement permission, and token membership via the API after upgrade. Existing local rows stay readable on the leader (the FK chain in local SQLite is intact) — they just won't replicate to followers until re-created.
312+
313+
The seed runs only on the Raft leader, gated by `WaitForLeader(30s)`. It is idempotent — re-running on the same leader is a no-op (each proposal is rejected as `"organization name already exists"` and the seed counts it as skipped rather than retrying).
314+
315+
### Prometheus counters
316+
317+
Per node, alongside the Phase A token counters:
318+
319+
```
320+
arc_cluster_rbac_apply_create_organization_total
321+
arc_cluster_rbac_apply_update_organization_total
322+
arc_cluster_rbac_apply_delete_organization_total
323+
arc_cluster_rbac_apply_create_team_total
324+
arc_cluster_rbac_apply_update_team_total
325+
arc_cluster_rbac_apply_delete_team_total
326+
arc_cluster_rbac_apply_create_role_total
327+
arc_cluster_rbac_apply_update_role_total
328+
arc_cluster_rbac_apply_delete_role_total
329+
arc_cluster_rbac_apply_create_measurement_permission_total
330+
arc_cluster_rbac_apply_delete_measurement_permission_total
331+
arc_cluster_rbac_apply_add_token_to_team_total
332+
arc_cluster_rbac_apply_remove_token_from_team_total
333+
arc_cluster_rbac_rejected_total
334+
arc_cluster_rbac_cascade_rejected_total
335+
```
336+
337+
`arc_cluster_rbac_rejected_total` is a **single counter aggregating applier-side validation failures across all 13 RBAC command types** — empty names, missing parent IDs, malformed permission strings, UNIQUE collisions. Same security-alerting semantics as `arc_cluster_auth_rejected_total`: non-zero growth means somebody is proposing malformed RBAC commands; alert on growth.
338+
339+
`arc_cluster_rbac_cascade_rejected_total` counts proposer-side cascade-cap refusals (see [Cascade-on-delete soft cap](#cascade-on-delete-soft-cap) above). Non-zero growth means operators are issuing cascades larger than `cluster.rbac.max_cascade_descendants`. Alert if you'd rather raise the cap than have operators retry after manual cleanup.
340+
341+
In a healthy cluster every node sees the same monotonic count for each `apply_*` counter. Per-node divergence indicates a node missing applies (network partition, FSM stall).
342+
343+
### Required configuration
344+
345+
RBAC replication itself requires no new env vars — it is gated by the same `cluster.enabled = true` + Enterprise license + `cluster.shared_secret` that gate Phase A token replication.
346+
347+
One **optional** knob is documented above: `cluster.rbac.max_cascade_descendants` (env `ARC_CLUSTER_RBAC_MAX_CASCADE_DESCENDANTS`, default `50000`) caps cluster-mode `DeleteOrganization` / `DeleteTeam` cascades. See [Cascade-on-delete soft cap](#cascade-on-delete-soft-cap).
348+
235349
## Token Management
236350

237351
All token management endpoints require **admin** authentication.

0 commit comments

Comments
 (0)