Commit 453dd1d
authored
🤖 feat: surface entitlements and gate provisioner reconciliation (#67)
## Summary
Surface license tier and entitlements on `CoderControlPlane.status`, and
gate
`CoderProvisioner` reconciliation on the `external_provisioner_daemons`
entitlement so unlicensed setups short-circuit cleanly.
## Background
External provisioner daemons are license-gated in Coder. Before this
change,
`CoderProvisioner` reconciliation could attempt key/deployment work even
when
the deployment was not entitled, causing noisy failures and wasted
reconcile
work.
## Implementation
- Added control-plane status fields for:
- `licenseTier`
- `entitlementsLastChecked`
- `externalProvisionerDaemonsEntitlement`
- Added control-plane entitlement reconciliation via
`/api/v2/entitlements`
using operator credentials, with best-effort tier derivation.
- Added `CoderProvisionerConditionExternalProvisionersEntitled` and
enforced an
entitlement gate in provisioner reconciliation:
- fast-path from control-plane status
- fallback API query via bootstrap client when status is unavailable
- Extended bootstrap client interface/SDK implementation with
`Entitlements(ctx, coderURL, sessionToken)`.
- Added a `CoderControlPlane` watch + `spec.controlPlaneRef.name` index
in the
provisioner controller for quicker entitlement re-evaluation after
control
plane status changes.
- Updated API docs and regenerated code/manifests for status schema
updates.
## Validation
- `make codegen`
- `make manifests`
- `make docs-reference`
- `make verify-vendor`
- `make test`
- `make build`
- `make lint`
- `make docs-build`
## Risks
- Moderate: reconciliation flow changed in both control-plane and
provisioner
controllers.
- Main risk is false-negative entitlement gating if entitlement APIs are
unavailable; mitigated by explicit condition reasons, retries, and
fallback
query behavior.
---
<details>
<summary>📋 Implementation Plan</summary>
# Plan: Surface license tier/entitlements on `CoderControlPlane` and
gate `CoderProvisioner`
## Context / Why
External provisioner daemons are a paid Coder feature. Today,
`CoderProvisionerReconciler` will attempt to create provisioner keys and
deploy `provisionerd` pods even when the target Coder deployment is not
entitled, leading to confusing failures and wasted reconciliation work.
We recently merged license reconciliation into the `CoderControlPlane`
controller (it already tracks when a license was last applied). We can
extend this to also surface **license tier/type** and the **external
provisioner entitlement** on `CoderControlPlane.status`, then use that
as a fast path for provisioner reconciliation.
Goals:
- `CoderControlPlane`: publish `status.licenseTier` (best-effort) and
`status.externalProvisionerDaemonsEntitlement` by querying
`/api/v2/entitlements` with the operator token.
- `CoderProvisioner`: short-circuit early when the control plane reports
`external_provisioner_daemons` is **not entitled**, and re-evaluate
automatically once it becomes entitled.
- Keep a fallback path where the provisioner controller queries
entitlements directly if the control plane hasn’t published them yet (or
operator access is disabled).
Non-goals (initially): deleting already-created provisioner
Deployments/keys when a license is removed; we can decide policy later.
## Evidence (repo + upstream SDK)
- `CoderControlPlane` already has license reconciliation support and
records:
- `status.licenseLastApplied` and `status.licenseLastAppliedHash`
(`api/v1alpha1/codercontrolplane_types.go`).
- `reconcileLicense(...)` uploads license via `codersdk.AddLicense` and
sets `LicenseApplied` condition
(`internal/controller/codercontrolplane_controller.go`).
- `CoderProvisionerReconciler` currently:
- Fetches `CoderControlPlane`, reads bootstrap session token, then calls
`BootstrapClient.EnsureProvisionerKey(...)` and creates
Secret/RBAC/Deployment
(`internal/controller/coderprovisioner_controller.go`).
- Does **not** check entitlements today.
- Vendored `codersdk` exposes entitlements:
- `Client.Entitlements(ctx) (codersdk.Entitlements, error)`.
- Feature constant `codersdk.FeatureExternalProvisionerDaemons =
"external_provisioner_daemons"`.
- `Entitlement.Entitled()` is true for `entitled` and `grace_period`.
- Coder docs confirm Premium-only features that we can use as tier
signals:
- **Custom Roles** are Premium (`docs/admin/users/groups-roles.md` →
maps to `codersdk.FeatureCustomRoles`).
- **Multiple Organizations** are Premium
(`docs/admin/users/organizations.md` → maps to
`codersdk.FeatureMultipleOrganizations`).
## Implementation details
### 1) Surface license tier/type + provisioner entitlement on
`CoderControlPlane.status` (fast path)
**Files:**
- `api/v1alpha1/codercontrolplane_types.go`
- `internal/controller/codercontrolplane_controller.go`
- `internal/app/controllerapp/controllerapp.go`
- `internal/controller/codercontrolplane_controller_test.go`
**API additions (status fields)**
Add new, user-visible status fields so other controllers can make fast
decisions without extra Coder API calls:
```go
type CoderControlPlaneStatus struct {
...
// LicenseTier is a best-effort classification of the currently-applied license.
// Values: none, trial, enterprise, premium, unknown.
// +optional
LicenseTier string `json:"licenseTier,omitempty"`
// EntitlementsLastChecked is when the operator last queried /api/v2/entitlements.
// +optional
EntitlementsLastChecked *metav1.Time `json:"entitlementsLastChecked,omitempty"`
// ExternalProvisionerDaemonsEntitlement is the entitlement value for
// feature "external_provisioner_daemons".
// Values: entitled, grace_period, not_entitled, unknown.
// +optional
ExternalProvisionerDaemonsEntitlement string `json:"externalProvisionerDaemonsEntitlement,omitempty"`
}
```
**Controller logic**
- Introduce an `EntitlementsInspector` (or similar) interface on
`CoderControlPlaneReconciler` with an SDK-backed implementation using
`codersdk.Client.Entitlements`.
- Wire the SDK-backed implementation in `controllerapp.SetupControllers`
so production always reports these fields.
- Add a `reconcileEntitlements(...)` step (called after
`reconcileLicense(...)` so the values reflect the most recently applied
license) that:
- Runs even when `spec.licenseSecretRef` is nil (so status reflects
licenses applied out-of-band).
- Requires: `nextStatus.Phase == Ready`,
`nextStatus.OperatorAccessReady`, `nextStatus.OperatorTokenSecretRef !=
nil`, and `nextStatus.URL != ""`.
- Reads the operator token Secret and calls `/api/v2/entitlements`.
- Stamps `EntitlementsLastChecked=now`,
`ExternalProvisionerDaemonsEntitlement`, and `LicenseTier`.
Tier derivation should be explicit and documented. A practical
best-effort heuristic is:
```go
func licenseTierFromEntitlements(ent codersdk.Entitlements) string {
if !ent.HasLicense {
return "none"
}
if ent.Trial {
return "trial"
}
// Premium-only signals per docs: custom roles + multiple organizations.
for _, f := range []codersdk.FeatureName{codersdk.FeatureCustomRoles, codersdk.FeatureMultipleOrganizations} {
if feat, ok := ent.Features[f]; ok && feat.Entitlement.Entitled() {
return "premium"
}
}
return "enterprise"
}
```
Error handling guidance:
- Treat transient API failures as **Unknown** (don’t hard-block
forever); keep last-known values when available.
- Requeue after `operatorAccessRetryInterval` when entitlements cannot
be queried, so the status eventually updates when a license is installed
later.
> Note: adding status fields requires regenerating deepcopy + CRDs (see
Validation section).
### 2) Add a provisioner status condition for entitlement
**File:** `api/v1alpha1/coderprovisioner_types.go`
Add a new condition constant (name bikesheddable; keep it explicit):
```go
const (
...
// CoderProvisionerConditionExternalProvisionersEntitled indicates whether the
// referenced Coder deployment is entitled to run external provisioner daemons.
CoderProvisionerConditionExternalProvisionersEntitled = "ExternalProvisionersEntitled"
)
```
Rationale: keeps licensing failures distinct from bootstrap secret / key
failures.
### 3) Add an entitlements fetch helper to `internal/coderbootstrap`
**Files:**
- `internal/coderbootstrap/client.go` (interface + SDK implementation)
- `internal/controller/*_test.go` fakes implementing the interface
Extend `coderbootstrap.Client` with an entitlements method:
```go
type Client interface {
EnsureWorkspaceProxy(context.Context, RegisterWorkspaceProxyRequest) (RegisterWorkspaceProxyResponse, error)
EnsureProvisionerKey(context.Context, EnsureProvisionerKeyRequest) (EnsureProvisionerKeyResponse, error)
DeleteProvisionerKey(ctx context.Context, coderURL, sessionToken, orgName, keyName string) error
// Entitlements returns the deployment entitlements from /api/v2/entitlements.
Entitlements(ctx context.Context, coderURL, sessionToken string) (codersdk.Entitlements, error)
}
```
Implement on `SDKClient` using the existing authenticated-client helper
(reusing timeout + optional rate-limit bypass):
```go
func (c *SDKClient) Entitlements(ctx context.Context, coderURL, sessionToken string) (codersdk.Entitlements, error) {
if coderURL == "" { return codersdk.Entitlements{}, xerrors.New("coder URL is required") }
if sessionToken == "" { return codersdk.Entitlements{}, xerrors.New("session token is required") }
client, err := newAuthenticatedClient(coderURL, sessionToken)
if err != nil { return codersdk.Entitlements{}, err }
ent, err := withOptionalRateLimitBypass(ctx, func(requestCtx context.Context) (codersdk.Entitlements, error) {
return client.Entitlements(requestCtx)
})
if err != nil { return codersdk.Entitlements{}, xerrors.Errorf("get entitlements: %w", err) }
if ent.Features == nil {
// Fail-fast: the API contract should always include a features map.
return codersdk.Entitlements{}, xerrors.New("assertion failed: entitlements.features is nil")
}
return ent, nil
}
```
Why do this instead of direct `codersdk.New(...)` in the reconciler?
- Keeps all Coder-API concerns (timeouts, bypass ratelimit header) in
one place.
- Makes unit tests easy by stubbing `BootstrapClient.Entitlements`.
### 4) Short-circuit in `CoderProvisionerReconciler.Reconcile`
**File:** `internal/controller/coderprovisioner_controller.go`
Add a new entitlement gate immediately after we have:
- `controlPlane.Status.URL` (already required), and
- `sessionToken` (bootstrap credentials).
Fast path:
- If `controlPlane.status.entitlementsLastChecked` is set and
`controlPlane.status.externalProvisionerDaemonsEntitlement` is present
(and optionally not stale), use it to decide without calling the Coder
API.
Fallback:
- Otherwise call `r.BootstrapClient.Entitlements(...)` and evaluate
`external_provisioner_daemons` directly.
Pseudo-flow:
```go
// 1) Fast path: trust ControlPlane status if present.
if controlPlane.Status.EntitlementsLastChecked != nil && controlPlane.Status.ExternalProvisionerDaemonsEntitlement != "" {
switch controlPlane.Status.ExternalProvisionerDaemonsEntitlement {
case "entitled", "grace_period":
setCondition(provisioner, coderv1alpha1.CoderProvisionerConditionExternalProvisionersEntitled,
metav1.ConditionTrue, "Entitled", "Coder deployment is entitled to external provisioner daemons")
// Proceed.
case "not_entitled":
setCondition(provisioner, coderv1alpha1.CoderProvisionerConditionExternalProvisionersEntitled,
metav1.ConditionFalse, "NotEntitled", "Coder deployment is not entitled to external provisioner daemons")
_ = r.Status().Update(ctx, provisioner)
return ctrl.Result{RequeueAfter: 2 * time.Minute}, nil
default:
// Unknown → fall through to API check.
}
}
// 2) Fallback: query coderd.
ent, err := r.BootstrapClient.Entitlements(ctx, controlPlane.Status.URL, sessionToken)
if err != nil {
// Set ExternalProvisionersEntitled=False, Reason=EntitlementsQueryFailed and retry.
}
feature, ok := ent.Features[codersdk.FeatureExternalProvisionerDaemons]
if !ok || !feature.Entitlement.Entitled() {
// Not entitled → short-circuit with RequeueAfter.
}
setCondition(... True, "Entitled", "Coder deployment is entitled to external provisioner daemons")
```
Recommended condition reasons/messages:
- **NotEntitled**: “Coder deployment is not entitled to external
provisioner daemons; install a Premium/Enterprise license to enable
external provisioners.”
- **EntitlementsQueryFailed**: “Failed to query Coder entitlements;
retrying.”
- **NotSupported** (HTTP 404): “Coder deployment does not expose
/api/v2/entitlements; cannot verify license.” (treat as blocked,
requeue)
- **Forbidden** (HTTP 401/403): “Bootstrap token is not authorized to
read entitlements; retrying.”
Re-evaluation requirement:
- Use `RequeueAfter` when not entitled so the controller will
automatically re-check.
- (Optional) Add jitter to the requeue interval to avoid synchronized
thundering-herd.
Placement note:
- Keep the check **out of the deletion path** so finalizer cleanup
continues to be best-effort even in unlicensed states.
### 5) (Optional but recommended) Watch `CoderControlPlane` changes to
reconcile faster
**File:** `internal/controller/coderprovisioner_controller.go`
(`SetupWithManager`)
Add a watch on `CoderControlPlane` that enqueues `CoderProvisioner`
objects in the same namespace whose `spec.controlPlaneRef.name` matches
the updated control plane.
This reduces time-to-recover after a license install from “up to
RequeueAfter” to “immediate”.
Implementation sketch:
- Add an index field on `CoderProvisioner.spec.controlPlaneRef.name`.
- In `SetupWithManager`, add
`Watches(&coderv1alpha1.CoderControlPlane{},
handler.EnqueueRequestsFromMapFunc(...))`.
(If we do this, keep the periodic `RequeueAfter` anyway; it’s the safety
net.)
## Tests
### Unit/envtest: `CoderControlPlane` surfaces license tier +
entitlements
**File:** `internal/controller/codercontrolplane_controller_test.go`
Add coverage for the new status fields:
- Use a fake `EntitlementsInspector` (or whatever interface we
introduce) that returns a controlled `codersdk.Entitlements` payload.
- Assert `CoderControlPlane.status` is populated after reconcile when
the control plane is Ready and operator access is available:
- `entitlementsLastChecked` is set.
- `externalProvisionerDaemonsEntitlement` matches the entitlements
payload.
- `licenseTier` is derived correctly (e.g., `none` when
`has_license=false`, `trial` when `trial=true`, `premium` when
`custom_roles` or `multiple_organizations` are entitled, else
`enterprise`).
### Unit/envtest: `CoderProvisioner` short-circuit behavior
**File:** `internal/controller/coderprovisioner_controller_test.go`
Extend the shared `fakeBootstrapClient` (currently in
`coderworkspaceproxy_controller_test.go`) to implement:
```go
Entitlements(ctx context.Context, coderURL, sessionToken string) (codersdk.Entitlements, error)
```
Add test cases:
1. **Not entitled (fast path)**
- Set `controlPlane.status.entitlementsLastChecked` and
`controlPlane.status.externalProvisionerDaemonsEntitlement=not_entitled`.
- Assert:
- `BootstrapClient.Entitlements` is **not** called.
- `EnsureProvisionerKey` is **not** called.
- No key Secret / Deployment is created.
- Condition `ExternalProvisionersEntitled=False`.
- Reconcile returns `RequeueAfter > 0`.
2. **Not entitled (fallback API)**
- Leave the control plane entitlement fields unset/unknown.
- Fake entitlements returns
`features[external_provisioner_daemons].entitlement=not_entitled`.
- Assert the same short-circuit behavior.
3. **Entitled**
- Either set control plane status to `entitled/grace_period` or have the
fake entitlements return `entitled`.
- Assert existing behavior unchanged (key created, resources created)
and condition `ExternalProvisionersEntitled=True`.
4. **Entitlements API forbidden / error**
- Fake entitlements returns a `codersdk.Error` with 401/403 (or generic
error).
- Assert condition is set to False with
`Forbidden`/`EntitlementsQueryFailed` and the reconcile retries.
5. (If we implement the optional watch) verify indexer + mapping
function enqueues provisioners.
## Validation / Rollout
- Run: `make test`, `make build`, `make lint`.
- Because API types/CRDs change (new `CoderControlPlaneStatus` fields),
run `make codegen` and `make manifests`, and commit generated changes.
</details>
---
_Generated with [`mux`](https://github.com/coder/mux) • Model:
`openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: $1.23_
<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh
costs=1.23 -->1 parent 939c739 commit 453dd1d
14 files changed
Lines changed: 958 additions & 2 deletions
File tree
- api/v1alpha1
- config/crd/bases
- docs/reference/api
- internal
- app/controllerapp
- coderbootstrap
- controller
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
15 | 29 | | |
16 | 30 | | |
17 | 31 | | |
| |||
71 | 85 | | |
72 | 86 | | |
73 | 87 | | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
74 | 100 | | |
75 | 101 | | |
76 | 102 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
345 | 345 | | |
346 | 346 | | |
347 | 347 | | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
348 | 359 | | |
349 | 360 | | |
350 | 361 | | |
| |||
356 | 367 | | |
357 | 368 | | |
358 | 369 | | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
359 | 375 | | |
360 | 376 | | |
361 | 377 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
36 | 39 | | |
37 | 40 | | |
38 | 41 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| 95 | + | |
95 | 96 | | |
96 | 97 | | |
97 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
98 | 99 | | |
99 | 100 | | |
100 | 101 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
138 | 138 | | |
139 | 139 | | |
140 | 140 | | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
141 | 168 | | |
142 | 169 | | |
143 | 170 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
315 | 315 | | |
316 | 316 | | |
317 | 317 | | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
318 | 367 | | |
319 | 368 | | |
320 | 369 | | |
| |||
0 commit comments