Commit 450fb4e
authored
🤖 feat: reconcile CoderControlPlane licenses from Secret (#66)
## Summary
Add automatic Coder Enterprise license management to
`CoderControlPlane`.
## Background
Previously, the operator had no API surface for license configuration
and no reconcile logic to upload licenses after control plane bootstrap.
This change allows operators to point at a Secret and have the
controller apply licenses once the control plane and operator access are
ready, including rotation behavior.
## Implementation
- Added `spec.licenseSecretRef` to `CoderControlPlane`.
- Added status fields `licenseLastApplied` and `licenseLastAppliedHash`.
- Added default shared constant `DefaultLicenseSecretKey = "license"`.
- Added `LicenseApplied` condition type and condition updates in
reconcile paths.
- Implemented controller-side license reconciliation with:
- readiness/operator-access preconditions,
- Secret read + trim + SHA-256 hash idempotency,
- upload via SDK-backed `LicenseUploader`,
- 404 (`NotSupported`) and auth/error handling.
- Added field index + Secret watch for non-owned `licenseSecretRef`
Secrets.
- Wired production uploader in controller app setup.
- Added controller tests for:
- no-ref behavior,
- pending-before-ready,
- first apply + idempotency,
- Secret rotation,
- 404/not-supported behavior.
- Regenerated deepcopy, CRD, and API reference docs; updated sample
manifest.
## Validation
- `make verify-vendor`
- `make test`
- `make build`
- `make lint`
- `make codegen`
- `make manifests`
- `make docs-reference`
## Risks
- Moderate: touches reconciliation/status/watches in
`CoderControlPlane`.
- Mitigated by focused unit/envtest coverage for
apply/idempotency/rotation/error handling and by preserving existing
operator access behavior.
---
<details>
<summary>📋 Implementation Plan</summary>
# Plan: `CoderControlPlane` license Secret reference + automatic license
application
## Context / Why
We want the `coder-k8s` operator to manage Coder Enterprise licensing
automatically:
- Add `spec.licenseSecretRef` to `coder.com/v1alpha1.CoderControlPlane`
to reference a Secret key containing a Coder license JWT.
- Once the control plane is *actually up* (pods ready) and the operator
has bootstrap API access, the controller should call the Coder API to
upload/apply that license.
- If the Secret value changes, the controller should apply the new
license (license rotation).
- Replace the previous boolean `LicenseApplied` idea with an optional
timestamp: `status.licenseLastApplied`.
This aligns with existing patterns in the repo (SecretKeySelector usage,
operator-managed API token creation, reconcile-with-requeue behavior),
and avoids requiring users to manually run `coder licenses add` after
every deployment/rotation.
## Evidence (what we verified)
- `CoderControlPlane` type today has no license field; it already uses
`SecretKeySelector` in status for the operator token secret:
- `api/v1alpha1/codercontrolplane_types.go`
- `api/v1alpha1/types_shared.go`
- Control plane readiness is currently determined by
`deployment.Status.ReadyReplicas > 0`, stored in `status.phase`
(`Pending`→`Ready`). The controller also computes an in-cluster URL:
- `internal/controller/codercontrolplane_controller.go` (`desiredStatus`
sets `URL = http://<svc>.<ns>.svc.cluster.local:<port>`)
- The controller already owns/watches Secrets (for the operator token
secret) and has a `readSecretValue()` helper with strong assertions.
- The operator already vendors and uses
`github.com/coder/coder/v2/codersdk` via `internal/coderbootstrap`.
- Coder license upload is an enterprise-only API endpoint:
- `POST /api/v2/licenses` with JSON body `{ "license": "<jwt>" }`
- Auth header accepted by Coder: `Coder-Session-Token: <token>` (Bearer
token also supported)
- Success status: `201 Created`
- Uploading the same license twice is **not idempotent**: DB has a
unique constraint on `licenses.jwt`, and the server returns `500` on
duplicate insert.
- In OSS builds, the `/licenses` routes are not registered → `404 Not
Found`.
## Design decisions
### API surface
Add:
- `spec.licenseSecretRef` (optional) — reference to Secret name + key.
- `status.licenseLastApplied` (optional `metav1.Time`) — when the
operator last successfully uploaded the currently-observed license.
Additionally (needed for correctness / idempotency):
- `status.licenseLastAppliedHash` (optional string) — SHA-256 hex of the
trimmed license JWT that was last successfully applied.
Rationale: Coder rejects duplicate uploads (500). Without persisting a
stable identity (hash), the controller can’t safely be re-entrant and
would spam POSTs on every reconcile.
### Preconditions for applying a license
Only attempt license upload when:
1. `spec.licenseSecretRef != nil`.
2. `status.phase == Ready` (deployment has ≥1 ready replica).
3. `status.operatorAccessReady == true` AND
`status.operatorTokenSecretRef != nil`.
If operator access is disabled or not yet ready, we should not attempt
license application (no credentials).
### Rotation semantics
- On each reconcile (and on referenced Secret changes), read the Secret
value, compute hash, compare to `status.licenseLastAppliedHash`.
- If hashes differ, call the license upload endpoint.
- On success, set `status.licenseLastApplied = now` and update
`status.licenseLastAppliedHash`.
### Watching the referenced Secret
The controller currently only watches Secrets it owns
(`Owns(&corev1.Secret{})`). User-provided license Secrets are not owned,
so we must add an explicit watch for Secrets referenced by
`spec.licenseSecretRef`.
Use a field index + watch mapping so we *only* enqueue
`CoderControlPlane` reconciles for Secrets actually referenced by
control planes.
### Enterprise-only behavior
If the license API returns 404:
- Treat as “not supported” (likely OSS image).
- Set a Condition (see below) to False with reason `NotSupported`.
- Do **not** requeue aggressively (avoid infinite loops).
### Status Conditions (recommended)
`CoderControlPlaneStatus` already has `conditions []metav1.Condition`
but it’s not currently populated. Introduce a single condition type for
license:
- `type: LicenseApplied`
- `status: True|False|Unknown`
- reasons: `Applied`, `Pending`, `SecretMissing`, `Forbidden`,
`NotSupported`, `Error`
Keep messages stable to avoid noisy status updates.
<details>
<summary>Alternatives considered (kept short)</summary>
- **Always POST on every reconcile**: rejected because duplicate uploads
return 500 due to unique constraint.
- **Store the hash in an annotation instead of status**: workable, but
status is the more idiomatic place for “observed applied license
identity”; also avoids having to Update both metadata + status.
- **Decode JWT and compare UUID claim vs `GET /licenses` response**:
adds JWT parsing/validation complexity; storing a SHA-256 hash is
simpler and avoids relying on claim structure.
</details>
## Implementation details (concrete edits)
### 1) API / CRD changes
**Files:**
- `api/v1alpha1/codercontrolplane_types.go`
- `api/v1alpha1/types_shared.go`
- generated: `api/v1alpha1/zz_generated.deepcopy.go`
- generated: `config/crd/bases/coder.com_codercontrolplanes.yaml`
- generated docs: `docs/reference/api/codercontrolplane.md`
- sample: `config/samples/coder_v1alpha1_codercontrolplane.yaml`
**a) Add spec field**
```go
// CoderControlPlaneSpec defines the desired state of a CoderControlPlane.
type CoderControlPlaneSpec struct {
...
// LicenseSecretRef references a Secret key containing a Coder Enterprise
// license JWT. When set, the controller uploads the license to the Coder
// API after the control plane is ready, and uploads a new license if the
// referenced Secret value changes.
// +optional
LicenseSecretRef *SecretKeySelector `json:"licenseSecretRef,omitempty"`
}
```
**b) Add status fields**
```go
// CoderControlPlaneStatus defines the observed state of a CoderControlPlane.
type CoderControlPlaneStatus struct {
...
// LicenseLastApplied is the timestamp of the most recent successful
// license upload performed by the operator. Nil means no license has been
// applied by the operator.
// +optional
LicenseLastApplied *metav1.Time `json:"licenseLastApplied,omitempty"`
// LicenseLastAppliedHash is the SHA-256 hex hash of the trimmed license JWT
// that LicenseLastApplied refers to. This prevents duplicate uploads.
// +optional
LicenseLastAppliedHash string `json:"licenseLastAppliedHash,omitempty"`
}
```
**c) Add a default key constant**
In `api/v1alpha1/types_shared.go` (or a new shared constants file):
```go
const DefaultLicenseSecretKey = "license"
```
Controller will treat empty `licenseSecretRef.key` as
`DefaultLicenseSecretKey`.
**d) Regenerate generated artifacts**
- `make codegen`
- `make manifests`
- `make docs-reference`
### 2) Controller changes
**File:** `internal/controller/codercontrolplane_controller.go`
**a) Reconciler fields (for testability)**
Add an interface that can be faked in tests:
```go
type LicenseUploader interface {
AddLicense(ctx context.Context, coderURL, sessionToken, licenseJWT string) error
}
// Production implementation uses codersdk.
type sdkLicenseUploader struct{}
```
Add to reconciler:
```go type CoderControlPlaneReconciler struct {
client.Client
Scheme *runtime.Scheme
OperatorAccessProvisioner coderbootstrap.OperatorAccessProvisioner
LicenseUploader LicenseUploader // optional; if nil, controller skips license reconciliation
}
```
Wire it in `internal/app/controllerapp/controllerapp.go`:
```go
reconciler := &controller.CoderControlPlaneReconciler{
Client: client,
Scheme: managerScheme,
OperatorAccessProvisioner: coderbootstrap.NewPostgresOperatorAccessProvisioner(),
LicenseUploader: controller.NewSDKLicenseUploader(),
}
```
**b) Reconcile flow changes**
After `reconcileOperatorAccess` and before `reconcileStatus`, call
`reconcileLicense`:
```go
operatorResult, err := r.reconcileOperatorAccess(...)
...
licenseResult, err := r.reconcileLicense(ctx, coderControlPlane, &nextStatus)
...
if err := r.reconcileStatus(...); err != nil { ... }
return mergeResults(operatorResult, licenseResult), nil
```
Where `mergeResults` chooses a non-zero requeue request
deterministically (e.g., prefer the shorter `RequeueAfter` if both set).
**c) Implement `reconcileLicense`**
Shape:
```go
func (r *CoderControlPlaneReconciler) reconcileLicense(
ctx context.Context,
cp *coderv1alpha1.CoderControlPlane,
nextStatus *coderv1alpha1.CoderControlPlaneStatus,
) (ctrl.Result, error)
```
Logic:
1. Defensive nil checks + validate inputs.
2. If `cp.Spec.LicenseSecretRef == nil`: clear/leave license condition
as Unknown; return.
3. If `nextStatus.Phase != Ready`: set condition False (Pending);
return.
4. If `!nextStatus.OperatorAccessReady ||
nextStatus.OperatorTokenSecretRef == nil`: set condition False
(Pending); return.
5. Read operator token via `readSecretValue(namespace,
nextStatus.OperatorTokenSecretRef.Name, key)`.
6. Read license JWT via `readSecretValue(namespace,
cp.Spec.LicenseSecretRef.Name, resolvedKey)`.
- `licenseJWT = strings.TrimSpace(licenseJWT)`; error if empty.
7. Compute `hash := sha256hex(licenseJWT)`.
8. If `hash == nextStatus.LicenseLastAppliedHash` and
`nextStatus.LicenseLastApplied != nil`: consider applied; set condition
True; return.
9. Call uploader:
- `err := r.LicenseUploader.AddLicense(ctx, nextStatus.URL, token,
licenseJWT)`
- If 404: condition False reason `NotSupported`; return without
aggressive requeue.
- If 401/403: condition False reason `Forbidden`; requeue after
`operatorAccessRetryInterval`.
- Other errors: condition False reason `Error`; requeue after
`operatorAccessRetryInterval`.
10. On success: set
- `now := metav1.Now(); nextStatus.LicenseLastApplied = &now`
- `nextStatus.LicenseLastAppliedHash = hash`
- condition True reason `Applied`
**d) Implement SDK uploader**
Use the same HTTP client setup pattern as
`internal/coderbootstrap/SDKClient` (dedicated transport clone,
timeout).
Pseudo-shape:
```go
func (u *sdkLicenseUploader) AddLicense(ctx context.Context, coderURL, sessionToken, licenseJWT string) error {
parsed, err := url.Parse(coderURL)
...
c := codersdk.New(parsed)
c.SetSessionToken(sessionToken)
c.HTTPClient = &http.Client{Timeout: 30 * time.Second, Transport: http.DefaultTransport.(*http.Transport).Clone()}
_, err = c.AddLicense(ctx, codersdk.AddLicenseRequest{License: licenseJWT})
return err
}
```
**e) Watch referenced license Secrets**
In `SetupWithManager`:
1. Add field indexer:
- key: `.spec.licenseSecretRef.name`
2. Add a watch on `corev1.Secret` events that maps to
`CoderControlPlane` requests via that index.
Sketch:
```go
const licenseSecretNameIndex = ".spec.licenseSecretRef.name"
if err := mgr.GetFieldIndexer().IndexField(ctx, &coderv1alpha1.CoderControlPlane{}, licenseSecretNameIndex, func(obj client.Object) []string {
cp := obj.(*coderv1alpha1.CoderControlPlane)
if cp.Spec.LicenseSecretRef == nil { return nil }
name := strings.TrimSpace(cp.Spec.LicenseSecretRef.Name)
if name == "" { return nil }
return []string{name}
}); err != nil { ... }
builder.Watches(
&corev1.Secret{},
handler.EnqueueRequestsFromMapFunc(func(ctx context.Context, obj client.Object) []reconcile.Request {
secret := obj.(*corev1.Secret)
var list coderv1alpha1.CoderControlPlaneList
if err := r.List(ctx, &list, client.InNamespace(secret.Namespace), client.MatchingFields{licenseSecretNameIndex: secret.Name}); err != nil {
return nil
}
...
}),
)
```
### 3) Tests
**File:** `internal/controller/codercontrolplane_controller_test.go`
Add a `fakeLicenseUploader` similar to `fakeOperatorAccessProvisioner`.
Test cases (table-driven preferred):
1. **No ref → no action**: `LicenseSecretRef=nil` results in no uploader
calls.
2. **Pending phase → no action**: with ref but
`deployment.Status.ReadyReplicas=0`, uploader not called.
3. **Ready + operator access ready → applies once**:
- create cp with `ExtraEnv` containing `CODER_PG_CONNECTION_URL` value
- fake operator access provisioner returns token
- create license Secret with key `license` and some value
- manually set deployment status readyReplicas to 1
- reconcile; assert uploader called once
- fetch cp; assert `status.licenseLastApplied != nil` and
`status.licenseLastAppliedHash != ""`
4. **Idempotent on re-reconcile**: second reconcile with same Secret
value does not call uploader again.
5. **Rotation**: update Secret data to a new value; reconcile; uploader
called again; hash updates.
6. **OSS / 404 handling**: uploader returns a `*codersdk.Error` with
status 404; controller sets condition reason `NotSupported` and does not
tight-loop.
### 4) Documentation + samples
- Update `config/samples/coder_v1alpha1_codercontrolplane.yaml` to
include an example:
```yaml
spec:
licenseSecretRef:
name: coder-license
key: license
```
- Run `make docs-reference` so `docs/reference/api/codercontrolplane.md`
includes the new fields.
## Validation / completion checklist
Run (in this repo root):
- `make codegen`
- `make manifests`
- `make docs-reference`
- `make test`
- `make build`
- `make lint`
Definition of done:
- New CRD fields are present and documented.
- Operator uploads the license exactly once per distinct Secret value
after control plane readiness.
- Secret rotation triggers a new upload (no duplicates / no 500 loops).
- 404 (OSS) is handled gracefully.
- Unit/integration tests cover success + rotation + idempotency.
</details>
---
_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking:
`xhigh` • Cost: `$1.21`_
<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh
costs=1.21 -->1 parent d68b694 commit 450fb4e
9 files changed
Lines changed: 1348 additions & 6 deletions
File tree
- api/v1alpha1
- config
- crd/bases
- samples
- docs/reference/api
- internal
- app/controllerapp
- controller
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
| 14 | + | |
13 | 15 | | |
14 | 16 | | |
15 | 17 | | |
| |||
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
35 | 42 | | |
36 | 43 | | |
37 | 44 | | |
| |||
56 | 63 | | |
57 | 64 | | |
58 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
59 | 74 | | |
60 | 75 | | |
61 | 76 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
8 | 10 | | |
9 | 11 | | |
10 | 12 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
226 | 226 | | |
227 | 227 | | |
228 | 228 | | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
229 | 244 | | |
230 | 245 | | |
231 | 246 | | |
| |||
330 | 345 | | |
331 | 346 | | |
332 | 347 | | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
333 | 359 | | |
334 | 360 | | |
335 | 361 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| 34 | + | |
| 35 | + | |
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
91 | 92 | | |
92 | 93 | | |
| 94 | + | |
93 | 95 | | |
94 | 96 | | |
95 | 97 | | |
| |||
0 commit comments