Skip to content

Commit 450fb4e

Browse files
authored
🤖 feat: reconcile CoderControlPlane licenses from Secret (#66)
## Summary Add automatic Coder Enterprise license management to `CoderControlPlane`. ## Background Previously, the operator had no API surface for license configuration and no reconcile logic to upload licenses after control plane bootstrap. This change allows operators to point at a Secret and have the controller apply licenses once the control plane and operator access are ready, including rotation behavior. ## Implementation - Added `spec.licenseSecretRef` to `CoderControlPlane`. - Added status fields `licenseLastApplied` and `licenseLastAppliedHash`. - Added default shared constant `DefaultLicenseSecretKey = "license"`. - Added `LicenseApplied` condition type and condition updates in reconcile paths. - Implemented controller-side license reconciliation with: - readiness/operator-access preconditions, - Secret read + trim + SHA-256 hash idempotency, - upload via SDK-backed `LicenseUploader`, - 404 (`NotSupported`) and auth/error handling. - Added field index + Secret watch for non-owned `licenseSecretRef` Secrets. - Wired production uploader in controller app setup. - Added controller tests for: - no-ref behavior, - pending-before-ready, - first apply + idempotency, - Secret rotation, - 404/not-supported behavior. - Regenerated deepcopy, CRD, and API reference docs; updated sample manifest. ## Validation - `make verify-vendor` - `make test` - `make build` - `make lint` - `make codegen` - `make manifests` - `make docs-reference` ## Risks - Moderate: touches reconciliation/status/watches in `CoderControlPlane`. - Mitigated by focused unit/envtest coverage for apply/idempotency/rotation/error handling and by preserving existing operator access behavior. --- <details> <summary>📋 Implementation Plan</summary> # Plan: `CoderControlPlane` license Secret reference + automatic license application ## Context / Why We want the `coder-k8s` operator to manage Coder Enterprise licensing automatically: - Add `spec.licenseSecretRef` to `coder.com/v1alpha1.CoderControlPlane` to reference a Secret key containing a Coder license JWT. - Once the control plane is *actually up* (pods ready) and the operator has bootstrap API access, the controller should call the Coder API to upload/apply that license. - If the Secret value changes, the controller should apply the new license (license rotation). - Replace the previous boolean `LicenseApplied` idea with an optional timestamp: `status.licenseLastApplied`. This aligns with existing patterns in the repo (SecretKeySelector usage, operator-managed API token creation, reconcile-with-requeue behavior), and avoids requiring users to manually run `coder licenses add` after every deployment/rotation. ## Evidence (what we verified) - `CoderControlPlane` type today has no license field; it already uses `SecretKeySelector` in status for the operator token secret: - `api/v1alpha1/codercontrolplane_types.go` - `api/v1alpha1/types_shared.go` - Control plane readiness is currently determined by `deployment.Status.ReadyReplicas > 0`, stored in `status.phase` (`Pending`→`Ready`). The controller also computes an in-cluster URL: - `internal/controller/codercontrolplane_controller.go` (`desiredStatus` sets `URL = http://<svc>.<ns>.svc.cluster.local:<port>`) - The controller already owns/watches Secrets (for the operator token secret) and has a `readSecretValue()` helper with strong assertions. - The operator already vendors and uses `github.com/coder/coder/v2/codersdk` via `internal/coderbootstrap`. - Coder license upload is an enterprise-only API endpoint: - `POST /api/v2/licenses` with JSON body `{ "license": "<jwt>" }` - Auth header accepted by Coder: `Coder-Session-Token: <token>` (Bearer token also supported) - Success status: `201 Created` - Uploading the same license twice is **not idempotent**: DB has a unique constraint on `licenses.jwt`, and the server returns `500` on duplicate insert. - In OSS builds, the `/licenses` routes are not registered → `404 Not Found`. ## Design decisions ### API surface Add: - `spec.licenseSecretRef` (optional) — reference to Secret name + key. - `status.licenseLastApplied` (optional `metav1.Time`) — when the operator last successfully uploaded the currently-observed license. Additionally (needed for correctness / idempotency): - `status.licenseLastAppliedHash` (optional string) — SHA-256 hex of the trimmed license JWT that was last successfully applied. Rationale: Coder rejects duplicate uploads (500). Without persisting a stable identity (hash), the controller can’t safely be re-entrant and would spam POSTs on every reconcile. ### Preconditions for applying a license Only attempt license upload when: 1. `spec.licenseSecretRef != nil`. 2. `status.phase == Ready` (deployment has ≥1 ready replica). 3. `status.operatorAccessReady == true` AND `status.operatorTokenSecretRef != nil`. If operator access is disabled or not yet ready, we should not attempt license application (no credentials). ### Rotation semantics - On each reconcile (and on referenced Secret changes), read the Secret value, compute hash, compare to `status.licenseLastAppliedHash`. - If hashes differ, call the license upload endpoint. - On success, set `status.licenseLastApplied = now` and update `status.licenseLastAppliedHash`. ### Watching the referenced Secret The controller currently only watches Secrets it owns (`Owns(&corev1.Secret{})`). User-provided license Secrets are not owned, so we must add an explicit watch for Secrets referenced by `spec.licenseSecretRef`. Use a field index + watch mapping so we *only* enqueue `CoderControlPlane` reconciles for Secrets actually referenced by control planes. ### Enterprise-only behavior If the license API returns 404: - Treat as “not supported” (likely OSS image). - Set a Condition (see below) to False with reason `NotSupported`. - Do **not** requeue aggressively (avoid infinite loops). ### Status Conditions (recommended) `CoderControlPlaneStatus` already has `conditions []metav1.Condition` but it’s not currently populated. Introduce a single condition type for license: - `type: LicenseApplied` - `status: True|False|Unknown` - reasons: `Applied`, `Pending`, `SecretMissing`, `Forbidden`, `NotSupported`, `Error` Keep messages stable to avoid noisy status updates. <details> <summary>Alternatives considered (kept short)</summary> - **Always POST on every reconcile**: rejected because duplicate uploads return 500 due to unique constraint. - **Store the hash in an annotation instead of status**: workable, but status is the more idiomatic place for “observed applied license identity”; also avoids having to Update both metadata + status. - **Decode JWT and compare UUID claim vs `GET /licenses` response**: adds JWT parsing/validation complexity; storing a SHA-256 hash is simpler and avoids relying on claim structure. </details> ## Implementation details (concrete edits) ### 1) API / CRD changes **Files:** - `api/v1alpha1/codercontrolplane_types.go` - `api/v1alpha1/types_shared.go` - generated: `api/v1alpha1/zz_generated.deepcopy.go` - generated: `config/crd/bases/coder.com_codercontrolplanes.yaml` - generated docs: `docs/reference/api/codercontrolplane.md` - sample: `config/samples/coder_v1alpha1_codercontrolplane.yaml` **a) Add spec field** ```go // CoderControlPlaneSpec defines the desired state of a CoderControlPlane. type CoderControlPlaneSpec struct { ... // LicenseSecretRef references a Secret key containing a Coder Enterprise // license JWT. When set, the controller uploads the license to the Coder // API after the control plane is ready, and uploads a new license if the // referenced Secret value changes. // +optional LicenseSecretRef *SecretKeySelector `json:"licenseSecretRef,omitempty"` } ``` **b) Add status fields** ```go // CoderControlPlaneStatus defines the observed state of a CoderControlPlane. type CoderControlPlaneStatus struct { ... // LicenseLastApplied is the timestamp of the most recent successful // license upload performed by the operator. Nil means no license has been // applied by the operator. // +optional LicenseLastApplied *metav1.Time `json:"licenseLastApplied,omitempty"` // LicenseLastAppliedHash is the SHA-256 hex hash of the trimmed license JWT // that LicenseLastApplied refers to. This prevents duplicate uploads. // +optional LicenseLastAppliedHash string `json:"licenseLastAppliedHash,omitempty"` } ``` **c) Add a default key constant** In `api/v1alpha1/types_shared.go` (or a new shared constants file): ```go const DefaultLicenseSecretKey = "license" ``` Controller will treat empty `licenseSecretRef.key` as `DefaultLicenseSecretKey`. **d) Regenerate generated artifacts** - `make codegen` - `make manifests` - `make docs-reference` ### 2) Controller changes **File:** `internal/controller/codercontrolplane_controller.go` **a) Reconciler fields (for testability)** Add an interface that can be faked in tests: ```go type LicenseUploader interface { AddLicense(ctx context.Context, coderURL, sessionToken, licenseJWT string) error } // Production implementation uses codersdk. type sdkLicenseUploader struct{} ``` Add to reconciler: ```go type CoderControlPlaneReconciler struct { client.Client Scheme *runtime.Scheme OperatorAccessProvisioner coderbootstrap.OperatorAccessProvisioner LicenseUploader LicenseUploader // optional; if nil, controller skips license reconciliation } ``` Wire it in `internal/app/controllerapp/controllerapp.go`: ```go reconciler := &controller.CoderControlPlaneReconciler{ Client: client, Scheme: managerScheme, OperatorAccessProvisioner: coderbootstrap.NewPostgresOperatorAccessProvisioner(), LicenseUploader: controller.NewSDKLicenseUploader(), } ``` **b) Reconcile flow changes** After `reconcileOperatorAccess` and before `reconcileStatus`, call `reconcileLicense`: ```go operatorResult, err := r.reconcileOperatorAccess(...) ... licenseResult, err := r.reconcileLicense(ctx, coderControlPlane, &nextStatus) ... if err := r.reconcileStatus(...); err != nil { ... } return mergeResults(operatorResult, licenseResult), nil ``` Where `mergeResults` chooses a non-zero requeue request deterministically (e.g., prefer the shorter `RequeueAfter` if both set). **c) Implement `reconcileLicense`** Shape: ```go func (r *CoderControlPlaneReconciler) reconcileLicense( ctx context.Context, cp *coderv1alpha1.CoderControlPlane, nextStatus *coderv1alpha1.CoderControlPlaneStatus, ) (ctrl.Result, error) ``` Logic: 1. Defensive nil checks + validate inputs. 2. If `cp.Spec.LicenseSecretRef == nil`: clear/leave license condition as Unknown; return. 3. If `nextStatus.Phase != Ready`: set condition False (Pending); return. 4. If `!nextStatus.OperatorAccessReady || nextStatus.OperatorTokenSecretRef == nil`: set condition False (Pending); return. 5. Read operator token via `readSecretValue(namespace, nextStatus.OperatorTokenSecretRef.Name, key)`. 6. Read license JWT via `readSecretValue(namespace, cp.Spec.LicenseSecretRef.Name, resolvedKey)`. - `licenseJWT = strings.TrimSpace(licenseJWT)`; error if empty. 7. Compute `hash := sha256hex(licenseJWT)`. 8. If `hash == nextStatus.LicenseLastAppliedHash` and `nextStatus.LicenseLastApplied != nil`: consider applied; set condition True; return. 9. Call uploader: - `err := r.LicenseUploader.AddLicense(ctx, nextStatus.URL, token, licenseJWT)` - If 404: condition False reason `NotSupported`; return without aggressive requeue. - If 401/403: condition False reason `Forbidden`; requeue after `operatorAccessRetryInterval`. - Other errors: condition False reason `Error`; requeue after `operatorAccessRetryInterval`. 10. On success: set - `now := metav1.Now(); nextStatus.LicenseLastApplied = &now` - `nextStatus.LicenseLastAppliedHash = hash` - condition True reason `Applied` **d) Implement SDK uploader** Use the same HTTP client setup pattern as `internal/coderbootstrap/SDKClient` (dedicated transport clone, timeout). Pseudo-shape: ```go func (u *sdkLicenseUploader) AddLicense(ctx context.Context, coderURL, sessionToken, licenseJWT string) error { parsed, err := url.Parse(coderURL) ... c := codersdk.New(parsed) c.SetSessionToken(sessionToken) c.HTTPClient = &http.Client{Timeout: 30 * time.Second, Transport: http.DefaultTransport.(*http.Transport).Clone()} _, err = c.AddLicense(ctx, codersdk.AddLicenseRequest{License: licenseJWT}) return err } ``` **e) Watch referenced license Secrets** In `SetupWithManager`: 1. Add field indexer: - key: `.spec.licenseSecretRef.name` 2. Add a watch on `corev1.Secret` events that maps to `CoderControlPlane` requests via that index. Sketch: ```go const licenseSecretNameIndex = ".spec.licenseSecretRef.name" if err := mgr.GetFieldIndexer().IndexField(ctx, &coderv1alpha1.CoderControlPlane{}, licenseSecretNameIndex, func(obj client.Object) []string { cp := obj.(*coderv1alpha1.CoderControlPlane) if cp.Spec.LicenseSecretRef == nil { return nil } name := strings.TrimSpace(cp.Spec.LicenseSecretRef.Name) if name == "" { return nil } return []string{name} }); err != nil { ... } builder.Watches( &corev1.Secret{}, handler.EnqueueRequestsFromMapFunc(func(ctx context.Context, obj client.Object) []reconcile.Request { secret := obj.(*corev1.Secret) var list coderv1alpha1.CoderControlPlaneList if err := r.List(ctx, &list, client.InNamespace(secret.Namespace), client.MatchingFields{licenseSecretNameIndex: secret.Name}); err != nil { return nil } ... }), ) ``` ### 3) Tests **File:** `internal/controller/codercontrolplane_controller_test.go` Add a `fakeLicenseUploader` similar to `fakeOperatorAccessProvisioner`. Test cases (table-driven preferred): 1. **No ref → no action**: `LicenseSecretRef=nil` results in no uploader calls. 2. **Pending phase → no action**: with ref but `deployment.Status.ReadyReplicas=0`, uploader not called. 3. **Ready + operator access ready → applies once**: - create cp with `ExtraEnv` containing `CODER_PG_CONNECTION_URL` value - fake operator access provisioner returns token - create license Secret with key `license` and some value - manually set deployment status readyReplicas to 1 - reconcile; assert uploader called once - fetch cp; assert `status.licenseLastApplied != nil` and `status.licenseLastAppliedHash != ""` 4. **Idempotent on re-reconcile**: second reconcile with same Secret value does not call uploader again. 5. **Rotation**: update Secret data to a new value; reconcile; uploader called again; hash updates. 6. **OSS / 404 handling**: uploader returns a `*codersdk.Error` with status 404; controller sets condition reason `NotSupported` and does not tight-loop. ### 4) Documentation + samples - Update `config/samples/coder_v1alpha1_codercontrolplane.yaml` to include an example: ```yaml spec: licenseSecretRef: name: coder-license key: license ``` - Run `make docs-reference` so `docs/reference/api/codercontrolplane.md` includes the new fields. ## Validation / completion checklist Run (in this repo root): - `make codegen` - `make manifests` - `make docs-reference` - `make test` - `make build` - `make lint` Definition of done: - New CRD fields are present and documented. - Operator uploads the license exactly once per distinct Secret value after control plane readiness. - Secret rotation triggers a new upload (no duplicates / no 500 loops). - 404 (OSS) is handled gracefully. - Unit/integration tests cover success + rotation + idempotency. </details> --- _Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `$1.21`_ <!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=1.21 -->
1 parent d68b694 commit 450fb4e

9 files changed

Lines changed: 1348 additions & 6 deletions

File tree

api/v1alpha1/codercontrolplane_types.go

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ const (
1010
CoderControlPlanePhasePending = "Pending"
1111
// CoderControlPlanePhaseReady indicates at least one control plane pod is ready.
1212
CoderControlPlanePhaseReady = "Ready"
13+
// CoderControlPlaneConditionLicenseApplied indicates whether the operator uploaded the configured license.
14+
CoderControlPlaneConditionLicenseApplied = "LicenseApplied"
1315
)
1416

1517
// CoderControlPlaneSpec defines the desired state of a CoderControlPlane.
@@ -32,6 +34,11 @@ type CoderControlPlaneSpec struct {
3234
// OperatorAccess configures bootstrap API access to the coderd instance.
3335
// +kubebuilder:default={}
3436
OperatorAccess OperatorAccessSpec `json:"operatorAccess,omitempty"`
37+
// LicenseSecretRef references a Secret key containing a Coder Enterprise
38+
// license JWT. When set, the controller uploads the license after the
39+
// control plane is ready and re-uploads when the Secret value changes.
40+
// +optional
41+
LicenseSecretRef *SecretKeySelector `json:"licenseSecretRef,omitempty"`
3542
}
3643

3744
// OperatorAccessSpec configures the controller-managed coderd operator user.
@@ -56,6 +63,14 @@ type CoderControlPlaneStatus struct {
5663
OperatorTokenSecretRef *SecretKeySelector `json:"operatorTokenSecretRef,omitempty"`
5764
// OperatorAccessReady reports whether operator API access bootstrap succeeded.
5865
OperatorAccessReady bool `json:"operatorAccessReady,omitempty"`
66+
// LicenseLastApplied is the timestamp of the most recent successful
67+
// operator-managed license upload.
68+
// +optional
69+
LicenseLastApplied *metav1.Time `json:"licenseLastApplied,omitempty"`
70+
// LicenseLastAppliedHash is the SHA-256 hex hash of the trimmed license JWT
71+
// that LicenseLastApplied refers to.
72+
// +optional
73+
LicenseLastAppliedHash string `json:"licenseLastAppliedHash,omitempty"`
5974
// Phase is a high-level readiness indicator.
6075
Phase string `json:"phase,omitempty"`
6176
// Conditions are Kubernetes-standard conditions for this resource.

api/v1alpha1/types_shared.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ import corev1 "k8s.io/api/core/v1"
55
const (
66
// DefaultTokenSecretKey is the default key used for proxy session tokens.
77
DefaultTokenSecretKey = "token"
8+
// DefaultLicenseSecretKey is the default key used for Coder license JWTs.
9+
DefaultLicenseSecretKey = "license"
810
)
911

1012
// ServiceSpec defines the Service configuration reconciled by the operator.

api/v1alpha1/zz_generated.deepcopy.go

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

config/crd/bases/coder.com_codercontrolplanes.yaml

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,21 @@ spec:
226226
type: object
227227
x-kubernetes-map-type: atomic
228228
type: array
229+
licenseSecretRef:
230+
description: |-
231+
LicenseSecretRef references a Secret key containing a Coder Enterprise
232+
license JWT. When set, the controller uploads the license after the
233+
control plane is ready and re-uploads when the Secret value changes.
234+
properties:
235+
key:
236+
description: Key is the key inside the Secret data map.
237+
type: string
238+
name:
239+
description: Name is the Kubernetes Secret name.
240+
type: string
241+
required:
242+
- name
243+
type: object
229244
operatorAccess:
230245
default: {}
231246
description: OperatorAccess configures bootstrap API access to the
@@ -330,6 +345,17 @@ spec:
330345
- type
331346
type: object
332347
type: array
348+
licenseLastApplied:
349+
description: |-
350+
LicenseLastApplied is the timestamp of the most recent successful
351+
operator-managed license upload.
352+
format: date-time
353+
type: string
354+
licenseLastAppliedHash:
355+
description: |-
356+
LicenseLastAppliedHash is the SHA-256 hex hash of the trimmed license JWT
357+
that LicenseLastApplied refers to.
358+
type: string
333359
observedGeneration:
334360
description: ObservedGeneration tracks the spec generation this status
335361
reflects.

config/samples/coder_v1alpha1_codercontrolplane.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,6 @@ metadata:
55
namespace: default
66
spec:
77
image: "ghcr.io/coder/coder-k8s:main"
8+
licenseSecretRef:
9+
name: coder-license
10+
key: license

docs/reference/api/codercontrolplane.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
| `extraEnv` | [EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.35/#envvar-v1-core) array | ExtraEnv are injected into the Coder control plane container. |
2121
| `imagePullSecrets` | [LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.35/#localobjectreference-v1-core) array | ImagePullSecrets are used by the pod to pull private images. |
2222
| `operatorAccess` | [OperatorAccessSpec](#operatoraccessspec) | OperatorAccess configures bootstrap API access to the coderd instance. |
23+
| `licenseSecretRef` | [SecretKeySelector](#secretkeyselector) | LicenseSecretRef references a Secret key containing a Coder Enterprise license JWT. When set, the controller uploads the license after the control plane is ready and re-uploads when the Secret value changes. |
2324

2425
## Status
2526

@@ -30,6 +31,8 @@
3031
| `url` | string | URL is the in-cluster URL for the control plane service. |
3132
| `operatorTokenSecretRef` | [SecretKeySelector](#secretkeyselector) | OperatorTokenSecretRef points to the Secret key containing the `coder-k8s-operator` API token. |
3233
| `operatorAccessReady` | boolean | OperatorAccessReady reports whether operator API access bootstrap succeeded. |
34+
| `licenseLastApplied` | [Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.35/#time-v1-meta) | LicenseLastApplied is the timestamp of the most recent successful operator-managed license upload. |
35+
| `licenseLastAppliedHash` | string | LicenseLastAppliedHash is the SHA-256 hex hash of the trimmed license JWT that LicenseLastApplied refers to. |
3336
| `phase` | string | Phase is a high-level readiness indicator. |
3437
| `conditions` | [Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.35/#condition-v1-meta) array | Conditions are Kubernetes-standard conditions for this resource. |
3538

internal/app/controllerapp/controllerapp.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,8 +88,10 @@ func SetupControllers(mgr manager.Manager) error {
8888

8989
reconciler := &controller.CoderControlPlaneReconciler{
9090
Client: client,
91+
APIReader: mgr.GetAPIReader(),
9192
Scheme: managerScheme,
9293
OperatorAccessProvisioner: coderbootstrap.NewPostgresOperatorAccessProvisioner(),
94+
LicenseUploader: controller.NewSDKLicenseUploader(),
9395
}
9496
if err := reconciler.SetupWithManager(mgr); err != nil {
9597
return fmt.Errorf("unable to create controller: %w", err)

0 commit comments

Comments
 (0)