Commit c360b6c
authored
🤖 feat: bootstrap coderd operator access from CoderControlPlane (#49)
## Summary
This PR bootstraps durable coderd API access from `CoderControlPlane` by
provisioning a controller-owned system user and token directly in coderd
Postgres, then storing the minted token in a managed Kubernetes Secret.
## Background
We need in-cluster components to authenticate to the coderd instance
created by `CoderControlPlane` without manual setup. The implementation
preserves first-user setup UX by creating a `coder-k8s-operator`
**system** user (`is_system=true`) and managing token issuance in the
backing database.
## Implementation
- Added `spec.operatorAccess` fields (`disabled`,
`generatedTokenSecretName`) and status fields (`operatorTokenSecretRef`,
`operatorAccessReady`) to `CoderControlPlane` API.
- Implemented
`internal/coderbootstrap/PostgresOperatorAccessProvisioner` to:
- upsert `coder-k8s-operator` as a system owner user,
- ensure organization-admin membership across organizations,
- rotate a stable-named API token with `coder:all` scope and return
plaintext token.
- Wired provisioner into `CoderControlPlaneReconciler` and controller
app startup.
- Extended reconcile flow to resolve `CODER_PG_CONNECTION_URL` from
`spec.extraEnv` (literal or `secretKeyRef`), create/update managed token
Secret, and update status readiness/ref fields.
- Added controller tests for missing PG URL, disabled operator access,
literal URL provisioning, and secretRef URL provisioning.
- Added provisioner unit tests for request validation and token-part
generation behavior.
- Regenerated/updated deepcopy, CRD manifests, API reference docs, and
vendored postgres dependency (`github.com/lib/pq`).
## Validation
- `make verify-vendor`
- `make test`
- `make build`
- `make lint`
## Risks
- **Medium:** Direct DB provisioning depends on coderd schema invariants
for users/memberships/api keys. This is mitigated with explicit
assertions, idempotent upserts, and controller retry behavior on
transient provisioning failures.
---
<details>
<summary>📋 Implementation Plan</summary>
# Plan: Bootstrap coderd API access from `CoderControlPlane`
## Context / Why
We want the **`CoderControlPlane` reconciler** to establish durable API
access to the **coderd instance it deploys**, so other in-cluster
components (e.g. the aggregated API server in a later PR) can
authenticate to coderd and perform CRUD.
Key requirements:
- The operator must **create/ensure a `coder-k8s-operator` Coder user by
default**.
- There must be an **option to disable** this behavior.
- It must **not disrupt the first-user (initial setup) experience** when
a human opens the Coder URL for the first time.
- After the operator user exists, mint an API token scoped to
**`coder:all`** for now.
- **Non-goal for this PR:** wiring the aggregated API server to use the
token.
### Reality check / constraint (validated against upstream Coder)
Coder does **not** expose a supported, unauthenticated REST “admin
backdoor” that would let the operator create users/tokens after the
first user exists.
However, upstream Coder’s first-user check explicitly **excludes system
users** (`is_system=true`). That means we can safely provision a
`coder-k8s-operator` *system* user without disabling the first-user
setup UI.
So the default bootstrap approach becomes:
- **Do not** call `CreateFirstUser` automatically (preserve first-user
UX).
- As soon as the control plane is configured with a Postgres URL
(`CODER_PG_CONNECTION_URL`) and the DB schema is ready, connect to
Postgres and upsert:
- a `coder-k8s-operator` system user with site role `owner`
- org membership in all orgs (role `organization-admin`)
- an API token with scope `coder:all`
- Store that token in a Kubernetes Secret owned by the
`CoderControlPlane`.
## Evidence (repo + upstream Coder)
- `internal/controller/codercontrolplane_controller.go`: currently
reconciles only Deployment/Service and overwrites status each reconcile
(we’ll need to preserve operator-access fields).
- `internal/controller/workspaceproxy_controller.go`: shows the
established pattern of generating/storing a token in a Secret owned by
the CR.
- `CODER_PG_CONNECTION_URL` is the canonical env var for coderd’s
Postgres connection URL (see
`examples/cloudnativepg/codercontrolplane.yaml` and the bundled Coder
docs under `.mux/skills/coder-docs/`).
- Upstream Coder (`coder/coder`):
- first-user detection excludes `is_system=true` users (so system users
do not block the setup wizard)
- API tokens live in `api_keys` and are formatted `{id}-{secret}` with
`hashed_secret = sha256(secret)`; long-lived tokens use
`login_type='token'`, include `scopes={'coder:all'}`, and require a
non-empty `allow_list` (default `*:*`)
- org membership is stored in `organization_members`, and the upstream
`create-admin-user` flow adds admins to all orgs
- dev scripts (`scripts/develop.sh`) bootstrap the *human* first user
via `coder login --first-user-*` (API path); system users (e.g.
`prebuilds` in `coderd/database/migrations/000308_system_user.up.sql`)
are inserted directly in Postgres with `is_system=true`, matching this
plan’s approach for `coder-k8s-operator`.
## Proposed design
### 1) API/CRD additions (`coder.com/v1alpha1`)
Add an operator-access block to the `CoderControlPlane` spec, **enabled
by default**.
```go
// api/v1alpha1/codercontrolplane_types.go
type CoderControlPlaneSpec struct {
...
// OperatorAccess configures bootstrap API access to the coderd instance.
// +kubebuilder:default={}
OperatorAccess OperatorAccessSpec `json:"operatorAccess,omitempty"`
}
type OperatorAccessSpec struct {
// Disabled turns off creation/management of the `coder-k8s-operator` user
// and its API token.
// +kubebuilder:default=false
Disabled bool `json:"disabled,omitempty"`
// GeneratedTokenSecretName controls where the controller stores the minted
// `coder-k8s-operator` API token (key: DefaultTokenSecretKey).
// If unset, defaults to "<controlplane-name>-operator-token" (with safe truncation).
GeneratedTokenSecretName string `json:"generatedTokenSecretName,omitempty"`
}
```
Bootstrap input: the controller needs the coderd Postgres URL so it can
provision the operator user/token directly in the DB.
- It will resolve the URL from `spec.extraEnv` entry
`CODER_PG_CONNECTION_URL` (either a literal `value` or
`valueFrom.secretKeyRef`).
- If the env var is missing/unresolvable, the controller will surface a
not-ready condition and keep requeuing.
Add status fields so later components can discover the token location.
```go
// api/v1alpha1/codercontrolplane_types.go
type CoderControlPlaneStatus struct {
...
// OperatorTokenSecretRef points to the Secret key containing the
// `coder-k8s-operator` API token.
OperatorTokenSecretRef *SecretKeySelector `json:"operatorTokenSecretRef,omitempty"`
// OperatorAccessReady reports whether the operator has successfully minted
// a token.
OperatorAccessReady bool `json:"operatorAccessReady,omitempty"`
}
```
Notes:
- Keep naming consistent with `WorkspaceProxy`
(`GeneratedTokenSecretName`, `SecretKeySelector`).
- We can later replace `OperatorAccessReady` with full
`metav1.Condition` usage, but start minimal.
### 2) Operator access provisioning via Postgres (no bootstrap token)
Instead of trying to bootstrap via the coderd REST API, we provision the
operator user/token **directly in Postgres**.
Implement a small provisioner that:
1. Resolves the Postgres URL from `CODER_PG_CONNECTION_URL` in
`spec.extraEnv` (supports literal `value` or `valueFrom.secretKeyRef`).
2. Connects to Postgres and upserts:
- system user `coder-k8s-operator` (`users.is_system=true`,
`users.login_type='none'`, `users.rbac_roles={'owner'}`)
- org membership for **all orgs**
(`organization_members.roles={'organization-admin'}`)
- a long-lived API token row in `api_keys` with:
- token format `{id}-{secret}`
- `hashed_secret = sha256(secret)`
- `login_type='token'`
- `scopes={'coder:all'}`
- a non-empty `allow_list` (default `*:*`)
3. Returns the token string so the controller can store it in a
Kubernetes Secret.
Because the user is created with `is_system=true`, it does **not**
affect the first-user setup wizard.
Implementation notes:
- Prefer reusing upstream token helpers from
`github.com/coder/coder/v2/coderd/apikey` (Generate/HashSecret, random
string lengths/charset) to avoid drifting from Coder’s constraints.
- Use a transaction and idempotent upserts keyed by `username` (user)
and a stable token identifier (e.g. `token_name='coder-k8s-operator'`).
Suggested interface shape (for wiring + tests):
```go
// internal/coderbootstrap/operator_access_postgres.go
type OperatorAccessProvisioner interface {
EnsureOperatorToken(ctx context.Context, req EnsureOperatorTokenRequest) (string, error)
}
type EnsureOperatorTokenRequest struct {
PostgresURL string
OperatorUsername string
OperatorEmail string
// TokenName is written to `api_keys.token_name`.
TokenName string
// TokenLifetime controls expires_at / lifetime_seconds.
TokenLifetime time.Duration
}
```
(Scopes are hardcoded to `coder:all` for now; we’ll scope down later.)
<details>
<summary>DB upsert sketch (for the provisioner)</summary>
- Upsert user by username (`users.username='coder-k8s-operator'`):
- `is_system=true`, `login_type='none'`, `status='active'`,
`rbac_roles={'owner'}`, `hashed_password='none'`
- Ensure org membership for all orgs:
- `INSERT ... SELECT id FROM organizations ...` into
`organization_members` with role `organization-admin` (upsert on
`(organization_id,user_id)`)
- Create token row in `api_keys`:
- generate `{id}-{secret}` (id len 10, secret len 22, `[0-9A-Za-z]`),
store `hashed_secret=sha256(secret)`
- set `login_type='token'`, `token_name='coder-k8s-operator'`,
`scopes={'coder:all'}`, `allow_list={'*:*'}`,
`expires_at=now()+lifetime`, `lifetime_seconds=...`
</details>
### 3) Controller changes (`CoderControlPlaneReconciler`)
Update `internal/controller/codercontrolplane_controller.go` to:
- Add RBAC for secrets (needed to read `CODER_PG_CONNECTION_URL` when
it’s a Secret ref, and to write the generated operator token Secret):
-
`+kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch;delete`
- Add an injectable provisioner (so controller tests don’t need a real
Postgres instance):
```go
type CoderControlPlaneReconciler struct {
client.Client
Scheme *runtime.Scheme
OperatorAccessProvisioner coderbootstrap.OperatorAccessProvisioner
}
```
- Reconcile order:
1. Reconcile Deployment
2. Reconcile Service
3. Compute base status (`ObservedGeneration`, `ReadyReplicas`, `URL`,
`Phase`)
4. If operator access is enabled, run `reconcileOperatorAccess(...)`
(DB-based; it can succeed even while the Deployment is Pending)
5. Update status once with both base fields + operator-access fields
(avoid overwriting operator fields the way `reconcileStatus` currently
does).
#### `reconcileOperatorAccess` behavior (DB-based)
Pseudo-flow:
```go
if cp.Spec.OperatorAccess.Disabled {
// Stop managing operator access. (Optional: delete the managed Secret.)
clear status.OperatorAccessReady / OperatorTokenSecretRef
return
}
// Operator access is DB-based, so we can provision even while the Deployment is Pending.
// If the DB schema/migrations aren’t ready yet, treat that as a retryable error and requeue.
// If we already have a managed operator token secret with a token, stop.
operatorSecretName := resolveGeneratedTokenSecretName(cp) // default <cp-name>-operator-token
if existing, err := readSecretValue(...operatorSecretName...); err == nil && existing != "" {
set status.OperatorTokenSecretRef and OperatorAccessReady=true
return
}
// No need to wait for the human first-user flow: system users (`is_system=true`) don't affect it.
pgURL, err := resolvePostgresURLFromExtraEnv(ctx, cp) // reads CODER_PG_CONNECTION_URL
if err != nil {
status.OperatorAccessReady = false
return ctrl.Result{RequeueAfter: 30s}, nil
}
// Provision system user + API token directly in Postgres.
token, err := r.OperatorAccessProvisioner.EnsureOperatorToken(ctx, coderbootstrap.EnsureOperatorTokenRequest{PostgresURL: pgURL, ...})
if err != nil {
return ctrl.Result{RequeueAfter: 30s}, nil
}
ensureTokenSecret(ctx, cp, operatorSecretName, DefaultTokenSecretKey, token)
status.OperatorTokenSecretRef = &SecretKeySelector{Name: operatorSecretName, Key: DefaultTokenSecretKey}
status.OperatorAccessReady = true
```
Notes:
- Create the operator as a **system** user (`is_system=true`) with
`rbac_roles={'owner'}` and `login_type='none'`.
- Upstream first-user setup ignores system users, so this does **not**
disrupt the initial setup UI.
- Ensure org membership for all orgs (`organization_members`) with role
`organization-admin`.
- Token row in `api_keys` should be created with:
- `login_type='token'`
- `scopes={'coder:all'}`
- `allow_list={'*:*'}` (schema constraint requires non-empty)
- Use stable defaults (unless we later add config fields):
- username: `coder-k8s-operator`
- email: `coder-k8s-operator@coder-k8s.invalid`
- token_name: `coder-k8s-operator`
- token_lifetime: e.g. 365d
### 4) Wiring (controller app)
Update `internal/app/controllerapp/controllerapp.go` to construct and
pass a real Postgres-backed provisioner into the control plane
reconciler (mirrors how `WorkspaceProxyReconciler` gets a bootstrap
client today):
```go
reconciler := &controller.CoderControlPlaneReconciler{
Client: client,
Scheme: managerScheme,
OperatorAccessProvisioner: coderbootstrap.NewPostgresOperatorAccessProvisioner(),
}
```
The provisioner should encapsulate:
- resolving/using a Postgres driver (`database/sql`)
- generating token `{id}-{secret}` using upstream helpers
(`coderd/apikey`)
- idempotent upserts into `users`, `organization_members`, and
`api_keys`
### 5) Tests
#### Unit tests (envtest controller tests)
Update/add tests under `internal/controller/` (use a fake provisioner so
we don’t need a real Postgres instance):
- `TestReconcile_DefaultOperatorAccess_MissingPostgresURL`
- Create `CoderControlPlane` with no `operatorAccess` block (defaults
apply) and without `CODER_PG_CONNECTION_URL`.
- Assert: no operator token Secret created; status shows
`OperatorAccessReady=false`; reconcile returns `RequeueAfter`.
- `TestReconcile_OperatorAccess_Disabled`
- Set `spec.operatorAccess.disabled: true`.
- Assert: controller never calls the provisioner; does not create
Secret.
-
`TestReconcile_OperatorAccess_ResolvesPostgresURLAndCreatesTokenSecret`
- Fake provisioner `EnsureOperatorToken` returns `token-value`.
- Provide `CODER_PG_CONNECTION_URL` in `spec.extraEnv` (cover both
literal and `secretKeyRef` cases).
- Assert: operator token Secret exists + ownerRef set; status references
it; fake provisioner observed the expected Postgres URL.
Implementation detail: add a small `fakeOperatorAccessProvisioner`
(similar to `fakeBootstrapClient` in workspace proxy tests) so tests
don’t need live coderd/Postgres.
### 6) Generated artifacts + docs
Because the CRD API types change:
- Run `make codegen` (deepcopy).
- Run `make manifests` (CRDs + RBAC).
- Run `make docs-reference` (updates
`docs/reference/api/codercontrolplane.md`).
Consider adding a small snippet to an existing “deploy controller” doc
or sample YAML documenting the output token Secret:
- Ensure `CODER_PG_CONNECTION_URL` is provided via `spec.extraEnv`
(value or `secretKeyRef`).
- (Optional) Complete the first-user setup in the UI — it remains
available even after the operator provisions its system user.
- The controller will create/maintain Secret
`<controlplane-name>-operator-token` (key `token`) containing the
`coder-k8s-operator` API token.
## Validation (local)
- `make test`
- `make build`
- `make lint`
- `make verify-vendor` (will include new vendored packages due to
Postgres/token helpers)
---
<details>
<summary>Appendix: why we don’t auto-create the first user</summary>
Coder’s first-user experience relies on *no users existing*.
Automatically creating `coder-k8s-operator` as the first user would flip
the UI into “login” mode immediately and block the initial setup flow.
The chosen approach provisions `coder-k8s-operator` as a *system* user
(`is_system=true`) and creates a `coder:all` token directly in Postgres.
System users are excluded from the first-user check, so the setup wizard
remains available.
</details>
## Implementation strategy (agents)
Recommendation: **have a single agent implement this end-to-end**. This
change spans API types, controller logic, DB/token provisioning logic,
tests, and generated artifacts (`codegen`, `manifests`,
`docs-reference`). A single implementer avoids merge conflicts and
“regen drift”.
If you *do* want to split it across multiple agents, a reasonable
breakdown is:
- **Agent A (API surface + generated output):** update
`api/v1alpha1/codercontrolplane_types.go`, then run/update `make
codegen`, `make manifests`, `make docs-reference`.
- **Agent B (Postgres provisioner):** implement
`internal/coderbootstrap/operator_access_postgres.go` (DB upserts +
token generation via upstream helpers), with focused unit tests for
SQL/idempotency.
- **Agent C (controller integration):** wire the provisioner into
`internal/controller/codercontrolplane_controller.go`, resolve
`CODER_PG_CONNECTION_URL` from `spec.extraEnv`, manage the output
Secret, and preserve status fields.
- **Agent D (controller tests):** add envtest coverage in
`internal/controller/codercontrolplane_controller_test.go` using a fake
provisioner.
Coordination note: only one agent should be responsible for the final
regeneration/validation pass to keep the branch consistent.
</details>
---
_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking:
`xhigh` • Cost: `$3.91`_
<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh
costs=3.91 -->1 parent db4eb3d commit c360b6c
41 files changed
Lines changed: 9507 additions & 12 deletions
File tree
- api/v1alpha1
- config/crd/bases
- docs/reference/api
- internal
- app/controllerapp
- coderbootstrap
- controller
- vendor
- github.com/lib/pq
- oid
- scram
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
32 | 45 | | |
33 | 46 | | |
34 | 47 | | |
| |||
39 | 52 | | |
40 | 53 | | |
41 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
42 | 59 | | |
43 | 60 | | |
44 | 61 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
226 | 226 | | |
227 | 227 | | |
228 | 228 | | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
229 | 245 | | |
230 | 246 | | |
231 | 247 | | |
| |||
319 | 335 | | |
320 | 336 | | |
321 | 337 | | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
322 | 356 | | |
323 | 357 | | |
324 | 358 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
| 32 | + | |
30 | 33 | | |
31 | 34 | | |
32 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
525 | 525 | | |
526 | 526 | | |
527 | 527 | | |
| 528 | + | |
| 529 | + | |
528 | 530 | | |
529 | 531 | | |
530 | 532 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
89 | | - | |
90 | | - | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
91 | 92 | | |
92 | 93 | | |
93 | 94 | | |
| |||
0 commit comments