Skip to content

Commit e142bce

Browse files
authored
🤖 feat: wire aggregated API server to codersdk backend (#50)
## Summary Wire the aggregated API server (`--app=aggregated-apiserver`) to a real Coder deployment backend via `codersdk`, replacing the hardcoded in-memory storage with codersdk-backed REST storage for `CoderTemplate` and `CoderWorkspace` resources. ## Background The aggregated API server mode previously served stub data from hardcoded maps. This PR makes it a real Kubernetes façade over a Coder deployment, so that Kubernetes clients can query and mutate Coder templates/workspaces using standard K8s REST semantics. Key design decisions (v1): - **Namespace == CoderControlPlane namespace**: Resources remain namespace-scoped; the namespace identifies which CoderControlPlane backs the request. - **Deterministic name formats**: Templates use `<org>.<template-name>`, Workspaces use `<org>.<user>.<workspace-name>` — the dot separator works because Coder names forbid dots while K8s object names allow them. - **Admin token auth**: A single `--coder-session-token` is used (no per-request impersonation in v1). - **Start/stop via Update**: `CoderWorkspace.spec.running` drives workspace transitions. ## Implementation ### New packages - **`internal/aggregated/coder/`** — Backend helpers: SDK client factory (`config.go`), K8s↔Coder name parsing (`names.go`), codersdk→K8s error mapping (`errors.go`), and `ClientProvider` interface with `StaticClientProvider` (`provider.go`). - **`internal/aggregated/convert/`** — Pure conversion functions between `codersdk.Template`/`codersdk.Workspace` and the K8s aggregated API types. ### Modified packages - **`api/aggregation/v1alpha1/types.go`** — Expanded `CoderTemplate{Spec,Status}` and `CoderWorkspace{Spec,Status}` with codersdk-aligned fields (organization, templateName, running, build status, etc.). - **`internal/aggregated/storage/`** — Rewrote `template.go` and `workspace.go` from hardcoded maps to codersdk-backed CRUD: - Templates: `GET` → `TemplateByName`, `LIST` → `Templates`, `CREATE` → `CreateTemplate`, `DELETE` → `DeleteTemplate` - Workspaces: `GET` → `WorkspaceByOwnerAndName`, `LIST` → `Workspaces`, `CREATE` → `CreateUserWorkspace` + optional stop build, `UPDATE` → start/stop via `CreateWorkspaceBuild`, `DELETE` → delete transition build - **`internal/app/apiserverapp/apiserverapp.go`** — Added `Options` struct and `RunWithOptions`, changed `NewAPIGroupInfo` to accept a `ClientProvider`, updated OpenAPI definitions. - **`app_dispatch.go`** — Added `--coder-session-token`, `--coder-url`, `--coder-request-timeout` CLI flags wired into `RunWithOptions`. ### Tests - Unit tests for all new packages (names, errors, config, provider, converters). - Rewrote `storage_test.go` with httptest-backed Coder API mock covering all CRUD operations. - Added `integration_test.go` that bootstraps the full aggregated API server and verifies HTTP responses through the complete stack. ## Risks - **Breaking change for `Run(ctx)`**: The zero-arg `Run` function now wraps `RunWithOptions` with empty options, which will fail without Coder configuration. This is intentional — the real entrypoint is now `RunWithOptions` via CLI dispatch. - **Legacy template fields**: `CoderTemplateSpec.Running` and `CoderTemplateStatus.AutoShutdown` are retained temporarily for compatibility with existing callers (e.g., MCP tools). These should be removed in a follow-up once callers are migrated. - **No coderdtest integration**: `coderdtest` is not vendored (massive dep tree). Storage tests use httptest mocks instead. True coderdtest integration is deferred. - **No WATCH semantics**: Only poll-based operations (GET/LIST/CREATE/UPDATE/DELETE) are implemented. --- <details> <summary>📋 Implementation Plan</summary> # Wire aggregated API server → Coder (`codersdk`) backend ## Context / Why `coder-k8s` has an **aggregated API server mode** (`--app=aggregated-apiserver`) that currently serves the API group `aggregation.coder.com/v1alpha1`, but its storage is **hardcoded in-memory**. We want that API server to become a real “Kubernetes façade” over a running Coder deployment, backed by `github.com/coder/coder/v2/codersdk`, so Kubernetes clients can **query and mutate** Coder templates/workspaces (and eventually other Coder resources) using standard K8s REST semantics. A key dev/test requirement is that we can run **`coderdtest` + envtest** to exercise the end-to-end path: `k8s client → kube-apiserver (envtest) → API aggregation proxy → coder-k8s aggregated-apiserver → codersdk → coderdtest` ## Goals (first milestone) 1. Replace hardcoded storage with `codersdk`-backed storage for: - `CoderTemplate` (`codertemplates`) - `CoderWorkspace` (`coderworkspaces`) 2. Support the core REST verbs: - Templates: `GET`, `LIST`, `CREATE`, `DELETE` (optionally `UPDATE` meta) - Workspaces: `GET`, `LIST`, `CREATE`, `UPDATE` (drives start/stop), `DELETE` (drives delete transition) 3. Add aggregated-apiserver configuration (flags/env) for: - Coder URL - Coder session token (admin token in dev) - Coder request timeout (optional) 4. Define a deterministic K8s identity scheme for locating Coder objects (see “Chosen semantics”). 5. Add tests that prove storage → codersdk behavior using `coderdtest`. 6. Add (or scaffold) an envtest + APIService-based integration test that proxies requests through the envtest kube-apiserver. ## Non-goals (for first milestone) - Per-user auth / impersonation (requestheader authn, OIDC token exchange, etc.). - Efficient `WATCH` semantics / informer-grade semantics. - Exhaustive mapping of every Coder API field into K8s types. ## Evidence (repo + upstream facts) - Aggregated apiserver bootstraps and installs storage in: - `internal/app/apiserverapp/apiserverapp.go` (`NewAPIGroupInfo` registers `coderworkspaces` and `codertemplates`). - Current storage is hardcoded read-only, implements only `rest.Getter` + `rest.Lister`: - `internal/aggregated/storage/workspace.go` - `internal/aggregated/storage/template.go` - API types are minimal stubs: - `api/aggregation/v1alpha1/types.go` - `codersdk` is already vendored and in use: - `internal/coderbootstrap/client.go` imports `github.com/coder/coder/v2/codersdk` - `go.mod` requires `github.com/coder/coder/v2 v2.30.0` - Relevant `codersdk` methods/signatures (vendored): - Workspaces: - `(*codersdk.Client).Workspaces(ctx, codersdk.WorkspaceFilter) (codersdk.WorkspacesResponse, error)` in `vendor/.../codersdk/workspaces.go` - `(*codersdk.Client).WorkspaceByOwnerAndName(ctx, owner, name string, params codersdk.WorkspaceOptions) (codersdk.Workspace, error)` in `vendor/.../codersdk/workspaces.go` - `(*codersdk.Client).CreateUserWorkspace(ctx, user string, req codersdk.CreateWorkspaceRequest) (codersdk.Workspace, error)` in `vendor/.../codersdk/organizations.go` - `(*codersdk.Client).CreateWorkspaceBuild(ctx, workspaceID uuid.UUID, req codersdk.CreateWorkspaceBuildRequest) (codersdk.WorkspaceBuild, error)` in `vendor/.../codersdk/workspaces.go` - Templates: - `(*codersdk.Client).TemplatesByOrganization(ctx, orgID uuid.UUID) ([]codersdk.Template, error)` in `vendor/.../codersdk/organizations.go` - `(*codersdk.Client).TemplateByName(ctx, orgID uuid.UUID, name string) (codersdk.Template, error)` in `vendor/.../codersdk/organizations.go` - `(*codersdk.Client).CreateTemplate(ctx, orgID uuid.UUID, req codersdk.CreateTemplateRequest) (codersdk.Template, error)` in `vendor/.../codersdk/organizations.go` - `(*codersdk.Client).DeleteTemplate(ctx, templateID uuid.UUID) error` in `vendor/.../codersdk/templates.go` - `coderdtest` exists upstream and provides in-memory coderd + helpers: - `coderd/coderdtest` from `coder/coder` tag `v2.30.0` (confirmed via `web_fetch`). - Kubernetes validates CRD instance names as **DNS-1123 subdomains** (dots allowed) — see `vendor/k8s.io/apimachinery/pkg/util/validation/validation.go` (`dns1123SubdomainFmt`). ## Chosen semantics (to keep v1 implementable) These are explicit “v1” choices to avoid design deadlocks. 1. **Resources remain namespace-scoped; namespace == CoderControlPlane namespace** - `CoderTemplate` and `CoderWorkspace` remain **namespaced** in the aggregated API (`NamespaceScoped() == true`). - `metadata.namespace` is interpreted as the namespace that contains the backing `coder.com/v1alpha1 CoderControlPlane` instance. - Storage uses `genericapirequest.NamespaceValue(ctx)` to select the control plane and build a `codersdk.Client` for that namespace. 2. **Namespace is NOT the Coder organization** - The Coder organization is explicit per object (encoded in `metadata.name`, and optionally mirrored in `spec.organization`). 3. **Deterministic name formats (encode Coder lookup keys)** - Templates: `<org>.<template-name>` - Workspaces: `<org>.<user>.<workspace-name>` - We use `.` as the separator because Coder “names” are alphanumeric-with-hyphens (no dots) (`codersdk.NameValid`), while Kubernetes object names allow dots. - Kubernetes validates CRD instance names as **DNS-1123 subdomains**: lowercase only; segments separated by dots must be valid DNS labels (no empty segments / no consecutive dots); max 253 chars; must start/end with an alphanumeric. 4. **Admin token, but no default owner** - The aggregated-apiserver is configured with a single `--coder-session-token` (admin token in dev/test); we do not implement per-request user impersonation in v1. - Storage never assumes `me`; it derives the target user from the workspace object name and can list across all workspaces visible to the admin token. 5. **`CoderWorkspace.spec.running` drives start/stop** - `UPDATE` compares desired `.spec.running` to current state and performs `CreateWorkspaceBuild` with `transition=start|stop`. <details> <summary>Why these semantics?</summary> - K8s `GET` requests provide only `{namespace, name}`. Using namespace to pick the backing CoderControlPlane, and encoding `{org,user}` into the object name, makes `GET` deterministic without extra indices/caches. - This also keeps the org dimension explicit without overloading Kubernetes namespaces. </details> --- # Implementation Plan (swarm-friendly) ## 0) Repo-wide prep / guardrails **Owner:** 1 agent - Add a short design doc comment to the aggregated storage package explaining the v1 semantics (namespaced resources keyed by control plane namespace, composite name formats, admin token, no per-request impersonation) so future changes don’t silently break assumptions. - Ensure every new exported function includes defensive nil/empty checks consistent with existing code (see `assertion failed:` style in `internal/aggregated/storage/*`). ## 1) API type expansion (if we want meaningful CREATE) **Owner:** 1–2 agents ### 1.1 Update `api/aggregation/v1alpha1/types.go` Add fields needed to create/drive resources. Resource scope: - Keep both resources **namespaced** (no scope change). Adding explicit `// +kubebuilder:resource:scope=Namespaced` markers is optional but recommended for clarity. Suggested shapes (keep minimal; everything optional except create-critical fields): ```go // CoderTemplateSpec defines the desired state of a CoderTemplate. type CoderTemplateSpec struct { // Organization is the target Coder organization name (explicit; not derived from namespace). // Must match the `<organization>` prefix in `metadata.name` (see “Chosen semantics”). Organization string `json:"organization,omitempty"` // VersionID is the Coder template version UUID used on creation. // Required for CREATE. VersionID string `json:"versionID,omitempty"` DisplayName string `json:"displayName,omitempty"` Description string `json:"description,omitempty"` Icon string `json:"icon,omitempty"` Deprecated bool `json:"deprecated,omitempty"` } type CoderTemplateStatus struct { ID string `json:"id,omitempty"` // Coder UUID OrganizationName string `json:"organizationName,omitempty"` ActiveVersionID string `json:"activeVersionID,omitempty"` UpdatedAt *metav1.Time `json:"updatedAt,omitempty"` } // CoderWorkspaceSpec defines the desired state of a CoderWorkspace. type CoderWorkspaceSpec struct { // Organization is the target Coder organization name (explicit; not derived from namespace). Organization string `json:"organization,omitempty"` // TemplateName resolves via TemplateByName(organization, templateName). TemplateName string `json:"templateName,omitempty"` // Optional: override and pin to a template version. TemplateVersionID string `json:"templateVersionID,omitempty"` Running bool `json:"running"` // Optional start/stop tuning. TTLMillis *int64 `json:"ttlMillis,omitempty"` AutostartSchedule *string `json:"autostartSchedule,omitempty"` } type CoderWorkspaceStatus struct { ID string `json:"id,omitempty"` // Coder UUID OwnerName string `json:"ownerName,omitempty"` OrganizationName string `json:"organizationName,omitempty"` TemplateName string `json:"templateName,omitempty"` LatestBuildID string `json:"latestBuildID,omitempty"` LatestBuildStatus string `json:"latestBuildStatus,omitempty"` AutoShutdown *metav1.Time `json:"autoShutdown,omitempty"` LastUsedAt *metav1.Time `json:"lastUsedAt,omitempty"` } ``` ### 1.2 Regenerate generated artifacts **Must do in Exec mode after type changes**: - `make codegen` - `make manifests` - `make docs-reference` (if API reference docs exist/are required) ### 1.3 Update OpenAPI definitions used by aggregated-apiserver `internal/app/apiserverapp/apiserverapp.go` currently manually builds a very small OpenAPI schema for spec/status. Update it to reflect the new fields (or, if we decide to defer schema accuracy, at minimum ensure required create fields are present in the schema). --- ## 2) Add a codersdk-backed “backend” helper package **Owner:** 1 agent Create `internal/aggregated/coder/` (new package) with the shared plumbing that storage needs. ### 2.1 Options / factory File: `internal/aggregated/coder/config.go` ```go type Config struct { CoderURL *url.URL SessionToken string RequestTimeout time.Duration } func NewSDKClient(cfg Config) (*codersdk.Client, error) { // defensive checks // codersdk.New(cfg.CoderURL) // client.SetSessionToken(cfg.SessionToken) // ensure HTTPClient != nil and set Timeout } ``` Notes: - Don’t mutate a shared `*codersdk.Client`’s token at runtime; treat it as immutable. ### 2.2 K8s name parsing (org/user keys) File: `internal/aggregated/coder/names.go` ```go // Template object name format: "<org>.<template-name>". func ParseTemplateName(name string) (org string, template string, err error) { // defensive checks: non-empty name, exactly one '.', no empty segments // validate full name is a DNS-1123 subdomain (CRD object name rules; dots allowed) // validate each segment with codersdk.NameValid and require lowercase } // Workspace object name format: "<org>.<user>.<workspace-name>". func ParseWorkspaceName(name string) (org string, user string, workspace string, err error) { // defensive checks: non-empty name, exactly two '.', no empty segments // validate full name is a DNS-1123 subdomain // validate each segment with codersdk.NameValid and require lowercase } func BuildTemplateName(org, template string) string { return strings.ToLower(org + "." + template) } func BuildWorkspaceName(org, user, workspace string) string { return strings.ToLower(org + "." + user + "." + workspace) } ``` Org resolution uses the parsed `org` (and/or `spec.organization`) and calls: - `orgObj, err := sdk.OrganizationByName(ctx, orgName)` ### 2.3 Error mapping File: `internal/aggregated/coder/errors.go` - Convert `*codersdk.Error` (status codes) into Kubernetes-style errors: - 404 → `apierrors.NewNotFound` - 403 → `apierrors.NewForbidden` - 409 → `apierrors.NewAlreadyExists` / `apierrors.NewConflict` (choose based on operation) - others → `apierrors.NewInternalError` Also: wrap unknown errors with context and retain the original error for logs. ### 2.4 Resolve `CoderControlPlane` per request namespace (namespace → Coder URL) File: `internal/aggregated/coder/provider.go` (or similar) We need a single place that translates the **Kubernetes request namespace** (which is the **control plane namespace**) into a `*codersdk.Client` configured for that control plane. Suggested interface: ```go type ClientProvider interface { ClientForNamespace(ctx context.Context, namespace string) (*codersdk.Client, error) } ``` Suggested concrete implementation (names are illustrative): ```go type ControlPlaneClientProvider struct { KubeClient client.Client // controller-runtime client, talks to the main kube-apiserver // Global defaults SessionToken string RequestTimeout time.Duration // Optional fallback for dev (when no CoderControlPlane exists) DefaultCoderURL *url.URL // Optional selector (if multiple CoderControlPlanes exist in a namespace) ControlPlaneName string // Optional cache: namespace -> *codersdk.Client // (protect with mutex) } ``` Algorithm for `ClientForNamespace`: 1. Validate `namespace != ""`. 2. Resolve the backing `CoderControlPlane` in that namespace: - If `ControlPlaneName != ""`: `Get(namespace, ControlPlaneName)`. - Else: `List` CoderControlPlanes in namespace and require exactly 1. 3. Read `controlPlane.Status.URL` and parse as `*url.URL`. - If empty/unparseable and `DefaultCoderURL != nil`, use fallback. - Else return a clear error (`ServiceUnavailable` / `BadRequest`). 4. Construct `codersdk.New(url)` and set the fixed session token + HTTP timeout. RBAC impact (cluster): aggregated-apiserver service account must be able to `get/list/watch` `codercontrolplanes` across namespaces. --- ## 3) Implement conversion layer (codersdk ↔ aggregated API types) **Owner:** 1–2 agents Create `internal/aggregated/convert/` with pure conversion functions. ### 3.1 Templates File: `internal/aggregated/convert/template.go` ```go func TemplateToK8s(namespace string, t codersdk.Template) *aggregationv1alpha1.CoderTemplate { // Name := coder.BuildTemplateName(t.OrganizationName, t.Name) // "<org>.<template-name>" // Namespace := namespace (CoderControlPlane namespace) // UID := t.ID.String() // ResourceVersion := strconv.FormatInt(t.UpdatedAt.UnixNano(), 10) // Spec.Organization := t.OrganizationName // Status fields from t } func TemplateCreateRequestFromK8s(obj *aggregationv1alpha1.CoderTemplate, templateName string) (codersdk.CreateTemplateRequest, error) { // templateName parsed from metadata.name "<org>.<template-name>" // parse obj.Spec.VersionID (uuid) // map display name, description, icon } ``` ### 3.2 Workspaces File: `internal/aggregated/convert/workspace.go` ```go func WorkspaceToK8s(namespace string, w codersdk.Workspace) *aggregationv1alpha1.CoderWorkspace { // Name := coder.BuildWorkspaceName(w.OrganizationName, w.OwnerName, w.Name) // "<org>.<user>.<workspace-name>" // Namespace := namespace (CoderControlPlane namespace) // UID := w.ID.String() // Spec.Organization := w.OrganizationName // Status includes owner, template, latest build info, last used, etc. // Spec.Running inferred from latest build transition/status } func WorkspaceCreateRequestFromK8s(obj *aggregationv1alpha1.CoderWorkspace, workspaceName string, templateID uuid.UUID) (codersdk.CreateWorkspaceRequest, error) { // requires workspaceName (parsed from metadata.name "<org>.<user>.<workspace-name>") // requires templateID // optional TemplateVersionID // optional TTL/autostart } ``` Add unit tests for converters (no network). --- ## 4) Replace hardcoded storage with codersdk-backed storage **Owner:** 2–3 agents (templates vs workspaces can be parallel) ### 4.1 Shared storage base patterns Keep the defensive style and interface assertions. Add new fields to both storages: - `provider coder.ClientProvider` (resolves `*codersdk.Client` from the request namespace / control plane) Update constructors to accept dependencies: - `storage.NewWorkspaceStorage(provider coder.ClientProvider) *WorkspaceStorage` - `storage.NewTemplateStorage(provider coder.ClientProvider) *TemplateStorage` Namespace semantics: - Keep `NamespaceScoped() == true`. - Treat `genericapirequest.NamespaceValue(ctx)` as the **CoderControlPlane namespace**. - For v1, require `namespace != ""` for all operations (including `List`). If the request namespace is empty (cluster-wide list), return a clear `BadRequest` explaining that `-A/--all-namespaces` is not supported yet. - (Optional follow-up) Add cross-namespace list support by listing all `CoderControlPlane` objects cluster-wide and aggregating results. ### 4.2 TemplateStorage: implement CRUD File: `internal/aggregated/storage/template.go` Implement interfaces: - `rest.Getter` - `rest.Lister` - `rest.Creater` - `rest.GracefulDeleter` - (Optional) `rest.Updater` for meta updates using `UpdateTemplateMeta` Concrete mapping (namespaced; namespace == control plane namespace; name format `"<org>.<template-name>"`): - In every method, resolve the backend client using the request namespace: - `ns := genericapirequest.NamespaceValue(ctx)` - `sdk, err := provider.ClientForNamespace(ctx, ns)` - **List** - If `ns == ""`, return `BadRequest` (v1 does not support `-A/--all-namespaces`). - `tpls, err := sdk.Templates(ctx, codersdk.TemplateFilter{})`. - Convert each template to a K8s object: - `metadata.namespace = ns` - `metadata.name = coder.BuildTemplateName(t.OrganizationName, t.Name)` - `spec.organization = t.OrganizationName` - **Get** - Parse `{name}` using `coder.ParseTemplateName` → `(orgName, templateName)`. - Resolve org: `org, err := sdk.OrganizationByName(ctx, orgName)`. - Fetch template: `tpl, err := sdk.TemplateByName(ctx, org.ID, templateName)`. - Return `convert.TemplateToK8s(ns, tpl)`. - **Create** - Parse `obj.Name` → `(orgName, templateName)`. - Validate `obj.Spec.Organization` (if set) matches `orgName` (else set it). - Resolve org: `org, err := sdk.OrganizationByName(ctx, orgName)`. - Build `codersdk.CreateTemplateRequest` from spec + parsed templateName, then `created, err := sdk.CreateTemplate(ctx, org.ID, req)`. - Return `convert.TemplateToK8s(ns, created)`. - **Delete** - Parse `{name}` → `(orgName, templateName)`. - Resolve org; fetch template by name; call `sdk.DeleteTemplate(ctx, tpl.ID)`. - Return `&metav1.Status{Status: "Success"}` (or the deleted object). ### 4.3 WorkspaceStorage: implement CRUD + start/stop via Update File: `internal/aggregated/storage/workspace.go` Implement interfaces: - `rest.Getter` - `rest.Lister` - `rest.Creater` - `rest.Updater` - `rest.GracefulDeleter` Concrete mapping (namespaced; namespace == control plane namespace; name format `"<org>.<user>.<workspace-name>"`): - In every method, resolve the backend client using the request namespace: - `ns := genericapirequest.NamespaceValue(ctx)` - `sdk, err := provider.ClientForNamespace(ctx, ns)` - **List** - If `ns == ""`, return `BadRequest` (v1 does not support `-A/--all-namespaces`). - `wres, err := sdk.Workspaces(ctx, codersdk.WorkspaceFilter{})`. - Convert each workspace to a K8s object: - `metadata.namespace = ns` - `metadata.name = coder.BuildWorkspaceName(w.OrganizationName, w.OwnerName, w.Name)` - `spec.organization = w.OrganizationName` - **Get** - Parse `{name}` using `coder.ParseWorkspaceName` → `(orgName, userName, workspaceName)`. - Fetch workspace: `ws, err := sdk.WorkspaceByOwnerAndName(ctx, userName, workspaceName, codersdk.WorkspaceOptions{})`. - If `ws.OrganizationName != orgName`, return NotFound. - Return `convert.WorkspaceToK8s(ns, ws)`. - **Create** - Parse `obj.Name` → `(orgName, userName, workspaceName)`. - Validate `obj.Spec.Organization` (if set) matches `orgName` (else set it). - Require `obj.Spec.TemplateName`. - Resolve org: `org, err := sdk.OrganizationByName(ctx, orgName)`. - Resolve template in that org: `tpl, err := sdk.TemplateByName(ctx, org.ID, obj.Spec.TemplateName)`. - Build `codersdk.CreateWorkspaceRequest`: - `Name = workspaceName` - `TemplateID = tpl.ID` (or `TemplateVersionID` if provided) - optional TTL/autostart from spec - Call `created, err := sdk.CreateUserWorkspace(ctx, userName, req)`. - If `.spec.running == false`, immediately queue a stop build: - `sdk.CreateWorkspaceBuild(ctx, created.ID, codersdk.CreateWorkspaceBuildRequest{Transition: stop, TemplateVersionID: created.LatestBuild.TemplateVersionID})` - Return `convert.WorkspaceToK8s(ns, created)`. - **Update** (key behavior) - Parse `{name}` → `(orgName, userName, workspaceName)` and fetch current workspace by owner+name. - Decode desired object via `objInfo.UpdatedObject(ctx, oldObj)`. - Forbid changing identity fields (`metadata.name`, `spec.organization`, `spec.templateName`). - If `running` changes, queue build transition start/stop. - If TTL/autostart changed, call `UpdateWorkspaceTTL` / `UpdateWorkspaceAutostart`. - Return the latest workspace representation. - **Delete** - Parse `{name}` → `(orgName, userName, workspaceName)`; fetch workspace; if org mismatch return NotFound; queue build: - `sdk.CreateWorkspaceBuild(ctx, ws.ID, codersdk.CreateWorkspaceBuildRequest{Transition: delete, TemplateVersionID: ws.LatestBuild.TemplateVersionID})` - Return `&metav1.Status{Status: "Success"}` (or the deleted object). ### 4.4 Defensive programming requirements - `assertion failed:` on nil storage, nil ctx, missing required spec fields. - Validate composite name formats (`<org>.<template-name>`, `<org>.<user>.<workspace-name>`) and required spec fields early with clear error messages. ### 4.5 Update existing tests impacted by the refactor - Rewrite or replace `internal/aggregated/storage/storage_test.go` (it currently assumes hardcoded maps and zero-arg constructors). - Preferred: replace with the `coderdtest` integration tests described in §7.2. - Alternative (faster): use an `httptest.Server` that implements only the required `/api/v2/...` endpoints, similar to `internal/coderbootstrap/client_test.go`. - Update `internal/app/apiserverapp/apiserverapp_test.go` to pass a dummy `coder.ClientProvider` into `NewAPIGroupInfo` once its signature changes. --- ## 5) Wire storage dependencies into aggregated-apiserver bootstrap **Owner:** 1 agent ### 5.1 Add apiserver options File: `internal/app/apiserverapp/apiserverapp.go` - Introduce: ```go type Options struct { // SecureServingPort is used when Listener is nil. // Default: DefaultSecureServingPort. SecureServingPort int // Listener is optional but allows tests to bind to 127.0.0.1:0 and then // discover the chosen port for APIService Endpoints. Listener net.Listener // KubeConfig is used to build a Kubernetes client for resolving // coder.com/v1alpha1 CoderControlPlane objects. // If nil, use the standard in-cluster / KUBECONFIG discovery. KubeConfig *rest.Config // Optional selector when multiple CoderControlPlanes exist in a namespace. ControlPlaneName string // Optional fallback when the control plane status URL is not available. DefaultCoderURL string // Coder auth. CoderSessionToken string CoderRequestTimeout time.Duration } func RunWithOptions(ctx context.Context, opts Options) error ``` - `Run(ctx)` can remain as a thin wrapper that reads env vars for backwards compatibility, or we can update callsites to always use `RunWithOptions`. ### 5.2 `NewAPIGroupInfo` must accept storage deps Change signature to accept a `coder.ClientProvider` (or a small dep struct containing it), and update: - `internal/app/apiserverapp/apiserverapp_test.go` Example: ```go func NewAPIGroupInfo(scheme *runtime.Scheme, codecs serializer.CodecFactory, provider coder.ClientProvider) (*genericapiserver.APIGroupInfo, error) { apiGroupInfo.VersionedResourcesStorageMap[...]=map[string]rest.Storage{ "coderworkspaces": storage.NewWorkspaceStorage(provider), "codertemplates": storage.NewTemplateStorage(provider), } } ``` --- ## 6) CLI flag plumbing **Owner:** 1 agent File: `app_dispatch.go` Add flags to the existing `FlagSet` (define them regardless of app mode; only enforce for aggregated-apiserver): - `--coder-session-token` (admin token for the backing coderd) - `--coder-request-timeout` (default `30s`) - `--coder-url` (optional fallback when the namespace’s CoderControlPlane has no `status.url`) - `--coder-control-plane-name` (optional selector when multiple control planes exist in a namespace) Also document/validate the required object name formats: - Templates: `<org>.<template-name>` - Workspaces: `<org>.<user>.<workspace-name>` Update the aggregated-apiserver dispatch to call `apiserverapp.RunWithOptions(ctx, opts)`. Update tests in `main_test.go` if signature changes (the dispatch tests currently monkeypatch `runAggregatedAPIServerApp func(context.Context) error`). --- ## 7) Testing strategy ### 7.1 Unit tests (fast) **Owner:** 1 agent - `internal/aggregated/convert/*_test.go`: - Validate conversion for templates/workspaces. - `internal/aggregated/coder/errors_test.go`: - Validate mapping of `codersdk.NewTestError` → `apierrors.IsNotFound/IsForbidden/...`. ### 7.2 Storage integration tests with `coderdtest` (primary correctness signal) **Owner:** 1–2 agents Add tests in `internal/aggregated/storage/*_integration_test.go` (or similar) that: (Imports you will need once you add these tests: `github.com/coder/coder/v2/coderd/coderdtest` and likely `github.com/coder/coder/v2/provisioner/echo`. Adding these will expand `vendor/` after `make vendor` / `make verify-vendor`.) 1. Starts in-memory coderd: - `client := coderdtest.New(t, &coderdtest.Options{IncludeProvisionerDaemon: true})` - `coderdtest.CreateFirstUser(t, client)` to authenticate `client`. 2. Resolve/ensure an organization to use in `spec.organization`: - `org, err := client.OrganizationByName(ctx, "default")` (or `CreateOrganization` if needed). 3. Creates a template version + template using coderdtest helpers: - `version := coderdtest.CreateTemplateVersion(t, client, orgID, &echo.Responses{...})` - `coderdtest.AwaitTemplateVersionJobCompleted(t, client, version.ID)` - `template := coderdtest.CreateTemplate(t, client, orgID, version.ID)` 4. Instantiate a test `coder.ClientProvider` that returns the authenticated `client` for a chosen control-plane namespace (e.g. `coder-123`). 5. Exercises the REST methods directly: - Use the composite object name formats from “Chosen semantics”: - Template object name: `"<org>.<template-name>"` - Workspace object name: `"<org>.<user>.<workspace-name>"` - Ensure the objects you pass to `Create/Update` include `metadata.namespace = <control-plane-namespace>` and required spec fields (`spec.organization`, `spec.templateName`, etc.). - Template: `Create` → `Get` → `List` → `Delete` - Workspace: `Create` → `Get` → `List` → `Update` (toggle running) → `Delete` Ensure contexts use a namespaced request context, e.g. `genericapirequest.WithNamespace(context.Background(), "coder-123")`. ### 7.3 End-to-end: envtest + APIService proxy (stretch but matches requirement) **Owner:** 1–2 agents Goal: route requests through envtest kube-apiserver’s aggregation layer. Implementation suggestion: - Put this in a dedicated test package (e.g. `internal/app/apiserverapp/e2e_test.go` or `internal/aggregated/e2e/`) so it can spin up its **own** envtest environment without inheriting the controller suite’s CRD installation. - Use `envtest.Environment{}` with `CRDDirectoryPaths` empty (or pointing to a directory that excludes the `aggregation.coder.com_*` CRDs). Steps: 1. Start envtest **without installing** the `aggregation.coder.com` CRDs (otherwise CRDs “own” the group and proxying won’t work). 2. Start `apiserverapp` on a dynamic port (listener `127.0.0.1:0`). 3. In envtest cluster, create: - Namespace `coder-k8s-system` (or similar) - Service `coder-k8s-aggregated-apiserver` in that namespace - Endpoints pointing to `127.0.0.1:<port>` - `apiregistration.k8s.io/v1 APIService` named `v1alpha1.aggregation.coder.com` with: - `spec.service` pointing to that service - `spec.insecureSkipTLSVerify: true` (test-only) 4. Use a dynamic client built from envtest’s `rest.Config` to call: - `POST /apis/aggregation.coder.com/v1alpha1/namespaces/<control-plane-namespace>/coderworkspaces` - request body includes `metadata.name: "<org>.<user>.<workspace-name>"` and `spec.organization: "<org>"` - verify side effects in coderdtest (workspace exists) Implementation detail: gate this test with a build tag or `t.Skip` if the envtest kube-apiserver does not support aggregation in this environment. --- ## 8) Validation / CI readiness **Owner:** 1 agent (final pass) Run, in order: - `make verify-vendor` (especially after adding `coderdtest` test deps) - `make test` - `make build` - `make lint` If API structs changed: - `make codegen` - `make manifests` --- # Agent task breakdown (parallel execution) 1. **Agent A (Types/OpenAPI):** Section 1 + update OpenAPI definitions. 2. **Agent B (Backend helpers):** Section 2 (config/names/errors + control-plane client provider). 3. **Agent C (Conversion):** Section 3 + unit tests. 4. **Agent D (Template storage):** Section 4.2 + integration tests. 5. **Agent E (Workspace storage):** Section 4.3 + integration tests. 6. **Agent F (Bootstrap/flags/RBAC):** Sections 5–6 (and update `deploy/` manifests + RBAC). 7. **Agent G (Envtest proxy E2E):** Section 7.3. 8. **Agent H (Validation + polish):** Section 8. </details> --- _Generated with `mux` • Model: `anthropic:claude-opus-4-6` • Thinking: `xhigh` • Cost: `$0.86`_ <!-- mux-attribution: model=anthropic:claude-opus-4-6 thinking=xhigh costs=0.86 -->
1 parent fb21a66 commit e142bce

33 files changed

Lines changed: 5225 additions & 1139 deletions

.cspell.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
"words": [
55
"Diátaxis",
66
"GOFLAGS",
7+
"Millis",
78
"apiregistration",
89
"apiserverapp",
910
"apiserver",

api/aggregation/v1alpha1/types.go

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,21 +4,42 @@ import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
44

55
// CoderWorkspaceSpec defines the desired state of a CoderWorkspace.
66
type CoderWorkspaceSpec struct {
7-
// Running indicates whether the workspace should be running.
7+
// Organization is the Coder organization name.
8+
Organization string `json:"organization,omitempty"`
9+
10+
// TemplateName resolves via TemplateByName(organization, templateName).
11+
TemplateName string `json:"templateName,omitempty"`
12+
13+
// TemplateVersionID optionally pins to a specific template version.
14+
TemplateVersionID string `json:"templateVersionID,omitempty"`
15+
16+
// Running drives start/stop via CreateWorkspaceBuild.
817
Running bool `json:"running"`
18+
19+
TTLMillis *int64 `json:"ttlMillis,omitempty"`
20+
AutostartSchedule *string `json:"autostartSchedule,omitempty"`
921
}
1022

1123
// CoderWorkspaceStatus defines the observed state of a CoderWorkspace.
1224
type CoderWorkspaceStatus struct {
13-
// AutoShutdown is the next planned shutdown time for the workspace.
25+
ID string `json:"id,omitempty"`
26+
OwnerName string `json:"ownerName,omitempty"`
27+
OrganizationName string `json:"organizationName,omitempty"`
28+
TemplateName string `json:"templateName,omitempty"`
29+
30+
LatestBuildID string `json:"latestBuildID,omitempty"`
31+
LatestBuildStatus string `json:"latestBuildStatus,omitempty"`
32+
1433
AutoShutdown *metav1.Time `json:"autoShutdown,omitempty"`
34+
LastUsedAt *metav1.Time `json:"lastUsedAt,omitempty"`
1535
}
1636

1737
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
1838
// +kubebuilder:object:root=true
1939
// +kubebuilder:subresource:status
2040

2141
// CoderWorkspace is the schema for Coder workspace resources.
42+
// metadata.name is <organization>.<user>.<workspace-name>.
2243
type CoderWorkspace struct {
2344
metav1.TypeMeta `json:",inline"`
2445
metav1.ObjectMeta `json:"metadata,omitempty"`
@@ -39,13 +60,29 @@ type CoderWorkspaceList struct {
3960

4061
// CoderTemplateSpec defines the desired state of a CoderTemplate.
4162
type CoderTemplateSpec struct {
42-
// Running indicates whether the template should be marked as running.
43-
Running bool `json:"running"`
63+
// Organization is the Coder organization name (must match the organization prefix in metadata.name).
64+
Organization string `json:"organization"`
65+
66+
// VersionID is the Coder template version UUID used on creation (required for CREATE).
67+
VersionID string `json:"versionID"`
68+
69+
DisplayName string `json:"displayName,omitempty"`
70+
Description string `json:"description,omitempty"`
71+
Icon string `json:"icon,omitempty"`
72+
73+
// Running is a legacy flag retained temporarily for in-repo callers that still read template run-state directly.
74+
Running bool `json:"running,omitempty"`
4475
}
4576

4677
// CoderTemplateStatus defines the observed state of a CoderTemplate.
4778
type CoderTemplateStatus struct {
48-
// AutoShutdown is the next planned shutdown time for workspaces created by this template.
79+
ID string `json:"id,omitempty"`
80+
OrganizationName string `json:"organizationName,omitempty"`
81+
ActiveVersionID string `json:"activeVersionID,omitempty"`
82+
Deprecated bool `json:"deprecated,omitempty"`
83+
UpdatedAt *metav1.Time `json:"updatedAt,omitempty"`
84+
85+
// AutoShutdown is a legacy timestamp retained temporarily for in-repo callers that still surface template shutdown timestamps.
4986
AutoShutdown *metav1.Time `json:"autoShutdown,omitempty"`
5087
}
5188

@@ -54,6 +91,7 @@ type CoderTemplateStatus struct {
5491
// +kubebuilder:subresource:status
5592

5693
// CoderTemplate is the schema for Coder template resources.
94+
// metadata.name is <organization>.<template-name>.
5795
type CoderTemplate struct {
5896
metav1.TypeMeta `json:",inline"`
5997
metav1.ObjectMeta `json:"metadata,omitempty"`

api/aggregation/v1alpha1/zz_generated.deepcopy.go

Lines changed: 19 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

app_dispatch.go

Lines changed: 64 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,12 @@
11
package main
22

33
import (
4+
"context"
45
"flag"
56
"fmt"
7+
"net/url"
8+
"strings"
9+
"time"
610

711
ctrl "sigs.k8s.io/controller-runtime"
812

@@ -15,24 +19,79 @@ const supportedAppModes = "controller, aggregated-apiserver, mcp-http"
1519

1620
var (
1721
runControllerApp = controllerapp.Run
18-
runAggregatedAPIServerApp = apiserverapp.Run
19-
runMCPHTTPApp = mcpapp.RunHTTP
20-
setupSignalHandler = ctrl.SetupSignalHandler
22+
runAggregatedAPIServerApp = func(ctx context.Context, opts apiserverapp.Options) error {
23+
return apiserverapp.RunWithOptions(ctx, opts)
24+
}
25+
runMCPHTTPApp = mcpapp.RunHTTP
26+
setupSignalHandler = ctrl.SetupSignalHandler
2127
)
2228

2329
func run(args []string) error {
2430
fs := flag.NewFlagSet("coder-k8s", flag.ContinueOnError)
25-
var appMode string
31+
var (
32+
appMode string
33+
coderURL string
34+
coderSessionToken string
35+
coderNamespace string
36+
coderRequestTimeout time.Duration
37+
)
2638
fs.StringVar(&appMode, "app", "", "Application mode (controller, aggregated-apiserver, mcp-http)")
39+
fs.StringVar(
40+
&coderSessionToken,
41+
"coder-session-token",
42+
"",
43+
"Admin session token for the backing Coder deployment",
44+
)
45+
fs.StringVar(
46+
&coderURL,
47+
"coder-url",
48+
"",
49+
"Coder deployment URL (fallback when CoderControlPlane status URL is unavailable)",
50+
)
51+
fs.StringVar(
52+
&coderNamespace,
53+
"coder-namespace",
54+
"",
55+
"Restrict the aggregated API server to serve only this Kubernetes namespace",
56+
)
57+
fs.DurationVar(
58+
&coderRequestTimeout,
59+
"coder-request-timeout",
60+
30*time.Second,
61+
"Timeout for Coder SDK API requests",
62+
)
2763
if err := fs.Parse(args); err != nil {
2864
return err
2965
}
3066

67+
if coderURL != "" {
68+
parsedCoderURL, err := url.Parse(coderURL)
69+
if err != nil {
70+
return fmt.Errorf("assertion failed: invalid --coder-url %q: %w", coderURL, err)
71+
}
72+
if parsedCoderURL.Scheme == "" || parsedCoderURL.Host == "" {
73+
return fmt.Errorf(
74+
"assertion failed: invalid --coder-url %q: must include scheme and host (for example, https://coder.example.com)",
75+
coderURL,
76+
)
77+
}
78+
scheme := strings.ToLower(parsedCoderURL.Scheme)
79+
if scheme != "http" && scheme != "https" {
80+
return fmt.Errorf("assertion failed: invalid --coder-url %q: scheme must be http or https", coderURL)
81+
}
82+
}
83+
3184
switch appMode {
3285
case "controller":
3386
return runControllerApp(setupSignalHandler())
3487
case "aggregated-apiserver":
35-
return runAggregatedAPIServerApp(setupSignalHandler())
88+
opts := apiserverapp.Options{
89+
CoderURL: coderURL,
90+
CoderSessionToken: coderSessionToken,
91+
CoderNamespace: coderNamespace,
92+
CoderRequestTimeout: coderRequestTimeout,
93+
}
94+
return runAggregatedAPIServerApp(setupSignalHandler(), opts)
3695
case "mcp-http":
3796
return runMCPHTTPApp(setupSignalHandler())
3897
case "":

config/crd/bases/aggregation.coder.com_codertemplates.yaml

Lines changed: 34 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@ spec:
1717
- name: v1alpha1
1818
schema:
1919
openAPIV3Schema:
20-
description: CoderTemplate is the schema for Coder template resources.
20+
description: |-
21+
CoderTemplate is the schema for Coder template resources.
22+
metadata.name is <organization>.<template-name>.
2123
properties:
2224
apiVersion:
2325
description: |-
@@ -39,19 +41,45 @@ spec:
3941
spec:
4042
description: CoderTemplateSpec defines the desired state of a CoderTemplate.
4143
properties:
44+
description:
45+
type: string
46+
displayName:
47+
type: string
48+
icon:
49+
type: string
50+
organization:
51+
description: Organization is the Coder organization name (must match
52+
the organization prefix in metadata.name).
53+
type: string
4254
running:
43-
description: Running indicates whether the template should be marked
44-
as running.
55+
description: Running is a legacy flag retained temporarily for in-repo
56+
callers that still read template run-state directly.
4557
type: boolean
58+
versionID:
59+
description: VersionID is the Coder template version UUID used on
60+
creation (required for CREATE).
61+
type: string
4662
required:
47-
- running
63+
- organization
64+
- versionID
4865
type: object
4966
status:
5067
description: CoderTemplateStatus defines the observed state of a CoderTemplate.
5168
properties:
69+
activeVersionID:
70+
type: string
5271
autoShutdown:
53-
description: AutoShutdown is the next planned shutdown time for workspaces
54-
created by this template.
72+
description: AutoShutdown is a legacy timestamp retained temporarily
73+
for in-repo callers that still surface template shutdown timestamps.
74+
format: date-time
75+
type: string
76+
deprecated:
77+
type: boolean
78+
id:
79+
type: string
80+
organizationName:
81+
type: string
82+
updatedAt:
5583
format: date-time
5684
type: string
5785
type: object

config/crd/bases/aggregation.coder.com_coderworkspaces.yaml

Lines changed: 35 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,9 @@ spec:
1717
- name: v1alpha1
1818
schema:
1919
openAPIV3Schema:
20-
description: CoderWorkspace is the schema for Coder workspace resources.
20+
description: |-
21+
CoderWorkspace is the schema for Coder workspace resources.
22+
metadata.name is <organization>.<user>.<workspace-name>.
2123
properties:
2224
apiVersion:
2325
description: |-
@@ -39,20 +41,49 @@ spec:
3941
spec:
4042
description: CoderWorkspaceSpec defines the desired state of a CoderWorkspace.
4143
properties:
44+
autostartSchedule:
45+
type: string
46+
organization:
47+
description: Organization is the Coder organization name.
48+
type: string
4249
running:
43-
description: Running indicates whether the workspace should be running.
50+
description: Running drives start/stop via CreateWorkspaceBuild.
4451
type: boolean
52+
templateName:
53+
description: TemplateName resolves via TemplateByName(organization,
54+
templateName).
55+
type: string
56+
templateVersionID:
57+
description: TemplateVersionID optionally pins to a specific template
58+
version.
59+
type: string
60+
ttlMillis:
61+
format: int64
62+
type: integer
4563
required:
4664
- running
4765
type: object
4866
status:
4967
description: CoderWorkspaceStatus defines the observed state of a CoderWorkspace.
5068
properties:
5169
autoShutdown:
52-
description: AutoShutdown is the next planned shutdown time for the
53-
workspace.
5470
format: date-time
5571
type: string
72+
id:
73+
type: string
74+
lastUsedAt:
75+
format: date-time
76+
type: string
77+
latestBuildID:
78+
type: string
79+
latestBuildStatus:
80+
type: string
81+
organizationName:
82+
type: string
83+
ownerName:
84+
type: string
85+
templateName:
86+
type: string
5687
type: object
5788
type: object
5889
served: true

docs/reference/api/codertemplate.md

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,17 +13,25 @@
1313

1414
| Field | Type | Description |
1515
| --- | --- | --- |
16-
| `spec.running` | `bool` | Running indicates whether the template should be marked as running. |
16+
| `spec.organization` | `string` | Organization is the Coder organization name (must match the organization prefix in metadata.name). |
17+
| `spec.versionID` | `string` | VersionID is the Coder template version UUID used on creation (required for CREATE). |
18+
| `spec.displayName` | `string` | |
19+
| `spec.description` | `string` | |
20+
| `spec.icon` | `string` | |
21+
| `spec.running` | `bool` | Running is a legacy flag retained temporarily for in-repo callers that still read template run-state directly. |
1722

1823
## Status
1924

2025
| Field | Type | Description |
2126
| --- | --- | --- |
22-
| `status.autoShutdown` | `metav1.Time` | AutoShutdown is the next planned shutdown time for workspaces created by this template. |
27+
| `status.id` | `string` | |
28+
| `status.organizationName` | `string` | |
29+
| `status.activeVersionID` | `string` | |
30+
| `status.deprecated` | `bool` | |
31+
| `status.updatedAt` | `metav1.Time` | |
32+
| `status.autoShutdown` | `metav1.Time` | AutoShutdown is a legacy timestamp retained temporarily for in-repo callers that still surface template shutdown timestamps. |
2333

2434
## Source
2535

2636
- Go type: `api/aggregation/v1alpha1/types.go`
27-
- Storage implementation: `internal/aggregated/storage/template.go`
28-
2937
- APIService registration manifest: `deploy/apiserver-apiservice.yaml`

0 commit comments

Comments
 (0)