🤖 ci: replace e2e smoke test with CNPG + Templates merge-queue E2E (#76)

## Summary

Replace the existing lightweight Kind smoke test (`e2e-kind`) with a
full end-to-end test that provisions CloudNativePG, a real
`CoderControlPlane`, the aggregated API server, and validates
`CoderTemplate` creation via `kubectl`.

## Background

The previous `e2e-kind` job only verified that a `CoderControlPlane` CR
could be created (no real Postgres, no Coder deployment, no aggregated
API). This left significant integration gaps uncovered by CI.

This PR wires up the full stack in the merge queue (`merge_group` event)
while keeping the job as a fast no-op on PRs so the required check stays
green without wasting resources.

## Implementation

### `config/e2e/deployment.yaml`
- Changed `--app=controller` → `--app=all` to run both controller and
aggregated API server in the same pod.
- Added HTTPS port 6443 (used by the `APIService` to reach the
aggregated API server).

### `config/e2e/codertemplate.yaml` (new)
- Minimal `CoderTemplate` manifest for the E2E test to apply via
`kubectl`.
- Uses the `default.<name>` naming convention with a trivial Terraform
`null_resource`.

### `.github/workflows/ci.yaml` (`e2e-kind` job)
- Job name updated to `E2E (Kind + CNPG + Templates)` (id `e2e-kind`
unchanged for branch protection).
- PR runs: instant skip with `::notice::` message.
- Merge-queue runs: full E2E flow:
  1. Kind cluster + build + load image
  2. Apply CRDs, RBAC, APIService, and deployment
  3. Wait for operator + APIService availability
  4. Install CloudNativePG 1.25.0, provision PostgreSQL
5. Apply CloudNativePG example manifests (CoderControlPlane + Postgres)
  6. Wait for Coder readiness + operator access bootstrap
  7. Apply `CoderTemplate` and verify it exists via `kubectl`
- Failure diagnostics step dumps pods, logs, and resource state on
failure.

## Validation
- YAML validity: all 3 modified/new files parse cleanly
- `actionlint`: no errors in CI workflow
- Spot-checked: job id, PR skip gate, merge_group gates, failure
diagnostics, publish-main dependency all correct

## Risks
- **CloudNativePG download**: fetches CNPG manifest from GitHub raw at a
pinned URL; could fail if GitHub is down or the URL changes. Mitigation:
pinned to `release-1.25/releases/cnpg-1.25.0.yaml`.
- **Timeout tuning**: merge-queue E2E has generous timeouts (10min for
Postgres/Coder). May need adjustment based on runner performance.

---

<details>
<summary>📋 Implementation Plan</summary>

# Plan: Merge-queue E2E (Kind + CloudNativePG + CoderTemplates)

## Context / Why
We want an end-to-end test that can run in GitHub’s merge queue
(`merge_group`) to ensure:

1. `coder-k8s` (controller) can reconcile a **real** `CoderControlPlane`
backed by PostgreSQL.
2. The in-cluster **aggregated API server** is available.
3. We can use **`kubectl`** to create and retrieve **`CoderTemplate`**
resources (served by the aggregated API server and backed by the Coder
API).

This replaces the current Kind “smoke” test that only verifies a
`CoderControlPlane` CR can be created.

## Evidence (repo facts)
- Existing CI + current Kind smoke job: `.github/workflows/ci.yaml` (job
`e2e-kind`).
- Current in-cluster e2e Deployment runs controller-only:
`config/e2e/deployment.yaml`.
- Aggregated API server can be enabled in Kind using
`APIService.insecureSkipTLSVerify: true`:
  - `deploy/apiserver-service.yaml`
  - `deploy/apiserver-apiservice.yaml`
- CloudNativePG example manifests already exist and are suitable to
reuse:
  - `examples/cloudnativepg/00-namespace.yaml`
  - `examples/cloudnativepg/cnpg-cluster.yaml`
  - `examples/cloudnativepg/codercontrolplane.yaml`
- Aggregated API server selects a `CoderControlPlane` in the request
namespace and reads operator URL/token from status+Secret (Explore
report: “CoderTemplate Aggregated API Usage Guide”).
- `CoderTemplate` create path can upload `spec.files` as a zip, create a
template version, then create the template (no synchronous wait in
codersdk): `internal/aggregated/storage/template.go`.

## Implementation details

### 1) Enable aggregated API server in the e2e Deployment
**File:** `config/e2e/deployment.yaml`

Change e2e deployment from controller-only to `--app=all` and expose the
HTTPS port used by the APIService.

Proposed diff shape:

```yaml
containers:
  - name: manager
    image: ghcr.io/coder/coder-k8s:e2e
    args: ["--app=all"] # was --app=controller
    ports:
      - containerPort: 8081
        name: health
      - containerPort: 6443
        name: https
```

Rationale: the `deploy/apiserver-service.yaml` targets port **6443** on
pods labeled `app: coder-k8s`.

---

### 2) Add an E2E `CoderTemplate` manifest
**New file:** `config/e2e/codertemplate.yaml`

Create a minimal template in the same namespace as the E2E
`CoderControlPlane` (the existing CNPG example uses `namespace: coder`).

```yaml
apiVersion: aggregation.coder.com/v1alpha1
kind: CoderTemplate
metadata:
  name: default.e2e-template
  namespace: coder
spec:
  organization: default
  displayName: "E2E Template"
  description: "Created by coder-k8s merge-queue e2e"
  files:
    main.tf: |
      terraform {
        required_version = ">= 1.0"
      }
      resource "null_resource" "example" {}
```

Notes:
- `metadata.name` **must** be `<org>.<template-name>` and
`spec.organization` must match the `<org>` prefix (enforced
server-side).
- Keeping the template minimal avoids turning this into a
workspace/provisioner E2E.

---

### 3) Replace the CI Kind smoke job with full CNPG + templates E2E
**File:** `.github/workflows/ci.yaml`

#### 3.1 Keep the job identity stable
Keep the job id `e2e-kind` (and ideally keep the `name:` similar) so
that existing branch protection / required checks don’t unexpectedly
change.

#### 3.2 Runtime control: run *full* E2E in merge queue only
Full CNPG + Coder bring-up will be materially slower than the current
smoke test.

Recommended approach:
- Keep the `e2e-kind` job **running** on PRs (so the check exists), but
make it a fast no-op.
- Run the full E2E only when `github.event_name == 'merge_group'`.

This satisfies “only run it in the merge queue” while still allowing the
check to be configured as **required**.

Implementation pattern:

```yaml
  e2e-kind:
    name: E2E (Kind + CNPG + Templates)
    ...
    steps:
      - name: Skip full E2E outside merge queue
        if: github.event_name != 'merge_group'
        run: |
          echo "Skipping full E2E on ${GITHUB_EVENT_NAME}; it runs in merge queue (merge_group)."

      - name: Checkout
        if: github.event_name == 'merge_group'
        uses: actions/checkout@...

      # all heavy steps also gated on merge_group
```

If you *do* want some signal on PRs, a compromise is to keep the old
smoke steps under `pull_request` and the full E2E under `merge_group`.

#### 3.3 Full merge-queue E2E steps (merge_group only)
Replace the existing “apply sample CR and verify it exists” with:

1. Create Kind cluster (existing)
2. Build + load `ghcr.io/coder/coder-k8s:e2e` image (existing)
3. Apply base manifests (existing):
   - `kubectl apply -f config/e2e/namespace.yaml`
   - `kubectl apply -f config/crd/bases/`
   - `kubectl apply -f config/rbac/`
4. Enable aggregated API (new):
   - `kubectl apply -f deploy/apiserver-service.yaml`
   - `kubectl apply -f deploy/apiserver-apiservice.yaml`
5. Deploy `coder-k8s` (updated `config/e2e/deployment.yaml`)
6. Wait for operator:
- `kubectl wait --for=condition=Available deploy/coder-k8s -n
coder-system --timeout=120s`
- `kubectl wait --for=condition=Available
apiservice/v1alpha1.aggregation.coder.com --timeout=180s`
7. Install CloudNativePG operator (pinned version) + wait:
- `kubectl apply --server-side -f
https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.25/releases/cnpg-1.25.0.yaml`
- `kubectl wait --for=condition=Available deploy/cnpg-controller-manager
-n cnpg-system --timeout=180s`
8. Provision Postgres + CoderControlPlane using existing repo examples:
   - `kubectl apply -f examples/cloudnativepg/`
- `kubectl -n coder wait --for=condition=Ready cluster/coder-db
--timeout=10m`
9. Wait for Coder to come up and for operator access bootstrap:
   - `kubectl -n coder rollout status deployment/coder --timeout=10m`
- `kubectl -n coder wait --for=jsonpath='{.status.phase}'=Ready
codercontrolplane/coder --timeout=10m`
- `kubectl -n coder wait
--for=jsonpath='{.status.operatorAccessReady}'=true
codercontrolplane/coder --timeout=10m`
   - Verify the token Secret exists:
     ```bash
TOKEN_SECRET=$(kubectl -n coder get codercontrolplane coder -o
jsonpath='{.status.operatorTokenSecretRef.name}')
     test -n "$TOKEN_SECRET"
     kubectl -n coder get secret "$TOKEN_SECRET"
     ```
10. Create + retrieve a CoderTemplate via kubectl (new):
    - `kubectl apply -f config/e2e/codertemplate.yaml`
    - `kubectl -n coder get codertemplates`
    - `kubectl -n coder get codertemplate default.e2e-template -o yaml`

#### 3.4 Add failure diagnostics (high value for flake triage)
Add `if: failure()` steps to dump state:
- `kubectl get pods -A`
- `kubectl describe apiservice v1alpha1.aggregation.coder.com`
- `kubectl -n coder-system logs deploy/coder-k8s --tail=200`
- `kubectl -n coder describe codercontrolplane coder`
- `kubectl -n coder logs deploy/coder --tail=200` (best-effort)
- `kubectl -n cnpg-system logs deploy/cnpg-controller-manager
--tail=200`

---

### 4) Make it required for merge queue completion
This is a **repo settings** change (not a code change): mark the check
run name (e.g., `E2E (Kind + CNPG + Templates)`) as a **required status
check** on the protected branch that uses merge queue.

Keep the job id/name stable to avoid having to reconfigure required
checks.

---

## Validation plan (when implementing)
- Manually run the new job steps locally once (optional) using
`hack/kind-dev.sh` + `kubectl apply` commands.
- In CI, confirm on a test PR:
- PR event: `e2e-kind` completes quickly (skipped heavy steps) and is
green.
- Merge queue (`merge_group`): full E2E runs, creates
`CoderControlPlane` + `CoderTemplate`, and succeeds.

</details>

---
_Generated with `mux` • Model: `anthropic:claude-opus-4-6` • Thinking:
`xhigh` • Cost: `$1.34`_

<!-- mux-attribution: model=anthropic:claude-opus-4-6 thinking=xhigh
costs=1.34 -->

Author: ThomasK33

.github/workflows/ci.yaml

@@ -155,29 +155,40 @@ jobs:
         run: go build ./...
 
   e2e-kind:
-    name: E2E Smoke (Kind)
+    name: E2E (Kind + CNPG + Templates)
     needs: [changes, test]
     if: needs.changes.outputs.go == 'true' && (github.event_name != 'push' || github.actor != 'github-merge-queue[bot]')
     runs-on: depot-ubuntu-24.04-8
     timeout-minutes: 45
     steps:
+      # ---- PR fast-path: skip heavy E2E on pull requests ----
+      - name: Skip full E2E on pull requests
+        if: github.event_name == 'pull_request'
+        run: |
+          echo "::notice::Skipping full E2E on ${GITHUB_EVENT_NAME}; runs in merge queue and on push to main."
+
+      # ---- Full CNPG + Templates E2E (merge queue + push) ----
       - name: Checkout
+        if: github.event_name != 'pull_request'
         uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
         with:
           persist-credentials: false
 
       - name: Setup Go
+        if: github.event_name != 'pull_request'
         uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
         with:
           go-version-file: go.mod
           cache: true
 
       - name: Create Kind cluster
+        if: github.event_name != 'pull_request'
         uses: helm/kind-action@a1b0e391336a6ee6713a0583f8c6240d70863de3 # v1.12.0
         with:
           cluster_name: e2e
 
       - name: Build binary
+        if: github.event_name != 'pull_request'
         env:
           GOFLAGS: -mod=vendor
           CGO_ENABLED: "0"
@@ -186,33 +197,96 @@ jobs:
         run: go build -o coder-k8s ./
 
       - name: Build and load image
+        if: github.event_name != 'pull_request'
         run: |
           docker build -f Dockerfile.goreleaser -t ghcr.io/coder/coder-k8s:e2e .
           kind load docker-image ghcr.io/coder/coder-k8s:e2e --name e2e
 
       - name: Apply namespace, CRDs, and RBAC
+        if: github.event_name != 'pull_request'
         run: |
           kubectl apply -f config/e2e/namespace.yaml
           kubectl apply -f config/crd/bases/
           kubectl apply -f config/rbac/
 
-      - name: Deploy controller
+      - name: Enable aggregated API server
+        if: github.event_name != 'pull_request'
+        run: |
+          kubectl apply -f deploy/apiserver-service.yaml
+          kubectl apply -f deploy/apiserver-apiservice.yaml
+
+      - name: Deploy coder-k8s (controller + aggregated API server)
+        if: github.event_name != 'pull_request'
         run: kubectl apply -f config/e2e/deployment.yaml
 
-      - name: Wait for controller
-        run: kubectl wait --for=condition=Available deploy/coder-k8s -n coder-system --timeout=120s
+      - name: Wait for operator and APIService
+        if: github.event_name != 'pull_request'
+        run: |
+          kubectl wait --for=condition=Available deploy/coder-k8s -n coder-system --timeout=120s
+          kubectl wait --for=condition=Available apiservice/v1alpha1.aggregation.coder.com --timeout=180s
+
+      - name: Install CloudNativePG operator
+        if: github.event_name != 'pull_request'
+        run: |
+          kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/v1.25.0/releases/cnpg-1.25.0.yaml
+          kubectl wait --for=condition=Available deploy/cnpg-controller-manager -n cnpg-system --timeout=180s
+
+      - name: Provision PostgreSQL and CoderControlPlane
+        if: github.event_name != 'pull_request'
+        run: kubectl apply -f examples/cloudnativepg/
 
-      - name: Apply sample CR
-        run: kubectl apply -f config/samples/coder_v1alpha1_codercontrolplane.yaml
+      - name: Wait for PostgreSQL
+        if: github.event_name != 'pull_request'
+        run: kubectl -n coder wait --for=condition=Ready cluster/coder-db --timeout=600s
+
+      - name: Wait for Coder deployment
+        if: github.event_name != 'pull_request'
+        run: kubectl -n coder rollout status deployment/coder --timeout=600s
+
+      - name: Wait for CoderControlPlane readiness
+        if: github.event_name != 'pull_request'
+        run: |
+          kubectl -n coder wait --for=jsonpath='{.status.phase}'=Ready codercontrolplane/coder --timeout=600s
+          kubectl -n coder wait --for=jsonpath='{.status.operatorAccessReady}'=true codercontrolplane/coder --timeout=600s
+
+      - name: Verify operator token secret
+        if: github.event_name != 'pull_request'
+        run: |
+          TOKEN_SECRET=$(kubectl -n coder get codercontrolplane coder -o jsonpath='{.status.operatorTokenSecretRef.name}')
+          test -n "$TOKEN_SECRET"
+          kubectl -n coder get secret "$TOKEN_SECRET"
+
+      - name: Create CoderTemplate via kubectl
+        if: github.event_name != 'pull_request'
+        run: kubectl apply -f config/e2e/codertemplate.yaml
+
+      - name: Verify CoderTemplate exists
+        if: github.event_name != 'pull_request'
+        run: |
+          kubectl -n coder get codertemplates
+          kubectl -n coder get codertemplate default.e2e-template -o yaml
 
-      - name: Verify CR exists
+      # ---- Failure diagnostics ----
+      - name: Dump cluster state on failure
+        if: failure() && github.event_name != 'pull_request'
         run: |
-          kubectl get codercontrolplanes -A
-          COUNT=$(kubectl get codercontrolplanes -A -o json | jq '.items | length')
-          if [ "$COUNT" -lt 1 ]; then
-            echo "assertion failed: expected at least 1 CoderControlPlane resource" >&2
-            exit 1
-          fi
+          echo "=== Pods (all namespaces) ==="
+          kubectl get pods -A || true
+          echo ""
+          echo "=== APIService ==="
+          kubectl describe apiservice v1alpha1.aggregation.coder.com || true
+          echo ""
+          echo "=== coder-k8s logs ==="
+          kubectl -n coder-system logs deploy/coder-k8s --tail=200 || true
+          echo ""
+          echo "=== CoderControlPlane ==="
+          kubectl -n coder describe codercontrolplane coder || true
+          echo ""
+          echo "=== Coder deployment logs ==="
+          kubectl -n coder logs deploy/coder --tail=200 || true
+          echo ""
+          echo "=== CNPG controller logs ==="
+          kubectl -n cnpg-system logs deploy/cnpg-controller-manager --tail=200 || true
 
   terraform:
     name: Terraform (fmt/validate/tflint/trivy)

config/e2e/codertemplate.yaml

@@ -0,0 +1,15 @@
+apiVersion: aggregation.coder.com/v1alpha1
+kind: CoderTemplate
+metadata:
+  name: default.e2e-template
+  namespace: coder
+spec:
+  organization: default
+  displayName: "E2E Template"
+  description: "Created by coder-k8s merge-queue e2e"
+  files:
+    main.tf: |
+      terraform {
+        required_version = ">= 1.0"
+      }
+      resource "null_resource" "example" {}

config/e2e/deployment.yaml

@@ -19,13 +19,15 @@ spec:
       containers:
         - name: manager
           image: ghcr.io/coder/coder-k8s:e2e
-          # E2E tests validate controller mode only; all mode requires
-          # additional infrastructure (APIService, TLS) not provisioned here.
-          args: ["--app=controller"]
+          # E2E tests validate both controller and aggregated API server
+          # behavior in one pod for APIService-backed integration coverage.
+          args: ["--app=all"]
           imagePullPolicy: Never
           ports:
             - containerPort: 8081
               name: health
+            - containerPort: 6443
+              name: https
           livenessProbe:
             httpGet:
               path: /healthz