🤖 ci: replace e2e smoke test with CNPG + Templates merge-queue E2E

[ThomasK33]

Summary

Replace the existing lightweight Kind smoke test (e2e-kind) with a full end-to-end test that provisions CloudNativePG, a real CoderControlPlane, the aggregated API server, and validates CoderTemplate creation via kubectl.

Background

The previous e2e-kind job only verified that a CoderControlPlane CR could be created (no real Postgres, no Coder deployment, no aggregated API). This left significant integration gaps uncovered by CI.

This PR wires up the full stack in the merge queue (merge_group event) while keeping the job as a fast no-op on PRs so the required check stays green without wasting resources.

Implementation

config/e2e/deployment.yaml

• Changed --app=controller → --app=all to run both controller and aggregated API server in the same pod.
• Added HTTPS port 6443 (used by the APIService to reach the aggregated API server).

config/e2e/codertemplate.yaml (new)

• Minimal CoderTemplate manifest for the E2E test to apply via kubectl.
• Uses the default.<name> naming convention with a trivial Terraform null_resource.

.github/workflows/ci.yaml (e2e-kind job)

• Job name updated to E2E (Kind + CNPG + Templates) (id e2e-kind unchanged for branch protection).
• PR runs: instant skip with ::notice:: message.
• Merge-queue runs: full E2E flow:
  1. Kind cluster + build + load image
  2. Apply CRDs, RBAC, APIService, and deployment
  3. Wait for operator + APIService availability
  4. Install CloudNativePG 1.25.0, provision PostgreSQL
  5. Apply CloudNativePG example manifests (CoderControlPlane + Postgres)
  6. Wait for Coder readiness + operator access bootstrap
  7. Apply CoderTemplate and verify it exists via kubectl
• Failure diagnostics step dumps pods, logs, and resource state on failure.

Validation

• YAML validity: all 3 modified/new files parse cleanly
• actionlint: no errors in CI workflow
• Spot-checked: job id, PR skip gate, merge_group gates, failure diagnostics, publish-main dependency all correct

Risks

• CloudNativePG download: fetches CNPG manifest from GitHub raw at a pinned URL; could fail if GitHub is down or the URL changes. Mitigation: pinned to release-1.25/releases/cnpg-1.25.0.yaml.
• Timeout tuning: merge-queue E2E has generous timeouts (10min for Postgres/Coder). May need adjustment based on runner performance.

---

📋 Implementation Plan

Plan: Merge-queue E2E (Kind + CloudNativePG + CoderTemplates)

Context / Why

We want an end-to-end test that can run in GitHub’s merge queue (merge_group) to ensure:

1. coder-k8s (controller) can reconcile a real CoderControlPlane backed by PostgreSQL.
2. The in-cluster aggregated API server is available.
3. We can use kubectl to create and retrieve CoderTemplate resources (served by the aggregated API server and backed by the Coder API).

This replaces the current Kind “smoke” test that only verifies a CoderControlPlane CR can be created.

Evidence (repo facts)

• Existing CI + current Kind smoke job: .github/workflows/ci.yaml (job e2e-kind).
• Current in-cluster e2e Deployment runs controller-only: config/e2e/deployment.yaml.
• Aggregated API server can be enabled in Kind using APIService.insecureSkipTLSVerify: true:
  • deploy/apiserver-service.yaml
  • deploy/apiserver-apiservice.yaml
• CloudNativePG example manifests already exist and are suitable to reuse:
  • examples/cloudnativepg/00-namespace.yaml
  • examples/cloudnativepg/cnpg-cluster.yaml
  • examples/cloudnativepg/codercontrolplane.yaml
• Aggregated API server selects a CoderControlPlane in the request namespace and reads operator URL/token from status+Secret (Explore report: “CoderTemplate Aggregated API Usage Guide”).
• CoderTemplate create path can upload spec.files as a zip, create a template version, then create the template (no synchronous wait in codersdk): internal/aggregated/storage/template.go.

Implementation details

1) Enable aggregated API server in the e2e Deployment

File: config/e2e/deployment.yaml

Change e2e deployment from controller-only to --app=all and expose the HTTPS port used by the APIService.

Proposed diff shape:

containers:
  - name: manager
    image: ghcr.io/coder/coder-k8s:e2e
    args: ["--app=all"] # was --app=controller
    ports:
      - containerPort: 8081
        name: health
      - containerPort: 6443
        name: https

Rationale: the deploy/apiserver-service.yaml targets port 6443 on pods labeled app: coder-k8s.

---

2) Add an E2E CoderTemplate manifest

New file: config/e2e/codertemplate.yaml

Create a minimal template in the same namespace as the E2E CoderControlPlane (the existing CNPG example uses namespace: coder).

apiVersion: aggregation.coder.com/v1alpha1
kind: CoderTemplate
metadata:
  name: default.e2e-template
  namespace: coder
spec:
  organization: default
  displayName: "E2E Template"
  description: "Created by coder-k8s merge-queue e2e"
  files:
    main.tf: |
      terraform {
        required_version = ">= 1.0"
      }
      resource "null_resource" "example" {}

Notes:

• metadata.name must be <org>.<template-name> and spec.organization must match the <org> prefix (enforced server-side).
• Keeping the template minimal avoids turning this into a workspace/provisioner E2E.

---

3) Replace the CI Kind smoke job with full CNPG + templates E2E

File: .github/workflows/ci.yaml

3.1 Keep the job identity stable

Keep the job id e2e-kind (and ideally keep the name: similar) so that existing branch protection / required checks don’t unexpectedly change.

3.2 Runtime control: run full E2E in merge queue only

Full CNPG + Coder bring-up will be materially slower than the current smoke test.

Recommended approach:

• Keep the e2e-kind job running on PRs (so the check exists), but make it a fast no-op.
• Run the full E2E only when github.event_name == 'merge_group'.

This satisfies “only run it in the merge queue” while still allowing the check to be configured as required.

Implementation pattern:

  e2e-kind:
    name: E2E (Kind + CNPG + Templates)
    ...
    steps:
      - name: Skip full E2E outside merge queue
        if: github.event_name != 'merge_group'
        run: |
          echo "Skipping full E2E on ${GITHUB_EVENT_NAME}; it runs in merge queue (merge_group)."

      - name: Checkout
        if: github.event_name == 'merge_group'
        uses: actions/checkout@...

      # all heavy steps also gated on merge_group

If you do want some signal on PRs, a compromise is to keep the old smoke steps under pull_request and the full E2E under merge_group.

3.3 Full merge-queue E2E steps (merge_group only)

Replace the existing “apply sample CR and verify it exists” with:

1. Create Kind cluster (existing)
2. Build + load ghcr.io/coder/coder-k8s:e2e image (existing)
3. Apply base manifests (existing):
   • kubectl apply -f config/e2e/namespace.yaml
   • kubectl apply -f config/crd/bases/
   • kubectl apply -f config/rbac/
4. Enable aggregated API (new):
   • kubectl apply -f deploy/apiserver-service.yaml
   • kubectl apply -f deploy/apiserver-apiservice.yaml
5. Deploy coder-k8s (updated config/e2e/deployment.yaml)
6. Wait for operator:
   • kubectl wait --for=condition=Available deploy/coder-k8s -n coder-system --timeout=120s
   • kubectl wait --for=condition=Available apiservice/v1alpha1.aggregation.coder.com --timeout=180s
7. Install CloudNativePG operator (pinned version) + wait:
   • kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.25/releases/cnpg-1.25.0.yaml
   • kubectl wait --for=condition=Available deploy/cnpg-controller-manager -n cnpg-system --timeout=180s
8. Provision Postgres + CoderControlPlane using existing repo examples:
   • kubectl apply -f examples/cloudnativepg/
   • kubectl -n coder wait --for=condition=Ready cluster/coder-db --timeout=10m
9. Wait for Coder to come up and for operator access bootstrap:
   • kubectl -n coder rollout status deployment/coder --timeout=10m
   • kubectl -n coder wait --for=jsonpath='{.status.phase}'=Ready codercontrolplane/coder --timeout=10m
   • kubectl -n coder wait --for=jsonpath='{.status.operatorAccessReady}'=true codercontrolplane/coder --timeout=10m
   • Verify the token Secret exists:

     TOKEN_SECRET=$(kubectl -n coder get codercontrolplane coder -o jsonpath='{.status.operatorTokenSecretRef.name}')
     test -n "$TOKEN_SECRET"
     kubectl -n coder get secret "$TOKEN_SECRET"

10. Create + retrieve a CoderTemplate via kubectl (new):
    • kubectl apply -f config/e2e/codertemplate.yaml
    • kubectl -n coder get codertemplates
    • kubectl -n coder get codertemplate default.e2e-template -o yaml

3.4 Add failure diagnostics (high value for flake triage)

Add if: failure() steps to dump state:

• kubectl get pods -A
• kubectl describe apiservice v1alpha1.aggregation.coder.com
• kubectl -n coder-system logs deploy/coder-k8s --tail=200
• kubectl -n coder describe codercontrolplane coder
• kubectl -n coder logs deploy/coder --tail=200 (best-effort)
• kubectl -n cnpg-system logs deploy/cnpg-controller-manager --tail=200

---

4) Make it required for merge queue completion

This is a repo settings change (not a code change): mark the check run name (e.g., E2E (Kind + CNPG + Templates)) as a required status check on the protected branch that uses merge queue.

Keep the job id/name stable to avoid having to reconfigure required checks.

---

Validation plan (when implementing)

• Manually run the new job steps locally once (optional) using hack/kind-dev.sh + kubectl apply commands.
• In CI, confirm on a test PR:
  • PR event: e2e-kind completes quickly (skipped heavy steps) and is green.
  • Merge queue (merge_group): full E2E runs, creates CoderControlPlane + CoderTemplate, and succeeds.

---

Generated with mux • Model: anthropic:claude-opus-4-6 • Thinking: xhigh • Cost: $1.34

[ThomasK33]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 917309104d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed the feedback: narrowed the E2E skip gate from github.event_name != 'merge_group' to github.event_name == 'pull_request'. Now the full E2E runs on both merge queue and direct pushes to main.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 597d7398e3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed both review comments:

1. Narrowed E2E skip gate to pull_request only (full E2E runs on merge queue + push to main)
2. Pinned CNPG manifest URL to immutable v1.25.0 git tag instead of release-1.25 branch ref

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

https://mux.md/mwr3S#il8bS7awa2b9zg