test(e2e): setup initial e2e testing by Cali0707 · Pull Request #1228 · containers/kubernetes-mcp-server

Cali0707 · 2026-06-24T16:13:09Z

Currently this repo relies heavily on unit tests, although it does not test the server e2e on real clusters.

This adds support for e2e tests on top of e.g. a kind cluster, with a github e2e test action running a kind cluster for tests.

Further, more complex tests are future work

grokspawn · 2026-06-24T20:23:59Z

+
+.PHONY: gotestsum
+gotestsum:
+	@[ -f $(GOTESTSUM) ] || GOBIN=$(shell dirname $(GOTESTSUM)) go install gotest.tools/gotestsum@latest


Can we pin gotestsum here?

grokspawn · 2026-06-24T20:27:20Z

+  e2e:
+    name: Run E2E Tests
+    runs-on: ubuntu-latest
+    steps:


add a timeout here? default timeout for things like a port-forward is 6h, and that sounds crazy. Maybe 30 mins?

grokspawn · 2026-06-24T20:37:37Z

+      - name: Run e2e tests
+        env:
+          KUBECONFIG: ${{ github.workspace }}/_output/kubeconfig
+        run: make e2e-test


junit output is optional in the accompanying makefile, but if we want to use it for e2e we need to add JUNIT_FILE to the envs, and also include a report publication step like

- name: publish junit report if: always() uses: mikepenz/action-junit-report@v5 with: report_paths: _output/junit-e2e.xml

grokspawn · 2026-06-24T20:55:55Z

+func findFreePort() (int, error) {
+	ln, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		return 0, err
+	}
+	defer ln.Close()
+	return ln.Addr().(*net.TCPAddr).Port, nil
+}


nit only: we could use existing test.RandomPortAddress

grokspawn · 2026-06-24T20:59:25Z

+	var buf bytes.Buffer
+	for _, args := range [][]string{
+		{"get", "pods", "-n", namespace, "-o", "wide"},
+		{"describe", "pods", "-n", namespace, "-l", "app.kubernetes.io/instance=" + releaseName},
+		{"logs", "-n", namespace, "-l", "app.kubernetes.io/instance=" + releaseName, "--tail=50"},
+		{"get", "events", "-n", namespace, "--sort-by=.lastTimestamp"},
+	} {
+		cmd := exec.Command("kubectl", args...)
+		out, _ := cmd.CombinedOutput()
+		fmt.Fprintf(&buf, "\n--- kubectl %s ---\n%s", strings.Join(args, " "), out)


can we use client-go instead of shelling out? the suite already has KubeClient as a member.

manusa · 2026-06-25T10:48:25Z

Thx! Created an internal comment for further discussion

Signed-off-by: Calum Murray <cmurray@redhat.com>

manusa · 2026-07-01T12:44:09Z

+  pull_request:
+    branches: ['main']
+    paths-ignore:
+      - '.gitignore'
+      - 'LICENSE'
+      - '*.md'
+      - 'docs/**'


Do we want to run these tests for all PRs?
I understand that these are now free, but if they actually grow into real e2e tests that deal with a model it will have some costs.
Options to consider:

merge to main

on demand

scheduled nightly

Also flagged by review agent, I think we should definitely clarify the scope of the e2e tests now.

IMO the e2e tests would not have a model - that is for the evals to use

I think the e2e tests are to exercise all the paths that our evals don't - specifically things around deployment and integration with other things in clusters like auth, operators, mcp gateways

In this view, running e2e on every PR probably makes sense

WDYT?

Even if with no models, the multi-cluster strategies and so on, will definitely make this expensive (either cash or time) to run.
Also, if running with internal infra, a real e2e test with a small model such as gpt-oss-20b or a hosted one would make things even more reliable.
At this point it's OK to keep it for each PR, since we'll be iterating on the test infra itself, but I think that eventually that will be too costly.

manusa · 2026-07-02T11:03:37Z

+
+        dep = ServerDeployment(name, namespace, server_url)
+        dep._port_forward_proc = proc
+        deployments.append(dep)


Flagged by Agent (I have no idea about Python):

Partial-failure resource leak in deploy_server (test/e2e/conftest.py:125). deployments.append(dep) runs only after _helm_install and _wait_for_healthz both succeed. If either raises (a helm failure, or the 30s healthz timeout), the created namespace, the installed Helm release, and the already-spawned kubectl port-forward process are never registered for teardown. CI is shielded because the whole cluster is torn down in the always() step, but a local make e2e-test against a persistent cluster leaks namespaces and orphaned port-forward processes across reruns. (Independently flagged by two reviewers.)

I think this is correct, but what we may want (from my experience with knative's e2e tests) is:

clean up the port forward if it is present

do not clean up the ns or resources in it if there was a failure, so you can inspect afterwards and figure out what went wrong

Sounds good

manusa · 2026-07-02T11:04:38Z

+                return
+        except (urllib.error.URLError, OSError):
+            await asyncio.sleep(0.5)
+    raise TimeoutError(f"Server at {url}/healthz not reachable within {timeout}s")


Flagged by Agent (I have no idea about Python):

The most likely failure produces no diagnostics (test/e2e/conftest.py:265). _dump_pod_diagnostics is only called from _helm_install. When a pod comes up but /healthz never becomes reachable, _wait_for_healthz raises a bare TimeoutError with no logs/events, and the kubectl port-forward stderr is discarded. The diagnostics helper exists but the path that needs it most bypasses it.

manusa · 2026-07-02T11:08:25Z

+      - name: Create Kind cluster
+        run: make kind-create-cluster


Incorrectly flagged by Agent:

make kind-create-cluster installs nginx-ingress and applies cert-manager from a remote GitHub release URL (three 120s waits) plus a ClusterIssuer. The tests set ingress.enabled=false and use none of it, so every run pays minutes of setup and takes a hard dependency on github.com being reachable (a transient hiccup fails the whole job).

I understand that once we add proper tests the more complete cluster will be needed, so what the agent flagged is false positive.

manusa · 2026-07-02T11:12:20Z

Flagged by Agent:

The suite shells out to helm/kubectl CLI tools which might not be installed in the e2e runner (in GitHub, ubuntu-latest does ship these tools).
If we were to run the suite locally or in a different CI environment this might be problem. We already have means to install required tooling in the makefiles, we might want to extend that to also install a pinned version of kubectl and helm so that the suite can shell out to those specific binaries.

manusa · 2026-07-02T11:13:59Z

+.PHONY: e2e-image
+e2e-image: ## Build the e2e container image and load it into the Kind cluster
+	$(CONTAINER_ENGINE) build -t $(E2E_IMAGE) .
+	$(KIND) load docker-image $(E2E_IMAGE) --name $(KIND_CLUSTER_NAME)


Flagged by Agent (important for my setup):

kind load docker-image lacks the podman provider env (build/e2e.mk:20). build/kind.mk only exports KIND_EXPERIMENTAL_PROVIDER=podman inside its own recipes, so on a podman-only Linux box make e2e-image builds with podman but this line looks for the image under docker and fails. CI unaffected (docker).

Ah this may be the case - I use podman personally but have an alias pointing docker to podman, so I never ran into the problem

manusa · 2026-07-02T11:15:38Z

+        with:
+          go-version-file: go.mod
+
+      - name: Install uv


Flagged by Agent:

Redundant uv install (build/e2e.mk:14). The workflow installs uv via the pinned astral-sh/setup-uv action, but e2e-test runs $(UV) = _output/tools/bin/uv, which the action doesn't populate, so the curl | sh runs anyway and the action's binary is unused (only its ~/.cache/uv warming helps).

I believe we should preserve the make installed one aligned with https://github.com/containers/kubernetes-mcp-server/pull/1228/changes#r3512563843

manusa · 2026-07-02T11:17:16Z

+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,


Flagged by Agent (I have no idea about Python):

port-forward pipes are never drained (test/e2e/conftest.py:244). stdout/stderr=PIPE on a long-lived kubectl port-forward that logs per connection could fill the ~64KB buffer and stall. Consider DEVNULL.

manusa

I added a few comments
Overall, I think this is solid work that lays the grounds for the future e2e tests (as discussed internally).
I'd try to solve the already flagged problems in the scope of this PR. Nonetheless, I think we'll definitely keep iterating over the implementation in the coming days/weeks.

Good work, thx!

Signed-off-by: Calum Murray <cmurray@redhat.com>

manusa

LGTM, thx!

Merging as is.
In any of your future changes, it'd be nice to have:

Add timeout-minutes to the e2e job, the earlier timeout ask from @grokspawn is still open, and the job currently inherits the 6h default (kind-create, image build, and pytest are otherwise unbounded).
A few minor nits to fold in whenever you next touch this: _find_free_port has a TOCTOU race that'll flake once deployments run in parallel; _parse_image only handles the registry/repo:tag form (digests/registry-ports get mangled); config.pop("http") silently drops that section; and paths-ignore: '*.md' only matches top-level markdown.

Cali0707 force-pushed the e2e-test branch 4 times, most recently from 0d860e9 to 263ef6c Compare June 24, 2026 18:38

grokspawn reviewed Jun 24, 2026

View reviewed changes

Cali0707 added 3 commits June 29, 2026 12:46

test(e2e): add initial e2e test

4203540

Signed-off-by: Calum Murray <cmurray@redhat.com>

test(e2e): add make targets

877665c

Signed-off-by: Calum Murray <cmurray@redhat.com>

test(e2e): add kind e2e action

8512fc3

Signed-off-by: Calum Murray <cmurray@redhat.com>

Cali0707 force-pushed the e2e-test branch from 263ef6c to 8512fc3 Compare June 29, 2026 16:46

manusa reviewed Jul 1, 2026

View reviewed changes

Comment thread build/e2e.mk

manusa reviewed Jul 2, 2026

View reviewed changes

Comment thread test/e2e/pyproject.toml

manusa reviewed Jul 2, 2026

View reviewed changes

manusa approved these changes Jul 2, 2026

View reviewed changes

manusa added this to the 0.1.0 milestone Jul 2, 2026

Cali0707 mentioned this pull request Jul 6, 2026

feat(otel): logs are exported with otel now #1260

Merged

chore: address review comments

67126f5

Signed-off-by: Calum Murray <cmurray@redhat.com>

manusa approved these changes Jul 6, 2026

View reviewed changes

manusa merged commit 36c37e4 into containers:main Jul 6, 2026
12 checks passed

This was referenced Jul 6, 2026

chore(ci): set 30m timeout on e2e job #1267

Merged

chore(e2e): cleanup e2e framework #1268

Merged

openshift-ci-robot mentioned this pull request Jul 9, 2026

"NO-JIRA: chore: sync with upstream 2026-07-09 - 23 new commits" openshift/openshift-mcp-server#379

Merged

Uh oh!

Conversation

Cali0707 commented Jun 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manusa commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manusa left a comment

Choose a reason for hiding this comment

Uh oh!

manusa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants