Skip to content

Commit 92add3a

Browse files
andyatmiamiandyatmiami
committed
ci: scaffold e2e testing framework infrastructure
This commit establishes the foundational infrastructure for end-to-end testing of the workspaces components. While some of the scripts appear redundant with the developing/ directory, a deliberate decision was made to keep this logic mutually exclusive to be more flexible. For instance, the testing/ directory has a need to deploy a Gateway that is not necessary for developing. Keeping everything separate for now makes it easier to evolve independently. Changes include: - Add new `testing/` directory with Makefile and setup scripts: * `setup-kind.sh`: Automated Kind cluster creation and configuration * `setup-cert-manager.sh`: Cert-manager installation (v1.12.13 LTS) * `setup-istio.sh`: Istio service mesh installation with Gateway and TLS certificate provisioning via cert-manager * `check-kind-context.sh`: Safety check to prevent accidental deployment to non-Kind clusters * `sanity-check.sh`: Post-deploy verification including rollout status, TLS handshake (webhook), HTTP health endpoints, and Istio gateway routing for backend and frontend * `gateway.yaml`: Istio Gateway (HTTP + HTTPS) for kubeflow-gateway * `gateway-cert.yaml`: Self-signed ClusterIssuer and Certificate for gateway TLS termination * Makefile targets: setup-cluster, deploy-all, sanity-check, teardown-cluster, clean, and local-e2e (placeholder) - Add GitHub Actions workflow (`.github/workflows/ws-e2e-test.yml`): * Triggers on pushes to main branches and PRs affecting workspaces * Pipeline: setup-cluster -> deploy-all -> sanity-check -> local-e2e Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Andy Stoneberg <astonebe@redhat.com>
1 parent 4db85a2 commit 92add3a

14 files changed

Lines changed: 646 additions & 3 deletions

File tree

‎.github/workflows/ws-e2e-test.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Workspaces E2E Tests
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
push:
8+
branches:
9+
- main
10+
- notebooks-v2
11+
- v*-branch
12+
pull_request:
13+
paths:
14+
- 'workspaces/**'
15+
- 'releasing/version/VERSION'
16+
17+
jobs:
18+
e2e-test:
19+
runs-on: ubuntu-latest
20+
env:
21+
GO_VERSION: '1.24.13'
22+
defaults:
23+
run:
24+
working-directory: testing
25+
steps:
26+
- name: Checkout code
27+
uses: actions/checkout@v4
28+
29+
- name: Set up Go
30+
uses: actions/setup-go@v5
31+
with:
32+
go-version: ${{ env.GO_VERSION }}
33+
cache: false
34+
35+
- name: Generate cache key from Makefile
36+
id: cache-key
37+
run: |
38+
VERSION_HASH=$(make dependency-hash)
39+
echo "cache_key=testing-bin-${{ runner.os }}-go${{ env.GO_VERSION }}-$VERSION_HASH" >> $GITHUB_OUTPUT
40+
41+
- name: Cache testing/bin directory
42+
uses: actions/cache@v4
43+
id: cache-testing-bin
44+
with:
45+
path: testing/bin
46+
key: ${{ steps.cache-key.outputs.cache_key }}
47+
48+
- name: Setup cluster
49+
run: make setup-cluster
50+
51+
- name: Deploy components
52+
run: make deploy-all
53+
54+
- name: Sanity check
55+
run: make sanity-check
56+
57+
- name: Run local-e2e tests
58+
run: make local-e2e
59+

‎developing/scripts/kind.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,6 @@ kubeadmConfigPatches:
1313
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
1414
nodes:
1515
- role: control-plane
16-
image: kindest/node:v1.33.1@sha256:050072256b9a903bd914c0b2866828150cb229cea0efe5892e2b644d5dd3b34f
16+
image: kindest/node:v1.34.3@sha256:08497ee19eace7b4b5348db5c6a1591d7752b164530a36f855cb0f2bdcbadd48
1717
- role: worker
18-
image: kindest/node:v1.33.1@sha256:050072256b9a903bd914c0b2866828150cb229cea0efe5892e2b644d5dd3b34f
18+
image: kindest/node:v1.34.3@sha256:08497ee19eace7b4b5348db5c6a1591d7752b164530a36f855cb0f2bdcbadd48

‎developing/scripts/setup-kind.sh‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,10 @@ else
2626
fi
2727

2828
# Ensure kubectl context is set to the Kind cluster
29-
kubectl config use-context "kind-${CLUSTER_NAME}" || true
29+
kubectl config use-context "kind-${CLUSTER_NAME}" || {
30+
echo "ERROR: Failed to set kubectl context to kind-${CLUSTER_NAME}"
31+
exit 1
32+
}
3033

3134
# Configure StorageClasses with Notebooks labels and annotations
3235
echo "Configuring StorageClasses for the Notebooks UI..."

‎testing/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Binaries for programs and plugins
2+
bin/*

‎testing/Makefile‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
GIT_COMMIT := $(shell git rev-parse HEAD)
2+
GIT_TREE_STATE := $(shell test -n "`git status --porcelain`" && echo "-dirty" || echo "")
3+
4+
# Image URL to use all building/pushing image targets
5+
REGISTRY ?= ghcr.io/kubeflow/notebooks
6+
TAG ?= sha-$(GIT_COMMIT)$(GIT_TREE_STATE)
7+
8+
CONTROLLER_NAME ?= workspaces-controller
9+
CONTROLLER_IMG ?= $(REGISTRY)/$(CONTROLLER_NAME):$(TAG)
10+
11+
BACKEND_NAME ?= workspaces-backend
12+
BACKEND_IMG ?= $(REGISTRY)/$(BACKEND_NAME):$(TAG)
13+
14+
FRONTEND_NAME ?= workspaces-frontend
15+
FRONTEND_IMG ?= $(REGISTRY)/$(FRONTEND_NAME):$(TAG)
16+
17+
KIND_CLUSTER_NAME ?= local-e2e
18+
19+
# Setting SHELL to bash allows bash commands to be executed by recipes.
20+
# Options are set to exit when a recipe line exits non-zero or a piped command fails.
21+
SHELL = /usr/bin/env bash -o pipefail
22+
.SHELLFLAGS = -ec
23+
24+
25+
# Export KIND_EXPERIMENTAL_PROVIDER to honor it if set in user's environment
26+
# (e.g., KIND_EXPERIMENTAL_PROVIDER=podman for podman support)
27+
export KIND_EXPERIMENTAL_PROVIDER
28+
29+
##@ General
30+
31+
# The help target prints out all targets with their descriptions organized
32+
# beneath their categories. The categories are represented by '##@' and the
33+
# target descriptions by '##'. The awk command is responsible for reading the
34+
# entire set of makefiles included in this invocation, looking for lines of the
35+
# file as xyz: ## something, and then pretty-format the target and help. Then,
36+
# if there's a line with ##@ something, that gets pretty-printed as a category.
37+
# More info on the usage of ANSI control characters for terminal formatting:
38+
# https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_parameters
39+
# More info on the awk command:
40+
# http://linuxcommand.org/lc3_adv_awk.php
41+
42+
.PHONY: help
43+
help: ## Display this help.
44+
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
45+
46+
##@ Deployment
47+
48+
.PHONY: build-controller build-backend build-frontend build-all e2e sanity-check
49+
50+
deploy-controller: kind check-kind-context ## Build and deploy the controller.
51+
cd ../workspaces/controller && $(MAKE) docker-build IMG=$(CONTROLLER_IMG)
52+
$(KIND) load docker-image $(CONTROLLER_IMG) --name $(KIND_CLUSTER_NAME)
53+
cd ../workspaces/controller && $(MAKE) deploy IMG=$(CONTROLLER_IMG)
54+
55+
deploy-backend: kind check-kind-context ## Build and deploy the backend.
56+
cd ../workspaces/backend && $(MAKE) docker-build IMG=$(BACKEND_IMG)
57+
$(KIND) load docker-image $(BACKEND_IMG) --name $(KIND_CLUSTER_NAME)
58+
cd ../workspaces/backend && $(MAKE) deploy IMG=$(BACKEND_IMG)
59+
60+
deploy-frontend: kind check-kind-context ## Build and deploy the frontend.
61+
cd ../workspaces/frontend && $(MAKE) docker-build IMG=$(FRONTEND_IMG)
62+
$(KIND) load docker-image $(FRONTEND_IMG) --name $(KIND_CLUSTER_NAME)
63+
cd ../workspaces/frontend && $(MAKE) deploy IMG=$(FRONTEND_IMG)
64+
65+
deploy-all: deploy-controller deploy-backend deploy-frontend ## Deploy all components.
66+
67+
sanity-check: check-kind-context ## Verify all components are deployed and responding.
68+
@bash scripts/sanity-check.sh
69+
70+
local-e2e: ## Run e2e tests.
71+
@echo "TODO: Run e2e tests..."
72+
73+
74+
##@ Dependencies
75+
76+
## Location to install dependencies to
77+
LOCALBIN ?= $(shell pwd)/bin
78+
$(LOCALBIN):
79+
mkdir -p $(LOCALBIN)
80+
81+
## Tool Binaries
82+
KIND ?= $(LOCALBIN)/kind
83+
KUBECTL ?= kubectl
84+
ISTIOCTL ?= $(LOCALBIN)/istioctl
85+
86+
## Tool Versions
87+
KIND_VERSION ?= v0.30.0
88+
ISTIOCTL_VERSION ?= 1.27.3
89+
90+
.PHONY: kind
91+
kind: $(KIND) ## Download kind locally if necessary.
92+
$(KIND): $(LOCALBIN)
93+
$(call go-install-tool,$(KIND),sigs.k8s.io/kind,$(KIND_VERSION))
94+
95+
.PHONY: check-kubectl
96+
check-kubectl: ## Verify that kubectl is available in PATH.
97+
@if ! command -v $(KUBECTL) >/dev/null 2>&1; then \
98+
echo "✗ ERROR: kubectl is not installed or not found in PATH"; \
99+
echo " Please install kubectl: https://kubernetes.io/docs/tasks/tools/#kubectl"; \
100+
exit 1; \
101+
fi
102+
@echo "✓ kubectl found: $$($(KUBECTL) version --client 2>/dev/null || echo 'version check failed')"
103+
104+
.PHONY: istioctl
105+
istioctl: $(ISTIOCTL) ## Download istioctl locally if necessary.
106+
$(ISTIOCTL): $(LOCALBIN)
107+
$(call go-install-tool,$(ISTIOCTL),istio.io/istio/istioctl/cmd/istioctl,$(ISTIOCTL_VERSION),"-X istio.io/istio/pkg/version.buildVersion=$(ISTIOCTL_VERSION) -X istio.io/istio/pkg/version.buildTag=$(ISTIOCTL_VERSION) -X istio.io/istio/pkg/version.buildHub=docker.io/istio")
108+
109+
.PHONY: dependency-hash
110+
dependency-hash: ## Calculate hash of dependency versions for caching.
111+
@echo -e "KIND_VERSION=$(KIND_VERSION)\nISTIOCTL_VERSION=$(ISTIOCTL_VERSION)" | sha256sum | cut -d' ' -f1 | head -c 16
112+
113+
.PHONY: setup-cluster
114+
setup-cluster: check-kubectl kind istioctl ## Set up a complete kind cluster with cert-manager and Istio.
115+
@export PATH="$(LOCALBIN):$$PATH" && \
116+
bash scripts/setup-kind.sh && \
117+
bash scripts/setup-cert-manager.sh && \
118+
bash scripts/setup-istio.sh
119+
@echo "✓ Cluster setup complete"
120+
121+
.PHONY: teardown-cluster
122+
teardown-cluster: kind ## Delete the Kind cluster.
123+
@$(KIND) delete cluster --name $(KIND_CLUSTER_NAME) && \
124+
echo "✓ Cluster '$(KIND_CLUSTER_NAME)' deleted" || \
125+
echo "✗ Cluster '$(KIND_CLUSTER_NAME)' not found"
126+
127+
.PHONY: clean
128+
clean: ## Remove downloaded tool binaries.
129+
@rm -rf $(LOCALBIN)
130+
@echo "✓ Cleaned $(LOCALBIN)"
131+
132+
.PHONY: check-kind-context
133+
check-kind-context: check-kubectl ## Verify that the current kubectl context is a kind cluster.
134+
@bash scripts/check-kind-context.sh
135+
136+
# go-install-tool will 'go install' any package with custom target and name of binary, if it doesn't exist
137+
# $1 - target path with name of binary
138+
# $2 - package url which can be installed
139+
# $3 - specific version of package
140+
# $4 - (optional) extra ldflags to set with the installation
141+
define go-install-tool
142+
@[ -f "$(1)-$(3)" ] || { \
143+
set -e; \
144+
package=$(2)@$(3) ;\
145+
ldflags=$(4) ;\
146+
echo "Downloading $${package}" ;\
147+
rm -f $(1) || true ;\
148+
GOBIN=$(LOCALBIN) go install $${ldflags:+-ldflags "$${ldflags}"} $${package} ;\
149+
mv $(1) $(1)-$(3) ;\
150+
} ;\
151+
ln -sf $(1)-$(3) $(1)
152+
endef

‎testing/OWNERS‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
labels:
2+
- area/ci
3+
- area/v2
4+
approvers:
5+
- andyatmiami

‎testing/scripts/check-kind-context.sh‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/usr/bin/env bash
2+
3+
# Verify that the current kubectl context is a kind cluster.
4+
# Checks both the context name (kind-* prefix) and the server URL (localhost).
5+
6+
set -euo pipefail
7+
8+
current_context=$(kubectl config current-context 2>/dev/null) || {
9+
echo "Error: Unable to get current kubectl context. Is kubectl configured?"
10+
exit 1
11+
}
12+
13+
server_url=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}' 2>/dev/null) || {
14+
echo "Error: Unable to get cluster server URL for context '${current_context}'"
15+
exit 1
16+
}
17+
18+
context_check=0
19+
server_check=0
20+
21+
if echo "${current_context}" | grep -qE '^kind-'; then
22+
context_check=1
23+
fi
24+
25+
if echo "${server_url}" | grep -qE '(127\.0\.0\.1|localhost)'; then
26+
server_check=1
27+
fi
28+
29+
if [ ${context_check} -ne 1 ] || [ ${server_check} -ne 1 ]; then
30+
echo "✗ ERROR: Current context '${current_context}' does not appear to be a kind cluster!"
31+
if [ ${context_check} -ne 1 ]; then
32+
echo " ✗ Context name does not match kind-* pattern (got: ${current_context})"
33+
fi
34+
if [ ${server_check} -ne 1 ]; then
35+
echo " ✗ Server URL does not use localhost/127.0.0.1 (got: ${server_url})"
36+
fi
37+
exit 1
38+
fi

‎testing/scripts/gateway-cert.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
apiVersion: cert-manager.io/v1
2+
kind: ClusterIssuer
3+
metadata:
4+
name: selfsigned-issuer
5+
spec:
6+
selfSigned: {}
7+
---
8+
apiVersion: cert-manager.io/v1
9+
kind: Certificate
10+
metadata:
11+
name: gateway-tls
12+
namespace: istio-system
13+
spec:
14+
secretName: gateway-tls-secret
15+
duration: 8760h # 1 year
16+
issuerRef:
17+
name: selfsigned-issuer
18+
kind: ClusterIssuer
19+
dnsNames:
20+
- localhost
21+
- "*.local"
22+
- kubeflow.example.com

‎testing/scripts/gateway.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
apiVersion: networking.istio.io/v1beta1
2+
kind: Gateway
3+
metadata:
4+
name: kubeflow-gateway
5+
namespace: kubeflow
6+
spec:
7+
selector:
8+
app: istio-ingressgateway
9+
istio: ingressgateway
10+
servers:
11+
- port:
12+
number: 443
13+
name: https
14+
protocol: HTTPS
15+
tls:
16+
mode: SIMPLE
17+
credentialName: gateway-tls-secret
18+
hosts:
19+
- "*"

‎testing/scripts/kind.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
apiVersion: kind.x-k8s.io/v1alpha4
2+
kind: Cluster
3+
# This is needed in order to support projected volumes with service account tokens.
4+
kubeadmConfigPatches:
5+
- |
6+
apiVersion: kubeadm.k8s.io/v1beta3
7+
kind: ClusterConfiguration
8+
metadata:
9+
name: config
10+
apiServer:
11+
extraArgs:
12+
"service-account-issuer": "kubernetes.default.svc"
13+
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
14+
nodes:
15+
- role: control-plane
16+
image: kindest/node:v1.34.3@sha256:08497ee19eace7b4b5348db5c6a1591d7752b164530a36f855cb0f2bdcbadd48
17+
- role: worker
18+
image: kindest/node:v1.34.3@sha256:08497ee19eace7b4b5348db5c6a1591d7752b164530a36f855cb0f2bdcbadd48

0 commit comments

Comments
 (0)