Skip to content

Commit 24ce51e

Browse files
andyatmiamiclaudethesuperzapper
authored
ci: scaffold e2e testing framework infrastructure (#763)
* ci: scaffold e2e testing framework infrastructure This commit establishes the foundational infrastructure for end-to-end testing of the workspaces components. While some of the scripts appear redundant with the developing/ directory, a deliberate decision was made to keep this logic mutually exclusive to be more flexible. For instance, the testing/ directory has a need to deploy a Gateway that is not necessary for developing. Keeping everything separate for now makes it easier to evolve independently. Changes include: - Add new `testing/` directory with Makefile and setup scripts: * `setup-kind.sh`: Automated Kind cluster creation and configuration * `setup-cert-manager.sh`: Cert-manager installation (v1.12.13 LTS) * `setup-istio.sh`: Istio service mesh installation with Gateway and TLS certificate provisioning via cert-manager * `check-kind-context.sh`: Safety check to prevent accidental deployment to non-Kind clusters * `sanity-check.sh`: Post-deploy verification including rollout status, TLS handshake (webhook), HTTP health endpoints, and Istio gateway routing for backend and frontend * `gateway.yaml`: Istio Gateway (HTTP + HTTPS) for kubeflow-gateway * `gateway-cert.yaml`: Self-signed ClusterIssuer and Certificate for gateway TLS termination * Makefile targets: setup-cluster, deploy-all, sanity-check, teardown-cluster, clean, and local-e2e (placeholder) - Add GitHub Actions workflow (`.github/workflows/ws-e2e-test.yml`): * Triggers on pushes to main branches and PRs affecting workspaces * Pipeline: setup-cluster -> deploy-all -> sanity-check -> local-e2e Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Andy Stoneberg <astonebe@redhat.com> * chore: address PR feedback Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Andy Stoneberg <astonebe@redhat.com> * chore: address PR feedback Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Andy Stoneberg <astonebe@redhat.com> * mathew: 1 Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> * mathew: 2 Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> --------- Signed-off-by: Andy Stoneberg <astonebe@redhat.com> Signed-off-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Mathew Wicks <5735406+thesuperzapper@users.noreply.github.com>
1 parent b590b6d commit 24ce51e

20 files changed

Lines changed: 588 additions & 17 deletions

.github/workflows/ws-controller-test.yml

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -92,10 +92,9 @@ jobs:
9292
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
9393

9494
- name: Set up Kind
95-
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1
95+
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
9696
with:
97-
version: v0.23.0
98-
node_image: kindest/node:v1.25.16@sha256:5da57dfc290ac3599e775e63b8b6c49c0c85d3fec771cd7d55b45fae14b38d3b
97+
config: ./testing/kind-1-35.yaml
9998
cluster_name: kind
10099

101100
- name: Set up Go

.github/workflows/ws-e2e-test.yml

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
name: E2E Tests
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
push:
8+
branches:
9+
- main
10+
- notebooks-v2
11+
- v*-branch
12+
pull_request:
13+
paths:
14+
- workspaces/**
15+
16+
jobs:
17+
e2e-test:
18+
runs-on: ubuntu-latest
19+
defaults:
20+
run:
21+
working-directory: testing
22+
steps:
23+
- name: Checkout code
24+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
25+
26+
- name: Set up Kind
27+
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
28+
with:
29+
config: ./testing/kind-1-35.yaml
30+
cluster_name: "local-e2e"
31+
32+
- name: Install Cluster Dependencies
33+
run: make setup-cluster
34+
35+
- name: Deploy Kubeflow Workspaces
36+
run: make deploy-all
37+
38+
- name: Verify Deployment
39+
run: make sanity-check
40+
41+
- name: Run local-e2e tests
42+
run: make local-e2e

developing/scripts/kind.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,6 @@ kubeadmConfigPatches:
1313
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"
1414
nodes:
1515
- role: control-plane
16-
image: kindest/node:v1.33.1@sha256:050072256b9a903bd914c0b2866828150cb229cea0efe5892e2b644d5dd3b34f
16+
image: kindest/node:v1.35.0@sha256:452d707d4862f52530247495d180205e029056831160e22870e37e3f6c1ac31f
1717
- role: worker
18-
image: kindest/node:v1.33.1@sha256:050072256b9a903bd914c0b2866828150cb229cea0efe5892e2b644d5dd3b34f
18+
image: kindest/node:v1.35.0@sha256:452d707d4862f52530247495d180205e029056831160e22870e37e3f6c1ac31f

developing/scripts/setup-cert-manager.sh

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,16 @@
11
#!/usr/bin/env bash
2+
23
# Setup script for cert-manager
3-
# This script checks if cert-manager is installed and installs it if needed
4-
# Uses the same version as the e2e tests: v1.12.13 (LTS version)
4+
# This script checks if cert-manager is installed and installs it if needed.
55

66
set -euo pipefail
77

8-
# Use LTS version of cert-manager (matches e2e tests)
9-
CERT_MANAGER_VERSION="v1.12.13"
8+
# Use LTS version of cert-manager
9+
# NOTE: ensure the following files use the same version when updating:
10+
# - developing/scripts/setup-istio.sh
11+
# - testing/scripts/setup-istio.sh
12+
# - workspaces/controller/test/utils/certmanager.go
13+
CERT_MANAGER_VERSION="v1.20.2"
1014
CERT_MANAGER_URL="https://github.com/jetstack/cert-manager/releases/download/${CERT_MANAGER_VERSION}/cert-manager.yaml"
1115

1216
# Check if cert-manager is already installed
@@ -40,4 +44,4 @@ kubectl wait --for=condition=established crd/certificates.cert-manager.io --time
4044
kubectl wait --for=condition=established crd/issuers.cert-manager.io --timeout=60s
4145
kubectl wait --for=condition=established crd/clusterissuers.cert-manager.io --timeout=60s
4246

43-
echo "Cert-manager setup complete"
47+
echo "Cert-manager setup complete"

developing/scripts/setup-istio.sh

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
#!/usr/bin/env bash
2+
23
# Setup script for Istio service mesh
34
# This script checks if Istio is installed and installs it if needed.
45
# Gateway resources (namespace, TLS cert, Gateway) are managed by Tilt.
@@ -9,15 +10,12 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
910
DEVELOPING_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
1011
LOCALBIN="${DEVELOPING_DIR}/bin"
1112

12-
# Determine istioctl path - prefer LOCALBIN, fallback to PATH
13+
# Require istioctl from LOCALBIN to ensure a known version
1314
if [ -f "${LOCALBIN}/istioctl" ]; then
1415
ISTIOCTL="${LOCALBIN}/istioctl"
15-
elif command -v istioctl >/dev/null 2>&1; then
16-
ISTIOCTL="istioctl"
1716
else
1817
echo "ERROR: istioctl is not installed. Please install istioctl first:"
1918
echo " cd developing && make istioctl"
20-
echo " or visit: https://istio.io/latest/docs/setup/getting-started/#download"
2119
exit 1
2220
fi
2321

developing/scripts/setup-kind.sh

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,10 @@ else
2626
fi
2727

2828
# Ensure kubectl context is set to the Kind cluster
29-
kubectl config use-context "kind-${CLUSTER_NAME}" || true
29+
kubectl config use-context "kind-${CLUSTER_NAME}" || {
30+
echo "ERROR: Failed to set kubectl context to kind-${CLUSTER_NAME}"
31+
exit 1
32+
}
3033

3134
# Configure StorageClasses with Notebooks labels and annotations
3235
echo "Configuring StorageClasses for the Notebooks UI..."

testing/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Binaries for programs and plugins
2+
bin/

testing/Makefile

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
GIT_COMMIT := $(shell git rev-parse HEAD)
2+
GIT_TREE_STATE := $(shell test -n "`git status --porcelain`" && echo "-dirty" || echo "")
3+
4+
# Image URL to use all building/pushing image targets
5+
REGISTRY ?= ghcr.io/kubeflow/notebooks
6+
TAG ?= sha-$(GIT_COMMIT)$(GIT_TREE_STATE)
7+
8+
CONTROLLER_NAME ?= workspaces-controller
9+
CONTROLLER_IMG ?= $(REGISTRY)/$(CONTROLLER_NAME):$(TAG)
10+
11+
BACKEND_NAME ?= workspaces-backend
12+
BACKEND_IMG ?= $(REGISTRY)/$(BACKEND_NAME):$(TAG)
13+
14+
FRONTEND_NAME ?= workspaces-frontend
15+
FRONTEND_IMG ?= $(REGISTRY)/$(FRONTEND_NAME):$(TAG)
16+
17+
KIND_CLUSTER_NAME ?= local-e2e
18+
19+
# Setting SHELL to bash allows bash commands to be executed by recipes.
20+
# Options are set to exit when a recipe line exits non-zero or a piped command fails.
21+
SHELL = /usr/bin/env bash -o pipefail
22+
.SHELLFLAGS = -ec
23+
24+
# Export KIND_EXPERIMENTAL_PROVIDER to honor it if set in user's environment
25+
# (e.g., KIND_EXPERIMENTAL_PROVIDER=podman for podman support)
26+
export KIND_EXPERIMENTAL_PROVIDER
27+
28+
##@ General
29+
30+
# The help target prints out all targets with their descriptions organized
31+
# beneath their categories. The categories are represented by '##@' and the
32+
# target descriptions by '##'. The awk command is responsible for reading the
33+
# entire set of makefiles included in this invocation, looking for lines of the
34+
# file as xyz: ## something, and then pretty-format the target and help. Then,
35+
# if there's a line with ##@ something, that gets pretty-printed as a category.
36+
# More info on the usage of ANSI control characters for terminal formatting:
37+
# https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_parameters
38+
# More info on the awk command:
39+
# http://linuxcommand.org/lc3_adv_awk.php
40+
41+
.PHONY: help
42+
help: ## Display this help.
43+
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
44+
45+
##@ Deployment
46+
47+
.PHONY: setup-cluster
48+
setup-cluster: check-kubectl check-kind ## Set up a complete kind cluster with cert-manager and Istio.
49+
bash scripts/setup-kind.sh
50+
bash scripts/setup-cert-manager.sh
51+
bash scripts/setup-istio.sh
52+
53+
.PHONY: teardown-cluster
54+
teardown-cluster: check-kind ## Delete the Kind cluster.
55+
kind delete cluster --name $(KIND_CLUSTER_NAME)
56+
57+
.PHONY: deploy-controller
58+
deploy-controller: check-kind check-kind-context ## Build and deploy the controller.
59+
$(MAKE) -C ../workspaces/controller docker-build IMG=$(CONTROLLER_IMG)
60+
kind load docker-image $(CONTROLLER_IMG) --name $(KIND_CLUSTER_NAME)
61+
$(MAKE) -C ../workspaces/controller deploy IMG=$(CONTROLLER_IMG)
62+
63+
.PHONY: deploy-backend
64+
deploy-backend: check-kind check-kind-context ## Build and deploy the backend.
65+
$(MAKE) -C ../workspaces/backend docker-build IMG=$(BACKEND_IMG)
66+
kind load docker-image $(BACKEND_IMG) --name $(KIND_CLUSTER_NAME)
67+
$(MAKE) -C ../workspaces/backend deploy IMG=$(BACKEND_IMG)
68+
69+
.PHONY: deploy-frontend
70+
deploy-frontend: check-kind check-kind-context ## Build and deploy the frontend.
71+
$(MAKE) -C ../workspaces/frontend docker-build IMG=$(FRONTEND_IMG)
72+
kind load docker-image $(FRONTEND_IMG) --name $(KIND_CLUSTER_NAME)
73+
$(MAKE) -C ../workspaces/frontend deploy IMG=$(FRONTEND_IMG)
74+
75+
.PHONY: deploy-all
76+
deploy-all: deploy-controller deploy-backend deploy-frontend ## Deploy all components.
77+
78+
##@ E2E Tests
79+
80+
.PHONY: sanity-check
81+
sanity-check: check-kind-context ## Verify all components are deployed and responding.
82+
@bash scripts/sanity-check.sh
83+
84+
.PHONY: local-e2e
85+
local-e2e: ## Run e2e tests.
86+
@echo "..."
87+
@echo "TODO: there are no e2e tests yet, they will be defined in Cypress..."
88+
@echo "..."
89+
90+
##@ Dependencies
91+
92+
## Location to install dependencies to
93+
LOCALBIN ?= $(shell pwd)/bin
94+
$(LOCALBIN):
95+
mkdir -p $(LOCALBIN)
96+
97+
.PHONY: check-kind
98+
check-kind: ## Verify that kind is available in PATH.
99+
@command -v kind >/dev/null 2>&1 || { \
100+
echo "ERROR: kind is not found in PATH. Please install kind."; \
101+
exit 1; \
102+
}
103+
104+
.PHONY: check-kind-context
105+
check-kind-context: ## Verify that the current kubectl context is set to the expected kind cluster.
106+
@{ \
107+
current_context=$$(kubectl config current-context); \
108+
expected_context="kind-$(KIND_CLUSTER_NAME)"; \
109+
if [ "$$current_context" != "$$expected_context" ]; then \
110+
echo "ERROR: Current kubectl context is '$$current_context', but expected '$$expected_context'."; \
111+
echo "ERROR: switch to the correct context using: kubectl config use-context $$expected_context"; \
112+
exit 1; \
113+
fi; \
114+
}
115+
116+
.PHONY: check-kubectl
117+
check-kubectl: ## Verify that kubectl is available in PATH.
118+
@command -v kubectl >/dev/null 2>&1 || { \
119+
echo "ERROR: kubectl is not found in PATH. Please install kubectl."; \
120+
exit 1; \
121+
}
122+
123+
.PHONY: clean
124+
clean: ## Remove downloaded tool binaries.
125+
rm -rf $(LOCALBIN)
126+
@echo "INFO: '$(LOCALBIN)' successfully cleaned."

testing/OWNERS

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
labels:
2+
- area/ci
3+
- area/v2
4+
approvers:
5+
- andyatmiami

testing/istio-cni.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
apiVersion: install.istio.io/v1alpha1
2+
kind: IstioOperator
3+
spec:
4+
profile: default
5+
components:
6+
cni:
7+
## NOTE: we use istio CNI mode because otherwise setting pod-security.kubernetes.io/enforce=baseline on a
8+
## namespace would prevent any pod with an istio sidecar from running, as the sidecar would be privileged.
9+
enabled: true
10+
namespace: kube-system

0 commit comments

Comments
 (0)