Skip to content

Commit cfc79cb

Browse files
scotwellsclaude
scotwells
and
claude
committed
test(e2e): shared multi-cluster e2e harness
Add the shared e2e environment helper plus the kind and Chainsaw configuration and kubeconfig/cluster-secret scripts that stand up a multi-cluster control plane, so federated scheduling can be exercised end to end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 667ec25 commit cfc79cb

6 files changed

Lines changed: 698 additions & 0 deletions

File tree

‎hack/e2e/kind-control-plane.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Kind cluster configuration for the compute-control-plane management cluster.
2+
#
3+
# extraPortMappings exposes port 32443 on the macOS host so that the Karmada
4+
# API server NodePort service (nodePort: 32443) is accessible at
5+
# https://localhost:32443 without any additional port-forwarding.
6+
#
7+
# This matches KARMADA_API_NODEPORT in Taskfile.yaml.
8+
9+
kind: Cluster
10+
apiVersion: kind.x-k8s.io/v1alpha4
11+
nodes:
12+
- role: control-plane
13+
extraPortMappings:
14+
- containerPort: 32443 # Karmada API server NodePort
15+
hostPort: 32443
16+
protocol: TCP
17+
listenAddress: "127.0.0.1"

‎hack/e2e/make-internal-kubeconfig.sh‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/usr/bin/env bash
2+
# make-internal-kubeconfig.sh <input-kubeconfig> <output-kubeconfig> <kind-cluster-name>
3+
#
4+
# Produces a kubeconfig variant that uses the Kind node's Docker container IP
5+
# instead of localhost. This variant is stored in Karmada so the controller
6+
# manager (running inside Docker) can reach member cluster API servers across
7+
# the kind bridge network.
8+
#
9+
# Background: Kind maps each cluster's API server to a random localhost port
10+
# on the developer machine. Inside Docker containers, "localhost" refers to the
11+
# container's own loopback — not the host. We therefore swap the server address
12+
# to the Kind control-plane container's Docker bridge IP (e.g. 172.18.0.x) and
13+
# set insecure-skip-tls-verify because the node certificate does not include
14+
# the Docker bridge IP in its SANs.
15+
#
16+
# Usage:
17+
# hack/e2e/make-internal-kubeconfig.sh \
18+
# tmp/e2e/kubeconfigs/pop-dfw.yaml \
19+
# tmp/e2e/kubeconfigs/pop-dfw-internal.yaml \
20+
# compute-pop-dfw
21+
22+
set -euo pipefail
23+
24+
INPUT="${1:?usage: $0 <input-kubeconfig> <output-kubeconfig> <kind-cluster-name>}"
25+
OUTPUT="${2:?usage: $0 <input-kubeconfig> <output-kubeconfig> <kind-cluster-name>}"
26+
CLUSTER_NAME="${3:?usage: $0 <input-kubeconfig> <output-kubeconfig> <kind-cluster-name>}"
27+
28+
CONTAINER_NAME="${CLUSTER_NAME}-control-plane"
29+
30+
# Resolve the container's Docker bridge IP.
31+
DOCKER_IP=$(docker inspect \
32+
-f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' \
33+
"${CONTAINER_NAME}" 2>/dev/null || true)
34+
35+
if [ -z "${DOCKER_IP}" ]; then
36+
echo "ERROR: Could not resolve Docker IP for container '${CONTAINER_NAME}'." >&2
37+
echo " Is the Kind cluster '${CLUSTER_NAME}' running?" >&2
38+
exit 1
39+
fi
40+
41+
echo " ${CLUSTER_NAME}: Docker IP ${DOCKER_IP} → ${OUTPUT}"
42+
43+
python3 - "${INPUT}" "${OUTPUT}" "${DOCKER_IP}" <<'PYEOF'
44+
import sys, yaml
45+
46+
src, dst, docker_ip = sys.argv[1], sys.argv[2], sys.argv[3]
47+
48+
with open(src) as f:
49+
cfg = yaml.safe_load(f)
50+
51+
for cluster in cfg.get('clusters', []):
52+
# Kind API server always listens on port 6443 inside the container.
53+
cluster['cluster']['server'] = f'https://{docker_ip}:6443'
54+
# The node cert only covers localhost / 127.0.0.1, not the bridge IP.
55+
cluster['cluster']['insecure-skip-tls-verify'] = True
56+
cluster['cluster'].pop('certificate-authority-data', None)
57+
58+
with open(dst, 'w') as f:
59+
yaml.dump(cfg, f, default_flow_style=False)
60+
PYEOF

‎hack/e2e/patch-cluster-secret.sh‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
#!/usr/bin/env bash
2+
# patch-cluster-secret.sh <karmada-kubeconfig> <cluster-name> <internal-kubeconfig>
3+
#
4+
# After "karmadactl join", Karmada stores the member cluster's kubeconfig in a
5+
# Secret referenced by the Cluster object's spec.secretRef, and sets
6+
# spec.apiEndpoint to the localhost address it resolved from the external
7+
# kubeconfig. The Karmada controller manager runs inside Docker and cannot use
8+
# localhost to reach POP cell API servers.
9+
#
10+
# This script:
11+
# 1. Replaces the kubeconfig in the Secret with the Docker-IP variant so that
12+
# the Karmada controller can make API calls to the member cluster.
13+
# 2. Patches spec.apiEndpoint on the Cluster object so that health checks also
14+
# use the Docker bridge IP instead of localhost.
15+
#
16+
# Usage:
17+
# hack/e2e/patch-cluster-secret.sh \
18+
# tmp/e2e/kubeconfigs/karmada.yaml \
19+
# compute-pop-dfw \
20+
# tmp/e2e/kubeconfigs/pop-dfw-internal.yaml
21+
22+
set -euo pipefail
23+
24+
KARMADA_KUBECONFIG="${1:?usage: $0 <karmada-kubeconfig> <cluster-name> <internal-kubeconfig>}"
25+
CLUSTER_NAME="${2:?usage: $0 <karmada-kubeconfig> <cluster-name> <internal-kubeconfig>}"
26+
INTERNAL_KUBECONFIG="${3:?usage: $0 <karmada-kubeconfig> <cluster-name> <internal-kubeconfig>}"
27+
28+
# ------------------------------------------------------------------
29+
# Read the Cluster object's secretRef (name + namespace)
30+
# ------------------------------------------------------------------
31+
SECRET_NAME=$(kubectl \
32+
--kubeconfig="${KARMADA_KUBECONFIG}" \
33+
get cluster "${CLUSTER_NAME}" \
34+
-o jsonpath='{.spec.secretRef.name}' 2>/dev/null || true)
35+
36+
if [ -z "${SECRET_NAME}" ]; then
37+
echo "ERROR: Could not find spec.secretRef.name on cluster '${CLUSTER_NAME}'." >&2
38+
echo " Has karmadactl join completed successfully?" >&2
39+
exit 1
40+
fi
41+
42+
SECRET_NAMESPACE=$(kubectl \
43+
--kubeconfig="${KARMADA_KUBECONFIG}" \
44+
get cluster "${CLUSTER_NAME}" \
45+
-o jsonpath='{.spec.secretRef.namespace}' 2>/dev/null || true)
46+
47+
SECRET_NAMESPACE="${SECRET_NAMESPACE:-karmada-system}"
48+
49+
echo " Patching secret ${SECRET_NAMESPACE}/${SECRET_NAME} with Docker-IP kubeconfig..."
50+
51+
# ------------------------------------------------------------------
52+
# Replace the kubeconfig data in the secret
53+
# ------------------------------------------------------------------
54+
kubectl \
55+
--kubeconfig="${KARMADA_KUBECONFIG}" \
56+
create secret generic "${SECRET_NAME}" \
57+
--namespace="${SECRET_NAMESPACE}" \
58+
--from-file=kubeconfig="${INTERNAL_KUBECONFIG}" \
59+
--dry-run=client -o yaml \
60+
| kubectl \
61+
--kubeconfig="${KARMADA_KUBECONFIG}" \
62+
apply -f -
63+
64+
echo " Secret ${SECRET_NAMESPACE}/${SECRET_NAME} updated — Karmada controller will use Docker bridge IP"
65+
66+
# ------------------------------------------------------------------
67+
# Extract the Docker-IP server URL from the internal kubeconfig and
68+
# patch spec.apiEndpoint on the Cluster object so that Karmada's
69+
# cluster-status controller uses the same reachable address for health
70+
# checks. Without this patch the controller continues to probe the
71+
# localhost address stored by karmadactl join and the cluster never
72+
# transitions to Ready.
73+
# ------------------------------------------------------------------
74+
DOCKER_SERVER=$(kubectl \
75+
--kubeconfig="${INTERNAL_KUBECONFIG}" \
76+
config view --minify -o jsonpath='{.clusters[0].cluster.server}')
77+
78+
if [ -z "${DOCKER_SERVER}" ]; then
79+
echo "ERROR: Could not read server URL from ${INTERNAL_KUBECONFIG}" >&2
80+
exit 1
81+
fi
82+
83+
echo " Patching spec.apiEndpoint on cluster '${CLUSTER_NAME}' → ${DOCKER_SERVER}..."
84+
kubectl \
85+
--kubeconfig="${KARMADA_KUBECONFIG}" \
86+
patch cluster "${CLUSTER_NAME}" \
87+
--type=merge \
88+
-p "{\"spec\":{\"apiEndpoint\":\"${DOCKER_SERVER}\"}}"
89+
90+
echo " Cluster '${CLUSTER_NAME}' patched — health checks will now use Docker bridge IP"

‎test/e2e/chainsaw-config.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Chainsaw global configuration for the compute federation e2e test suite.
2+
#
3+
# Prerequisites
4+
# ─────────────
5+
# Run `task e2e:up` to create the Kind clusters and populate kubeconfigs under
6+
# tmp/e2e/kubeconfigs/ before running Chainsaw.
7+
#
8+
# Running
9+
# ───────
10+
# From the repository root via Taskfile (recommended):
11+
#
12+
# task e2e:test
13+
#
14+
# Or directly:
15+
#
16+
# KUBECONFIG=tmp/e2e/kubeconfigs/control-plane.yaml \
17+
# chainsaw test --config test/e2e/chainsaw-config.yaml test/e2e/
18+
#
19+
# The KUBECONFIG env var sets the "default" cluster (control-plane cell).
20+
# Additional clusters (downstream, pop-dfw, pop-ord) are declared below and
21+
# referenced by name in individual test steps via `cluster: downstream` etc.
22+
#
23+
# Kubeconfig paths below are relative to the working directory where Chainsaw is
24+
# invoked (the project root), NOT relative to this config file's location.
25+
apiVersion: chainsaw.kyverno.io/v1alpha1
26+
kind: Configuration
27+
metadata:
28+
name: chainsaw
29+
spec:
30+
timeouts:
31+
apply: 30s
32+
assert: 60s
33+
cleanup: 60s
34+
delete: 30s
35+
error: 30s
36+
exec: 30s
37+
clusters:
38+
# Downstream control plane. WorkloadDeployments, PropagationPolicies,
39+
# and Instance write-backs live here.
40+
downstream:
41+
kubeconfig: tmp/e2e/kubeconfigs/downstream.yaml
42+
# POP DFW cell — downstream member cluster labelled topology.datum.net/city-code=dfw.
43+
pop-dfw:
44+
kubeconfig: tmp/e2e/kubeconfigs/pop-dfw.yaml
45+
# POP ORD cell — downstream member cluster labelled topology.datum.net/city-code=ord.
46+
pop-ord:
47+
kubeconfig: tmp/e2e/kubeconfigs/pop-ord.yaml

0 commit comments

Comments
 (0)