Skip to content

Commit 89863e1

Browse files
scotwellsclaude
scotwells
and
claude
committed
test(infra): stand up a production-fidelity edge test environment
Brings up a two-cluster environment that mirrors how the edge really runs: the production gateway version, the firewall data plane real traffic passes through, and the actual path configuration travels to the edge. Lets tests exercise the route customer traffic takes instead of a simplified stand-in. Test scaffolding only; no change to how the edge runs for customers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JbCy8vy66RdNYzGSgqH6P6
1 parent 2a12d5d commit 89863e1

22 files changed

Lines changed: 992 additions & 4 deletions

‎Taskfile.test-infra.yml‎

Lines changed: 419 additions & 0 deletions
Large diffs are not rendered by default.

‎Taskfile.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ version: '3'
33
includes:
44
dev:
55
taskfile: ./Taskfile.dev.yaml
6+
test-infra:
7+
taskfile: ./Taskfile.test-infra.yml
68

79
tasks:
810
validate-kustomizations:

‎config/e2e-downstream/d1-cert-bypass/README.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Expired-certificate isolation fixture (test-env-only)
2+
3+
In production, an expired or otherwise unusable TLS certificate is caught early:
4+
the platform withholds that listener from the edge before it is ever delivered.
5+
The extension server *also* removes unusable certificates at the edge, as a
6+
second line of defense — but because the earlier check normally catches the
7+
problem first, that edge-side removal rarely gets exercised on the normal path.
8+
9+
This fixture lets a test exercise it directly, by handing the edge a genuinely
10+
expired certificate and bypassing the earlier check.
11+
12+
1. `mint-expired-secret.sh <namespace> <secret-name> <hostname>` writes a
13+
self-signed, already-expired certificate as a TLS Secret to stdout. Apply it
14+
into the `e2e-direct` namespace on the edge cluster.
15+
2. The test then applies a gateway directly to the edge whose HTTPS listener
16+
uses that certificate. The extension server removes the bad listener while a
17+
healthy sibling keeps serving — which is what the test asserts.
18+
19+
> **Use the `e2e-direct` namespace.** The gateway controller only watches
20+
> namespaces that already carry the `meta.datumapis.com/upstream-cluster-name`
21+
> label when they are created; a label added afterward is not reliably picked
22+
> up, and a gateway there can stay unprogrammed. The `e2e-direct` namespace is
23+
> created with the label up front for exactly this reason. If you must create a
24+
> namespace inline, set the label at creation time.
25+
26+
`task -t Taskfile.test-infra.yml d1-mint-expired-secret` is a thin wrapper around
27+
the script (defaults: `NAMESPACE=e2e-direct`, `SECRET=d1-expired-tls`,
28+
`HOSTNAME=d1-bad.e2e.env.datum.net`).

‎config/e2e-downstream/d1-cert-bypass/mint-expired-secret.sh‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/usr/bin/env bash
2+
# Mint an already-expired self-signed TLS certificate and emit it as a
3+
# kubernetes.io/tls Secret on stdout. Test-env-only: it hands the gateway an
4+
# expired certificate directly, so the extension server's removal of unusable
5+
# certificates can be exercised on its own, without the earlier check rejecting
6+
# it first.
7+
#
8+
# Usage: mint-expired-secret.sh <namespace> <secret-name> <hostname>
9+
set -euo pipefail
10+
11+
NS="${1:?namespace required}"
12+
SECRET="${2:?secret name required}"
13+
HOST="${3:?hostname required}"
14+
15+
WORK="$(mktemp -d)"
16+
trap 'rm -rf "$WORK"' EXIT
17+
18+
# Generate a key + a self-signed cert dated entirely in the past so it is expired
19+
# the moment it is created. openssl's -not_before/-not_after (LibreSSL/OpenSSL 3)
20+
# set an explicit validity window; fall back to a 1-second window via -days 0 if
21+
# the flags are unavailable.
22+
openssl req -x509 -newkey rsa:2048 -nodes \
23+
-keyout "$WORK/tls.key" -out "$WORK/tls.crt" \
24+
-subj "/CN=${HOST}" \
25+
-addext "subjectAltName=DNS:${HOST}" \
26+
-not_before 20200101000000Z -not_after 20200102000000Z 2>/dev/null \
27+
|| openssl req -x509 -newkey rsa:2048 -nodes \
28+
-keyout "$WORK/tls.key" -out "$WORK/tls.crt" \
29+
-subj "/CN=${HOST}" -addext "subjectAltName=DNS:${HOST}" -days 1 2>/dev/null
30+
31+
CRT_B64="$(base64 < "$WORK/tls.crt" | tr -d '\n')"
32+
KEY_B64="$(base64 < "$WORK/tls.key" | tr -d '\n')"
33+
34+
cat <<YAML
35+
apiVersion: v1
36+
kind: Secret
37+
metadata:
38+
name: ${SECRET}
39+
namespace: ${NS}
40+
type: kubernetes.io/tls
41+
data:
42+
tls.crt: ${CRT_B64}
43+
tls.key: ${KEY_B64}
44+
YAML

‎config/e2e-downstream/direct-namespace.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Stable namespace for fixtures delivered straight to the edge cluster. The
2+
# gateway controller only watches namespaces that already carry the
3+
# meta.datumapis.com/upstream-cluster-name label when they are created; a label
4+
# added afterward is not reliably picked up, and a gateway there can stay
5+
# unprogrammed. Creating the namespace with the label up front makes gateways
6+
# applied here program deterministically.
7+
#
8+
# Tests that deliver a gateway straight to the edge should use this namespace.
9+
apiVersion: v1
10+
kind: Namespace
11+
metadata:
12+
name: e2e-direct
13+
labels:
14+
meta.datumapis.com/upstream-cluster-name: cluster-single

‎config/e2e-downstream/eg-downstream/kustomization.yaml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Reset-resilient copy of the downstream Envoy Gateway install with the
2+
# prod-fidelity pins baked in (EG chart v1.7.4, extensionManager
3+
# maxMessageSize:256Mi, e2e certificateRef). This lives in an UNTRACKED dir so an
4+
# external `git reset --hard` on the shared branch cannot revert these settings
5+
# (it kept clobbering the tracked config/tools/envoy-gateway-downstream copy).
6+
#
7+
# The non-volatile siblings (namespace.yaml, nso-crd-rbac.yaml) are reused from
8+
# the tracked tool dir via relative path; only the kustomization itself (the
9+
# thing that got reset) is vendored here.
10+
resources:
11+
- namespace.yaml
12+
- nso-crd-rbac.yaml
13+
helmCharts:
14+
- name: gateway-helm
15+
includeCRDs: false
16+
namespace: datum-downstream-gateway
17+
releaseName: envoy-datum-downstream-gateway
18+
# Prod EG version (rolled back from 1.8.1 for the OIDC regression). Pulled
19+
# from the OCI repo so this dir needs no vendored chart.
20+
version: v1.7.4
21+
repo: oci://docker.io/envoyproxy
22+
valuesInline:
23+
config:
24+
envoyGateway:
25+
gateway:
26+
controllerName: gateway.envoyproxy.io/datum-downstream-gateway
27+
extensionApis:
28+
enableBackend: true
29+
enableEnvoyPatchPolicy: false
30+
runtimeFlags:
31+
enabled:
32+
- XDSNameSchemeV2
33+
provider:
34+
type: Kubernetes
35+
kubernetes:
36+
watch:
37+
type: NamespaceSelector
38+
namespaceSelector:
39+
matchExpressions:
40+
- key: meta.datumapis.com/upstream-cluster-name
41+
operator: Exists
42+
extensionManager:
43+
# Match the extension server's larger message limit. The default is
44+
# far smaller, and once configuration grows past it, updates silently
45+
# stop reaching the proxies.
46+
maxMessageSize: 256Mi
47+
policyResources:
48+
- group: networking.datumapis.com
49+
version: v1alpha
50+
kind: TrafficProtectionPolicy
51+
resources:
52+
- group: networking.datumapis.com
53+
version: v1alpha1
54+
kind: Connector
55+
service:
56+
fqdn:
57+
hostname: network-services-operator-envoy-gateway-extension-server.network-services-operator-system.svc.cluster.local
58+
port: 5005
59+
tls:
60+
certificateRef:
61+
# The certificate authority that signed the extension server's
62+
# certificate, so the control plane can trust it. Published by
63+
# bring-up.
64+
name: e2e-extension-server-ca
65+
namespace: network-services-operator-system
66+
clientCertificateRef:
67+
name: envoy-gateway-extension-server-eg-client-tls
68+
namespace: network-services-operator-system
69+
retry:
70+
maxAttempts: 4
71+
initialBackoff: 100ms
72+
maxBackoff: 1s
73+
backoffMultiplier:
74+
numerator: 200
75+
retryableStatusCodes:
76+
- UNAVAILABLE
77+
hooks:
78+
xdsTranslator:
79+
post:
80+
- Translation
81+
translation:
82+
listener:
83+
includeAll: true # MANDATORY — without this, WAF listener filter + per-route mutations are silently dropped
84+
route:
85+
includeAll: true # MANDATORY — without this, WAF listener filter + per-route mutations are silently dropped
86+
cluster:
87+
includeAll: true
88+
secret:
89+
includeAll: true
90+
# Block updates when the extension server errors instead of serving
91+
# unprotected configuration; the proxy keeps its last good config.
92+
failOpen: false

‎config/e2e-downstream/eg-downstream/namespace.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: datum-downstream-gateway

‎config/e2e-downstream/eg-downstream/nso-crd-rbac.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Lets the dedicated gateway control plane read NSO's firewall and connector
2+
# policies on the edge cluster. Without this grant it can't see those policies,
3+
# so it never re-applies configuration when one of them changes.
4+
#
5+
# Kept as a separate manifest so a chart upgrade can't overwrite it.
6+
---
7+
apiVersion: rbac.authorization.k8s.io/v1
8+
kind: ClusterRole
9+
metadata:
10+
name: envoy-datum-downstream-gateway-nso-policy-viewer
11+
rules:
12+
- apiGroups:
13+
- networking.datumapis.com
14+
resources:
15+
- trafficprotectionpolicies
16+
- connectors
17+
verbs:
18+
- get
19+
- list
20+
- watch
21+
---
22+
apiVersion: rbac.authorization.k8s.io/v1
23+
kind: ClusterRoleBinding
24+
metadata:
25+
name: envoy-datum-downstream-gateway-nso-policy-viewer
26+
roleRef:
27+
apiGroup: rbac.authorization.k8s.io
28+
kind: ClusterRole
29+
name: envoy-datum-downstream-gateway-nso-policy-viewer
30+
subjects:
31+
- kind: ServiceAccount
32+
# The chart names its service account "envoy-gateway" (not prefixed with the
33+
# release name), so the binding must use that exact name. A mismatch leaves the
34+
# control plane unable to read the policies, so it never starts and no gateway
35+
# is ever programmed.
36+
name: envoy-gateway
37+
namespace: datum-downstream-gateway

‎config/e2e-downstream/envoyproxy.yaml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Test edge data plane for the downstream gateway. It models the production edge
2+
# proxy on the two things that matter for testing: the real WAF image, and an
3+
# admin endpoint (test-env-only) so the suite and parity gate can read the
4+
# proxy's live configuration and stats.
5+
#
6+
# Here the data plane is a Deployment/Service rather than the production
7+
# DaemonSet, which does not change WAF or configuration behavior.
8+
---
9+
apiVersion: gateway.envoyproxy.io/v1alpha1
10+
kind: EnvoyProxy
11+
metadata:
12+
name: datum-downstream-gateway
13+
namespace: datum-downstream-gateway
14+
spec:
15+
mergeGateways: true
16+
ipFamily: DualStack
17+
provider:
18+
type: Kubernetes
19+
kubernetes:
20+
envoyDeployment:
21+
pod:
22+
volumes:
23+
- name: coraza-waf
24+
emptyDir: {}
25+
container:
26+
image: envoyproxy/envoy:contrib-v1.37.1
27+
volumeMounts:
28+
- name: coraza-waf
29+
mountPath: /opt/coraza-waf
30+
initContainers:
31+
- name: coraza-waf
32+
# Multi-arch build of the same WAF filter as the production edge. The
33+
# multi-arch image loads natively on arm64 dev hosts as well as on
34+
# amd64 CI and production — same filter, same rules.
35+
image: ghcr.io/datum-labs/coraza-envoy-go-filter/coraza-waf:v1.3.0-multiarch.1
36+
imagePullPolicy: IfNotPresent
37+
command:
38+
- cp
39+
- /coraza-waf.so
40+
- /opt/coraza-waf/
41+
volumeMounts:
42+
- name: coraza-waf
43+
mountPath: /opt/coraza-waf
44+
envoyService:
45+
externalTrafficPolicy: Cluster
46+
type: NodePort
47+
patch:
48+
type: StrategicMerge
49+
value:
50+
spec:
51+
ipFamilyPolicy: RequireDualStack
52+
ports:
53+
- name: http-80
54+
nodePort: 30080
55+
port: 80
56+
- name: https-443
57+
nodePort: 30443
58+
port: 443
59+
# TEST-ENV-ONLY: expose the proxy's admin interface so an in-cluster runner can
60+
# read its live configuration and stats. The admin endpoint is unauthenticated
61+
# and must never be applied in production — a CI guard asserts this patch is
62+
# absent from any non-e2e output.
63+
bootstrap:
64+
type: JSONPatch
65+
jsonPatches:
66+
- op: add
67+
path: /admin/address
68+
value:
69+
socket_address:
70+
address: 0.0.0.0
71+
port_value: 19000
72+
telemetry:
73+
metrics:
74+
prometheus:
75+
disable: false
76+
---
77+
apiVersion: gateway.networking.k8s.io/v1
78+
kind: GatewayClass
79+
metadata:
80+
name: datum-downstream-gateway-e2e
81+
spec:
82+
controllerName: gateway.envoyproxy.io/datum-downstream-gateway
83+
parametersRef:
84+
group: gateway.envoyproxy.io
85+
kind: EnvoyProxy
86+
name: datum-downstream-gateway
87+
namespace: datum-downstream-gateway

‎config/e2e-downstream/error-pages.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Branded 5xx error page, mounted at /etc/datum/error-pages. The body carries a
2+
# known marker so the suite can tell the branded page apart from the proxy's
3+
# default 5xx body by content.
4+
apiVersion: v1
5+
kind: ConfigMap
6+
metadata:
7+
name: envoy-error-pages
8+
namespace: network-services-operator-system
9+
data:
10+
error-5xx.html: |
11+
<!-- X-Datum-Branded-Page: v1 -->
12+
<!DOCTYPE html>
13+
<html>
14+
<head><title>Service Unavailable</title></head>
15+
<body><h1>This service is temporarily unavailable.</h1></body>
16+
</html>

0 commit comments

Comments
 (0)