Skip to content

Commit 3eda2ea

Browse files
ecvecv
committed
fix: install downstream WAF data plane in e2e CI
The E2E Tests workflow is red because the chainsaw test trafficprotectionpolicy-enforce asserts Gateway waf-gw reaches Programmed=True, but the test requests gatewayClassName datum-downstream-gateway and nothing in CI programs that class. The stock prepare-infra-cluster installs only the vanilla Envoy Gateway and the NSO CRDs, never the Coraza WAF data plane the test needs. Wire the WAF data plane into the infra-cluster setup so the datum-downstream-gateway class is served end to end on kind-nso-infra. Key changes: - Add make target downstream-waf-dataplane (a prepare-infra-cluster dependency) that installs the downstream Envoy Gateway instance, the extension server, and the datum-downstream-gateway GatewayClass plus its Coraza EnvoyProxy, loading the operator image into nso-infra. - Add config/e2e-downstream overlay: reuses the config/extension-server base and resolves its mTLS placeholders with a self-signed CA chain (cert-manager Issuer issues both server and client certs; the CA Secret is mounted directly, avoiding trust-manager). Drops the ServiceMonitor and pins replicas to one for the single-node cluster. - Resolve the envoy-gateway-downstream extensionManager placeholders to the extension server FQDN and the self-signed CA Secret. - Serve the datum-downstream-gateway GatewayClass with the downstream controllerName so the test's hardcoded class matches the installed data plane; config/e2e/config.yaml is left untouched since its datum-downstream-gateway-e2e value drives the separate gateway tests. The test's stale "skipped by CI" note is corrected to reflect that prepare-infra-cluster now installs the data plane.
1 parent 411d117 commit 3eda2ea

8 files changed

Lines changed: 200 additions & 9 deletions

File tree

‎Makefile‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,14 +153,19 @@ set-image-controller: manifests kustomize
153153
cd config/manager && $(KUSTOMIZE) edit set image ghcr.io/datum-cloud/network-services-operator=${IMG}
154154

155155
.PHONY: prepare-infra-cluster
156-
prepare-infra-cluster: cert-manager envoy-gateway external-dns downstream-crds
156+
prepare-infra-cluster: cert-manager envoy-gateway external-dns downstream-crds downstream-waf-dataplane
157157

158158
.PHONY: downstream-crds
159159
downstream-crds: ## Install NSO CRDs on the downstream (infra) cluster that the replicator mirrors into it.
160160
$(KUBECTL) apply -f config/crd/bases/networking.datumapis.com_connectors.yaml
161161
$(KUBECTL) apply -f config/crd/bases/networking.datumapis.com_httpproxies.yaml
162162
$(KUBECTL) apply -f config/crd/bases/networking.datumapis.com_trafficprotectionpolicies.yaml
163163

164+
.PHONY: downstream-waf-dataplane
165+
downstream-waf-dataplane: kustomize load-image-infra ## Install the Coraza WAF data plane (downstream EG instance + extension server + datum-downstream-gateway GatewayClass) on the infra cluster.
166+
$(KUSTOMIZE) build --enable-helm config/tools/envoy-gateway-downstream | kubectl apply --server-side=true --force-conflicts -f -
167+
$(KUSTOMIZE) build config/e2e-downstream | kubectl apply --server-side=true --force-conflicts -f -
168+
164169
.PHONY: prepare-e2e
165170
prepare-e2e: chainsaw set-image-controller cert-manager load-image-all deploy-e2e
166171

@@ -174,6 +179,10 @@ load-image-all: load-image-operator
174179
load-image-operator: docker-build kind
175180
$(KIND) load docker-image $(IMG) -n nso-standard
176181

182+
.PHONY: load-image-infra
183+
load-image-infra: docker-build kind
184+
$(KIND) load docker-image $(IMG) -n nso-infra
185+
177186
.PHONY: cert-manager
178187
cert-manager: cmctl
179188
$(KUSTOMIZE) build --enable-helm config/tools/cert-manager | kubectl apply --server-side=true --force-conflicts -f -

‎config/e2e-downstream/extension-server-config.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: apiserver.config.datumapis.com/v1alpha1
2+
kind: NetworkServicesOperator
3+
gateway:
4+
coraza:
5+
routeBaseDirectives:
6+
- Include @crs-setup-conf
7+
- Include @recommended-conf
8+
- SecResponseBodyLimitAction ProcessPartial
9+
- SecResponseBodyLimit 32000
10+
errorPage:
11+
enabled: true
12+
bodyPath: /etc/datum/error-pages/error-5xx.html

‎config/e2e-downstream/kustomization.yaml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
namespace: datum-downstream-gateway
5+
6+
resources:
7+
- namespace.yaml
8+
- ../extension-server
9+
- pki.yaml
10+
- waf-gateway.yaml
11+
12+
configMapGenerator:
13+
- name: nso-extension-server-config
14+
files:
15+
- config.yaml=extension-server-config.yaml
16+
options:
17+
disableNameSuffixHash: true
18+
19+
patches:
20+
- target:
21+
kind: ServiceMonitor
22+
name: envoy-gateway-extension-server-metrics
23+
patch: |
24+
$patch: delete
25+
apiVersion: monitoring.coreos.com/v1
26+
kind: ServiceMonitor
27+
metadata:
28+
name: envoy-gateway-extension-server-metrics
29+
- target:
30+
kind: Certificate
31+
name: envoy-gateway-extension-server-eg-client-tls
32+
patch: |
33+
- op: replace
34+
path: /spec/issuerRef/kind
35+
value: Issuer
36+
- op: replace
37+
path: /spec/issuerRef/name
38+
value: envoy-gateway-extension-server
39+
- target:
40+
kind: Deployment
41+
name: envoy-gateway-extension-server
42+
patch: |
43+
apiVersion: apps/v1
44+
kind: Deployment
45+
metadata:
46+
name: envoy-gateway-extension-server
47+
spec:
48+
replicas: 1
49+
template:
50+
spec:
51+
containers:
52+
- name: envoy-gateway-extension-server
53+
env:
54+
- name: SERVER_CONFIG
55+
value: /config/config.yaml
56+
volumeMounts:
57+
- name: server-config
58+
mountPath: /config
59+
readOnly: true
60+
volumes:
61+
- name: tls
62+
csi:
63+
volumeAttributes:
64+
csi.cert-manager.io/issuer-kind: Issuer
65+
csi.cert-manager.io/issuer-name: envoy-gateway-extension-server
66+
csi.cert-manager.io/dns-names: "envoy-gateway-extension-server.datum-downstream-gateway.svc,envoy-gateway-extension-server.datum-downstream-gateway.svc.cluster.local"
67+
- name: tls-ca
68+
configMap: null
69+
secret:
70+
secretName: envoy-gateway-extension-server-ca
71+
items:
72+
- key: ca.crt
73+
path: ca.crt
74+
- name: server-config
75+
configMap:
76+
name: nso-extension-server-config

‎config/e2e-downstream/namespace.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: datum-downstream-gateway

‎config/e2e-downstream/pki.yaml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
apiVersion: cert-manager.io/v1
3+
kind: ClusterIssuer
4+
metadata:
5+
name: envoy-gateway-extension-server-selfsigned
6+
spec:
7+
selfSigned: {}
8+
---
9+
apiVersion: cert-manager.io/v1
10+
kind: Certificate
11+
metadata:
12+
name: envoy-gateway-extension-server-ca
13+
spec:
14+
isCA: true
15+
commonName: envoy-gateway-extension-server-ca
16+
secretName: envoy-gateway-extension-server-ca
17+
privateKey:
18+
algorithm: ECDSA
19+
size: 256
20+
issuerRef:
21+
name: envoy-gateway-extension-server-selfsigned
22+
kind: ClusterIssuer
23+
group: cert-manager.io
24+
---
25+
apiVersion: cert-manager.io/v1
26+
kind: Issuer
27+
metadata:
28+
name: envoy-gateway-extension-server
29+
spec:
30+
ca:
31+
secretName: envoy-gateway-extension-server-ca

‎config/e2e-downstream/waf-gateway.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
apiVersion: gateway.envoyproxy.io/v1alpha1
3+
kind: EnvoyProxy
4+
metadata:
5+
name: datum-downstream-gateway
6+
spec:
7+
mergeGateways: true
8+
ipFamily: DualStack
9+
provider:
10+
type: Kubernetes
11+
kubernetes:
12+
envoyDeployment:
13+
pod:
14+
volumes:
15+
- name: coraza-waf
16+
emptyDir: {}
17+
container:
18+
image: envoyproxy/envoy:contrib-v1.35.0
19+
volumeMounts:
20+
- name: coraza-waf
21+
mountPath: /opt/coraza-waf
22+
initContainers:
23+
- name: coraza-waf
24+
image: ghcr.io/datum-labs/coraza-envoy-go-filter/coraza-waf:v1.3.0-alpha10
25+
imagePullPolicy: IfNotPresent
26+
command:
27+
- cp
28+
- /app/coraza-waf.so
29+
- /opt/coraza-waf/
30+
volumeMounts:
31+
- name: coraza-waf
32+
mountPath: /opt/coraza-waf
33+
envoyService:
34+
externalTrafficPolicy: Cluster
35+
type: NodePort
36+
patch:
37+
type: StrategicMerge
38+
value:
39+
spec:
40+
ipFamilyPolicy: RequireDualStack
41+
ports:
42+
- name: http-80
43+
nodePort: 30080
44+
port: 80
45+
- name: https-443
46+
nodePort: 30443
47+
port: 443
48+
---
49+
apiVersion: gateway.networking.k8s.io/v1
50+
kind: GatewayClass
51+
metadata:
52+
name: datum-downstream-gateway
53+
spec:
54+
controllerName: gateway.envoyproxy.io/datum-downstream-gateway
55+
parametersRef:
56+
group: gateway.envoyproxy.io
57+
kind: EnvoyProxy
58+
name: datum-downstream-gateway
59+
namespace: datum-downstream-gateway

‎config/tools/envoy-gateway-downstream/kustomization.yaml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -39,16 +39,17 @@ helmCharts:
3939
kind: Connector
4040
service:
4141
fqdn:
42-
hostname: network-services-operator-envoy-gateway-extension-server.network-services-operator-system.svc.cluster.local
42+
hostname: envoy-gateway-extension-server.datum-downstream-gateway.svc.cluster.local
4343
port: 5005
4444
tls:
45+
# EG reads the CA from tls.crt of this Secret; the self-signed CA
46+
# Certificate's Secret carries the CA cert there.
4547
certificateRef:
46-
# Placeholder — an overlay must patch this to the Secret holding the CA that issued the server cert.
47-
name: placeholder-ca
48-
namespace: placeholder-namespace
48+
name: envoy-gateway-extension-server-ca
49+
namespace: datum-downstream-gateway
4950
clientCertificateRef:
5051
name: envoy-gateway-extension-server-eg-client-tls
51-
namespace: network-services-operator-system
52+
namespace: datum-downstream-gateway
5253
retry:
5354
maxAttempts: 4
5455
initialBackoff: 100ms

‎test/e2e/trafficprotectionpolicy-enforce/chainsaw-test.yaml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,8 @@ metadata:
1212
#
1313
# Precondition: the downstream (nso-infra) must have the WAF data plane wired up
1414
# (extension-server + the extensionManager Envoy Gateway registered on the
15-
# `datum-downstream-gateway` GatewayClass, with the Coraza filter). The stock
16-
# `make prepare-infra-cluster` does NOT install this; this test is skipped by
17-
# CI environments without it (the Gateway never reaches Programmed=True).
15+
# `datum-downstream-gateway` GatewayClass, with the Coraza filter).
16+
# `make prepare-infra-cluster` installs this via `make downstream-waf-dataplane`.
1817
spec:
1918
cluster: nso-infra
2019
# EG only reconciles namespaces carrying this label.

0 commit comments

Comments
 (0)