Skip to content

Commit fb89033

Browse files
andreademurtasandreademurtas
authored
feat(xds): support SNI based dynamic forward proxy for TLSRoute (#9184)
* feat(xds): support SNI based dynamic forward proxy for TLSRoute Allow a Backend of type DynamicResolver to be referenced by a TLSRoute so Envoy Gateway can act as an SNI based dynamic forward proxy. The upstream host is resolved from the TLS SNI of the incoming connection and the raw TLS bytes are forwarded unchanged (passthrough). - Add the envoy.filters.network.sni_dynamic_forward_proxy network filter to the TCP filter chain of dynamic resolver routes, sharing the same DNS cache as the dynamic forward proxy cluster (fixed upstream port 443). - Ensure the tls_inspector listener filter is present so the SNI is available. - Deny loopback SNIs with a network RBAC filter to mitigate SSRF, mirroring the loopback protection of the HTTP dynamic forward proxy. - Reject a dynamic resolver combined with multiple backendRefs on a TLSRoute. Fixes #8787 Signed-off-by: Andrea De Murtas <56048157+andreademurtas@users.noreply.github.com> * fix(xds): restrict SNI dynamic forward proxy to TLS passthrough A dynamic resolver backend forwards the connection based on the SNI and requires TLS passthrough. When the parent listener terminates TLS, Envoy consumes the downstream handshake before tcp_proxy and would forward the decrypted stream to the SNI host on port 443, breaking the intended passthrough forwarding. Reject a dynamic resolver backend on a TLS terminate listener during translation (ResolvedRefs=False) and gate the SNI dynamic forward proxy filter to passthrough routes only. Addresses review feedback on #8787. Signed-off-by: Andrea De Murtas <56048157+andreademurtas@users.noreply.github.com> * test(e2e): add TLSRoute SNI dynamic forward proxy e2e test Add an end-to-end test verifying that a TLSRoute referencing a DynamicResolver backend forwards based on the SNI. A dedicated TLS passthrough backend serves a self-signed certificate whose SAN matches its in-cluster FQDN, so the SNI used to route also resolves the upstream and is validated by the client. Signed-off-by: Andrea De Murtas <56048157+andreademurtas@users.noreply.github.com> --------- Signed-off-by: Andrea De Murtas <56048157+andreademurtas@users.noreply.github.com>
1 parent 8782e63 commit fb89033

19 files changed

Lines changed: 1148 additions & 1 deletion

‎internal/gatewayapi/route.go‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1485,6 +1485,25 @@ func (t *Translator) processTLSRouteParentRefs(tlsRoute *TLSRouteContext, resour
14851485
// - etc.
14861486
}
14871487

1488+
// A route can only have a single destination if that destination is a dynamic resolver,
1489+
// because combining a dynamic resolver with other backends doesn't make sense.
1490+
hasDynamicResolver := false
1491+
for _, ds := range destSettings {
1492+
if ds.IsDynamicResolver {
1493+
hasDynamicResolver = true
1494+
break
1495+
}
1496+
}
1497+
if hasDynamicResolver && len(destSettings) > 1 {
1498+
resolveErrs.Add(status.NewRouteStatusError(
1499+
errors.New("dynamic resolver is not supported for multiple backendRefs"),
1500+
status.RouteReasonInvalidBackendRef,
1501+
))
1502+
// Drop the destinations so neither a dynamic forward proxy cluster nor a regular
1503+
// cluster is produced from an invalid combination of backends.
1504+
destSettings = nil
1505+
}
1506+
14881507
routeStatus := GetRouteStatus(tlsRoute)
14891508
if !resolveErrs.Empty() {
14901509
status.SetRouteStatusCondition(routeStatus,
@@ -1524,7 +1543,21 @@ func (t *Translator) processTLSRouteParentRefs(tlsRoute *TLSRouteContext, resour
15241543
if irListener != nil {
15251544
var tlsConfig *ir.TLS
15261545
if irListener.TLS != nil {
1527-
// Listener is in terminate mode.
1546+
// Listener is in terminate mode. A dynamic resolver backend forwards the connection
1547+
// based on the SNI and requires TLS passthrough, so it cannot be used with a listener
1548+
// that terminates TLS (Envoy would forward the decrypted stream instead).
1549+
if hasDynamicResolver {
1550+
routeStatus := GetRouteStatus(tlsRoute)
1551+
status.SetRouteStatusCondition(routeStatus,
1552+
parentRef.routeParentStatusIdx,
1553+
tlsRoute.GetGeneration(),
1554+
gwapiv1.RouteConditionResolvedRefs,
1555+
metav1.ConditionFalse,
1556+
gwapiv1.RouteReasonUnsupportedValue,
1557+
"Dynamic resolver backend is only supported with TLS passthrough listeners",
1558+
)
1559+
continue
1560+
}
15281561
tlsConfig = &ir.TLS{
15291562
Terminate: irListener.TLS,
15301563
}

‎internal/gatewayapi/testdata/tlsroute-dynamic-resolver-with-multiple-backends.in.yaml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
gateways:
2+
- apiVersion: gateway.networking.k8s.io/v1
3+
kind: Gateway
4+
metadata:
5+
namespace: envoy-gateway
6+
name: gateway-1
7+
spec:
8+
gatewayClassName: envoy-gateway-class
9+
listeners:
10+
- name: tls
11+
protocol: TLS
12+
port: 443
13+
tls:
14+
mode: Passthrough
15+
allowedRoutes:
16+
namespaces:
17+
from: All
18+
tlsRoutes:
19+
- apiVersion: gateway.networking.k8s.io/v1alpha2
20+
kind: TLSRoute
21+
metadata:
22+
namespace: default
23+
name: tlsroute-1
24+
spec:
25+
parentRefs:
26+
- namespace: envoy-gateway
27+
name: gateway-1
28+
hostnames:
29+
- "foo.com"
30+
rules:
31+
- backendRefs:
32+
- group: gateway.envoyproxy.io
33+
kind: Backend
34+
name: backend-dynamic-resolver
35+
- name: service-1
36+
port: 8080
37+
backends:
38+
- apiVersion: gateway.envoyproxy.io/v1alpha1
39+
kind: Backend
40+
metadata:
41+
name: backend-dynamic-resolver
42+
namespace: default
43+
spec:
44+
type: DynamicResolver

‎internal/gatewayapi/testdata/tlsroute-dynamic-resolver-with-multiple-backends.out.yaml‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
backends:
2+
- apiVersion: gateway.envoyproxy.io/v1alpha1
3+
kind: Backend
4+
metadata:
5+
name: backend-dynamic-resolver
6+
namespace: default
7+
spec:
8+
type: DynamicResolver
9+
status:
10+
conditions:
11+
- lastTransitionTime: null
12+
message: The Backend was accepted
13+
reason: Accepted
14+
status: "True"
15+
type: Accepted
16+
gateways:
17+
- apiVersion: gateway.networking.k8s.io/v1
18+
kind: Gateway
19+
metadata:
20+
name: gateway-1
21+
namespace: envoy-gateway
22+
spec:
23+
gatewayClassName: envoy-gateway-class
24+
listeners:
25+
- allowedRoutes:
26+
namespaces:
27+
from: All
28+
name: tls
29+
port: 443
30+
protocol: TLS
31+
tls:
32+
mode: Passthrough
33+
status:
34+
listeners:
35+
- attachedRoutes: 1
36+
conditions:
37+
- lastTransitionTime: null
38+
message: Sending translated listener configuration to the data plane
39+
reason: Programmed
40+
status: "True"
41+
type: Programmed
42+
- lastTransitionTime: null
43+
message: Listener has been successfully translated
44+
reason: Accepted
45+
status: "True"
46+
type: Accepted
47+
- lastTransitionTime: null
48+
message: Listener references have been resolved
49+
reason: ResolvedRefs
50+
status: "True"
51+
type: ResolvedRefs
52+
name: tls
53+
supportedKinds:
54+
- group: gateway.networking.k8s.io
55+
kind: TLSRoute
56+
infraIR:
57+
envoy-gateway/gateway-1:
58+
proxy:
59+
listeners:
60+
- name: envoy-gateway/gateway-1/tls
61+
ports:
62+
- containerPort: 10443
63+
name: tls-443
64+
protocol: TLS
65+
servicePort: 443
66+
metadata:
67+
labels:
68+
gateway.envoyproxy.io/owning-gateway-name: gateway-1
69+
gateway.envoyproxy.io/owning-gateway-namespace: envoy-gateway
70+
ownerReference:
71+
kind: GatewayClass
72+
name: envoy-gateway-class
73+
name: envoy-gateway/gateway-1
74+
namespace: envoy-gateway-system
75+
tlsRoutes:
76+
- apiVersion: gateway.networking.k8s.io/v1alpha2
77+
kind: TLSRoute
78+
metadata:
79+
name: tlsroute-1
80+
namespace: default
81+
spec:
82+
hostnames:
83+
- foo.com
84+
parentRefs:
85+
- name: gateway-1
86+
namespace: envoy-gateway
87+
rules:
88+
- backendRefs:
89+
- group: gateway.envoyproxy.io
90+
kind: Backend
91+
name: backend-dynamic-resolver
92+
- name: service-1
93+
port: 8080
94+
status:
95+
parents:
96+
- conditions:
97+
- lastTransitionTime: null
98+
message: Route is accepted
99+
reason: Accepted
100+
status: "True"
101+
type: Accepted
102+
- lastTransitionTime: null
103+
message: dynamic resolver is not supported for multiple backendRefs
104+
reason: InvalidBackendRef
105+
status: "False"
106+
type: ResolvedRefs
107+
controllerName: gateway.envoyproxy.io/gatewayclass-controller
108+
parentRef:
109+
name: gateway-1
110+
namespace: envoy-gateway
111+
xdsIR:
112+
envoy-gateway/gateway-1:
113+
accessLog:
114+
json:
115+
- path: /dev/stdout
116+
globalResources:
117+
proxyServiceCluster:
118+
metadata:
119+
kind: Service
120+
name: envoy-envoy-gateway-gateway-1-196ae069
121+
namespace: envoy-gateway-system
122+
sectionName: "8080"
123+
name: envoy-gateway/gateway-1
124+
settings:
125+
- addressType: IP
126+
endpoints:
127+
- host: 7.6.5.4
128+
port: 8080
129+
zone: zone1
130+
metadata:
131+
kind: Service
132+
name: envoy-envoy-gateway-gateway-1-196ae069
133+
namespace: envoy-gateway-system
134+
sectionName: "8080"
135+
name: envoy-gateway/gateway-1
136+
protocol: TCP
137+
readyListener:
138+
address: 0.0.0.0
139+
ipFamily: IPv4
140+
path: /ready
141+
port: 19003
142+
tcp:
143+
- address: 0.0.0.0
144+
externalPort: 443
145+
metadata:
146+
kind: Gateway
147+
name: gateway-1
148+
namespace: envoy-gateway
149+
sectionName: tls
150+
name: envoy-gateway/gateway-1/tls
151+
port: 10443
152+
routes:
153+
- destination:
154+
metadata:
155+
kind: TLSRoute
156+
name: tlsroute-1
157+
namespace: default
158+
name: tlsroute/default/tlsroute-1/rule/-1
159+
metadata:
160+
kind: TLSRoute
161+
name: tlsroute-1
162+
namespace: default
163+
name: tlsroute/default/tlsroute-1
164+
tls:
165+
inspector:
166+
snis:
167+
- foo.com

‎internal/gatewayapi/testdata/tlsroute-dynamic-resolver-with-tls-terminate.in.yaml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
gateways:
2+
- apiVersion: gateway.networking.k8s.io/v1
3+
kind: Gateway
4+
metadata:
5+
namespace: envoy-gateway
6+
name: gateway-1
7+
spec:
8+
gatewayClassName: envoy-gateway-class
9+
listeners:
10+
- name: tls
11+
protocol: TLS
12+
port: 90
13+
tls:
14+
certificateRefs:
15+
- group: ""
16+
kind: Secret
17+
name: tls-secret-1
18+
mode: Terminate
19+
allowedRoutes:
20+
namespaces:
21+
from: All
22+
tlsRoutes:
23+
- apiVersion: gateway.networking.k8s.io/v1alpha2
24+
kind: TLSRoute
25+
metadata:
26+
namespace: default
27+
name: tlsroute-1
28+
spec:
29+
parentRefs:
30+
- namespace: envoy-gateway
31+
name: gateway-1
32+
sectionName: tls
33+
hostnames:
34+
- "foo.com"
35+
rules:
36+
- backendRefs:
37+
- group: gateway.envoyproxy.io
38+
kind: Backend
39+
name: backend-dynamic-resolver
40+
41+
backends:
42+
- apiVersion: gateway.envoyproxy.io/v1alpha1
43+
kind: Backend
44+
metadata:
45+
name: backend-dynamic-resolver
46+
namespace: default
47+
spec:
48+
type: DynamicResolver
49+
50+
secrets:
51+
- apiVersion: v1
52+
kind: Secret
53+
metadata:
54+
namespace: envoy-gateway
55+
name: tls-secret-1
56+
type: kubernetes.io/tls
57+
data:
58+
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREVENDQWZXZ0F3SUJBZ0lVRUZNaFA5ZUo5WEFCV3NRNVptNmJSazJjTE5Rd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZqRVVNQklHQTFVRUF3d0xabTl2TG1KaGNpNWpiMjB3SGhjTk1qUXdNakk1TURrek1ERXdXaGNOTXpRdwpNakkyTURrek1ERXdXakFXTVJRd0VnWURWUVFEREF0bWIyOHVZbUZ5TG1OdmJUQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFKbEk2WXhFOVprQ1BzNnBDUXhickNtZWl4OVA1RGZ4OVJ1NUxENFQKSm1kVzdJS2R0UVYvd2ZMbXRzdTc2QithVGRDaldlMEJUZmVPT1JCYlIzY1BBRzZFbFFMaWNsUVVydW4zcStncwpKcEsrSTdjSStqNXc4STY4WEg1V1E3clZVdGJ3SHBxYncrY1ZuQnFJVU9MaUlhdGpJZjdLWDUxTTF1RjljZkVICkU0RG5jSDZyYnI1OS9SRlpCc2toeHM1T3p3Sklmb2hreXZGd2V1VHd4Sy9WcGpJKzdPYzQ4QUJDWHBOTzlEL3EKRWgrck9hdWpBTWNYZ0hRSVRrQ2lpVVRjVW82TFNIOXZMWlB0YXFmem9acTZuaE1xcFc2NUUxcEF3RjNqeVRUeAphNUk4SmNmU0Zqa2llWjIwTFVRTW43TThVNHhIamFvL2d2SDBDQWZkQjdSTFUyc0NBd0VBQWFOVE1GRXdIUVlEClZSME9CQllFRk9SQ0U4dS8xRERXN2loWnA3Y3g5dFNtUG02T01COEdBMVVkSXdRWU1CYUFGT1JDRTh1LzFERFcKN2loWnA3Y3g5dFNtUG02T01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQgpBRnQ1M3pqc3FUYUg1YThFMmNodm1XQWdDcnhSSzhiVkxNeGl3TkdqYm1FUFJ6K3c2TngrazBBOEtFY0lEc0tjClNYY2k1OHU0b1didFZKQmx6YS9adWpIUjZQMUJuT3BsK2FveTc4NGJiZDRQMzl3VExvWGZNZmJCQ20xdmV2aDkKQUpLbncyWnRxcjRta2JMY3hFcWxxM3NCTEZBUzlzUUxuS05DZTJjR0xkVHAyYm9HK3FjZ3lRZ0NJTTZmOEVNdgpXUGlmQ01NR3V6Sy9HUkY0YlBPL1lGNDhld0R1M1VlaWgwWFhkVUFPRTlDdFVhOE5JaGMxVVBhT3pQcnRZVnFyClpPR2t2L0t1K0I3OGg4U0VzTzlYclFjdXdiT25KeDZLdFIrYWV5a3ZBcFhDUTNmWkMvYllLQUFSK1A4QUpvUVoKYndJVW1YaTRnajVtK2JLUGhlK2lyK0U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0=
59+
tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQ2QwZlBDYWtweE1nUnUKT0VXQjFiQk5FM3ZseW55aTZWbkV2VWF1OUhvakR2UHVPTFJIaGI4MmoyY1ovMHhnL1lKR09LelBuV2JERkxGNApHdWh3dDRENmFUR0xYNklPODEwTDZ0SXZIWGZNUXRJS2VwdTZ3K3p1WVo4bG1yejB1RjZlWEtqamVIbHhyb2ZrCnVNekM3OUVaU0lYZlZlczJ1SmdVRSs4VGFzSDUzQ2Y4MFNSRGlIeEdxckttdVNjWCtwejBreGdCZ1VWYTVVS20KUWdTZDFmVUxLOUEwNXAxOXkrdURPM204bVhRNkxVQ0N1STFwZHNROGFlNS9zamlxa0VjWlJjMTdWYVgxWjVVaQpvcGZnNW9SY05VTG9VTHNiek9aNTR0YlVDUmdSV2VLbGZxaElINEZ6OUlkVlUyR3dFdEdhMmV6TjgyMVBaQ3QzCjZhbVRIelJsQWdNQkFBRUNnZ0VBWTFGTUlLNDVXTkVNUHJ6RTZUY3NNdVV2RkdhQVZ4bVk5NW5SMEtwajdvb3IKY21CVys2ZXN0TTQ4S1AwaitPbXd3VFpMY29Cd3VoWGN0V1Bob1lXcDhteWUxRUlEdjNyaHRHMDdocEQ1NGg2dgpCZzh3ejdFYStzMk9sT0N6UnlKNzBSY281YlhjWDNGaGJjdnFlRWJwaFFyQnpOSEtLMjZ4cmZqNWZIT3p6T1FGCmJHdUZ3SDVic3JGdFhlajJXM3c4eW90N0ZQSDV3S3RpdnhvSWU5RjMyOXNnOU9EQnZqWnpiaG1LVTArckFTK1kKRGVield2bFJyaEUrbXVmQTN6M0N0QXhDOFJpNzNscFNoTDRQQWlvcG1SUXlxZXRXMjYzOFFxcnM0R3hnNzhwbApJUXJXTmNBc2s3Slg5d3RZenV6UFBXSXRWTTFscFJiQVRhNTJqdFl2NVFLQmdRRE5tMTFtZTRYam1ZSFV2cStZCmFTUzdwK2UybXZEMHVaOU9JeFluQnBWMGkrckNlYnFFMkE1Rm5hcDQ5Yld4QTgwUElldlVkeUpCL2pUUkoxcVMKRUpXQkpMWm1LVkg2K1QwdWw1ZUtOcWxFTFZHU0dCSXNpeE9SUXpDZHBoMkx0UmtBMHVjSVUzY3hiUmVMZkZCRQpiSkdZWENCdlNGcWd0VDlvZTFldVpMVmFOd0tCZ1FERWdENzJENk81eGIweEQ1NDQ1M0RPMUJhZmd6aThCWDRTCk1SaVd2LzFUQ0w5N05sRWtoeXovNmtQd1owbXJRcE5CMzZFdkpKZFVteHdkU2MyWDhrOGcxMC85NVlLQkdWQWoKL3d0YVZYbE9WeEFvK0ZSelpZeFpyQ29uWWFSMHVwUzFybDRtenN4REhlZU9mUVZUTUgwUjdZN0pnbTA5dXQ4SwplanAvSXZBb1F3S0JnQjNaRWlRUWhvMVYrWjBTMlpiOG5KS0plMy9zMmxJTXFHM0ZkaS9RS3Q0eWViQWx6OGY5ClBZVXBzRmZEQTg5Z3grSU1nSm5sZVptdTk2ZnRXSjZmdmJSenllN216TG5zZU05TXZua1lHbGFGWmJRWnZubXMKN3ZoRmtzY3dHRlh4d21GMlBJZmU1Z3pNMDRBeVdjeTFIaVhLS2dNOXM3cGsxWUdyZGowZzdacmRBb0dCQUtLNApDR3MrbkRmMEZTMFJYOWFEWVJrRTdBNy9YUFhtSG5YMkRnU1h5N0Q4NTRPaWdTTWNoUmtPNTErbVNJejNQbllvCk41T1FXM2lHVVl1M1YvYmhnc0VSUzM1V2xmRk9BdDBzRUR5bjF5SVdXcDF5dG93d3BUNkVvUXVuZ2NYZjA5RjMKS1NROXowd3M4VmsvRWkvSFVXcU5LOWFXbU51cmFaT0ZqL2REK1ZkOUFvR0FMWFN3dEE3K043RDRkN0VEMURSRQpHTWdZNVd3OHFvdDZSdUNlNkpUY0FnU3B1MkhNU3JVY2dXclpiQnJZb09FUnVNQjFoMVJydk5ybU1qQlM0VW9FClgyZC8vbGhpOG1wL2VESWN3UDNRa2puanBJRFJWMFN1eWxrUkVaZURKZjVZb3R6eDdFdkJhbzFIbkQrWEg4eUIKVUtmWGJTaHZKVUdhRmgxT3Q1Y3JoM1k9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K

0 commit comments

Comments
 (0)