Skip to content

Commit 1ab7da9

Browse files
zac-nixonzac-nixon
committed
Add Remote Infrastructure documentation
Signed-off-by: Zachary Nixon <nixozach@amazon.com>
1 parent e4b57e6 commit 1ab7da9

2 files changed

Lines changed: 283 additions & 0 deletions

File tree

‎site/content/en/latest/tasks/extensibility/_index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,4 @@ Envoy Gateway provides several ways to extend its functionality beyond the built
1717
- [External Processing](ext-proc) - Call external gRPC services during request processing
1818
- [Lua Extensions](lua) - Write lightweight scripting extensions
1919
- [Dynamic Modules](dynamic-modules) - Load custom C++ modules at runtime for advanced extensibility
20+
- [Remote Infrastructure Provider](remote-infrastructure-provider) - Defer Envoy data plane management to a custom infrastructure provider.

‎site/content/en/latest/tasks/extensibility/remote-infrastructure-provider.md‎

Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,282 @@
1+
---
2+
title: 'Remote Infrastructure Provider'
3+
---
4+
5+
This task explains how to configure Envoy Gateway to defer Envoy proxy and rate
6+
limit infrastructure management to a remote service of your own. With the Remote
7+
infrastructure provider, Envoy Gateway translates Gateway API resources into its
8+
[Infrastructure IR](/community/design/infrastructure-ir) and forwards it over
9+
gRPC to a provider you operate. The provider is then responsible for reconciling
10+
the data plane.
11+
12+
This is the right choice when the built-in Kubernetes infrastructure provider
13+
doesn't fit your environment. Common cases include running proxies in a
14+
different cluster from the control plane, deploying onto a non-Kubernetes
15+
substrate (VMs, an internal scheduler, etc.), or imposing custom organizational
16+
rules on how proxy workloads are shaped.
17+
18+
**Note:** Most users should use the built-in Kubernetes infrastructure provider.
19+
The Remote provider is an extension point; running it means owning the lifecycle
20+
of every proxy and rate limit deployment yourself.
21+
22+
## Security Warning
23+
24+
{{% alert title="Security Warning" color="warning" %}} The remote infrastructure
25+
provider is a privileged component. It receives the complete Infrastructure IR
26+
for every Gateway managed by Envoy Gateway and is trusted to faithfully
27+
reconcile it. A compromised or misconfigured provider can stop creating proxies,
28+
deploy proxies with attacker-controlled configuration, or expose the data plane
29+
to networks it shouldn't reach.
30+
31+
When deploying a remote infrastructure provider, you should:
32+
33+
- Run the provider as a sidecar of `envoy-gateway`, or otherwise on a private
34+
channel that is not reachable from untrusted networks. The reference
35+
deployment uses a Unix domain socket on a shared volume.
36+
- Restrict the provider's permissions to the minimum needed to manage the proxy
37+
data plane.
38+
- Audit the provider on the same cadence as Envoy Gateway itself; bumping Envoy
39+
Gateway may change the IR shape. {{% /alert %}}
40+
41+
## How it works
42+
43+
When the Remote provider is enabled, Envoy Gateway defers infrastructure
44+
management using an gRPC client to send updates to your provider. For every
45+
reconcile that mutates a proxy or rate limit deployment, Envoy Gateway issues a
46+
corresponding RPC carrying the JSON-serialized IR.
47+
48+
The service contract is defined in `proto/remoteinfra/service.proto`:
49+
50+
```proto
51+
service EnvoyGatewayRemoteInfrastructureProvider {
52+
rpc CreateOrUpdateProxyInfra(CreateOrUpdateProxyInfraRequest) returns (CreateOrUpdateProxyInfraResponse);
53+
rpc DeleteProxyInfra(DeleteProxyInfraRequest) returns (DeleteProxyInfraResponse);
54+
rpc CreateOrUpdateRateLimitInfra(CreateOrUpdateRateLimitInfraRequest) returns (CreateOrUpdateRateLimitInfraResponse);
55+
rpc DeleteRateLimitInfra(DeleteRateLimitInfraRequest) returns (DeleteRateLimitInfraResponse);
56+
}
57+
```
58+
59+
The proxy RPCs carry the IR as raw JSON in the `ir_bytes` field; the rate limit
60+
RPCs are parameterless. The provider is expected to be idempotent — Envoy
61+
Gateway calls the `CreateOrUpdate` RPCs every time the desired state changes and
62+
may retry on transient errors.
63+
64+
## Configuration
65+
66+
The Remote provider is selected under `provider.custom.infrastructure` in the
67+
`EnvoyGateway` config. The `service` field is required and uses the standard
68+
[ExtensionService] shape, so you can reach the provider over a Unix socket, a
69+
hostname, or an IP.
70+
71+
```yaml
72+
apiVersion: gateway.envoyproxy.io/v1alpha1
73+
kind: EnvoyGateway
74+
gateway:
75+
controllerName: gateway.envoyproxy.io/gatewayclass-controller
76+
provider:
77+
type: Custom
78+
custom:
79+
resource:
80+
type: Kubernetes
81+
infrastructure:
82+
type: Remote
83+
remote:
84+
service:
85+
unix:
86+
path: /var/run/remote-infra/server.sock
87+
```
88+
89+
To dial a remote service over the network instead, use `fqdn` or `ip`:
90+
91+
```yaml
92+
service:
93+
fqdn:
94+
hostname: remote-infra.envoy-gateway-system.svc.cluster.local
95+
port: 5005
96+
tls:
97+
certificateRef:
98+
name: remote-infra-ca
99+
namespace: envoy-gateway-system
100+
```
101+
102+
When TLS is configured, Envoy Gateway verifies the server certificate against
103+
the CA in the referenced Secret. Add `clientCertificateRef` if your provider
104+
requires mTLS.
105+
106+
### Rate limit service address
107+
108+
The Remote provider tells Envoy Gateway _what_ rate limit infrastructure to
109+
stand up via the `CreateOrUpdateRateLimitInfra` RPC, but Envoy proxies still
110+
need to know _where_ to send rate limit checks. Set `rateLimit.url` on the
111+
`EnvoyGateway` config to the address your provider exposes:
112+
113+
```yaml
114+
apiVersion: gateway.envoyproxy.io/v1alpha1
115+
kind: EnvoyGateway
116+
rateLimit:
117+
url: my-hosted-rl-service.com:443
118+
```
119+
120+
If `rateLimit.url` is unset, Envoy Gateway falls back to the address of the
121+
Service the Kubernetes infrastructure manager would have created.
122+
123+
## Quickstart
124+
125+
This walks through deploying the example provider in `examples/remote-infra` as
126+
a native sidecar of the `envoy-gateway` Deployment, communicating over a Unix
127+
domain socket. The example provider reconciles a `Deployment` and a `Service`
128+
per Gateway and is intended as a starting point — not a production-ready
129+
provider.
130+
131+
### Prerequisites
132+
133+
{{< boilerplate prerequisites >}}
134+
135+
### Build and load the example provider image
136+
137+
The example lives in `examples/remote-infra/` and ships with a Makefile target
138+
that builds a multi-arch image tagged `envoyproxy/gateway-remote-infra:latest`.
139+
140+
```shell
141+
make -C examples/remote-infra docker-buildx
142+
```
143+
144+
If you're running on `kind`, load the image into the cluster so the sidecar can
145+
pull it:
146+
147+
```shell
148+
kind load docker-image --name envoy-gateway envoyproxy/gateway-remote-infra:latest
149+
```
150+
151+
### Inject the provider as a sidecar
152+
153+
The provider needs to share a volume with `envoy-gateway` so they can talk over
154+
a UDS, and it needs to start before `envoy-gateway` does. Native sidecars give
155+
us both for free on Kubernetes 1.29+.
156+
157+
```shell
158+
cat <<'EOF' | kubectl patch deployment envoy-gateway -n envoy-gateway-system --patch-file /dev/stdin
159+
apiVersion: apps/v1
160+
kind: Deployment
161+
metadata:
162+
name: envoy-gateway
163+
namespace: envoy-gateway-system
164+
spec:
165+
template:
166+
spec:
167+
securityContext:
168+
runAsNonRoot: true
169+
runAsUser: 65532
170+
runAsGroup: 65532
171+
# fsGroup ensures the shared emptyDir is group-owned by 65532 so the
172+
# non-root sidecar can create the UDS socket file in it.
173+
fsGroup: 65532
174+
volumes:
175+
- name: remote-infra-socket
176+
emptyDir: {}
177+
containers:
178+
- name: envoy-gateway
179+
volumeMounts:
180+
- name: remote-infra-socket
181+
mountPath: /var/run/remote-infra
182+
initContainers:
183+
- name: remote-infra
184+
image: envoyproxy/gateway-remote-infra:latest
185+
imagePullPolicy: IfNotPresent
186+
restartPolicy: Always
187+
args:
188+
- --socket-path=/var/run/remote-infra/server.sock
189+
env:
190+
- name: NAMESPACE
191+
valueFrom:
192+
fieldRef:
193+
fieldPath: metadata.namespace
194+
# Gates startup of envoy-gateway: it will not start until the UDS
195+
# socket exists.
196+
startupProbe:
197+
exec:
198+
command: ["test", "-S", "/var/run/remote-infra/server.sock"]
199+
periodSeconds: 1
200+
failureThreshold: 30
201+
volumeMounts:
202+
- name: remote-infra-socket
203+
mountPath: /var/run/remote-infra
204+
EOF
205+
```
206+
207+
### Switch Envoy Gateway to the Remote provider
208+
209+
Patch the `envoy-gateway-config` ConfigMap to dial the sidecar over the shared
210+
socket:
211+
212+
```shell
213+
cat <<EOF | kubectl apply -f -
214+
apiVersion: v1
215+
kind: ConfigMap
216+
metadata:
217+
name: envoy-gateway-config
218+
namespace: envoy-gateway-system
219+
data:
220+
envoy-gateway.yaml: |
221+
apiVersion: gateway.envoyproxy.io/v1alpha1
222+
kind: EnvoyGateway
223+
gateway:
224+
controllerName: gateway.envoyproxy.io/gatewayclass-controller
225+
provider:
226+
type: Custom
227+
custom:
228+
resource:
229+
type: Kubernetes
230+
infrastructure:
231+
type: Remote
232+
remote:
233+
service:
234+
unix:
235+
path: /var/run/remote-infra/server.sock
236+
EOF
237+
```
238+
239+
Restart Envoy Gateway so the new config is picked up:
240+
241+
```shell
242+
kubectl rollout restart -n envoy-gateway-system deployment/envoy-gateway
243+
kubectl rollout status -n envoy-gateway-system deployment/envoy-gateway --timeout=2m
244+
```
245+
246+
### Verify
247+
248+
Apply a Gateway and an HTTPRoute, then watch the example provider's logs to
249+
confirm it received the IR:
250+
251+
```shell
252+
kubectl logs -n envoy-gateway-system deployment/envoy-gateway -c remote-infra
253+
```
254+
255+
You should see `Creating proxy infra [...]` lines as Gateways are reconciled.
256+
The example provider creates a `Deployment` and `Service` for each Gateway in
257+
the same namespace as Envoy Gateway; check them with:
258+
259+
```shell
260+
kubectl get deploy,svc -n envoy-gateway-system -l app.kubernetes.io/managed-by=envoy-gateway
261+
```
262+
263+
## Writing your own provider
264+
265+
The example in `examples/remote-infra/` is a useful starting point but only
266+
renders a `Deployment` and a `Service`. A production-grade provider will
267+
typically need to:
268+
269+
- **Be idempotent.** Envoy Gateway calls `CreateOrUpdate` whenever the IR
270+
changes, which in practice can be many times per second during churn. The
271+
example uses a server-side apply pattern; you should do the same or its
272+
equivalent for your data plane.
273+
- **Plan for IR evolution.** The IR is JSON for forward-compatibility, but new
274+
fields will be added as Envoy Gateway evolves. Design your provider to ignore
275+
unknown fields rather than fail on them.
276+
- **Communicate data plane status**. Envoy Gateway needs metadata about your
277+
data plane, such as the DNS address to find the fleet at. Make sure your
278+
provider communicates that information back via a Service object. The example
279+
infra server provides a basic example.
280+
281+
The IR types are defined in `internal/ir/`. The `ir.Infra` value Envoy Gateway
282+
sends is documented by the Go types `ir.ProxyInfra` and `ir.RateLimitInfra`.

0 commit comments

Comments
 (0)