You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# kro Integration (providing LoadBalancer as a Service)
9
9
10
-
This guide demonstrates how to use [kro](https://kro.run/) and [Envoy Gateway](https://gateway.envoyproxy.io/) to offer a "LoadBalancer as a Service" API in multi-cluster environments.
11
-
12
-
When operating multiple clusters on-premises, managing load balancer infrastructure separately for each cluster becomes operationally expensive. This integration enables a centralized load balancer cluster that serves all tenant clusters connected via converged networking solutions. By running load balancer resources in a dedicated load balancer cluster, organizations can simplify operations for application teams who can self-service load balancers without managing the underlying infrastructure and enforce consistent security policies and configuration across gateways.
10
+
This guide demonstrates how to use [kro](https://kro.run/) and
11
+
[Envoy Gateway](https://gateway.envoyproxy.io/) to offer a "LoadBalancer as a Service" API in
12
+
multi-cluster environments.
13
+
14
+
When operating multiple clusters on-premises, managing load balancer infrastructure separately
15
+
for each cluster becomes operationally expensive. This integration enables a centralized load
16
+
balancer cluster that serves all tenant clusters connected via converged networking solutions.
17
+
By running load balancer resources in a dedicated load balancer cluster, organizations can simplify
18
+
operations for application teams who can self-service load balancers without managing the
19
+
underlying infrastructure and enforce consistent security policies and configuration across
20
+
gateways.
13
21
14
22
15
23
16
-
In this guide in the consumer cluster we create a simple `LoadBalancer` object, and the provider automatically provisions an Envoy Gateway infrastructure and related `Gateway` and `HTTPRoute` to expose the service between two kind clusters.
24
+
In this guide in the consumer cluster we create a simple `LoadBalancer` object, and the provider
25
+
automatically provisions an Envoy Gateway infrastructure and related `Gateway` and `HTTPRoute` to
26
+
expose the service between two kind clusters.
17
27
18
-
This example includes support for syncing **custom configuration (via ConfigMaps or Secrets)** from consumer clusters to the provider.
28
+
This example includes support for syncing **custom configuration (via ConfigMaps or Secrets)** from
29
+
consumer clusters to the provider.
19
30
20
31
!!! note
21
-
For this to work end-to-end, the consumer's service (`backend`) must be reachable from the provider cluster (e.g., via multi-cluster networking).
32
+
For this to work end-to-end, the consumer's service (`backend`) must be reachable from the
33
+
provider cluster (e.g., via multi-cluster networking).
22
34
23
-
In this example, we simulate multi-cluster networking by exposing the consumer's backend service via NodePort and creating corresponding Service/EndpointSlice in the provider cluster. In production, you would use proper multi-cluster networking solutions like Submariner or Cilium cluster mesh.
35
+
In this example, we simulate multi-cluster networking by exposing the consumer's backend
36
+
service via NodePort and creating corresponding Service/EndpointSlice in the provider cluster.
37
+
In production, you would use proper multi-cluster networking solutions like Submariner or
38
+
Cilium cluster mesh.
24
39
25
40
## Prerequisites
26
41
27
-
In this integration guide, we will be using `kubectl bind dev` command to provision two kind clusters. More details on the command you can find [here](../../developers/dev-environments.md).
42
+
In this integration guide, we will be using `kubectl bind dev` command to provision two kind
43
+
clusters. More details on the command you can find [here](../../developers/dev-environments.md).
28
44
29
45
***Provider Cluster:** Runs kro, Envoy Gateway, and the kube-bind backend.
30
46
***Consumer Cluster:** Runs the kube-bind konnector.
31
47
32
48
## Setup On The Provider Cluster
33
49
34
-
The following sections will guide you through the one-time setup that is required for providing LoadBalancer as a Service using kro and kube-bind.
50
+
The following sections will guide you through the one-time setup that is required for providing
51
+
LoadBalancer as a Service using kro and kube-bind.
35
52
36
53
### Install Envoy Gateway
37
54
@@ -88,14 +105,14 @@ metadata:
88
105
kubernetes.io/service-name: backend
89
106
addressType: IPv4
90
107
ports:
91
-
- name: http
92
-
port: 30080
93
-
protocol: TCP
108
+
- name: http
109
+
port: 30080
110
+
protocol: TCP
94
111
endpoints:
95
-
- addresses:
96
-
- ${CONSUMER_NODE_IP}
97
-
conditions:
98
-
ready: true
112
+
- addresses:
113
+
- ${CONSUMER_NODE_IP}
114
+
conditions:
115
+
ready: true
99
116
EOF
100
117
```
101
118
@@ -105,14 +122,15 @@ kro allows you to define custom APIs (`ResourceGraphDefinition`) and map them to
Create a kro `ResourceGraphDefinition` that defines the API `loadbalancers.networking.kro.run`.
115
-
This definition includes referencing a `ConfigMap` for custom routing rules (e.g., adding headers). The same way user could reference a `Secret` with `Certificate` to setup TLS.
132
+
This definition includes referencing a `ConfigMap` for custom routing rules (e.g., adding headers).
133
+
The same way user could reference a `Secret` with `Certificate` to setup TLS.
Create an `APIServiceExportTemplate`. Crucially, we add **PermissionClaims** to allow the provider to read the `ConfigMaps` that the consumer will create and reference.
227
+
Create an `APIServiceExportTemplate`. Crucially, we add **PermissionClaims** to allow the provider
228
+
to read the ConfigMaps that the consumer will create and reference.
210
229
211
230
```yaml
212
231
kubectl apply -f - <<EOF
@@ -225,22 +244,25 @@ spec:
225
244
resource: configmaps
226
245
selector:
227
246
references:
228
-
- resource: loadbalancers
229
-
group: networking.kro.run
230
-
jsonPath:
231
-
name: 'spec.configMapRef'
247
+
- resource: loadbalancers
248
+
group: networking.kro.run
249
+
jsonPath:
250
+
name: 'spec.configMapRef'
232
251
scope: Namespaced
233
252
EOF
234
253
```
235
254
236
255
## Setup on the Consumer Cluster
237
256
238
-
Now that everything is set up, users can begin to bind to your backend and begin consuming the new API.
257
+
Now that everything is set up, users can begin to bind to your backend and begin consuming the
258
+
new API.
239
259
240
260
### Login to kube-bind
241
261
242
262
!!! note
243
-
When you run `kubectl bind dev create`, **save the output** as it contains important information (like cluster IPs and configuration) that will be needed throughout this guide. This output is unique for each user and environment.
263
+
When you run `kubectl bind dev create`, **save the output** as it contains important
264
+
information (like cluster IPs and configuration) that will be needed throughout this guide.
265
+
This output is unique for each user and environment.
Once the binding is active, you can create `LoadBalancer` resources in your consumer cluster, and you will get `LoadBalancer` objects synced from the provider cluster.
275
+
Once the binding is active, you can create `LoadBalancer` resources in your consumer cluster, and
276
+
you will get `LoadBalancer` objects synced from the provider cluster.
254
277
255
278
```bash
256
279
kubectl bind
@@ -277,10 +300,10 @@ kubectl apply -f - <<EOF
277
300
apiVersion: v1
278
301
kind: ConfigMap
279
302
metadata:
280
-
name: my-lb-config
281
-
namespace: default
303
+
name: my-lb-config
304
+
namespace: default
282
305
data:
283
-
custom-header: "hello-kube-bind"
306
+
custom-header: "hello-kube-bind"
284
307
EOF
285
308
```
286
309
@@ -354,20 +377,21 @@ kubectl apply -f - <<EOF
354
377
apiVersion: networking.kro.run/v1alpha1
355
378
kind: LoadBalancer
356
379
metadata:
357
-
name: my-lb
358
-
namespace: default
380
+
name: my-lb
381
+
namespace: default
359
382
spec:
360
-
domain: "www.example.com"
361
-
configMapRef: "my-lb-config"
362
-
targetService: "backend"
363
-
targetServiceNamespace: "default"
364
-
targetPort: 30080
383
+
domain: "www.example.com"
384
+
configMapRef: "my-lb-config"
385
+
targetService: "backend"
386
+
targetServiceNamespace: "default"
387
+
targetPort: 30080
365
388
EOF
366
389
```
367
390
368
391
### Observe the Provisioning
369
392
370
-
**Provider Side:** kube-bind syncs the `ConfigMap` back to the provider namespace. kro creates the `Gateway`, `HTTPRoute` and `ReferenceGrant`, and Envoy Gateway provisions the load balancer.
393
+
**Provider Side:** kube-bind syncs the ConfigMap back to the provider namespace. kro creates the
394
+
Gateway, Route and ReferenceGrant, and Envoy Gateway provisions the load balancer.
371
395
372
396
**Consumer Side:** The status is updated with the provider status.
373
397
@@ -405,12 +429,17 @@ my-lb-config 1 15s
405
429
406
430
## Appendix
407
431
408
-
Test the connection with provisioned load balancer and verify that `hello-kube-bind` header was added from the `ConfigMap`.
432
+
Test the connection with provisioned load balancer and verify that `hello-kube-bind` header was
433
+
added from the ConfigMap.
409
434
410
435
!!! note
411
-
For the basic check in this example, we will do port-forward. To be able to use `LoadBalancer` service IP in the kind cluster you would need to setup additional measures like [metalb](https://github.com/metallb/metallb) or [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind).
436
+
For the basic check in this example, we will do port-forward. To be able to use LoadBalancer
437
+
service IP in the kind cluster you would need to setup additional measures like
0 commit comments