Skip to content

Commit 4da9818

Browse files
committed
Update multi-cluster guide doc with obs config and troubleshooting
1 parent fb822d2 commit 4da9818

2 files changed

Lines changed: 140 additions & 2 deletions

File tree

docs/getting-started/multi-cluster.mdx

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@ This guide walks you through step-by-step instructions for deploying OpenChoreo
1313

1414
## Prerequisites
1515

16-
- [Docker](https://docs.docker.com/get-docker/) v20.10+ installed and running
16+
- **Docker** – Just have it installed on your machine, and you're good to go.
17+
- We recommend using [Docker Engine version 26.0+](https://docs.docker.com/engine/release-notes/26.0/).
18+
- Allocate at least **8 GB RAM** and **4 CPU** cores to Docker (or the VM running Docker).
1719
- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) v0.20+ installed
1820
- [kubectl](https://kubernetes.io/docs/tasks/tools/) v1.32+ installed
1921
- [Helm](https://helm.sh/docs/intro/install/) v3.12+ installed
@@ -317,6 +319,20 @@ helm upgrade data-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-pla
317319

318320
**Important Security Note**: The observability plane collects data from outside clusters without encryption in this setup. For production environments, we recommend implementing proper TLS encryption and network security measures.
319321

322+
After updating the FluentBit configuration, restart the FluentBit pods to apply the new settings:
323+
324+
```bash
325+
# Restart FluentBit pods in Build Plane
326+
kubectl rollout restart daemonset/fluent-bit -n openchoreo-build-plane --context kind-openchoreo-bp
327+
328+
# Restart FluentBit pods in Data Plane
329+
kubectl rollout restart daemonset/fluent-bit -n openchoreo-data-plane --context kind-openchoreo-dp
330+
331+
# Verify FluentBit pods are running
332+
kubectl get pods -n openchoreo-build-plane --context kind-openchoreo-bp | grep fluent
333+
kubectl get pods -n openchoreo-data-plane --context kind-openchoreo-dp | grep fluent
334+
```
335+
320336
Verify FluentBit is sending logs to OpenSearch:
321337

322338
```bash
@@ -329,6 +345,49 @@ kubectl exec -n openchoreo-observability-plane opensearch-0 --context kind-openc
329345

330346
If the indices exist and the count is greater than 0, FluentBit is successfully collecting and storing logs.
331347

348+
#### Configure Observer Integration
349+
350+
Configure the DataPlane and BuildPlane to use the observer service. For multi-cluster setup, we need to expose the observer service via NodePort for cross-cluster communication.
351+
352+
First, expose the observer service with a NodePort:
353+
354+
```bash
355+
# Patch the observer service to use NodePort
356+
kubectl patch svc observer -n openchoreo-observability-plane --type='json' \
357+
-p='[{"op": "replace", "path": "/spec/type", "value": "NodePort"}, {"op": "add", "path": "/spec/ports/0/nodePort", "value": 30880}]' \
358+
--context kind-openchoreo-op
359+
```
360+
361+
Then configure the DataPlane and BuildPlane to use the observer service via NodePort:
362+
363+
```bash
364+
# Configure DataPlane to use observer service via NodePort
365+
kubectl patch dataplane default -n default --type merge \
366+
-p '{"spec":{"observer":{"url":"http://openchoreo-op-control-plane:30880","authentication":{"basicAuth":{"username":"dummy","password":"dummy"}}}}}' \
367+
--context kind-openchoreo-cp
368+
369+
# Configure BuildPlane to use observer service via NodePort
370+
kubectl patch buildplane default -n default --type merge \
371+
-p '{"spec":{"observer":{"url":"http://openchoreo-op-control-plane:30880","authentication":{"basicAuth":{"username":"dummy","password":"dummy"}}}}}' \
372+
--context kind-openchoreo-cp
373+
```
374+
375+
This configuration enables:
376+
- Application logs to appear in Backstage portal
377+
- Enhanced logging and monitoring across build and data planes
378+
- Integration with the observability plane for comprehensive platform monitoring
379+
- Centralized log publishing and access through the observer service
380+
381+
Verify the observer configuration:
382+
383+
```bash
384+
# Check DataPlane observer config
385+
kubectl get dataplane default -n default -o jsonpath='{.spec.observer}' --context kind-openchoreo-cp | jq '.'
386+
387+
# Check BuildPlane observer config
388+
kubectl get buildplane default -n default -o jsonpath='{.spec.observer}' --context kind-openchoreo-cp | jq '.'
389+
```
390+
332391
### 5. Install OpenChoreo Backstage Portal (Optional)
333392

334393
Install the OpenChoreo Backstage developer portal to provide a unified developer experience across your multi-cluster OpenChoreo platform. Backstage serves as a centralized hub where developers can discover, manage, and monitor all their services and components.
@@ -439,6 +498,16 @@ kubectl cluster-info --context kind-openchoreo-op
439498
kubectl get pods -n openchoreo-observability-plane --context kind-openchoreo-op
440499
```
441500

501+
## Troubleshooting
502+
503+
### Kind Cluster Creation Failures
504+
505+
If you encounter the error `failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"` when creating clusters, this typically indicates resource constraints or system limits.
506+
507+
- Ensure you have sufficient CPU and memory allocated to Docker (or the VM running Docker). We recommend at least **8 GB RAM** and **4 CPU** cores.
508+
- Increase inotify limits as described in https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files
509+
510+
442511
## Next Steps
443512

444513
After completing this multi-cluster setup you can:

versioned_docs/version-v0.3.x/getting-started/multi-cluster.mdx

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@ This guide walks you through step-by-step instructions for deploying OpenChoreo
1313

1414
## Prerequisites
1515

16-
- [Docker](https://docs.docker.com/get-docker/) v20.10+ installed and running
16+
- **Docker** – Just have it installed on your machine, and you're good to go.
17+
- We recommend using [Docker Engine version 26.0+](https://docs.docker.com/engine/release-notes/26.0/).
18+
- Allocate at least **8 GB RAM** and **4 CPU** cores to Docker (or the VM running Docker).
1719
- [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) v0.20+ installed
1820
- [kubectl](https://kubernetes.io/docs/tasks/tools/) v1.32+ installed
1921
- [Helm](https://helm.sh/docs/intro/install/) v3.12+ installed
@@ -317,6 +319,20 @@ helm upgrade data-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-pla
317319

318320
**Important Security Note**: The observability plane collects data from outside clusters without encryption in this setup. For production environments, we recommend implementing proper TLS encryption and network security measures.
319321

322+
After updating the FluentBit configuration, restart the FluentBit pods to apply the new settings:
323+
324+
```bash
325+
# Restart FluentBit pods in Build Plane
326+
kubectl rollout restart daemonset/fluent-bit -n openchoreo-build-plane --context kind-openchoreo-bp
327+
328+
# Restart FluentBit pods in Data Plane
329+
kubectl rollout restart daemonset/fluent-bit -n openchoreo-data-plane --context kind-openchoreo-dp
330+
331+
# Verify FluentBit pods are running
332+
kubectl get pods -n openchoreo-build-plane --context kind-openchoreo-bp | grep fluent
333+
kubectl get pods -n openchoreo-data-plane --context kind-openchoreo-dp | grep fluent
334+
```
335+
320336
Verify FluentBit is sending logs to OpenSearch:
321337

322338
```bash
@@ -329,6 +345,49 @@ kubectl exec -n openchoreo-observability-plane opensearch-0 --context kind-openc
329345

330346
If the indices exist and the count is greater than 0, FluentBit is successfully collecting and storing logs.
331347

348+
#### Configure Observer Integration
349+
350+
Configure the DataPlane and BuildPlane to use the observer service. For multi-cluster setup, we need to expose the observer service via NodePort for cross-cluster communication.
351+
352+
First, expose the observer service with a NodePort:
353+
354+
```bash
355+
# Patch the observer service to use NodePort
356+
kubectl patch svc observer -n openchoreo-observability-plane --type='json' \
357+
-p='[{"op": "replace", "path": "/spec/type", "value": "NodePort"}, {"op": "add", "path": "/spec/ports/0/nodePort", "value": 30880}]' \
358+
--context kind-openchoreo-op
359+
```
360+
361+
Then configure the DataPlane and BuildPlane to use the observer service via NodePort:
362+
363+
```bash
364+
# Configure DataPlane to use observer service via NodePort
365+
kubectl patch dataplane default -n default --type merge \
366+
-p '{"spec":{"observer":{"url":"http://openchoreo-op-control-plane:30880","authentication":{"basicAuth":{"username":"dummy","password":"dummy"}}}}}' \
367+
--context kind-openchoreo-cp
368+
369+
# Configure BuildPlane to use observer service via NodePort
370+
kubectl patch buildplane default -n default --type merge \
371+
-p '{"spec":{"observer":{"url":"http://openchoreo-op-control-plane:30880","authentication":{"basicAuth":{"username":"dummy","password":"dummy"}}}}}' \
372+
--context kind-openchoreo-cp
373+
```
374+
375+
This configuration enables:
376+
- Application logs to appear in Backstage portal
377+
- Enhanced logging and monitoring across build and data planes
378+
- Integration with the observability plane for comprehensive platform monitoring
379+
- Centralized log publishing and access through the observer service
380+
381+
Verify the observer configuration:
382+
383+
```bash
384+
# Check DataPlane observer config
385+
kubectl get dataplane default -n default -o jsonpath='{.spec.observer}' --context kind-openchoreo-cp | jq '.'
386+
387+
# Check BuildPlane observer config
388+
kubectl get buildplane default -n default -o jsonpath='{.spec.observer}' --context kind-openchoreo-cp | jq '.'
389+
```
390+
332391
### 5. Install OpenChoreo Backstage Portal (Optional)
333392

334393
Install the OpenChoreo Backstage developer portal to provide a unified developer experience across your multi-cluster OpenChoreo platform. Backstage serves as a centralized hub where developers can discover, manage, and monitor all their services and components.
@@ -439,6 +498,16 @@ kubectl cluster-info --context kind-openchoreo-op
439498
kubectl get pods -n openchoreo-observability-plane --context kind-openchoreo-op
440499
```
441500

501+
## Troubleshooting
502+
503+
### Kind Cluster Creation Failures
504+
505+
If you encounter the error `failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"` when creating clusters, this typically indicates resource constraints or system limits.
506+
507+
- Ensure you have sufficient CPU and memory allocated to Docker (or the VM running Docker). We recommend at least **8 GB RAM** and **4 CPU** cores.
508+
- Increase inotify limits as described in https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files
509+
510+
442511
## Next Steps
443512

444513
After completing this multi-cluster setup you can:

0 commit comments

Comments
 (0)