Skip to content

Commit 1631ccf

Browse files
committed
Add "Arbiter" document to Architecture
Add "Arbiter" document to the Architecture section of the Apeiro documentation. Signed-off-by: Zac Dover <zac.dover@clyso.com>
1 parent f6d3249 commit 1631ccf

1 file changed

Lines changed: 196 additions & 0 deletions

File tree

docs/architecture/arbiter.md

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
---
2+
title: Arbiter
3+
---
4+
5+
# External Arbiter Operator
6+
7+
## About This Project
8+
9+
The external-arbiter-operator works with Rook-provisioned Ceph clusters to
10+
deploy external arbiters (monitors) that are not managed by Rook but that
11+
participate in consensus.
12+
13+
The operator also monitors the remote cluster to verify its availability and
14+
ensure that the tenant has sufficient permissions to handle the deployment of
15+
Arbiter.
16+
17+
## Requirements and Setup
18+
19+
### Required Tools
20+
21+
The following tools are required on your development machine:
22+
23+
- `sed`
24+
- `openssl`
25+
- `make`
26+
- `git`
27+
- `golang`
28+
- `lima` (or another method to provision Kubernetes locally, such as Minikube)
29+
- `kubectl`
30+
- `docker` (or any compatible container engine, such as Podman)
31+
- `helm`
32+
33+
The remaining dependencies are provisioned via Go tools, including the
34+
Kubebuilder toolset.
35+
36+
## Quick Start
37+
38+
What follows is a quick walkthrough on how to prepare the environment, run the
39+
operator locally, and deploy an external monitor.
40+
41+
### Clone and Setup
42+
43+
```bash
44+
# Clone [Rook repository](https://github.com/rook/rook) if not already done
45+
make deps
46+
47+
# Create OSD for Ceph
48+
limactl disk create osd --size=8G
49+
50+
# Create VM instance
51+
limactl create --name=k8s ./contrib/vm.yaml
52+
53+
# Start VM
54+
limactl start k8s
55+
56+
# Use kubeconfig provided by VM
57+
export KUBECONFIG="${HOME}/.lima/k8s/copied-from-guest/kubeconfig.yaml"
58+
```
59+
60+
### Install Prerequisites
61+
62+
```bash
63+
# Install cert-manager
64+
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml
65+
66+
# Install Rook operator
67+
kubectl apply -f ./rook/deploy/examples/crds.yaml
68+
kubectl apply -f ./rook/deploy/examples/common.yaml
69+
kubectl apply -f ./rook/deploy/examples/operator.yaml
70+
kubectl apply -f ./rook/deploy/examples/csi-operator.yaml
71+
72+
# Create Ceph cluster
73+
kubectl apply -f ./rook/deploy/examples/cluster-test.yaml
74+
75+
# (Optional) Install Ceph toolbox
76+
kubectl apply -f ./rook/deploy/examples/toolbox.yaml
77+
```
78+
79+
### Build and Install Operator
80+
81+
```bash
82+
# Build image
83+
limactl shell k8s sudo nerdctl --namespace k8s.io build \
84+
-t localhost:5000/cobaltcore-dev/external-arbiter-operator:latest \
85+
-f ./Dockerfile .
86+
87+
# Dry run operator install via Helm
88+
helm install --dry-run --create-namespace --namespace arbiter-operator \
89+
--values ./contrib/charts/external-arbiter-operator/local.yaml \
90+
arbiter-operator ./contrib/charts/external-arbiter-operator
91+
92+
# Install operator via Helm chart
93+
helm install --create-namespace --namespace arbiter-operator \
94+
--values ./contrib/charts/external-arbiter-operator/local.yaml \
95+
arbiter-operator ./contrib/charts/external-arbiter-operator
96+
```
97+
98+
### Configure and Deploy Arbiter
99+
100+
```bash
101+
# Create namespace, user, role, rolebinding, kubeconfig and secret for arbiter
102+
./hack/configure-k8s-user.sh
103+
104+
# Create secret with remote cluster access configuration
105+
kubectl apply -f ./contrib/k8s/examples/secret.yaml -n arbiter-operator
106+
107+
# Create remote cluster resource
108+
kubectl apply -f ./contrib/k8s/examples/remote-cluster.yaml -n arbiter-operator
109+
110+
# Create remote arbiter resource
111+
kubectl apply -f ./contrib/k8s/examples/remote-arbiter.yaml -n arbiter-operator
112+
113+
# Watch until Arbiter is ready
114+
kubectl get remotearbiter -n arbiter-operator -w
115+
116+
# Check that Arbiter has joined quorum
117+
kubectl exec deployment/rook-ceph-tools -n rook-ceph -it -- ceph mon dump
118+
```
119+
120+
### Cleanup
121+
122+
```bash
123+
# Remove Helm chart
124+
helm uninstall --namespace arbiter-operator arbiter-operator
125+
126+
# Stop VM
127+
limactl stop k8s
128+
129+
# Delete VM
130+
limactl delete k8s
131+
```
132+
133+
## Make Goals
134+
135+
Useful make commands for development:
136+
137+
```bash
138+
# Build binary
139+
make
140+
141+
# Prettify project, run linters, etc.
142+
make pretty
143+
144+
# Run tests
145+
make test
146+
147+
# Regenerate Kubernetes resources
148+
make gen
149+
150+
# Copy CRD definitions to Helm chart
151+
make helm
152+
```
153+
154+
## Configuration
155+
156+
### Deployment Configuration
157+
158+
Deployment manifests are managed by Helm. The `values.yaml` file lists all
159+
available configuration options.
160+
161+
### Resource Configuration
162+
163+
The following example resources are provided:
164+
165+
- `secret.yaml` - Kubeconfig secret for arbiter installation
166+
- `remote-cluster.yaml` - RemoteCluster resource definition
167+
- `remote-arbiter.yaml` - RemoteArbiter resource definition
168+
169+
## How to Run
170+
171+
### Prerequisites
172+
173+
Before running the operator, ensure the following conditions are met:
174+
175+
1. A Ceph cluster operated by Rook is already up and running on the source
176+
Kubernetes cluster
177+
1. Resources (pods, services) from the target (arbiter) cluster are reachable
178+
from the source (operator/Rook) cluster and vice versa
179+
180+
### Deployment Steps
181+
182+
1. Create a user on the target cluster.
183+
1. Create the target namespace on the target cluster.
184+
1. Grant the user permissions to manage deployments, secrets, configmaps, their
185+
statuses, and finalizers.
186+
1. Provision the target user kubeconfig on the source cluster via secret.
187+
1. Deploy the operator on the source cluster.
188+
1. Create a RemoteCluster resource on the source cluster, referencing the target
189+
user kubeconfig secret.
190+
1. Create a RemoteArbiter resource on the source cluster, referencing the
191+
RemoteCluster.
192+
1. Watch until resources are ready.
193+
1. Verify that the arbiter has joined the quorum by running `ceph mon dump`.
194+
195+
## See Also
196+
[https://github.com/cobaltcore-dev/external-arbiter-operator?tab=readme-ov-file](Arbiter Repository)

0 commit comments

Comments
 (0)