Skip to content

arbiter.md

Latest commit

 

History

History
198 lines (140 loc) · 5.13 KB

arbiter.md

File metadata and controls

198 lines (140 loc) · 5.13 KB
Raw
title Arbiter

Arbiter

About This Project

The external-arbiter-operator (Arbiter) works with Rook-provisioned Ceph clusters to deploy external arbiters (monitors) that are not managed by Rook but that participate in consensus.

The operator also monitors the remote cluster to verify its availability and ensure that the tenant has sufficient permissions to handle the deployment of Arbiter.

Requirements and Setup

Required Tools

The following tools are required on your development machine:

  • sed
  • openssl
  • make
  • git
  • golang
  • lima (or another method to provision Kubernetes locally, such as Minikube)
  • kubectl
  • docker (or any compatible container engine, such as Podman)
  • helm

The remaining dependencies are provisioned via Go tools, including the Kubebuilder toolset.

Quick Start

What follows is a quick walkthrough on how to prepare the environment, run the operator locally, and deploy an external monitor.

Clone and Setup

# Clone the Rook repository: https://github.com/rook/rook 

#Run `make deps`:
make deps

# Create OSD for Ceph
limactl disk create osd --size=8G

# Create VM instance
limactl create --name=k8s ./contrib/vm.yaml

# Start VM
limactl start k8s

# Use kubeconfig provided by VM
export KUBECONFIG="${HOME}/.lima/k8s/copied-from-guest/kubeconfig.yaml"

Install Prerequisites

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml

# Install Rook operator
kubectl apply -f ./rook/deploy/examples/crds.yaml
kubectl apply -f ./rook/deploy/examples/common.yaml
kubectl apply -f ./rook/deploy/examples/operator.yaml
kubectl apply -f ./rook/deploy/examples/csi-operator.yaml

# Create Ceph cluster
kubectl apply -f ./rook/deploy/examples/cluster-test.yaml

# (Optional) Install Ceph toolbox
kubectl apply -f ./rook/deploy/examples/toolbox.yaml

Build and Install Operator

# Build image
limactl shell k8s sudo nerdctl --namespace k8s.io build \
  -t localhost:5000/cobaltcore-dev/external-arbiter-operator:latest \
  -f ./Dockerfile .

# Dry run operator install via Helm
helm install --dry-run --create-namespace --namespace arbiter-operator \
  --values ./contrib/charts/external-arbiter-operator/local.yaml \
  arbiter-operator ./contrib/charts/external-arbiter-operator

# Install operator via Helm chart
helm install --create-namespace --namespace arbiter-operator \
  --values ./contrib/charts/external-arbiter-operator/local.yaml \
  arbiter-operator ./contrib/charts/external-arbiter-operator

Configure and Deploy Arbiter

# Create namespace, user, role, rolebinding, kubeconfig and secret for arbiter
./hack/configure-k8s-user.sh

# Create secret with remote cluster access configuration
kubectl apply -f ./contrib/k8s/examples/secret.yaml -n arbiter-operator

# Create remote cluster resource
kubectl apply -f ./contrib/k8s/examples/remote-cluster.yaml -n arbiter-operator

# Create remote arbiter resource
kubectl apply -f ./contrib/k8s/examples/remote-arbiter.yaml -n arbiter-operator

# Watch until Arbiter is ready
kubectl get remotearbiter -n arbiter-operator -w

# Check that Arbiter has joined quorum
kubectl exec deployment/rook-ceph-tools -n rook-ceph -it -- ceph mon dump

Cleanup

# Remove Helm chart
helm uninstall --namespace arbiter-operator arbiter-operator

# Stop VM
limactl stop k8s

# Delete VM
limactl delete k8s

Make Goals

Useful make commands for development:

# Build binary
make

# Prettify project, run linters, etc.
make pretty

# Run tests
make test

# Regenerate Kubernetes resources
make gen

# Copy CRD definitions to Helm chart
make helm

Configuration

Deployment Configuration

Deployment manifests are managed by Helm. The values.yaml file lists all available configuration options.

Resource Configuration

The following example resources are provided:

  • secret.yaml - Kubeconfig secret for arbiter installation
  • remote-cluster.yaml - RemoteCluster resource definition
  • remote-arbiter.yaml - RemoteArbiter resource definition

How to Run

Prerequisites

Before running the operator, ensure the following conditions are met:

  1. A Ceph cluster operated by Rook is already up and running on the source Kubernetes cluster
  2. Resources (pods, services) from the target (arbiter) cluster are reachable from the source (operator/Rook) cluster and vice versa

Deployment Steps

  1. Create a user on the target cluster.
  2. Create the target namespace on the target cluster.
  3. Grant the user permissions to manage deployments, secrets, configmaps, their statuses, and finalizers.
  4. Provision the target user kubeconfig on the source cluster via secret.
  5. Deploy the operator on the source cluster.
  6. Create a RemoteCluster resource on the source cluster, referencing the target user kubeconfig secret.
  7. Create a RemoteArbiter resource on the source cluster, referencing the RemoteCluster.
  8. Watch until resources are ready.
  9. Verify that the arbiter has joined the quorum by running ceph mon dump.

See Also

Arbiter Repository