From 8763fae37961a1e7c69e43b405fa2aecc58e68d3 Mon Sep 17 00:00:00 2001 From: Anthony Floeder Date: Fri, 5 Dec 2025 13:16:49 -0600 Subject: [PATCH 1/2] Document the new HA cluster setup and fencing agent Signed-off-by: Anthony Floeder --- docs/guides/ha-cluster/readme.md | 428 +++++++++++++++++++++++++++---- 1 file changed, 373 insertions(+), 55 deletions(-) diff --git a/docs/guides/ha-cluster/readme.md b/docs/guides/ha-cluster/readme.md index 1f88a993..5f313bc2 100644 --- a/docs/guides/ha-cluster/readme.md +++ b/docs/guides/ha-cluster/readme.md @@ -1,92 +1,410 @@ --- -authors: Nate Thornton +authors: Tony Floeder categories: setup --- -# High Availability Cluster +# High Availability Cluster Setup for Rabbit Nodes -NNF software supports provisioning of Red Hat GFS2 (Global File System 2) storage. Per RedHat: -> GFS2 allows multiple nodes to share storage at a block level as if the storage were connected locally to each cluster node. GFS2 cluster file system requires a cluster infrastructure. +NNF software supports provisioning of Red Hat GFS2 (Global File System 2) storage. GFS2 allows multiple nodes to share storage at a block level as if the storage were connected locally to each cluster node. To use GFS2, each Rabbit node and its associated compute nodes must form a high-availability cluster using Pacemaker and Corosync. -Therefore, in order to use GFS2, the NNF node and its associated compute nodes must form a high availability cluster. +## Table of Contents + +- [Background: Pacemaker and Corosync](#background-pacemaker-and-corosync) +- [Cluster Architecture](#cluster-architecture) +- [Cluster Setup](#cluster-setup) +- [Quorum Configuration](#quorum-configuration) + - [Why the Rabbit Needs 17 Votes](#why-the-rabbit-needs-17-votes) + - [Configuring Quorum Votes](#configuring-quorum-votes) +- [Fencing with fence_recorder](#fencing-with-fence_recorder) + - [How It Works](#how-it-works) + - [Installation](#installation) + - [STONITH Configuration](#stonith-configuration) + - [Configuration Options](#configuration-options) + - [Verifying Configuration](#verifying-configuration) +- [Request/Response Protocol](#requestresponse-protocol) + - [Request File Format](#request-file-format) + - [Response File Format](#response-file-format) + - [Atomic File Writing](#atomic-file-writing) +- [Log Files](#log-files) +- [Troubleshooting](#troubleshooting) + - [Check STONITH Status](#check-stonith-status) + - [Common Issues](#common-issues) + - [Testing Fence Operations](#testing-fence-operations) +- [GFS2 File System Configuration](#gfs2-file-system-configuration) +- [References](#references) + +## Background: Pacemaker and Corosync + +**Corosync** provides the cluster communication layer—it handles node membership, messaging between nodes, and quorum decisions. When a node becomes unresponsive, Corosync detects this and informs Pacemaker. + +**Pacemaker** is the cluster resource manager. It decides where resources run, handles failover when nodes fail, and coordinates fencing (STONITH) to protect shared storage from corruption. + +**Fencing (STONITH)** ensures that when a node fails, it is forcibly removed from the cluster before other nodes access its shared resources. Without fencing, a failed node could corrupt shared storage if it continues writing after the cluster assumes it's dead. + +For comprehensive documentation, see: + +- [Red Hat: Overview of High Availability](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_overview-of-high-availability-configuring-and-managing-high-availability-clusters) +- [Red Hat: Fencing in a Red Hat High Availability Cluster](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_configuring-fencing-configuring-and-managing-high-availability-clusters) + +## Cluster Architecture + +Each Rabbit node forms a separate HA cluster with its 16 compute nodes: + +```text +┌─────────────────────────────────────────────────────────────────────────┐ +│ Rabbit Cluster (17 nodes) │ +│ │ +│ ┌──────────────┐ │ +│ │ rabbit-node-1│ │ +│ │ (Rabbit) │ │ +│ └──────────────┘ │ +│ │ │ +│ ┌───────────┬───────────┼───────────┬────────────┐ │ +│ │ │ │ │ │ │ +│ ▼ ▼ ▼ ▼ ▼ │ +│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ │ +│ │compute-1│ │compute-2│ │ · · · │ │compute-15│ │compute-16│ │ +│ └─────────┘ └─────────┘ └─────────┘ └──────────┘ └──────────┘ │ +│ │ +│ All 17 nodes communicate via Corosync │ +└─────────────────────────────────────────────────────────────────────────┘ +``` ## Cluster Setup -Red Hat provides instructions for [creating a high availability cluster with Pacemaker](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters#doc-wrapper), including instructions for [installing cluster software](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters#proc_installing-cluster-software-creating-high-availability-cluster) and -[creating a high availability cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters). When following these instructions, each of the high availability clusters that are created should be named after the hostname of the NNF node. In the Red Hat examples the cluster name is `my_cluster`. +Red Hat provides comprehensive instructions for cluster setup: + +- [Installing Cluster Software](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters#proc_installing-cluster-software-creating-high-availability-cluster) +- [Creating a High Availability Cluster](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_creating-high-availability-cluster-configuring-and-managing-high-availability-clusters) + +Each cluster should be named after the Rabbit node hostname. The following examples use `rabbit-node-1` with compute nodes numbered 1-16: + +```bash +# On each node, install cluster packages +dnf install pcs pacemaker fence-agents-all + +# Enable and start pcsd +systemctl enable --now pcsd + +# Set hacluster password (same on all nodes) +echo 'password' | passwd --stdin hacluster + +# From the Rabbit node, authenticate all 17 nodes +pcs host auth rabbit-node-1 \ + rabbit-compute-1 rabbit-compute-2 rabbit-compute-3 rabbit-compute-4 \ + rabbit-compute-5 rabbit-compute-6 rabbit-compute-7 rabbit-compute-8 \ + rabbit-compute-9 rabbit-compute-10 rabbit-compute-11 rabbit-compute-12 \ + rabbit-compute-13 rabbit-compute-14 rabbit-compute-15 rabbit-compute-16 \ + -u hacluster -p password + +# Create the cluster with all nodes +pcs cluster setup rabbit-node-1 \ + rabbit-node-1 \ + rabbit-compute-1 rabbit-compute-2 rabbit-compute-3 rabbit-compute-4 \ + rabbit-compute-5 rabbit-compute-6 rabbit-compute-7 rabbit-compute-8 \ + rabbit-compute-9 rabbit-compute-10 rabbit-compute-11 rabbit-compute-12 \ + rabbit-compute-13 rabbit-compute-14 rabbit-compute-15 rabbit-compute-16 + +pcs cluster start --all +pcs cluster enable --all +``` + +## Quorum Configuration + +Quorum determines how many nodes must agree before the cluster can take action. By default, each node gets one vote and quorum requires a majority. However, in the Rabbit cluster architecture, the Rabbit node owns all NVMe storage—compute nodes only access storage through the Rabbit. This creates an asymmetric relationship that requires special quorum configuration. + +### Why the Rabbit Needs 17 Votes + +The Rabbit node must **always** have quorum because: + +1. **Storage ownership**: The Rabbit owns all NVMe namespaces. If compute nodes could fence the Rabbit, they would lose access to all storage anyway. + +2. **Preventing split-brain**: If the Rabbit becomes isolated from compute nodes (network partition), compute nodes should not be able to form their own quorum and attempt to fence the Rabbit. + +3. **Fencing authority**: Only the Rabbit (via NNF software) can properly detach storage from failed compute nodes. Compute nodes cannot fence each other directly. + +With 17 votes for the Rabbit and 1 vote per compute node: + +| Scenario | Votes | Quorum (17 required) | Result | +|----------|-------|----------------------|--------| +| Rabbit alone | 17 | ✓ Yes | Rabbit can fence any compute node | +| All 16 computes, no Rabbit | 16 | ✗ No | Computes cannot take action | +| Rabbit + any computes | 17+ | ✓ Yes | Normal operation | + +### Configuring Quorum Votes + +After cluster setup, configure voting weights: + +```bash +# Stop the cluster to modify corosync.conf +pcs cluster stop --all + +# Edit corosync.conf on the Rabbit node to set votes +# The nodelist section should look like: +# +# nodelist { +# node { +# ring0_addr: rabbit-node-1 +# nodeid: 1 +# quorum_votes: 17 +# } +# node { +# ring0_addr: rabbit-compute-1 +# nodeid: 2 +# quorum_votes: 1 +# } +# ... (repeat for all compute nodes) +# } + +# After editing, sync the configuration to all nodes +pcs cluster sync + +# Start the cluster +pcs cluster start --all + +# Verify quorum configuration +corosync-quorumtool +``` + +The output should show the Rabbit with 17 votes and each compute with 1 vote, for a total of 33 votes and quorum at 17. + +## Fencing with fence_recorder + +The `fence_recorder` agent coordinates fencing with external NNF software using a request/response file pattern. When Pacemaker decides to fence a compute node, `fence_recorder`: + +1. Writes a fence request file +2. Waits for the NNF software to process the request and write a response +3. Returns success or failure to Pacemaker +This allows the NNF software to perform storage cleanup (detaching NVMe namespaces) before the fence operation completes. -## Fencing Agents +> **Note:** Although the fence action is named "reboot", the NNF software does **not** actually reboot the compute node. Instead, the Rabbit's NNF software detaches all NVMe namespaces from the target compute node, preventing it from accessing any shared storage. This is sufficient for GFS2's requirements—the failed node can no longer corrupt shared data, regardless of whether it is still running. -Fencing is the process of restricting and releasing access to resources that a failed cluster node may have access to. Since a failed node may be unresponsive, an external device must exist that can restrict access to shared resources of that node, or to issue a hard reboot of the node. More information can be found form Red Hat: [1.2.1 Fencing](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_overview-of-high-availability-configuring-and-managing-high-availability-clusters#fencing). +### How It Works -HPE hardware implements software known as the Hardware System Supervisor (HSS), which itself conforms to the SNIA Redfish/Swordfish standard. This provides the means to manage hardware outside the host OS. +```text +┌──────────────────────┐ +│ Pacemaker/Corosync │ +│ (Cluster Manager) │ +└──────────┬───────────┘ + │ Calls fence_recorder + ▼ +┌──────────────────────┐ ┌───────────────────────────────┐ +│ fence_recorder │──────▶│ Request File │ +│ (Fence Agent) │ │ requests/-.json │ +└──────────────────────┘ └───────────────────────────────┘ + │ │ + │ Waits for response │ NNF software reads request + │ ▼ + │ ┌───────────────────────────────┐ + │ │ NNF Software │ + │ │ - Detaches storage │ + │ │ - Updates node status │ + │ └───────────────────────────────┘ + │ │ + ▼ │ Writes response +┌───────────────────────────────┐ │ +│ Response File │◀────────────┘ +│ responses/-.json│ +└───────────────────────────────┘ + │ + │ Exit 0 (success) or 1 (failure) + ▼ +┌──────────────────────┐ +│ Pacemaker │ +└──────────────────────┘ +``` + +### Installation + +Install `fence_recorder` on all nodes in the cluster: -### NNF Fencing +```bash +# Copy the agent to each node +sudo cp fence_recorder /usr/sbin/fence_recorder +sudo chmod 755 /usr/sbin/fence_recorder -#### Source -The NNF Fencing agent is available at https://github.com/NearNodeFlash/fence-agents under the `nnf` branch. +# Create the request/response directories on each rabbit node +sudo mkdir -p /localdisk/fence-recorder/{requests,responses} +sudo chmod 755 /localdisk/fence-recorder/{requests,responses} -```shell -git clone https://github.com/NearNodeFlash/fence-agents --branch nnf +# Create log directory +sudo mkdir -p /var/log/cluster +sudo chmod 755 /var/log/cluster ``` -#### Build -Refer to the `NNF.md file` at the root directory of the fence-agents repository. The fencing agents must be installed on every node in the cluster. +### STONITH Configuration -#### Setup -Configure the NNF agent with the following parameters: +Create a STONITH resource for each compute node. Run these commands from any node in the cluster: -| Argument | Definition | -| -------- | ---------- | -| `kubernetes-service-host=[ADDRESS]` | The IP address of the kubeapi server | -| `kubernetes-service-port=[PORT]` | The listening port of the kubeapi server | -| `service-token-file=[PATH]` | The location of the service token file. The file must be present on all nodes within the cluster | -| `service-cert-file=[PATH]` | The location of the service certificate file. The file must be present on all nodes within the cluster | -| `nnf-node-name=[NNF-NODE-NAME]` | Name of the NNF node as it is appears in the System Configuration | -| `api-version=[VERSION]` | The API Version of the NNF Node resource. Defaults to "v1alpha1" | +```bash +# Create STONITH resources for all 16 compute nodes (1-16) +for i in $(seq 1 16); do + pcs stonith create compute-${i}-fence-recorder fence_recorder \ + port=compute-${i} \ + pcmk_host_list=compute-${i} \ + request_dir=/localdisk/fence-recorder/requests \ + response_dir=/localdisk/fence-recorder/responses \ + log_dir=/var/log/cluster \ + op monitor interval=120s timeout=10s +done -The token and certificate can be found in the Kubernetes Secrets resource for the nnf-system/nnf-fencing-agent ServiceAccount. This provides RBAC rules to limit the fencing agent to only the Kubernetes resources it needs access to. +# Enable fencing +pcs property set stonith-enabled=true +``` -For example, setting up the NNF fencing agent on `rabbit-node-1` with a kubernetes service API running at `192.168.0.1:6443` and the service token and certificate copied to `/etc/nnf/fence/`. This needs to be run on one node in the cluster. +Alternatively, create resources individually: +```bash +# Example: Create STONITH for rabbit-compute-1 +pcs stonith create compute-1-fence-recorder fence_recorder \ + port=rabbit-compute-1 \ + pcmk_host_list=compute-1 \ + request_dir=/localdisk/fence-recorder/requests \ + response_dir=/localdisk/fence-recorder/responses \ + log_dir=/var/log/cluster \ + op monitor interval=120s timeout=10s ``` -pcs stonith create rabbit-node-1 fence_nnf pcmk_host_list=rabbit-node-1 kubernetes-service-host=192.168.0.1 kubernetes-service-port=6443 service-token-file=/etc/nnf/fence/service.token service-cert-file=/etc/nnf/fence/service.cert nnf-node-name=rabbit-node-1 + +### Configuration Options + +| Option | Default | Description | +|--------|---------|-------------| +| `port` | (required) | Target node name to fence | +| `pcmk_host_list` | (required) | Node this STONITH resource can fence | +| `request_dir` | `/var/run/fence_recorder/requests` | Directory for fence request files | +| `response_dir` | `/var/run/fence_recorder/responses` | Directory for fence response files | +| `log_dir` | `/var/log/cluster` | Directory for log files | + +### Verifying Configuration + +```bash +# Check STONITH status +pcs stonith status + +# View STONITH configuration +pcs stonith config + +# Test that fence_recorder can generate metadata +fence_recorder --action metadata ``` -#### Recovery -Since the NNF node is connected to 16 compute blades, careful coordination around fencing of a NNF node is required to minimize the impact of the outage. When a Rabbit node is fenced, the corresponding DWS Storage resource (`storages.dws.cray.hpe.com`) status changes. The workload manager must observe this change and follow the procedure below to recover from the fencing status. +## Request/Response Protocol -1. Observed the `storage.Status` changed and that `storage.Status.RequiresReboot == True` -2. Set the `storage.Spec.State := Disabled` -4. Wait for a change to the Storage status `storage.Status.State == Disabled` -5. Reboot the NNF node -6. Set the `storage.Spec.State := Enabled` -7. Wait for `storage.Status.State == Enabled` +### Request File Format -### Compute Fencing +When a fence operation is triggered, `fence_recorder` writes a JSON request file: -The [Redfish fencing agent](https://github.com/ClusterLabs/fence-agents/tree/main/agents/redfish) from [ClusterLabs](https://github.com/ClusterLabs/fence-agents) should be used for Compute nodes in the cluster. It is also included at https://github.com/NearNodeFlash/fence-agents, and can be built at the same time as the NNF fencing agent. Configure the agent with the following parameters: +**Location**: `/-.json` -| Argument | Definition | -| -------- | ---------- | -| `ip=[ADDRESS]` | The IP address or hostname of the HSS controller | -| `port=80` | The Port of the HSS controller. Must be `80` | -| `systems-uri=/redfish/v1/Systems/1` | The URI of the Systems object. Must be `/redfish/v1/Systems/1` | -| `ssl-insecure=true` | Instructs the use of an insecure SSL exchange. Must be `true` | -| `username=[USER]` | The user name for connecting to the HSS controller | -| `password=[PASSWORD]` | the password for connecting to the HSS controller | +```json +{ + "request_id": "550e8400-e29b-41d4-a716-446655440000", + "timestamp": "2025-01-10T14:30:00-06:00", + "action": "reboot", + "target_node": "rabbit-compute-1", + "recorder_node": "rabbit-node-1" +} +``` -For example, setting up the Redfish fencing agent on `rabbit-compute-2` with the redfish service at `192.168.0.1`. This needs to be run on one node in the cluster. +### Response File Format -```shell -pcs stonith create rabbit-compute-2 fence_redfish pcmk_host_list=rabbit-compute-2 ip=192.168.0.1 systems-uri=/redfish/v1/Systems/1 username=root password=password ssl_insecure=true +The NNF software writes a response file after processing: + +**Location**: `/-.json` + +```json +{ + "request_id": "550e8400-e29b-41d4-a716-446655440000", + "success": true, + "action_performed": "reboot", + "target_node": "rabbit-compute-1", + "message": "Successfully fenced node", + "timestamp": "2025-01-10T14:30:15-06:00" +} ``` -### Dummy Fencing +### Atomic File Writing + +Both request and response files use atomic rename to ensure consumers only see complete files: + +1. Write to temporary file: `..tmp` +2. Close the file +3. Rename to final name: `` + +File watchers should ignore files starting with `.` (hidden/temporary files). + +## Log Files + +Log files are written to the configured log directory: + +| File | Format | Description | +|------|--------|-------------| +| `fence-events.log` | Timestamped text | Main operational log | +| `fence-events-readable.log` | Key=value | Grep-friendly format | +| `fence-events-detailed.jsonl` | JSON Lines | Machine-parseable format | + +## Troubleshooting + +### Check STONITH Status + +```bash +# View current status +pcs stonith status + +# Check for failed resources +pcs status + +# View detailed configuration +pcs stonith config +``` + +### Common Issues + +**STONITH resource stopped or failed:** + +```bash +# Clean up failed resource state +pcs resource cleanup + +# Check logs for errors +journalctl -u pacemaker | grep -i stonith +``` + +**Timeout waiting for response:** + +```bash +# Check if NNF software is processing requests +ls -la /localdisk/fence-recorder/requests/ +ls -la /localdisk/fence-recorder/responses/ + +# Check fence_recorder logs +tail -f /var/log/cluster/fence-events.log +``` + +**Module not found error:** +Ensure the fencing library path is correct in `/usr/sbin/fence_recorder`. The `sys.path.append` line should point to `/usr/share/fence`. + +### Testing Fence Operations + +```bash +# Test metadata generation +fence_recorder --action metadata + +# Test monitor action (non-destructive) +fence_recorder --action monitor -n rabbit-compute-1 \ + --request-dir=/localdisk/fence-recorder/requests \ + --response-dir=/localdisk/fence-recorder/responses +``` + +## GFS2 File System Configuration + +After the cluster is configured with fencing, you can configure GFS2 file systems. See Red Hat documentation: -The [dummy fencing agent](https://github.com/ClusterLabs/fence-agents/tree/main/agents/dummy) from ClusterLabs can be used for nodes in the cluster for an early access development system. +- [Configuring a GFS2 File System in a Cluster](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_configuring-gfs2-in-a-cluster-configuring-and-managing-high-availability-clusters) -## Configuring a GFS2 file system in a cluster +## References -Follow steps 1-8 of the procedure from Red Hat: [Configuring a GFS2 file system in a cluster](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_configuring-gfs2-in-a-cluster-configuring-and-managing-high-availability-clusters#doc-wrapper). +- [Red Hat: Configuring and Managing High Availability Clusters](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/) +- [Red Hat: Fencing in a Cluster](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_configuring-fencing-configuring-and-managing-high-availability-clusters) +- [ClusterLabs fence-agents](https://github.com/ClusterLabs/fence-agents) From 16cabbb899e7a1ffbb93269fc86c67eb7078299e Mon Sep 17 00:00:00 2001 From: Anthony Floeder Date: Fri, 5 Dec 2025 15:27:36 -0600 Subject: [PATCH 2/2] Add section explaining dynamically starting/stopping cluster management services Signed-off-by: Anthony Floeder --- docs/guides/ha-cluster/readme.md | 37 ++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/guides/ha-cluster/readme.md b/docs/guides/ha-cluster/readme.md index 5f313bc2..dea2b3ef 100644 --- a/docs/guides/ha-cluster/readme.md +++ b/docs/guides/ha-cluster/readme.md @@ -31,6 +31,7 @@ NNF software supports provisioning of Red Hat GFS2 (Global File System 2) storag - [Common Issues](#common-issues) - [Testing Fence Operations](#testing-fence-operations) - [GFS2 File System Configuration](#gfs2-file-system-configuration) +- [Dynamic Cluster Lifecycle](#dynamic-cluster-lifecycle) - [References](#references) ## Background: Pacemaker and Corosync @@ -403,6 +404,42 @@ After the cluster is configured with fencing, you can configure GFS2 file system - [Configuring a GFS2 File System in a Cluster](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/assembly_configuring-gfs2-in-a-cluster-configuring-and-managing-high-availability-clusters) +## Dynamic Cluster Lifecycle + +Unlike traditional HA clusters that run continuously on all nodes, NNF manages Pacemaker cluster services dynamically on compute nodes based on workflow requirements. Cluster services run **continuously on the Rabbit** but are **started and stopped dynamically on compute nodes**. + +### When Cluster Services Start on Compute Nodes + +NNF software starts Pacemaker/Corosync cluster services on compute nodes when: + +1. A workflow requests GFS2 storage +2. The NNF software provisions the GFS2 file system +3. Compute nodes need to mount the shared storage + +At this point, cluster services are started on participating compute nodes, they join the Rabbit's cluster, fencing is enabled, and the GFS2 file system is mounted. + +> See [Storage Profiles](https://nearnodeflash.github.io/latest/guides/storage-profiles/readme/) for more information about how to start cluster services using the `PreActivate` command. + +### When Cluster Services Stop on Compute Nodes + +When the GFS2 workflow completes: + +1. The GFS2 file system is unmounted from all compute nodes +2. Storage resources are cleaned up +3. Pacemaker/Corosync cluster services are stopped on compute nodes +4. The Rabbit continues running cluster services + +> See [Storage Profiles](https://nearnodeflash.github.io/latest/guides/storage-profiles/readme/) for more information about how to stop cluster services using the `PostDeactivate` command. + +This dynamic lifecycle means: + +- **Rabbit runs cluster services continuously**: Always ready to accept compute nodes +- **Compute node services are transient**: They run only for the duration of GFS2 workflows +- **Resource efficiency**: Cluster overhead on compute nodes is incurred only when needed +- **Reduced complexity**: No long-running cluster services to maintain on compute nodes between jobs + +> **Note:** The cluster configuration (node membership, quorum votes, STONITH resources) is set up during system provisioning and persists on all nodes. Only the cluster *services* on compute nodes are started and stopped dynamically. + ## References - [Red Hat: Configuring and Managing High Availability Clusters](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_high_availability_clusters/)