Skip to content

Add NNF_FENCE* environment variable information and change fence_reco… #306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ajfloeder merged 1 commit into from
Mar 12, 2026
Merged

Add NNF_FENCE* environment variable information and change fence_reco… #306

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 43 additions & 19 deletions docs/guides/ha-cluster/readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ NNF software supports provisioning of Red Hat GFS2 (Global File System 2) storag
- [Quorum Configuration](#quorum-configuration)
- [Why the Rabbit Needs 17 Votes](#why-the-rabbit-needs-17-votes)
- [Configuring Quorum Votes](#configuring-quorum-votes)
- [Fencing with fence_recorder](#fencing-with-fence_recorder)
- [Fencing with fence_dummy](#fencing-with-fence_dummy)
- [How It Works](#how-it-works)
- [Installation](#installation)
- [STONITH Configuration](#stonith-configuration)
Expand Down Expand Up @@ -169,9 +169,9 @@ corosync-quorumtool

The output should show the Rabbit with 17 votes and each compute with 1 vote, for a total of 33 votes and quorum at 17.

## Fencing with fence_recorder
## Fencing with fence_dummy

The `fence_recorder` agent coordinates fencing with external NNF software using a request/response file pattern. When Pacemaker decides to fence a compute node, `fence_recorder`:
The [`fence_dummy`](https://github.com/ClusterLabs/fence-agents/tree/main/agents/dummy) agent coordinates fencing with external NNF software using a request/response file pattern. When Pacemaker decides to fence a compute node, `fence_dummy`:

1. Writes a fence request file
2. Waits for the NNF software to process the request and write a response
Expand All @@ -188,10 +188,10 @@ This allows the NNF software to perform storage cleanup (detaching NVMe namespac
│ Pacemaker/Corosync │
│ (Cluster Manager) │
└──────────┬───────────┘
│ Calls fence_recorder
│ Calls fence_dummy
▼
┌──────────────────────┐ ┌───────────────────────────────┐
│ fence_recorder │──────▶│ Request File │
│ fence_dummy │──────▶│ Request File │
│ (Fence Agent) │ │ requests/<node>-<uuid>.json │
└──────────────────────┘ └───────────────────────────────┘
│ │
Expand All @@ -218,12 +218,12 @@ This allows the NNF software to perform storage cleanup (detaching NVMe namespac

### Installation

Install `fence_recorder` on all nodes in the cluster:
Install `fence_dummy` on all nodes in the cluster:

```bash
# Copy the agent to each node
sudo cp fence_recorder /usr/sbin/fence_recorder
sudo chmod 755 /usr/sbin/fence_recorder
sudo cp fence_dummy /usr/sbin/fence_dummy
sudo chmod 755 /usr/sbin/fence_dummy

# Create the request/response directories on each rabbit node
sudo mkdir -p /localdisk/fence-recorder/{requests,responses}
Expand All @@ -241,7 +241,7 @@ Create a STONITH resource for each compute node. Run these commands from any nod
```bash
# Create STONITH resources for all 16 compute nodes (1-16)
for i in $(seq 1 16); do
pcs stonith create compute-${i}-fence-recorder fence_recorder \
pcs stonith create compute-${i}-fence-dummy fence_dummy \
port=compute-${i} \
pcmk_host_list=compute-${i} \
request_dir=/localdisk/fence-recorder/requests \
Expand All @@ -258,7 +258,7 @@ Alternatively, create resources individually:

```bash
# Example: Create STONITH for rabbit-compute-1
pcs stonith create compute-1-fence-recorder fence_recorder \
pcs stonith create compute-1-fence-dummy fence_dummy \
port=rabbit-compute-1 \
pcmk_host_list=compute-1 \
request_dir=/localdisk/fence-recorder/requests \
Expand All @@ -273,10 +273,34 @@ pcs stonith create compute-1-fence-recorder fence_recorder \
|--------|---------|-------------|
| `port` | (required) | Target node name to fence |
| `pcmk_host_list` | (required) | Node this STONITH resource can fence |
| `request_dir` | `/var/run/fence_recorder/requests` | Directory for fence request files |
| `response_dir` | `/var/run/fence_recorder/responses` | Directory for fence response files |
| `request_dir` | `/var/run/fence_dummy/requests` | Directory for fence request files |
| `response_dir` | `/var/run/fence_dummy/responses` | Directory for fence response files |
| `log_dir` | `/var/log/cluster` | Directory for log files |

### NNF Software Configuration

The nnf-node-manager DaemonSet must be configured with the **same** request and response directories used by the `fence_dummy` agent. These are controlled by environment variables in the pod spec:

| Environment Variable | Default | Description |
|---------------------|---------|-------------|
| `NNF_FENCE_REQUEST_DIR` | `/localdisk/fence-recorder/requests` | Directory where fence agents write JSON request files |
| `NNF_FENCE_RESPONSE_DIR` | `/localdisk/fence-recorder/responses` | Directory where nnf-sos writes JSON response files |

These variables are set in the nnf-node-manager DaemonSet manifest (`config/manager/manager.yaml`):

```yaml
env:
- name: NNF_FENCE_REQUEST_DIR
value: "/localdisk/fence-recorder/requests"
- name: NNF_FENCE_RESPONSE_DIR
value: "/localdisk/fence-recorder/responses"
```

When unset or empty, the defaults shown above are used. The nnf-node-manager creates the directories automatically if they do not exist.

!!! important
The `request_dir` and `response_dir` values in the STONITH resource configuration **must match** the `NNF_FENCE_REQUEST_DIR` and `NNF_FENCE_RESPONSE_DIR` environment variables in the nnf-node-manager pod. If they differ, fence requests will not be detected.

### Verifying Configuration

```bash
Expand All @@ -286,15 +310,15 @@ pcs stonith status
# View STONITH configuration
pcs stonith config

# Test that fence_recorder can generate metadata
fence_recorder --action metadata
# Test that fence_dummy can generate metadata
fence_dummy --action metadata
```

## Request/Response Protocol

### Request File Format

When a fence operation is triggered, `fence_recorder` writes a JSON request file:
When a fence operation is triggered, `fence_dummy` writes a JSON request file:

**Location**: `<request_dir>/<node>-<uuid>.json`

Expand Down Expand Up @@ -379,21 +403,21 @@ journalctl -u pacemaker | grep -i stonith
ls -la /localdisk/fence-recorder/requests/
ls -la /localdisk/fence-recorder/responses/

# Check fence_recorder logs
# Check fence_dummy logs
tail -f /var/log/cluster/fence-events.log
```

**Module not found error:**
Ensure the fencing library path is correct in `/usr/sbin/fence_recorder`. The `sys.path.append` line should point to `/usr/share/fence`.
Ensure the fencing library path is correct in `/usr/sbin/fence_dummy`. The `sys.path.append` line should point to `/usr/share/fence`.

### Testing Fence Operations

```bash
# Test metadata generation
fence_recorder --action metadata
fence_dummy --action metadata

# Test monitor action (non-destructive)
fence_recorder --action monitor -n rabbit-compute-1 \
fence_dummy --action monitor -n rabbit-compute-1 \
--request-dir=/localdisk/fence-recorder/requests \
--response-dir=/localdisk/fence-recorder/responses
```
Expand Down
Loading