You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/ha-cluster/readme.md
+43-19Lines changed: 43 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ NNF software supports provisioning of Red Hat GFS2 (Global File System 2) storag
15
15
-[Quorum Configuration](#quorum-configuration)
16
16
-[Why the Rabbit Needs 17 Votes](#why-the-rabbit-needs-17-votes)
17
17
-[Configuring Quorum Votes](#configuring-quorum-votes)
18
-
-[Fencing with fence_recorder](#fencing-with-fence_recorder)
18
+
-[Fencing with fence_dummy](#fencing-with-fence_dummy)
19
19
-[How It Works](#how-it-works)
20
20
-[Installation](#installation)
21
21
-[STONITH Configuration](#stonith-configuration)
@@ -169,9 +169,9 @@ corosync-quorumtool
169
169
170
170
The output should show the Rabbit with 17 votes and each compute with 1 vote, for a total of 33 votes and quorum at 17.
171
171
172
-
## Fencing with fence_recorder
172
+
## Fencing with fence_dummy
173
173
174
-
The `fence_recorder` agent coordinates fencing with external NNF software using a request/response file pattern. When Pacemaker decides to fence a compute node, `fence_recorder`:
174
+
The [`fence_dummy`](https://github.com/ClusterLabs/fence-agents/tree/main/agents/dummy) agent coordinates fencing with external NNF software using a request/response file pattern. When Pacemaker decides to fence a compute node, `fence_dummy`:
175
175
176
176
1. Writes a fence request file
177
177
2. Waits for the NNF software to process the request and write a response
@@ -188,10 +188,10 @@ This allows the NNF software to perform storage cleanup (detaching NVMe namespac
|`pcmk_host_list`| (required) | Node this STONITH resource can fence |
276
-
|`request_dir`|`/var/run/fence_recorder/requests`| Directory for fence request files |
277
-
|`response_dir`|`/var/run/fence_recorder/responses`| Directory for fence response files |
276
+
|`request_dir`|`/var/run/fence_dummy/requests`| Directory for fence request files |
277
+
|`response_dir`|`/var/run/fence_dummy/responses`| Directory for fence response files |
278
278
|`log_dir`|`/var/log/cluster`| Directory for log files |
279
279
280
+
### NNF Software Configuration
281
+
282
+
The nnf-node-manager DaemonSet must be configured with the **same** request and response directories used by the `fence_dummy` agent. These are controlled by environment variables in the pod spec:
283
+
284
+
| Environment Variable | Default | Description |
285
+
|---------------------|---------|-------------|
286
+
|`NNF_FENCE_REQUEST_DIR`|`/localdisk/fence-recorder/requests`| Directory where fence agents write JSON request files |
287
+
|`NNF_FENCE_RESPONSE_DIR`|`/localdisk/fence-recorder/responses`| Directory where nnf-sos writes JSON response files |
288
+
289
+
These variables are set in the nnf-node-manager DaemonSet manifest (`config/manager/manager.yaml`):
290
+
291
+
```yaml
292
+
env:
293
+
- name: NNF_FENCE_REQUEST_DIR
294
+
value: "/localdisk/fence-recorder/requests"
295
+
- name: NNF_FENCE_RESPONSE_DIR
296
+
value: "/localdisk/fence-recorder/responses"
297
+
```
298
+
299
+
When unset or empty, the defaults shown above are used. The nnf-node-manager creates the directories automatically if they do not exist.
300
+
301
+
!!! important
302
+
The `request_dir` and `response_dir` values in the STONITH resource configuration **must match** the `NNF_FENCE_REQUEST_DIR` and `NNF_FENCE_RESPONSE_DIR` environment variables in the nnf-node-manager pod. If they differ, fence requests will not be detected.
303
+
280
304
### Verifying Configuration
281
305
282
306
```bash
@@ -286,15 +310,15 @@ pcs stonith status
286
310
# View STONITH configuration
287
311
pcs stonith config
288
312
289
-
# Test that fence_recorder can generate metadata
290
-
fence_recorder --action metadata
313
+
# Test that fence_dummy can generate metadata
314
+
fence_dummy --action metadata
291
315
```
292
316
293
317
## Request/Response Protocol
294
318
295
319
### Request File Format
296
320
297
-
When a fence operation is triggered, `fence_recorder` writes a JSON request file:
321
+
When a fence operation is triggered, `fence_dummy` writes a JSON request file:
|`perl`| Version string manipulation | Used for semver bump calculations |
58
+
|`sed`| Text processing ||
59
+
|`tput`| Terminal formatting ||
60
+
61
+
**Environment variables:**
62
+
63
+
-`GH_TOKEN` — A GitHub classic personal access token with `repo` scope. Required by `gh` and by the `release-push`, `create-pr`, `merge-pr`, and `tag-release` phases.
64
+
65
+
**SSH access:**
66
+
67
+
- Verify with `ssh -T git@github.com`. The release tool clones all repos via SSH.
68
+
32
69
## Assumptions
33
70
34
71
-`master` or `main` branch for each repository contains **tested** software and documentation ready to be released.
35
-
- You've installed the GitHub CLI tool, `gh`.
36
-
- This tool requires a GH_TOKEN environment variable containing a `repo` scope classic token.
37
72
38
73
## Steps
39
74
40
75
### Run the steps in this order
41
76
42
-
> **Note:** You almost always want to use the -R option to focus the `phase` activity to a specific repo.
77
+
> **Note:** You almost always want to use the `-R` option to focus the `phase` activity to a specific repo.
78
+
>
79
+
> Use `-B major|minor|patch` (default: `patch`) to control which part of the version number is bumped.
43
80
44
-
0.**List Repos:** Get the ordered list of repo names to use with -R option in subsequent steps. This is referred to as `repo-list`
81
+
0.**List Repos:** Get the ordered list of repo names to use with `-R` option in subsequent steps. This is referred to as `repo-list`
45
82
> **Pro tip:** Keep this list in a separate window for easy viewing
83
+
84
+
```bash
46
85
./release-all.sh -L
86
+
```
87
+
88
+
The output will show the repos in dependency order:
89
+
90
+
```
91
+
dws
92
+
lustre_csi_driver
93
+
lustre_fs_operator
94
+
nnf_mfu
95
+
nnf_ec
96
+
nnf_sos
97
+
nnf_dm
98
+
nnf_integration_test
99
+
nnf_deploy
100
+
nnf_doc
101
+
```
47
102
48
103
1. **Check Vendoring:** For each repo's master/main branch; determine whether any of them need to be re-vendored.
49
104
> **Note:** Ensure each repo is error-free before proceeding to the next repo in `repo-list`
@@ -61,21 +116,24 @@ other components.
61
116
```
62
117
63
118
3. **Generate Release:** For each repo in`repo-list`, proceed through the following steps in sequence before moving on to the next repo.
64
-
> **Note:** The next steps use the gh(1) GitHub CLI tool and require a GH_TOKEN environment variable containing a 'repo' scope classic token.
65
-
1. If the **Create Trial Release Branch** had no errors:
119
+
>**Note:** The next steps use the `gh` GitHub CLI tool and require a `GH_TOKEN` environment variable containing a `repo` scope classic token.
120
+
121
+
**Step 3a — Push the release branch.** Choose one of the following based on whether step 2 had merge conflicts:
122
+
123
+
- If the **Create Trial Release Branch** had **no errors**:
66
124
67
125
```bash
68
126
./release-all.sh -P release-push -R <repo>
69
127
```
70
128
71
-
2. If **Create Trial Release Branch** was unable to auto merge, manually fix and merge the release branch and re-run this phase on that existing branch:
129
+
- If **Create Trial Release Branch** was unable to auto merge, manually fix and merge the release branch, then re-run this phase on the existing branch:
72
130
73
131
```bash
74
-
cd workingspace/repo
132
+
cd workingspace/<repo>
75
133
# Manually merge the changes from master/main to the release branch
76
134
go mod tidy
77
135
go mod vendor
78
-
git status # confirm all issues have been address
136
+
git status # confirm all issues have been addressed
79
137
git add <all affected files>
80
138
git commit -s # take the default commit message, don't bother editing it.
**Step 3b** — Create PR for the pushed release branch:
90
148
91
-
```bash
92
-
./release-all.sh -P create-pr -R <repo>
93
-
```
149
+
```bash
150
+
./release-all.sh -P create-pr -R <repo>
151
+
```
94
152
95
-
4. Merge PR for the pushed release branch:
96
-
**Note: Do NOT manually merge the PR, let`release-all.sh` merge it.**
153
+
**Step 3c** — Merge PR for the pushed release branch:
97
154
98
-
```bash
99
-
./release-all.sh -P merge-pr -R <repo>
100
-
```
155
+
>**Warning:** Do NOT manually merge the PR. Let `release-all.sh` merge it. If you accidentally merge manually, the `tag-release` phase may not find the expected merge commit message — use `-x force-tag=vX.Y.Z` to recover (see step 3d).
101
156
102
-
5. Tag the release:
157
+
```bash
158
+
./release-all.sh -P merge-pr -R <repo>
159
+
```
103
160
104
-
```bash
105
-
./release-all.sh -P tag-release -R <repo>
106
-
```
161
+
**Step 3d** — Tag the release:
162
+
163
+
>**Important:** This creates an **annotated** git tag. The CI/CD workflow (`handle_release_tag.yaml`) verifies that the tag is annotated and will reject lightweight tags. After tagging, the CI/CD workflow automatically creates the GitHub release with auto-generated release notes and attaches build artifacts (e.g., `manifests.tar`for`nnf-deploy`).
164
+
165
+
```bash
166
+
./release-all.sh -P tag-release -R <repo>
167
+
```
168
+
169
+
If tagging fails because the most recent commit doesn't contain the expected merge message (e.g., due to a manual merge), use the force-tag override:
Finalize the release by updating the `nnf-deploy` release notes to include the release notes from all submodules that were modified by this release. This also updates the release notes for any submodule that has CRDs, to include information about each version of the CRD offered by that submodule. Do this after the release steps have been completed for all repositories, including the NearNodeFlash.github.io repository.
111
178
179
+
> **Note:** `final-release-notes.sh` requires `yq`, `gh`, `jq`, and `perl` to be installed, and `GH_TOKEN` to be set.
180
+
112
181
1. Generate complete release notes for the specified `nnf-deploy` release for review:
113
182
114
183
```bash
115
184
./final-release-notes.sh -r $NNF_RELEASE
116
185
```
117
186
118
-
2. Generate and commit the release notes to the specified `nnf-deploy` release:
187
+
2. Review the generated notes, then commit them:
119
188
120
189
```bash
121
190
./final-release-notes.sh -r $NNF_RELEASE -C
122
191
```
123
192
124
-
## Compare release manifests
193
+
## Verify the release
125
194
126
-
Compare the new NNF release manifest to a previous NNF release manifest. This can be useful fora variety of purposes. For example, this is a quick way to check for any problemsinthe release or to see which submodules were updated in the release.
195
+
After all repos are released and release notes are finalized, compare the new NNF release manifest to the previous release manifest. This is a recommended verification step to confirm the release contains the expected changes and catch any problems.
0 commit comments