|
| 1 | +# NNF Software Release Plan |
| 2 | + |
| 3 | +This document is the authoritative, self-contained guide for executing an NNF software release. All release phases are driven by `release-all.sh` in this directory, wrapped by the `nnf_cmd` helper function. The release process is the same regardless of how many repos have changes — the `release` phase automatically detects which repos have new commits since the last release and skips those that don't. An agent or human following this document should be able to execute the entire release without external context. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Repository Reference |
| 8 | + |
| 9 | +| Script Name | GitHub Repo | Owner | Default Branch | Notes | |
| 10 | +| --- | --- | --- | --- | --- | |
| 11 | +| `dws` | `DataWorkflowServices/dws` | DataWorkflowServices | `master` | | |
| 12 | +| `lustre_csi_driver` | `HewlettPackard/lustre-csi-driver` | HewlettPackard | `master` | | |
| 13 | +| `lustre_fs_operator` | `NearNodeFlash/lustre-fs-operator` | NearNodeFlash | `master` | | |
| 14 | +| `nnf_mfu` | `NearNodeFlash/nnf-mfu` | NearNodeFlash | `master` | Standalone, not a submodule | |
| 15 | +| `nnf_ec` | `NearNodeFlash/nnf-ec` | NearNodeFlash | `master` | Standalone, not a submodule | |
| 16 | +| `nnf_sos` | `NearNodeFlash/nnf-sos` | NearNodeFlash | `master` | Requires `-M` flag | |
| 17 | +| `nnf_dm` | `NearNodeFlash/nnf-dm` | NearNodeFlash | `master` | Vendors nnf-sos | |
| 18 | +| `nnf_integration_test` | `NearNodeFlash/nnf-integration-test` | NearNodeFlash | `master` | Vendors nnf-sos | |
| 19 | +| `nnf_deploy` | `NearNodeFlash/nnf-deploy` | NearNodeFlash | `master` | Has submodules for most repos | |
| 20 | +| `nnf_doc` | `NearNodeFlash/NearNodeFlash.github.io` | NearNodeFlash | **`main`** (not master) | Released last; tracks nnf-deploy version | |
| 21 | + |
| 22 | +**PR Reviewers:** Defined in `.github/CODEOWNERS`. Assign all code owners **except** the person creating the release. The `setup-release-env.sh` helper reads CODEOWNERS automatically and sets `$RELEASE_REVIEWERS`. |
| 23 | + |
| 24 | +**Dependency chain (release order):** |
| 25 | + |
| 26 | +```text |
| 27 | +dws → lustre_csi_driver → lustre_fs_operator → nnf_mfu → nnf_ec → nnf_sos → nnf_dm → nnf_integration_test → nnf_deploy → nnf_doc |
| 28 | +``` |
| 29 | + |
| 30 | +**Vendoring dependencies (peer modules in `go.mod`):** |
| 31 | + |
| 32 | +- `lustre_fs_operator` vendors `dws` |
| 33 | +- `nnf_sos` vendors `dws`, `lustre-fs-operator`, `nnf-ec` |
| 34 | +- `nnf_dm` vendors `dws`, `lustre-fs-operator`, `nnf-ec` (indirect), `nnf-sos` |
| 35 | +- `nnf_integration_test` vendors `dws`, `lustre-fs-operator`, `nnf-ec` (indirect), `nnf-sos` |
| 36 | + |
| 37 | +If an upstream repo changes, **all downstream repos that vendor it** may need revendoring before the release (see Step 2a). The vendoring check in Step 2 catches these automatically. |
| 38 | + |
| 39 | +--- |
| 40 | + |
| 41 | +## Release Execution Plan |
| 42 | + |
| 43 | +### Prerequisites (Pre-flight Checklist) |
| 44 | + |
| 45 | +1. Ensure these tools are installed: `gh`, `yq` (Go version v4.x), `jq`, `perl`, `git`, `make`, `tput`, `sed` |
| 46 | +2. Set `GH_TOKEN` env var with a GitHub **classic** token (not fine-grained) with `repo` scope. The token is 40 characters, starting with `ghp_`. |
| 47 | +3. Verify `gh` authentication: `gh auth status`. Confirm it shows the expected user and token scopes. |
| 48 | +4. Verify SSH access to all repo URLs (`ssh -T git@github.com`) |
| 49 | +5. Decide the release type (`-B major|minor|patch`). **Always pass `-B` explicitly** — do not rely on the script default, which could change. Most releases use `-B patch`. |
| 50 | +6. **Pre-release vendoring check:** If any upstream repo has had changes merged to master since the last release that affect downstream vendoring, ensure downstream repos have been revendored **before** starting the release. For example, if `nnf-sos` changed, then `nnf-dm` and `nnf-integration-test` (which vendor `nnf-sos`) must have their vendor directories updated via PRs merged to master. Phase 1's vendoring checks will catch any mismatches. |
| 51 | + |
| 52 | +### Phase 0: Fresh Clone |
| 53 | + |
| 54 | +> The release uses a **volatile** working directory that is deleted and recreated at the start of each release. Set `$RELEASE_WORKDIR` to the desired location (e.g., `RELEASE_WORKDIR=~/release-work`). |
| 55 | +
|
| 56 | +#### Step 0 — Clone nnf-deploy with submodules |
| 57 | + |
| 58 | +If `$RELEASE_WORKDIR` already exists, confirm with the user that it can be deleted. **If they say no, halt the release process.** |
| 59 | + |
| 60 | +```sh |
| 61 | +# Set the working directory for this release |
| 62 | +export RELEASE_WORKDIR=~/release-work # adjust as needed |
| 63 | + |
| 64 | +# If $RELEASE_WORKDIR exists, ask the user before proceeding. |
| 65 | +# If they decline, stop here. |
| 66 | + |
| 67 | +# Clean build artifacts before removing (rm -rf can fail on build outputs) |
| 68 | +if [ -d "$RELEASE_WORKDIR/nnf-deploy/workingspace" ]; then |
| 69 | + for repo in "$RELEASE_WORKDIR"/nnf-deploy/workingspace/*/; do |
| 70 | + ( cd "$repo" && make clean-bin 2>/dev/null || true ) |
| 71 | + done |
| 72 | +fi |
| 73 | + |
| 74 | +rm -rf "$RELEASE_WORKDIR" |
| 75 | +mkdir -p "$RELEASE_WORKDIR" |
| 76 | +cd "$RELEASE_WORKDIR" |
| 77 | +git clone --recurse-submodules git@github.com:NearNodeFlash/nnf-deploy.git |
| 78 | +cd nnf-deploy |
| 79 | +``` |
| 80 | + |
| 81 | +All subsequent steps run from `$RELEASE_WORKDIR/nnf-deploy`. |
| 82 | + |
| 83 | +### General Notes |
| 84 | + |
| 85 | +> **Helper functions:** After sourcing `setup-release-env.sh`, use `nnf_cmd <phase> <repo>` for all `release-all.sh` invocations. It automatically handles the `-B` flag, the `-M` flag for `nnf_sos`, and pipes through `cat` to avoid pager issues. Use `nnf_create_pr <repo>` for the `create-pr` phase (captures `$PR_NUMBER`) and `nnf_add_reviewers <repo>` to assign reviewers. |
| 86 | +> |
| 87 | +> **Manual invocations:** If you need to call `release-all.sh` directly, always pass `-B patch` (or `-B minor`/`-B major`) explicitly, add `-M` for `nnf_sos`, and append `2>&1 | cat`. |
| 88 | +
|
| 89 | +### Phase 1: Discovery & Validation |
| 90 | + |
| 91 | +#### Step 1 — Set up and list repos |
| 92 | + |
| 93 | +```sh |
| 94 | +cd tools/release-all |
| 95 | +``` |
| 96 | + |
| 97 | +Source the release environment helper to auto-compute versions, detect the release operator, and set reviewers: |
| 98 | + |
| 99 | +```sh |
| 100 | +source ./setup-release-env.sh -B patch # or -B minor / -B major |
| 101 | +``` |
| 102 | + |
| 103 | +This exports `$PREVIOUS_RELEASE`, `$NNF_RELEASE`, `$RELEASE_TYPE`, and `$RELEASE_REVIEWERS`, and defines helper functions `nnf_cmd`, `nnf_create_pr`, `nnf_add_reviewers`, and `nnf_gh_repo`. Confirm the printed summary with the user before proceeding. |
| 104 | + |
| 105 | +List repos: |
| 106 | + |
| 107 | +```sh |
| 108 | +./release-all.sh -L |
| 109 | +``` |
| 110 | + |
| 111 | +Save the ordered list: `dws`, `lustre_csi_driver`, `lustre_fs_operator`, `nnf_mfu`, `nnf_ec`, `nnf_sos`, `nnf_dm`, `nnf_integration_test`, `nnf_deploy`, `nnf_doc`. |
| 112 | + |
| 113 | +#### Step 2 — Check vendoring *(sequential, per repo)* |
| 114 | + |
| 115 | +> Must be error-free before proceeding. This runs on **all 10 repos** including `nnf_doc` — vendoring checks validate the current state of master/main, which is independent of the release branching. |
| 116 | +
|
| 117 | +```sh |
| 118 | +for repo in $(./release-all.sh -L); do |
| 119 | + nnf_cmd master "$repo" |
| 120 | +done |
| 121 | +``` |
| 122 | + |
| 123 | +> **Important:** Run vendoring checks on **all** repos, not just the ones you plan to release. This catches stale submodule pointers in `nnf-deploy` (e.g., a force-push on a submodule repo can leave the pointer at an orphaned commit SHA). |
| 124 | +
|
| 125 | +#### Step 2a — Fix stale vendoring *(if Step 2 fails)* |
| 126 | + |
| 127 | +If the vendoring check for a repo reports **"Peer modules are behind"**, the error output includes the exact `go get` commands needed to update the vendor. Follow this procedure for each failing repo, working **in dependency order** per the vendoring table above (e.g., fix `lustre_fs_operator` before `nnf_sos`, fix `nnf_sos` before `nnf_dm` or `nnf_integration_test`). |
| 128 | + |
| 129 | +For each failing repo: |
| 130 | + |
| 131 | +1. **Read the error output.** It contains the `go get` command(s) with the specific module paths and branch targets. |
| 132 | +2. **Clone the repo** into a temporary directory using `nnf_gh_repo` to resolve the GitHub `owner/repo` path. |
| 133 | +3. **Create a branch** named `update-vendor`. |
| 134 | +4. **Run the `go get` commands** from the error output, then run `go mod tidy` and `go mod vendor`. |
| 135 | +5. **Commit** all changes with a signed-off commit (`-s`) and message `"Update vendor dependencies"`. |
| 136 | +6. **Push** the branch and **create a PR** titled `"Update vendor dependencies"` with body `"Pre-release vendoring update."`. |
| 137 | +7. **Assign reviewers** from `$RELEASE_REVIEWERS`. |
| 138 | +8. **Wait for CI** and reviewer approval, then **merge** the PR. |
| 139 | +9. **Verify the fix against origin:** Re-clone (repeat Step 0) so you're working from what's actually on GitHub, then re-run the `nnf_cmd master <repo>` check starting from the repo you just fixed and continuing through all remaining downstream repos in order. The fixed repo must now pass. If a downstream repo fails, repeat this procedure (steps 1–9) for it. |
| 140 | +10. Return to `$RELEASE_WORKDIR/nnf-deploy/tools/release-all`. |
| 141 | + |
| 142 | +> **Dependency order matters:** An upstream repo's vendoring PR must be merged to master before any downstream repo that vendors it can be fixed. For example, if both `nnf_sos` and `nnf_dm` fail, merge the `nnf_sos` fix first — `nnf_dm` vendors `nnf_sos` and needs the updated master. |
| 143 | +> |
| 144 | +> **`nnf_deploy` submodule pointers:** Any submodule repo that has had commits merged to master since the last submodule pointer update — whether from development work or vendoring fixes — will cause `nnf_deploy`'s Step 2 check to fail with "Submodules are not up to date." The script catches this automatically. Fix it the same way: clone `nnf_deploy`, create a branch, run `git submodule update --remote` for the affected submodules, commit, PR, merge. Then re-clone and verify. |
| 145 | +
|
| 146 | +After fixing and verifying the last failing repo, **re-run Step 2 on all repos** as a final safety net to confirm everything passes end to end. |
| 147 | + |
| 148 | +### Phase 2: Branch Creation |
| 149 | + |
| 150 | +#### Step 3 — Create trial release branches *(sequential, per repo)* |
| 151 | + |
| 152 | +Process all repos **except `nnf_doc`**, which is deferred (see below): |
| 153 | + |
| 154 | +```sh |
| 155 | +for repo in $(./release-all.sh -L | grep -v nnf_doc); do |
| 156 | + nnf_cmd release "$repo" |
| 157 | +done |
| 158 | +``` |
| 159 | + |
| 160 | +Review output for merge conflicts before continuing. |
| 161 | + |
| 162 | +> **Note:** `nnf_mfu` and `nnf_ec` are standalone repos that often have no new commits between releases. If either reports "No new changes to release", this is normal — **skip that repo in all subsequent phases** (Steps 4a–4d and the `nnf_doc` sequence). Do not attempt to push, PR, merge, or tag a repo that had no changes. |
| 163 | +
|
| 164 | +#### `nnf_doc` deferral |
| 165 | + |
| 166 | +`nnf_doc` must be deferred until after `nnf_deploy` is tagged and its GitHub Release is published. The `nnf_doc` script updates `mkdocs.yml` with the latest `nnf-deploy` release version by querying GitHub Releases (not just tags). Follow this sequence: |
| 167 | + |
| 168 | +1. Complete Steps 3–4d for all repos through `nnf_deploy` |
| 169 | +2. **Wait ~60 seconds** for `nnf_deploy`'s "Handle Release Tag" GitHub Actions workflow to publish the GitHub Release |
| 170 | +3. Verify: `gh release view $NNF_RELEASE -R NearNodeFlash/nnf-deploy 2>&1 | cat` should succeed. If it fails with "release not found", wait another 30 seconds and retry. The workflow typically completes within 2 minutes; if it hasn't after 5 minutes, check the Actions tab on the `nnf-deploy` repo for errors. |
| 171 | +4. Then run `nnf_doc` through Steps 3–4d: |
| 172 | + |
| 173 | +```sh |
| 174 | +nnf_cmd release nnf_doc |
| 175 | +nnf_cmd release-push nnf_doc |
| 176 | +nnf_create_pr nnf_doc |
| 177 | +nnf_add_reviewers nnf_doc |
| 178 | +nnf_cmd merge-pr nnf_doc |
| 179 | +nnf_cmd tag-release nnf_doc |
| 180 | +``` |
| 181 | + |
| 182 | +> **Important:** `nnf_doc`'s repo is `NearNodeFlash/NearNodeFlash.github.io` and uses the `main` branch (not `master`). |
| 183 | +
|
| 184 | +### Phase 3: Release Generation |
| 185 | + |
| 186 | +*Complete steps 4a–4d for each repo sequentially before moving on to the next. Present the results of all four sub-steps to the user and wait for approval once per repo (after step 4d), rather than after each individual sub-step.* |
| 187 | + |
| 188 | +> **Why sequential order matters:** Phase 2 creates release branches (not tags). Phase 3 then processes each repo through push → PR → merge → tag **in dependency order**. By the time a downstream repo (e.g., `nnf_dm`) reaches `create-pr`, any upstream repo it depends on (e.g., `nnf_sos`) is already tagged. The script's `release` phase handles submodule pointers to the correct release branches automatically. |
| 189 | +
|
| 190 | +#### Step 4a — Push release branch |
| 191 | + |
| 192 | +*Only if no merge conflicts.* |
| 193 | + |
| 194 | +```sh |
| 195 | +nnf_cmd release-push <repo> |
| 196 | +``` |
| 197 | + |
| 198 | +#### Step 4a-alt — Merge conflict resolution |
| 199 | + |
| 200 | +*Only if merge conflicts exist.* **Stop and present the conflict details to the user.** Merge conflicts during a release are rare and require human judgment. Do not attempt to resolve them automatically. Wait for the user to provide guidance before proceeding. |
| 201 | + |
| 202 | +#### Step 4b — Create PR |
| 203 | + |
| 204 | +```sh |
| 205 | +nnf_create_pr <repo> |
| 206 | +``` |
| 207 | + |
| 208 | +The `create-pr` output includes the PR URL (e.g., `https://github.com/<owner>/<repo>/pull/123`). The `nnf_create_pr` function extracts the PR number and sets `$PR_NUMBER` automatically. |
| 209 | + |
| 210 | +> **Note:** The script does **not** assign PR reviewers automatically. After each `create-pr`, assign reviewers: |
| 211 | +
|
| 212 | +```sh |
| 213 | +nnf_add_reviewers <repo> |
| 214 | +``` |
| 215 | + |
| 216 | +#### Step 4c — Merge PR |
| 217 | + |
| 218 | +> Do NOT manually merge — let the tool do it. |
| 219 | +
|
| 220 | +```sh |
| 221 | +nnf_cmd merge-pr <repo> |
| 222 | +``` |
| 223 | + |
| 224 | +> **Note:** Before running `merge-pr`, ensure CI status checks on the PR have passed. If required checks are still running, `merge-pr` may fail. |
| 225 | +> |
| 226 | +> **Note:** `merge-pr` may produce no visible output even on success. Verify the merge completed: |
| 227 | +> |
| 228 | +> ```sh |
| 229 | +> gh api repos/<owner>/<repo>/pulls/<pr_number> 2>&1 | grep -o '"merged":[^,]*' |
| 230 | +> ``` |
| 231 | +> |
| 232 | +> Expected output: `"merged":true` |
| 233 | +
|
| 234 | +#### Step 4d — Tag the release |
| 235 | +
|
| 236 | +*Creates the annotated tag required by CI/CD.* |
| 237 | +
|
| 238 | +```sh |
| 239 | +nnf_cmd tag-release <repo> |
| 240 | +``` |
| 241 | +
|
| 242 | +> **Note:** `tag-release` output will include "Bypassed rule violations for refs/tags/...". This is expected. The repos have rulesets ("Auto-imported tag create protections" and "Auto-imported tag delete protections") that block creation, update, and deletion of `v*` tags by default — protecting release tags from unauthorized changes. Your account bypasses these rules because it has an admin or maintain role, which is in the ruleset's bypass list. The tags are created correctly. |
| 243 | +
|
| 244 | +### Phase 4: Finalize |
| 245 | + |
| 246 | +#### Step 5 — Finalize release notes |
| 247 | + |
| 248 | +*Run only after ALL repos, including `nnf_doc`, are released. Run from `tools/release-all/`.* |
| 249 | + |
| 250 | +```sh |
| 251 | +./final-release-notes.sh -r $NNF_RELEASE 2>&1 | cat # preview |
| 252 | +./final-release-notes.sh -r $NNF_RELEASE -C 2>&1 | cat # commit |
| 253 | +``` |
| 254 | + |
| 255 | +### Phase 5: Verification |
| 256 | + |
| 257 | +#### Step 6 — Compare release manifests |
| 258 | + |
| 259 | +*Run from `tools/release-all/`.* |
| 260 | + |
| 261 | +```sh |
| 262 | +./compare-releases.sh -i $PREVIOUS_RELEASE $NNF_RELEASE 2>&1 | cat |
| 263 | +``` |
| 264 | + |
| 265 | +The `-i` flag displays image version changes inline. Use `-d` to display the full diff. The diff file is always saved to `workingspace/manifest-<ver1>-to-<ver2>.diff`. |
| 266 | + |
| 267 | +#### Step 7 — Verify GitHub releases |
| 268 | + |
| 269 | +Check each repo for correct tag, release notes, and artifacts (`manifests.tar` + `manifests-kind.tar` on `nnf-deploy`). |
| 270 | + |
| 271 | + |
| 272 | + |
| 273 | +--- |
| 274 | + |
| 275 | +### Relevant files |
| 276 | + |
| 277 | +- `tools/release-all/release-all.sh` — Main orchestration script, all phases |
| 278 | +- `tools/release-all/final-release-notes.sh` — Release notes finalization |
| 279 | +- `tools/release-all/compare-releases.sh` — Manifest diff tool |
| 280 | +- `tools/release-all/README.md` — Minimal, needs expansion |
| 281 | +- `.github/workflows/handle_release_tag.yaml` — CI/CD that creates GitHub releases on tag push |
| 282 | +- `config/repositories.yaml` — Submodule version tracking updated by release-push phase |
| 283 | +- External docs source: NearNodeFlash/NearNodeFlash.github.io repo |
| 284 | + |
| 285 | +### Decisions |
| 286 | + |
| 287 | +- Release execution follows the strict 10-repo dependency order — no parallelization |
| 288 | +- `nnf-mfu` and `nnf-ec` are released standalone (positions 4 & 5) despite not being submodules |
| 289 | +- `nnf_doc` released last, version should match nnf-deploy |
| 290 | +- Default `-B patch` unless breaking changes present |
| 291 | + |
| 292 | +### Verification |
| 293 | + |
| 294 | +1. `./release-all.sh -L` confirms 10 repos in dependency order |
| 295 | +2. Zero errors from each `master` phase before proceeding |
| 296 | +3. GitHub release page shows correct annotated tag after each `tag-release` |
| 297 | +4. `compare-releases.sh` diff validates expected changes |
| 298 | +5. `final-release-notes.sh` output includes all submodule notes + CRD API info |
| 299 | +6. nnf-deploy GitHub release has manifests.tar and manifests-kind.tar |
| 300 | + |
| 301 | +### Further Considerations |
| 302 | + |
| 303 | +1. **Release type** — Is this patch, minor, or major? Determines `-B` flag. Recommend: default `patch` unless breaking changes. |
| 304 | +2. **Doc improvements timing** — Corrections applied first (branch `improve-release-docs` in NearNodeFlash.github.io), to be used as a guide during the release. Enhancements can follow after the release. |
| 305 | +3. **README expansion** — Add prerequisites and quick-start to `tools/release-all/README.md` inline; keep detailed guide external. |
0 commit comments