Address PR review feedback and dry-run findings

- Deduplicate Instructions.md (remove sections that duplicate release.agent.md)
- Strengthen core rules in release.agent.md with NEVER/ALWAYS/STOP emphasis
- Name release-all.sh as the central tool in agent file and plan intro
- Add .github/CODEOWNERS with 4 reviewers
- Read CODEOWNERS in setup-release-env.sh instead of hardcoded pool
- Add make clean-bin loop before rm -rf in Step 0
- Add gh auth status pre-flight check (prereq 3)
- Add retry guidance for nnf_deploy GitHub Release wait
- Add empty RELEASE_REVIEWERS warning guard

Signed-off-by: Anthony Floeder <anthony.floeder@hpe.com>

Author: ajfloeder

.github/CODEOWNERS

@@ -0,0 +1,3 @@
+# NNF release reviewers — used by the release agent to assign PR reviewers.
+# All code owners are assigned as reviewers except the person creating the release.
+* @matthew-richerson @bdevcich @roehrich-hpe @ajfloeder

.github/agents/release.agent.md

@@ -6,28 +6,29 @@ argument-hint: "Specify the release type (patch, minor, or major) — defaults t
 
 You are the NNF Release Agent. Your job is to execute NNF software releases by following the documented release process precisely, step by step, with user approval at each gate.
 
+The entire release process is orchestrated by `release-all.sh` (located in `tools/release-all/`). It handles cloning, branching, PR creation, merging, and tagging for all NNF repos. The helper function `nnf_cmd` wraps `release-all.sh` with the correct flags.
+
 ## Required Reading
 
 Before starting any release work, read these files completely:
 
 1. [Release Plan](../../tools/release-all/RELEASE-PLAN.md) — The authoritative, self-contained guide for releases (repo table, dependency chain, all phases)
-2. [Agent Instructions](../../tools/release-all/Instructions.md) — Behavioral guidelines: execution model, approval gates, error handling, key gotchas
+2. [Agent Instructions](../../tools/release-all/Instructions.md) — Error handling specifics, key gotchas, and guidance for updating the plan
 
 ## Core Rules
 
-- **Wait for user approval** after every step before proceeding.
-- After each step, state: what you did, whether it succeeded, and the evidence.
-- Run all commands in a terminal so the user can see them.
-- Append `2>&1 | cat` to all `release-all.sh` and `gh` commands.
-- Execute one phase per repo at a time — never parallelize across repos.
-- If a command fails, analyze the error and propose a fix. Do not retry blindly.
-- Never use `--force`, `--no-verify`, or destructive operations without asking.
-- If you discover gaps or errors in the plan, update RELEASE-PLAN.md and note the change.
+- **ALWAYS wait for explicit user approval before proceeding to the next step.** After every step, state what you did, whether it succeeded, and the evidence. **WAIT for the user.**
+- **ALWAYS run commands in a terminal** so the user can see them.
+- **ALWAYS append `2>&1 | cat`** to every `release-all.sh` and `gh` command — no exceptions.
+- **ONE repo at a time.** Execute one phase per repo sequentially. **NEVER parallelize** across repos.
+- **STOP on failure.** If a command fails, analyze the error and propose a fix. **Do NOT retry blindly.**
+- **NEVER use `--force`, `--no-verify`, or destructive operations** without asking the user first.
+- If you discover gaps or errors in the plan, **update RELEASE-PLAN.md** and tell the user what you changed.
 
 ## Workflow
 
 1. Ask the user: What is the release type? (`patch`, `minor`, or `major` — default `patch`)
 2. Read the Release Plan and Agent Instructions completely.
-3. Follow the plan step by step, using the todo tool to track progress. The plan starts with cloning into a fresh working directory, then sources `setup-release-env.sh -B <type>` to set up environment variables and helper functions. Present the computed values to the user for confirmation.
+3. Follow the plan step by step, using the todo tool to track progress.
 4. At each approval gate, present your evidence and wait.
 5. After all repos are released, run `final-release-notes.sh` and `compare-releases.sh`.

tools/release-all/Instructions.md

@@ -1,73 +1,11 @@
 # Agent Instructions for NNF Software Release
 
-> **Agent entry point:** In VS Code, invoke the `@release` agent from the chat picker (defined in `.github/agents/release.agent.md`). The agent automatically loads this file and `RELEASE-PLAN.md` as context.
+> This file supplements `RELEASE-PLAN.md` (the authoritative reference) with agent-specific guidance. **All procedural steps and gotchas are in the plan — do not duplicate them here.**
 
-## Overview
-
-Execute the release plan defined in `RELEASE-PLAN.md` (in this same directory). That document is the authoritative, self-contained reference for the release process. Read it completely before starting.
-
-## Goal
-
-Create a release of the NNF software while ensuring the release process documented in `RELEASE-PLAN.md` is correct and repeatable. If you discover gaps, errors, or missing details in the plan, update it.
-
-## Agent Behavior
-
-### Execution Model
-
-- Run all commands in a terminal window so the user can see them.
-- Append `2>&1 | cat` to all `release-all.sh` and `gh` commands to avoid pager issues.
-- Execute one phase per repo at a time. Do not parallelize across repos.
-- Complete steps 4a–4d for each repo before moving to the next.
-
-### Approval Gates
-
-- **Wait for the user to approve** after each step before proceeding.
-- After each step, clearly state:
-  1. What you did
-  2. Whether you believe it succeeded
-  3. The evidence (command output, exit codes, API responses)
-- The user will respond with "approved", "approved. continue", or will coach you if something is wrong.
-
-### Error Handling
-
-- If a command fails, do **not** retry blindly. Analyze the error, explain it, and propose a fix.
-- If `merge-pr` produces no output, verify via: `gh api repos/<owner>/<repo>/pulls/<pr_number> 2>&1 | grep -o '"merged":[^,]*'`
-- If `tag-release` shows "Bypassed rule violations for refs/tags/..." — this is expected (tag protection rulesets allow admin/maintain bypass).
-- If a vendoring check fails, investigate the submodule pointer or vendor directory before proceeding.
-- Never use `--force`, `--no-verify`, or destructive operations without asking the user first.
-
-### Release Process
-
-The release process is the same regardless of how many repos have changes. The `release` phase automatically detects which repos have new commits since the last release. Repos with no changes report "No new changes to release" and are skipped in subsequent phases. Simply run all phases on all repos — the script handles the rest.
-
-### Key Gotchas (Learned from Experience)
-
-- `nnf_sos` requires the `-M` flag on ALL phases (not just `master`).
-- `nnf_mfu` and `nnf_ec` often have no changes — "No new changes to release" is normal, skip them.
-- `nnf_doc` must wait until after `nnf_deploy`'s GitHub Release is published (~60s after tagging). Verify with: `gh release view $NNF_RELEASE -R NearNodeFlash/nnf-deploy`
-- `nnf_doc` uses the `main` branch (not `master`). Its repo is `NearNodeFlash/NearNodeFlash.github.io`.
-- When a dependency changes (e.g., `nnf-sos`), all downstream repos that vendor it (`nnf-dm`, `nnf-integration-test`) must be revendored and their PRs merged **before** starting the release. If the vendoring check (Step 2) fails with "Peer modules are behind", follow Step 2a in RELEASE-PLAN.md to fix it. The check output prints the exact `go get` commands needed.
-- Always run vendoring checks on ALL repos — this catches stale submodule pointers.
-- After completing all releases, finalize release notes with `final-release-notes.sh` and run `compare-releases.sh` for verification.
-
-### Updating the Plan
+## Updating the Plan
 
 If you encounter an issue not covered by RELEASE-PLAN.md, or if a step's success criteria are unclear:
 
 1. Resolve the issue with the user's guidance.
 2. Update RELEASE-PLAN.md with what you learned.
 3. Note the update to the user.
-
-## Quick Start
-
-```sh
-# 1. Read the plan
-cat RELEASE-PLAN.md
-
-# 2. Ask the user: release type? (patch/minor/major, default patch)
-
-# 3. Source the helper: source ./setup-release-env.sh -B patch
-#    Confirm the printed summary with the user
-
-# 4. Follow RELEASE-PLAN.md step by step, waiting for approval at each gate.
-```

tools/release-all/RELEASE-PLAN.md

@@ -1,6 +1,6 @@
 # NNF Software Release Plan
 
-This document is the authoritative, self-contained guide for executing an NNF software release. The release process is the same regardless of how many repos have changes — the `release` phase automatically detects which repos have new commits since the last release and skips those that don't. An agent or human following this document should be able to execute the entire release without external context.
+This document is the authoritative, self-contained guide for executing an NNF software release. All release phases are driven by `release-all.sh` in this directory, wrapped by the `nnf_cmd` helper function. The release process is the same regardless of how many repos have changes — the `release` phase automatically detects which repos have new commits since the last release and skips those that don't. An agent or human following this document should be able to execute the entire release without external context.
 
 ---
 
@@ -19,14 +19,7 @@ This document is the authoritative, self-contained guide for executing an NNF so
 | `nnf_deploy` | `NearNodeFlash/nnf-deploy` | NearNodeFlash | `master` | Has submodules for most repos |
 | `nnf_doc` | `NearNodeFlash/NearNodeFlash.github.io` | NearNodeFlash | **`main`** (not master) | Released last; tracks nnf-deploy version |
 
-**PR Reviewers:** Assign all members of the reviewer pool **except** the person creating the release:
-
-- `matthew-richerson`
-- `bdevcich`
-- `roehrich-hpe`
-- `ajfloeder`
-
-For example, if `ajfloeder` is running the release, the reviewers are `matthew-richerson`, `bdevcich`, `roehrich-hpe`.
+**PR Reviewers:** Defined in `.github/CODEOWNERS`. Assign all code owners **except** the person creating the release. The `setup-release-env.sh` helper reads CODEOWNERS automatically and sets `$RELEASE_REVIEWERS`.
 
 **Dependency chain (release order):**
 
@@ -51,9 +44,10 @@ If an upstream repo changes, **all downstream repos that vendor it** may need re
 
 1. Ensure these tools are installed: `gh`, `yq` (Go version v4.x), `jq`, `perl`, `git`, `make`, `tput`, `sed`
 2. Set `GH_TOKEN` env var with a GitHub **classic** token (not fine-grained) with `repo` scope. The token is 40 characters, starting with `ghp_`.
-3. Verify SSH access to all repo URLs (`ssh -T git@github.com`)
-4. Decide the release type (`-B major|minor|patch`). **Always pass `-B` explicitly** — do not rely on the script default, which could change. Most releases use `-B patch`.
-5. **Pre-release vendoring check:** If any upstream repo has had changes merged to master since the last release that affect downstream vendoring, ensure downstream repos have been revendored **before** starting the release. For example, if `nnf-sos` changed, then `nnf-dm` and `nnf-integration-test` (which vendor `nnf-sos`) must have their vendor directories updated via PRs merged to master. Phase 1's vendoring checks will catch any mismatches.
+3. Verify `gh` authentication: `gh auth status`. Confirm it shows the expected user and token scopes.
+4. Verify SSH access to all repo URLs (`ssh -T git@github.com`)
+5. Decide the release type (`-B major|minor|patch`). **Always pass `-B` explicitly** — do not rely on the script default, which could change. Most releases use `-B patch`.
+6. **Pre-release vendoring check:** If any upstream repo has had changes merged to master since the last release that affect downstream vendoring, ensure downstream repos have been revendored **before** starting the release. For example, if `nnf-sos` changed, then `nnf-dm` and `nnf-integration-test` (which vendor `nnf-sos`) must have their vendor directories updated via PRs merged to master. Phase 1's vendoring checks will catch any mismatches.
 
 ### Phase 0: Fresh Clone
 
@@ -69,6 +63,14 @@ export RELEASE_WORKDIR=~/release-work   # adjust as needed
 
 # If $RELEASE_WORKDIR exists, ask the user before proceeding.
 # If they decline, stop here.
+
+# Clean build artifacts before removing (rm -rf can fail on build outputs)
+if [ -d "$RELEASE_WORKDIR/nnf-deploy/workingspace" ]; then
+    for repo in "$RELEASE_WORKDIR"/nnf-deploy/workingspace/*/; do
+        ( cd "$repo" && make clean-bin 2>/dev/null || true )
+    done
+fi
+
 rm -rf "$RELEASE_WORKDIR"
 mkdir -p "$RELEASE_WORKDIR"
 cd "$RELEASE_WORKDIR"
@@ -134,11 +136,11 @@ For each failing repo:
 6. **Push** the branch and **create a PR** titled `"Update vendor dependencies"` with body `"Pre-release vendoring update."`.
 7. **Assign reviewers** from `$RELEASE_REVIEWERS`.
 8. **Wait for CI** and reviewer approval, then **merge** the PR.
-9. **Verify the fix against origin:** Re-clone (repeat Step 0) so you're working from what's actually on GitHub, then re-run Step 2 starting from the repo you just fixed and continuing through the remaining downstream repos. The fixed repo must pass. If a downstream repo fails, repeat this procedure for it.
+9. **Verify the fix against origin:** Re-clone (repeat Step 0) so you're working from what's actually on GitHub, then re-run the `nnf_cmd master <repo>` check starting from the repo you just fixed and continuing through all remaining downstream repos in order. The fixed repo must now pass. If a downstream repo fails, repeat this procedure (steps 1–9) for it.
 10. Return to `$RELEASE_WORKDIR/nnf-deploy/tools/release-all`.
 
 > **Dependency order matters:** An upstream repo's vendoring PR must be merged to master before any downstream repo that vendors it can be fixed. For example, if both `nnf_sos` and `nnf_dm` fail, merge the `nnf_sos` fix first — `nnf_dm` vendors `nnf_sos` and needs the updated master.
-
+>
 > **`nnf_deploy` submodule pointers:** Any submodule repo that has had commits merged to master since the last submodule pointer update — whether from development work or vendoring fixes — will cause `nnf_deploy`'s Step 2 check to fail with "Submodules are not up to date." The script catches this automatically. Fix it the same way: clone `nnf_deploy`, create a branch, run `git submodule update --remote` for the affected submodules, commit, PR, merge. Then re-clone and verify.
 
 After fixing and verifying the last failing repo, **re-run Step 2 on all repos** as a final safety net to confirm everything passes end to end.
@@ -157,15 +159,15 @@ done
 
 Review output for merge conflicts before continuing.
 
-> **Note:** `nnf_mfu` (and sometimes `nnf_ec`) may report "No new changes to release" if master has no commits since the last release. This is normal — skip the repo in subsequent phases.
+> **Note:** `nnf_mfu` and `nnf_ec` are standalone repos that often have no new commits between releases. If either reports "No new changes to release", this is normal — **skip that repo in all subsequent phases** (Steps 4a–4d and the `nnf_doc` sequence). Do not attempt to push, PR, merge, or tag a repo that had no changes.
 
 #### `nnf_doc` deferral
 
 `nnf_doc` must be deferred until after `nnf_deploy` is tagged and its GitHub Release is published. The `nnf_doc` script updates `mkdocs.yml` with the latest `nnf-deploy` release version by querying GitHub Releases (not just tags). Follow this sequence:
 
 1. Complete Steps 3–4d for all repos through `nnf_deploy`
 2. **Wait ~60 seconds** for `nnf_deploy`'s "Handle Release Tag" GitHub Actions workflow to publish the GitHub Release
-3. Verify: `gh release view $NNF_RELEASE -R NearNodeFlash/nnf-deploy 2>&1 | cat` should succeed
+3. Verify: `gh release view $NNF_RELEASE -R NearNodeFlash/nnf-deploy 2>&1 | cat` should succeed. If it fails with "release not found", wait another 30 seconds and retry. The workflow typically completes within 2 minutes; if it hasn't after 5 minutes, check the Actions tab on the `nnf-deploy` repo for errors.
 4. Then run `nnf_doc` through Steps 3–4d:
 
 ```sh
@@ -266,35 +268,7 @@ The `-i` flag displays image version changes inline. Use `-d` to display the ful
 
 Check each repo for correct tag, release notes, and artifacts (`manifests.tar` + `manifests-kind.tar` on `nnf-deploy`).
 
----
 
-## Part B: Documentation Improvements & Corrections
-
-### Corrections (issues found by comparing docs to actual scripts)
-
-> Items marked ✅ have been applied to `release-all.md` on branch `improve-release-docs`.
-> Items marked 🔲 remain open.
-
-1. ✅ **Missing prerequisites** — Added full Prerequisites section with tool table, `GH_TOKEN` env var, and SSH access check.
-2. ✅ **Missing `-B` flag** — Documented `-B major|minor|patch` (default: `patch`) in the Steps section.
-3. 🔲 **Missing `-w`, `-M` flags** — `-x force-tag` is now documented for error recovery, but `-w` (workspace dir override) and `-M` (multi-API-version bypass) are not yet documented. These are advanced/optional and could be an enhancement.
-4. ✅ **Step structure was confusing** — Restructured as steps 3a–3d with bold headers; 3a branching path made explicit.
-5. ✅ **Annotated tag requirement not called out** — Added Important note at step 3d explaining annotated tag requirement and CI/CD rejection of lightweight tags.
-6. ✅ **`repo-list` output never shown** — Added the full 10-repo dependency-ordered list under step 0.
-7. ✅ **nnf-mfu and nnf-ec distinction unclear** — Added *(standalone repo, not a submodule)* annotations and dependency-order list with rationale.
-8. ✅ **No error recovery guidance** — Added `-x force-tag=vX.Y.Z` recovery, manual merge warning on step 3c.
-9. ✅ **`compare-releases.sh` not framed as verification** — Renamed section to "Verify the release" and framed as a recommended verification step.
-10. ✅ **`final-release-notes.sh` prereqs not listed** — Added note listing `yq`, `gh`, `jq`, `perl`, and `GH_TOKEN` requirement.
-11. ✅ **No mention of CI/CD workflow** — Added explanation of `handle_release_tag.yaml` at step 3d (creates GitHub release, attaches artifacts).
-12. ✅ **Step numbering was wrong** — Renumbered as 3a–3d (was 1–5 sub-list); removed phantom step "e".
-
-### Enhancements
-
-1. **Add a pre-flight checklist section** before Step 1
-2. **Add a troubleshooting section** — tag rejected, yq version mismatch, GH_TOKEN expired, merge conflicts
-3. **Add "What happens under the hood"** — Brief description of what each phase does internally (updates kustomization images, Helm versions, Dockerfile refs, repositories.yaml)
-4. **Document full repo ordering with rationale** — Explain the dependency chain (dws → lustre → nnf-mfu/ec → nnf-sos → nnf-dm → integration-test → deploy → docs)
-5. **Expand `tools/release-all/README.md`** — Add prerequisites and quick-start
 
 ---
 

tools/release-all/setup-release-env.sh

@@ -59,11 +59,21 @@ NNF_RELEASE=$(echo "$PREVIOUS_RELEASE" | perl -F'/\./' -ane "$PERL_CMD")
 
 # --- Detect operator & reviewers --------------------------------------------
 RELEASE_USER=$(gh api user --jq '.login' 2>&1)
-REVIEWER_POOL="matthew-richerson,bdevcich,roehrich-hpe,ajfloeder"
+CODEOWNERS_FILE="../../.github/CODEOWNERS"
+if [[ ! -f "$CODEOWNERS_FILE" ]]; then
+    echo "ERROR: $CODEOWNERS_FILE not found. Run from tools/release-all/ inside an nnf-deploy clone." >&2
+    return 1
+fi
+REVIEWER_POOL=$(grep '^\*' "$CODEOWNERS_FILE" | tr -s ' ' | cut -d' ' -f2- | tr ' ' '\n' | sed 's/@//' | paste -sd,)
 export RELEASE_REVIEWERS
 RELEASE_REVIEWERS=$(echo "$REVIEWER_POOL" | tr ',' '\n' \
     | grep -v "$RELEASE_USER" | paste -sd,)
 
+if [[ -z "$RELEASE_REVIEWERS" ]]; then
+    echo "WARNING: RELEASE_REVIEWERS is empty — you are the only code owner in CODEOWNERS." >&2
+    echo "         You will need to assign reviewers manually for each PR." >&2
+fi
+
 # --- Helper: run release-all.sh with auto -M and -B -------------------------
 # Usage: nnf_cmd <phase> <repo>
 nnf_cmd() {