Split clinic preparation into its own runbook; correct private endpoint guidance (#171)

clinic-preparation.md is now the checklist for running a clinic
(week-before / day-before / morning / during / after), carrying the
July dummy-run lessons: the gateway_images flag lives in the Manage
repo's flags.production.yml and needs a Manage deploy to change; the
VM must be confirmed on and unable to sleep (it has been found switched
off, which also silently hangs deploys by queueing run commands
forever); accession numbers must be fresh per clinic; the relay path
is proven by traffic, not configuration.

Also: ship debug_toolkit.ps1 in the release package so it is already
on the VM; add deploy-hang troubleshooting to the onboarding runbook
(machine off, stale run-command queue); and add the manual-images-mode
gotcha to the promotion runbook.

Author: carlosmartinez

docs/deployment/runbooks/clinic-preparation.md

@@ -0,0 +1,82 @@
+# Clinic preparation
+
+> **When to use**: Preparing for a live clinic (or a dummy run) that uses the
+> gateway imaging flow. Work through it in order — the sections have lead times.
+
+---
+
+## The week before
+
+- [ ] **Software currency**: the intended gateway release is deployed to the site VM
+      (check the deploy pipeline history, or `current` junction target on the VM),
+      and any Manage changes the clinic depends on are in prod.
+- [ ] **Feature flag**: `gateway_images` is **on** for prod — it lives in
+      **`flags.production.yml` in the Manage repo**.
+- [ ] **Prod Manage records**: `Gateway` record with the machine's managed-identity
+      OID; `Relay` record whose **Setting matches the clinic's Setting**.
+- [ ] **Test data prepared with fresh accession numbers** — never reuse a previous
+      clinic's CSV: Manage's per-gateway accession uniqueness rejects reused values,
+      and gateway-side reuse serves stale participant data.
+- [ ] **First time at a site, or after a promotion**: run a full remote dummy run
+      first — one test participant end to end, glove-box exposure included.
+
+## The day before
+
+- [ ] **The VM is on and will stay on** — `az connectedmachine show … --query status`
+      returns **Connected**.
+- [ ] **The relay path passes traffic** — echo probe from the Manage container, or
+      a fresh successful check-in.
+- [ ] **Fresh appointments only** — any appointment that ever had manual image
+      details recorded stays in manual mode by design.
+- [ ] **Gateway worklist DB is clean** — no stale items from previous tests
+      (`Get-GwWorklist`, or the run-command SELECT; wipe if needed).
+- [ ] **People and access roster confirmed for the whole clinic window**:
+  - someone with a **prod Rubie login** including Django admin (Gateway actions)
+  - someone with **az / portal access** to the prod subscriptions
+  - the **hospital contact** who can operate the modality and physically reach the VM
+  - RDP to the VM if the hospital can provide it (live tails beat run-commands)
+- [ ] **Fallback agreed**: if the gateway flow breaks mid-clinic, the manual-images
+      flow carries the clinic and debugging happens without time pressure.
+- [ ] **No stale Arc run-commands queued** on the machine
+      (`az connectedmachine run-command list -o table`; delete non-terminal stragglers —
+      they block deploys and diagnostics).
+
+## The morning (~20 minutes before the first participant)
+
+On the VM (RDP or run-command):
+
+1. `Restart-Service Gateway-Relay` — fresh relay connection, freshly-started token
+   clock. Wait for **`Connected - waiting for worklist actions...`** in the log.
+2. `Get-GwHealth` — all four services **Running**, ports 104 and 11112 listening.
+3. After the first (test) participant is checked in on Manage: `Get-GwWorklist`
+   shows the item almost immediately — that proves Manage → Relay → VM
+   end to end.
+4. On the modality: query the worklist and confirm the participant is listed.
+
+From a laptop:
+
+1. `az relay hyco show … --query listenerCount` returns ≥ 1.
+2. Gateway actions in the Django admin: the test check-in shows **`confirmed`**
+   with an empty `last_error`.
+
+## During the clinic — passive monitoring
+
+On the VM, keep two windows open:
+
+```powershell
+Watch-GwLog PACS     # window 1 — see each C-STORE arrive
+Watch-GwLog Upload   # window 2 — see each upload to Manage
+```
+
+Elsewhere: the **Gateway actions** admin page (each check-in → `confirmed`), and
+App Insights **Live Metrics** if telemetry is up.
+
+After each participant: `Get-GwImages` shows the expected image count with
+`uploaded` equal to `images` within a minute or two.
+
+## Afterwards
+
+- [ ] Wipe test items from the gateway worklist DB; note any test studies/images
+      created in prod Rubie and clean up or record them.
+- [ ] Write up anything that surprised you — against this checklist — and fold the
+      lesson back into this document. That is how everything above got here.

docs/deployment/runbooks/live-clinic-debugging.md

@@ -30,10 +30,10 @@ in pipeline order.
 
 ## On the gateway VM
 
-Copy `debug_toolkit.ps1` to the VM (or paste it into an editor there), then:
+The toolkit ships with the deployed release:
 
 ```powershell
-. .\debug_toolkit.ps1
+. 'C:\Program Files\NHS\ManageBreastScreeningGateway\current\scripts\powershell\debug_toolkit.ps1'
 Get-GwHealth        # start here: services, ports, relay connection, disk
 ```
 
@@ -158,8 +158,15 @@ python scripts/diagnostics/relay_echo_probe.py \
 There is also `--bearer-token` mode (see the script's docstring) to test the AAD
 bearer path Manage uses in production. To run the probe from *inside* the Manage
 container (testing Manage's actual network/DNS/identity), see the container-side
-snippet in the script's docstring — during the July 2026 incident that variant is
-what exposed the relay hostname resolving to a private endpoint IP inside the VNet.
+snippet in the script's docstring.
+
+**Interpreting in-container DNS resolution**: prod Manage reaches the relay via a
+**private endpoint**, so the relay hostname resolving to a **private IP** inside
+the VNet is *correct*. During the July 2026 incident the PE silently blackholed
+all traffic **while showing Approved** — likely an Azure-side fault or race at
+PE creation; the interim fix was pointing DNS at the public endpoint, superseded
+by deleting and recreating the PE. The health question is never "which IP family"
+or "what does the PE state say" — it is **"does the echo come back"**.
 
 Rotate the namespace key after a debugging session — nothing in a deployed
 environment uses SAS, so rotation is free.
@@ -200,30 +207,3 @@ az monitor log-analytics query --workspace <workspace-customer-id> \
 Manage-side data checks (worklist item actually created and sent?) are quickest in
 the Django admin: **Gateway actions** — a healthy send is `status: confirmed`;
 `failed` rows carry `last_error` and the retry schedule.
-
-## Pre-flight checklist (run before the clinic starts)
-
-On the VM:
-
-1. `Get-GwHealth` — all four services **Running**, ports 104 and 11112 listening,
-   relay log shows "Connected - waiting for worklist actions", disk has headroom.
-2. `Get-GwWorklist` — after the first participant is checked in on Manage, the item
-   appears here within seconds. That one check proves Manage → Relay → VM end to end.
-3. On the modality: query the worklist and confirm the participant is listed.
-
-From the laptop:
-
-4. `az relay hyco show ... --query listenerCount` returns ≥ 1.
-5. App Insights errors query over `ago(12h)` is quiet (no crash loops overnight).
-
-## During the clinic — a useful passive setup
-
-On the VM, keep two windows open:
-
-```powershell
-Watch-GwLog PACS     # window 1 — see each C-STORE arrive
-Watch-GwLog Upload   # window 2 — see each upload to Manage
-```
-
-After each participant: `Get-GwImages` should show the expected image count with
-`uploaded` equal to `images` within a minute or two.

docs/deployment/runbooks/onboard-hospital-vm.md

@@ -284,6 +284,13 @@ If disconnected, re-run the script. It is safe to re-run — the agent will reco
 
 The Arc machine must appear in the resource group before Terraform can create the HC. Confirm the machine is **Connected** in the portal (Step 2 verify), then re-run the pipeline.
 
+### Deploy pipeline hangs with no output after the ring banner
+
+The deploy job prints the `--- Ring: … ---` banner and then goes silent for 30+ minutes. Possible causes:
+
+1. **The VM is powered off or asleep.** `az connectedmachine show -n <machine> -g <rg> --query status` — anything other than **Connected** means run commands queue indefinitely against an absent machine.
+2. **Stale run commands blocking the queue.** Run commands execute serially per machine, and diagnostic commands accumulate as persistent resources. `az connectedmachine run-command list -o table` — delete anything stuck in a non-terminal state.
+
 ### Gateway services not starting after deploy
 
 Check the deployment log on the VM:

docs/deployment/runbooks/promote-vm-preprod-to-prod.md

@@ -142,6 +142,8 @@ With the modality reconfigured and prod Rubie seeded:
 
 If step 2 fails, the Gateway action's `last_error` names the failing phase — see the [live clinic debugging runbook](./live-clinic-debugging.md) for the symptom → check mapping.
 
+Before the site's first real clinic on the promoted gateway, work through [Clinic preparation](./clinic-preparation.md).
+
 ---
 
 ## Gotchas
@@ -153,7 +155,8 @@ If step 2 fails, the Gateway action's `last_error` names the failing phase — s
 | Modality C-ECHO fails after promotion | Modality still targeting `..._PRE` AE titles | Reconfigure modality to `..._PROD` (Step 5) |
 | Hybrid Connection not created by Terraform | Arc machine not yet **Connected** in the prod RG | Confirm Step 3 verify, then re-run the infra pipeline |
 | Pre-prod relay still shows connection attempts | Orphaned pre-prod HC / registration | Complete Step 6 |
-| Rubie check-ins time out with `phase: connecting` while the listener is healthy | Rubie-side network/DNS — historically, the relay hostname resolving to a private endpoint IP inside the Rubie VNet | See [live clinic debugging](./live-clinic-debugging.md); verify the relay FQDN resolves to a **public** IP from the Rubie container |
+| Rubie check-ins time out with `phase: connecting` while the listener is healthy | Rubie-side network — most likely a **broken relay private endpoint**. Note the PE can blackhole all traffic **while showing Approved** | Delete the PE and let Terraform recreate it, then prove the path with the echo probe from the Rubie container — see [live clinic debugging](./live-clinic-debugging.md) |
+| The whole UI runs in manual-images mode | `gateway_images` flag off in the Manage repo's **`flags.production.yml`**, or Relay↔Setting mismatch, or the appointment already has a manual study | Flag change requires a Manage deploy (one-line PR, leave the other flags alone); Setting fix is one field in admin; otherwise use a fresh appointment |
 | Image uploads rejected with 403 after promotion | `Gateway.oid` still holds the old pre-prod identity | Step 4.4 — update to the new principal ID |
 | Rubie shows the manual-images flow instead of awaiting images | `Relay` record's Setting doesn't match the clinic's Setting, or the appointment already has a manual study | Fix the Setting FK in the admin; retest with a fresh appointment |
 | Sends fail after the gateway has been idle overnight | Relay listener connection died silently (known issue) | `Restart-Service Gateway-Relay`; restart before every clinic until the listener liveness fix ships |

scripts/bash/package_release.sh

@@ -62,7 +62,8 @@ echo ""
 # ── Validate required files ───────────────────────────────────────────────────
 
 REQUIRED_FILES=("src/" "sample_images/" "scripts/python/database.py" \
-    "scripts/powershell/maintenance.ps1" "pyproject.toml" "uv.lock" "README.md" "LICENCE.md")
+    "scripts/powershell/maintenance.ps1" "scripts/powershell/debug_toolkit.ps1" \
+    "pyproject.toml" "uv.lock" "README.md" "LICENCE.md")
 
 for item in "${REQUIRED_FILES[@]}"; do
     if [[ ! -e "${REPO_ROOT}/${item}" ]]; then

scripts/diagnostics/relay_echo_probe.py

@@ -33,7 +33,8 @@
 
     az containerapp exec -n <app-name> -g <resource-group> --command sh
 
-    # 1. What does the relay hostname resolve to? Must be a PUBLIC IP.
+    # 1. What does the relay hostname resolve to? In an environment using a
+    #    private endpoint (prod does), a PRIVATE IP is correct.
     /app/.venv/bin/python -c "import socket; print(socket.getaddrinfo(
         'relay-manbrs-prod.servicebus.windows.net', 443)[0][4])"