diff --git a/docs/deployment/runbooks/clinic-preparation.md b/docs/deployment/runbooks/clinic-preparation.md new file mode 100644 index 00000000..41005891 --- /dev/null +++ b/docs/deployment/runbooks/clinic-preparation.md @@ -0,0 +1,82 @@ +# Clinic preparation + +> **When to use**: Preparing for a live clinic (or a dummy run) that uses the +> gateway imaging flow. Work through it in order — the sections have lead times. + +--- + +## The week before + +- [ ] **Software currency**: the intended gateway release is deployed to the site VM + (check the deploy pipeline history, or `current` junction target on the VM), + and any Manage changes the clinic depends on are in prod. +- [ ] **Feature flag**: `gateway_images` is **on** for prod — it lives in + **`flags.production.yml` in the Manage repo**. +- [ ] **Prod Manage records**: `Gateway` record with the machine's managed-identity + OID; `Relay` record whose **Setting matches the clinic's Setting**. +- [ ] **Test data prepared with fresh accession numbers** — never reuse a previous + clinic's CSV: Manage's per-gateway accession uniqueness rejects reused values, + and gateway-side reuse serves stale participant data. +- [ ] **First time at a site, or after a promotion**: run a full remote dummy run + first — one test participant end to end, glove-box exposure included. + +## The day before + +- [ ] **The VM is on and will stay on** — `az connectedmachine show … --query status` + returns **Connected**. +- [ ] **The relay path passes traffic** — echo probe from the Manage container, or + a fresh successful check-in. +- [ ] **Fresh appointments only** — any appointment that ever had manual image + details recorded stays in manual mode by design. +- [ ] **Gateway worklist DB is clean** — no stale items from previous tests + (`Get-GwWorklist`, or the run-command SELECT; wipe if needed). +- [ ] **People and access roster confirmed for the whole clinic window**: + - someone with a **prod Rubie login** including Django admin (Gateway actions) + - someone with **az / portal access** to the prod subscriptions + - the **hospital contact** who can operate the modality and physically reach the VM + - RDP to the VM if the hospital can provide it (live tails beat run-commands) +- [ ] **Fallback agreed**: if the gateway flow breaks mid-clinic, the manual-images + flow carries the clinic and debugging happens without time pressure. +- [ ] **No stale Arc run-commands queued** on the machine + (`az connectedmachine run-command list -o table`; delete non-terminal stragglers — + they block deploys and diagnostics). + +## The morning (~20 minutes before the first participant) + +On the VM (RDP or run-command): + +1. `Restart-Service Gateway-Relay` — fresh relay connection, freshly-started token + clock. Wait for **`Connected - waiting for worklist actions...`** in the log. +2. `Get-GwHealth` — all four services **Running**, ports 104 and 11112 listening. +3. After the first (test) participant is checked in on Manage: `Get-GwWorklist` + shows the item almost immediately — that proves Manage → Relay → VM + end to end. +4. On the modality: query the worklist and confirm the participant is listed. + +From a laptop: + +1. `az relay hyco show … --query listenerCount` returns ≥ 1. +2. Gateway actions in the Django admin: the test check-in shows **`confirmed`** + with an empty `last_error`. + +## During the clinic — passive monitoring + +On the VM, keep two windows open: + +```powershell +Watch-GwLog PACS # window 1 — see each C-STORE arrive +Watch-GwLog Upload # window 2 — see each upload to Manage +``` + +Elsewhere: the **Gateway actions** admin page (each check-in → `confirmed`), and +App Insights **Live Metrics** if telemetry is up. + +After each participant: `Get-GwImages` shows the expected image count with +`uploaded` equal to `images` within a minute or two. + +## Afterwards + +- [ ] Wipe test items from the gateway worklist DB; note any test studies/images + created in prod Rubie and clean up or record them. +- [ ] Write up anything that surprised you — against this checklist — and fold the + lesson back into this document. That is how everything above got here. diff --git a/docs/deployment/runbooks/live-clinic-debugging.md b/docs/deployment/runbooks/live-clinic-debugging.md index dfd3e56d..482d05e4 100644 --- a/docs/deployment/runbooks/live-clinic-debugging.md +++ b/docs/deployment/runbooks/live-clinic-debugging.md @@ -30,10 +30,10 @@ in pipeline order. ## On the gateway VM -Copy `debug_toolkit.ps1` to the VM (or paste it into an editor there), then: +The toolkit ships with the deployed release: ```powershell -. .\debug_toolkit.ps1 +. 'C:\Program Files\NHS\ManageBreastScreeningGateway\current\scripts\powershell\debug_toolkit.ps1' Get-GwHealth # start here: services, ports, relay connection, disk ``` @@ -158,8 +158,15 @@ python scripts/diagnostics/relay_echo_probe.py \ There is also `--bearer-token` mode (see the script's docstring) to test the AAD bearer path Manage uses in production. To run the probe from *inside* the Manage container (testing Manage's actual network/DNS/identity), see the container-side -snippet in the script's docstring — during the July 2026 incident that variant is -what exposed the relay hostname resolving to a private endpoint IP inside the VNet. +snippet in the script's docstring. + +**Interpreting in-container DNS resolution**: prod Manage reaches the relay via a +**private endpoint**, so the relay hostname resolving to a **private IP** inside +the VNet is *correct*. During the July 2026 incident the PE silently blackholed +all traffic **while showing Approved** — likely an Azure-side fault or race at +PE creation; the interim fix was pointing DNS at the public endpoint, superseded +by deleting and recreating the PE. The health question is never "which IP family" +or "what does the PE state say" — it is **"does the echo come back"**. Rotate the namespace key after a debugging session — nothing in a deployed environment uses SAS, so rotation is free. @@ -200,30 +207,3 @@ az monitor log-analytics query --workspace \ Manage-side data checks (worklist item actually created and sent?) are quickest in the Django admin: **Gateway actions** — a healthy send is `status: confirmed`; `failed` rows carry `last_error` and the retry schedule. - -## Pre-flight checklist (run before the clinic starts) - -On the VM: - -1. `Get-GwHealth` — all four services **Running**, ports 104 and 11112 listening, - relay log shows "Connected - waiting for worklist actions", disk has headroom. -2. `Get-GwWorklist` — after the first participant is checked in on Manage, the item - appears here within seconds. That one check proves Manage → Relay → VM end to end. -3. On the modality: query the worklist and confirm the participant is listed. - -From the laptop: - -4. `az relay hyco show ... --query listenerCount` returns ≥ 1. -5. App Insights errors query over `ago(12h)` is quiet (no crash loops overnight). - -## During the clinic — a useful passive setup - -On the VM, keep two windows open: - -```powershell -Watch-GwLog PACS # window 1 — see each C-STORE arrive -Watch-GwLog Upload # window 2 — see each upload to Manage -``` - -After each participant: `Get-GwImages` should show the expected image count with -`uploaded` equal to `images` within a minute or two. diff --git a/docs/deployment/runbooks/onboard-hospital-vm.md b/docs/deployment/runbooks/onboard-hospital-vm.md index 31b0efe2..88ab59b4 100644 --- a/docs/deployment/runbooks/onboard-hospital-vm.md +++ b/docs/deployment/runbooks/onboard-hospital-vm.md @@ -284,6 +284,13 @@ If disconnected, re-run the script. It is safe to re-run — the agent will reco The Arc machine must appear in the resource group before Terraform can create the HC. Confirm the machine is **Connected** in the portal (Step 2 verify), then re-run the pipeline. +### Deploy pipeline hangs with no output after the ring banner + +The deploy job prints the `--- Ring: … ---` banner and then goes silent for 30+ minutes. Possible causes: + +1. **The VM is powered off or asleep.** `az connectedmachine show -n -g --query status` — anything other than **Connected** means run commands queue indefinitely against an absent machine. +2. **Stale run commands blocking the queue.** Run commands execute serially per machine, and diagnostic commands accumulate as persistent resources. `az connectedmachine run-command list -o table` — delete anything stuck in a non-terminal state. + ### Gateway services not starting after deploy Check the deployment log on the VM: diff --git a/docs/deployment/runbooks/promote-vm-preprod-to-prod.md b/docs/deployment/runbooks/promote-vm-preprod-to-prod.md index 44cd6d38..c78973d4 100644 --- a/docs/deployment/runbooks/promote-vm-preprod-to-prod.md +++ b/docs/deployment/runbooks/promote-vm-preprod-to-prod.md @@ -142,6 +142,8 @@ With the modality reconfigured and prod Rubie seeded: If step 2 fails, the Gateway action's `last_error` names the failing phase — see the [live clinic debugging runbook](./live-clinic-debugging.md) for the symptom → check mapping. +Before the site's first real clinic on the promoted gateway, work through [Clinic preparation](./clinic-preparation.md). + --- ## Gotchas @@ -153,7 +155,8 @@ If step 2 fails, the Gateway action's `last_error` names the failing phase — s | Modality C-ECHO fails after promotion | Modality still targeting `..._PRE` AE titles | Reconfigure modality to `..._PROD` (Step 5) | | Hybrid Connection not created by Terraform | Arc machine not yet **Connected** in the prod RG | Confirm Step 3 verify, then re-run the infra pipeline | | Pre-prod relay still shows connection attempts | Orphaned pre-prod HC / registration | Complete Step 6 | -| Rubie check-ins time out with `phase: connecting` while the listener is healthy | Rubie-side network/DNS — historically, the relay hostname resolving to a private endpoint IP inside the Rubie VNet | See [live clinic debugging](./live-clinic-debugging.md); verify the relay FQDN resolves to a **public** IP from the Rubie container | +| Rubie check-ins time out with `phase: connecting` while the listener is healthy | Rubie-side network — most likely a **broken relay private endpoint**. Note the PE can blackhole all traffic **while showing Approved** | Delete the PE and let Terraform recreate it, then prove the path with the echo probe from the Rubie container — see [live clinic debugging](./live-clinic-debugging.md) | +| The whole UI runs in manual-images mode | `gateway_images` flag off in the Manage repo's **`flags.production.yml`**, or Relay↔Setting mismatch, or the appointment already has a manual study | Flag change requires a Manage deploy (one-line PR, leave the other flags alone); Setting fix is one field in admin; otherwise use a fresh appointment | | Image uploads rejected with 403 after promotion | `Gateway.oid` still holds the old pre-prod identity | Step 4.4 — update to the new principal ID | | Rubie shows the manual-images flow instead of awaiting images | `Relay` record's Setting doesn't match the clinic's Setting, or the appointment already has a manual study | Fix the Setting FK in the admin; retest with a fresh appointment | | Sends fail after the gateway has been idle overnight | Relay listener connection died silently (known issue) | `Restart-Service Gateway-Relay`; restart before every clinic until the listener liveness fix ships | diff --git a/scripts/bash/package_release.sh b/scripts/bash/package_release.sh index 33d58681..f756dce8 100755 --- a/scripts/bash/package_release.sh +++ b/scripts/bash/package_release.sh @@ -62,7 +62,8 @@ echo "" # ── Validate required files ─────────────────────────────────────────────────── REQUIRED_FILES=("src/" "sample_images/" "scripts/python/database.py" \ - "scripts/powershell/maintenance.ps1" "pyproject.toml" "uv.lock" "README.md" "LICENCE.md") + "scripts/powershell/maintenance.ps1" "scripts/powershell/debug_toolkit.ps1" \ + "pyproject.toml" "uv.lock" "README.md" "LICENCE.md") for item in "${REQUIRED_FILES[@]}"; do if [[ ! -e "${REPO_ROOT}/${item}" ]]; then diff --git a/scripts/diagnostics/relay_echo_probe.py b/scripts/diagnostics/relay_echo_probe.py index 983d2acb..631ee4b7 100644 --- a/scripts/diagnostics/relay_echo_probe.py +++ b/scripts/diagnostics/relay_echo_probe.py @@ -33,7 +33,8 @@ az containerapp exec -n -g --command sh - # 1. What does the relay hostname resolve to? Must be a PUBLIC IP. + # 1. What does the relay hostname resolve to? In an environment using a + # private endpoint (prod does), a PRIVATE IP is correct. /app/.venv/bin/python -c "import socket; print(socket.getaddrinfo( 'relay-manbrs-prod.servicebus.windows.net', 443)[0][4])"