Merge pull request #157 from NHSDigital/docs/promote-preprod

Update gateway deployment/onboarding documentation

Author: floormind

docs/deployment/deployment-pipeline.md

@@ -1,7 +1,7 @@
 # Deployment Pipeline
 
 > **Scope**: End-to-end flow from a GitHub tag or main-branch push through Azure DevOps pipelines to an Arc Run Command executing on hospital gateway VMs.
-> **Related docs**: [Windows Service Deploy](./windows-service-deploy.md) | [Onboard Hospital VM](./runbooks/onboard-hospital-vm.md) | [Rollback Runbook](./runbooks/rollback.md)
+> **Related docs**: [Windows Service Deploy](./windows-service-deploy.md) | [Onboard Hospital VM](./runbooks/onboard-hospital-vm.md) | [Promote VM Pre-Prod → Prod](./runbooks/promote-vm-preprod-to-prod.md) | [Rollback Runbook](./runbooks/rollback.md)
 
 ---
 

docs/deployment/runbooks/live-clinic-debugging.md

@@ -0,0 +1,229 @@
+# Live clinic debugging
+
+> **When to use**: On-the-day support for a live clinic — checking worklist delivery,
+> image receipt and upload from the gateway VM, and tracing problems through the
+> Azure logs for both the gateway and Manage.
+
+Two vantage points:
+
+- **The gateway VM** (RDP) — direct access to the MWL/PACS SQLite databases, DICOM
+  storage and service logs. Use [`scripts/powershell/debug_toolkit.ps1`](../../../scripts/powershell/debug_toolkit.ps1).
+- **Your laptop** (Azure CLI / portal) — Application Insights for the gateway
+  services, container app logs for Manage, and Azure Relay connection health.
+
+## The pipeline at a glance
+
+```
+Manage (container app)
+   │  worklist item ─── Azure Relay (relay-manbrs-prod / hc-<machine>) ──► Gateway-Relay
+   │                                                                          │
+   │                                                              writes data\worklist.db
+   │                                                                          │
+Modality ── C-FIND (port 104) ──► Gateway-MWL ◄── reads worklist.db ──────────┘
+Modality ── C-STORE (port 11112) ──► Gateway-PACS ──► data\storage + data\pacs.db
+                                                              │
+Manage /api/v1/dicom ◄── HTTPS upload (managed identity) ── Gateway-Upload
+```
+
+A failure at any hop has a distinct signature — the sections below work through them
+in pipeline order.
+
+## On the gateway VM
+
+Copy `debug_toolkit.ps1` to the VM (or paste it into an editor there), then:
+
+```powershell
+. .\debug_toolkit.ps1
+Get-GwHealth        # start here: services, ports, relay connection, disk
+```
+
+Reference points:
+
+| Thing | Where |
+|---|---|
+| Install root | `C:\Program Files\NHS\ManageBreastScreeningGateway` |
+| Services (NSSM) | `Gateway-Relay`, `Gateway-MWL`, `Gateway-PACS`, `Gateway-Upload` |
+| Worklist DB (SQLite) | `data\worklist.db` — table `worklist_items` |
+| PACS DB (SQLite) | `data\pacs.db` — table `stored_instances` |
+| DICOM files | `data\storage\` (hash-based directory layout; use `Get-GwDicomFiles`) |
+| Service logs | `logs\Gateway-<Service>.log` (NSSM stdout+stderr) |
+| Deployment logs | `logs\deployments\deploy-*` |
+| Config | `.env` in the install root |
+| MWL | AET `RUBIE_MWL_PROD`, port 104 |
+| PACS | AET `RUBIE_PACS_PROD`, port 11112 |
+
+### Symptom → first command
+
+| Symptom | Check |
+|---|---|
+| Appointment checked in on Manage but not on the modality worklist | `Get-GwWorklist` — is the item in the DB? If **no**: relay hop failed → `Watch-GwLog Relay`, then the Azure Relay + Manage sections below. If **yes**: modality⇄MWL hop → `Watch-GwLog MWL` while the modality refreshes; confirm port 104 listening; scheduled_date is stored in **UTC** `YYYYMMDD` |
+| Exposure taken but images not on the VM | `Watch-GwLog PACS` while the modality sends; `Get-GwImages`; confirm port 11112 and that the modality is configured with AE title `RUBIE_PACS_PROD`, correct IP and port |
+| Images on VM but not appearing in Manage | `Get-GwUploadFailures` — look at `upload_error` and `upload_attempt_count`; `Watch-GwLog Upload`. Auth errors here usually mean the Arc managed identity / `Gateway.Access` app role or `Gateway.oid` in Manage is wrong |
+| Item status stuck at SCHEDULED after procedure started | `Get-GwWorklistItem <ACC>` — MPPS updates set `status` / `mpps_instance_uid`; check `Watch-GwLog MWL` for MPPS N-CREATE/N-SET messages |
+| Anything crashing / restart loops | `Search-GwLogs`; `Get-Service Gateway-*` (NSSM restarts failed services automatically — repeated banner lines in the log mean crash-looping) |
+
+## From your laptop — Azure
+
+```bash
+az account set --subscription "Breast Screening - Manage Breast Screening - Prod"
+```
+
+### Gateway service telemetry (Application Insights)
+
+All four VM services send OpenTelemetry to `ai-mbsgw-prod-arc-uks`
+(resource group `rg-mbsgw-prod-uks-arc-enabled-servers`). `cloud_RoleName` values:
+`relay-listener`, `mwl-server`, `pacs-server`, `upload-listener`.
+
+```bash
+# Errors and warnings in the last hour, all services
+az monitor app-insights query \
+  --app ai-mbsgw-prod-arc-uks -g rg-mbsgw-prod-uks-arc-enabled-servers \
+  --analytics-query "traces
+    | where timestamp > ago(1h) and severityLevel >= 2
+    | project timestamp, cloud_RoleName, message
+    | order by timestamp desc" -o table
+
+# Exceptions with stack traces
+az monitor app-insights query \
+  --app ai-mbsgw-prod-arc-uks -g rg-mbsgw-prod-uks-arc-enabled-servers \
+  --analytics-query "exceptions
+    | where timestamp > ago(4h)
+    | project timestamp, cloud_RoleName, type, outerMessage
+    | order by timestamp desc" -o table
+
+# Everything one service said in a window (e.g. relay listener around a failure)
+az monitor app-insights query \
+  --app ai-mbsgw-prod-arc-uks -g rg-mbsgw-prod-uks-arc-enabled-servers \
+  --analytics-query "traces
+    | where timestamp > ago(1h) and cloud_RoleName == 'relay-listener'
+    | project timestamp, message | order by timestamp desc" -o table
+```
+
+Portal equivalent: the App Insights resource → **Logs**, or **Live Metrics** for
+real-time output during the clinic.
+
+> Telemetry is a no-op if `APPLICATIONINSIGHTS_CONNECTION_STRING` wasn't set at
+> deploy time — if queries return nothing at all, fall back to the VM logs and
+> check the deploy log for the "Application Insights resource not found" warning.
+
+### Azure Relay connection health
+
+The namespace is `relay-manbrs-prod` in `rg-manbrs-prod-uks`; the Hull hybrid
+connection is `hc-<arc-machine-name>`.
+
+```bash
+# Does the HC exist / listener count (listenerCount > 0 means the VM is connected)
+az relay hyco show --namespace-name relay-manbrs-prod -g rg-manbrs-prod-uks \
+  -n hc-gw-hull-university-teaching-hospitals-nhs-trust-rwa-01 \
+  --query "{name:name, listenerCount:listenerCount}"
+
+# Namespace metrics: listeners and sender connections over the morning
+az monitor metrics list \
+  --resource "$(az relay namespace show -g rg-manbrs-prod-uks -n relay-manbrs-prod --query id -o tsv)" \
+  --metric "ListenerConnections-Success" "SenderConnections-Success" "ListenerConnections-ClientError" \
+  --interval PT5M --offset 4h -o table
+```
+
+`listenerCount: 0` means `Gateway-Relay` on the VM is not connected — check the
+service on the VM (`Watch-GwLog Relay`; look for "Connected - waiting for worklist
+actions"). Sender client errors while listeners are healthy point at the Manage side
+(its managed identity / `AZURE_RELAY_CLIENT_ID`).
+
+> **Beware the zombie listener**: `listenerCount: 1` does not prove the listener is
+> *functional* — a connection that died without a clean close can stay registered
+> while rendezvous dispatches go nowhere. The echo probe below is the truth test.
+
+### The echo probe — test the relay → gateway path without Manage
+
+[`scripts/diagnostics/relay_echo_probe.py`](../../../scripts/diagnostics/relay_echo_probe.py)
+connects to the hybrid connection **as a sender** (exactly like Manage does), sends an
+`echo` action — which the relay listener answers without touching its database — and
+waits for the reply. It needs only Python, `pip install websockets`, and a credential;
+no Azure login, runnable from any laptop.
+
+```bash
+python scripts/diagnostics/relay_echo_probe.py \
+  --namespace relay-manbrs-prod.servicebus.windows.net \
+  --hc hc-gw-hull-university-teaching-hospitals-nhs-trust-rwa-01 \
+  --key '<RootManageSharedAccessKey value>'   # namespace-level key, NOT the HC's listen-only key
+```
+
+| Result | Meaning |
+|---|---|
+| `RESPONSE` in <1s | Relay and gateway listener fully healthy → the fault is on Manage's side |
+| `TIMEOUT` waiting for reply | Sender accepted but listener never answered → zombie listener or gateway-side fault; restart `Gateway-Relay` and retry |
+| `CONNECT FAILED: HTTP 401` | Bad key — the per-HC `listen` policy key cannot send; use the namespace root key |
+| `CONNECT FAILED: HTTP 404` | Wrong HC name, or no listener registered at all |
+
+There is also `--bearer-token` mode (see the script's docstring) to test the AAD
+bearer path Manage uses in production. To run the probe from *inside* the Manage
+container (testing Manage's actual network/DNS/identity), see the container-side
+snippet in the script's docstring — during the July 2026 incident that variant is
+what exposed the relay hostname resolving to a private endpoint IP inside the VNet.
+
+Rotate the namespace key after a debugging session — nothing in a deployed
+environment uses SAS, so rotation is free.
+
+### Manage application logs
+
+Discover the names once (they come from the shared naming module, so don't guess):
+
+```bash
+az containerapp list -g rg-manbrs-prod-uks --query "[].name" -o tsv
+az monitor log-analytics workspace list -g rg-manbrs-prod-uks --query "[].{name:name, id:customerId}" -o table
+```
+
+Then either live-stream:
+
+```bash
+az containerapp logs show -n <app-name> -g rg-manbrs-prod-uks --type console --follow --tail 50
+```
+
+or query Log Analytics (better for searching):
+
+```bash
+# Worklist-send activity and errors from the gateway app code
+az monitor log-analytics query --workspace <workspace-customer-id> \
+  --analytics-query "ContainerAppConsoleLogs_CL
+    | where TimeGenerated > ago(2h)
+    | where Log_s has_any ('relay', 'GatewayAction', 'worklist', 'ERROR', 'Traceback')
+    | project TimeGenerated, Log_s | order by TimeGenerated desc" -o table
+
+# Incoming DICOM uploads from the gateway (the /api/v1/dicom endpoint)
+az monitor log-analytics query --workspace <workspace-customer-id> \
+  --analytics-query "ContainerAppConsoleLogs_CL
+    | where TimeGenerated > ago(2h)
+    | where Log_s has 'dicom'
+    | project TimeGenerated, Log_s | order by TimeGenerated desc" -o table
+```
+
+Manage-side data checks (worklist item actually created and sent?) are quickest in
+the Django admin: **Gateway actions** — a healthy send is `status: confirmed`;
+`failed` rows carry `last_error` and the retry schedule.
+
+## Pre-flight checklist (run before the clinic starts)
+
+On the VM:
+
+1. `Get-GwHealth` — all four services **Running**, ports 104 and 11112 listening,
+   relay log shows "Connected - waiting for worklist actions", disk has headroom.
+2. `Get-GwWorklist` — after the first participant is checked in on Manage, the item
+   appears here within seconds. That one check proves Manage → Relay → VM end to end.
+3. On the modality: query the worklist and confirm the participant is listed.
+
+From the laptop:
+
+4. `az relay hyco show ... --query listenerCount` returns ≥ 1.
+5. App Insights errors query over `ago(12h)` is quiet (no crash loops overnight).
+
+## During the clinic — a useful passive setup
+
+On the VM, keep two windows open:
+
+```powershell
+Watch-GwLog PACS     # window 1 — see each C-STORE arrive
+Watch-GwLog Upload   # window 2 — see each upload to Manage
+```
+
+After each participant: `Get-GwImages` should show the expected image count with
+`uploaded` equal to `images` within a minute or two.

docs/deployment/runbooks/onboard-hospital-vm.md

@@ -15,7 +15,36 @@
 - [ ] Trust ODS code confirmed via the [ODS portal](https://odsportal.nhsbsa.nhs.uk/)
 - [ ] NHS region confirmed (`nw` | `neyh` | `mids` | `eoe` | `lon` | `se` | `sw`)
 - [ ] Deployment ring agreed with the programme team
-- [ ] `arc-onboarding-spn-client-id` retrieved from Entra ID; new `arc-onboarding-spn-client-secret` generated with 1-day expiry and shared with the hospital trust IT team before the onboarding call
+- [ ] Onboarding engineer has the permissions listed below
+- [ ] Temporary onboarding secret created (Step 0) ready to share on the onboarding call
+
+## Who can run this
+
+The engineer driving the onboarding needs, for the target environment:
+
+| Requirement | Why |
+| --- | --- |
+| Membership of `screening_mbsgw_<env>` (PIM-activated) | Access to the gateway spoke subscription |
+| PIM roles activated for **both** the Core Services hub and the Manage Breast Screening spoke subscriptions | Deployment tooling runs against the hub subscription and writes to the spoke, so both are needed during onboarding |
+| **Owner** of the `spn-manbrs-web-api-<env>` enterprise application *and* its app registration | Required to run `assign-arc-app-roles` (Step 3) |
+| Rights to create client secrets on the `spn-azure-arc-onboarding-screening-<env>` app registration | Step 0 |
+
+> The pipeline's managed identity additionally requires the **Monitoring Contributor** role. This is assigned by the environment's infrastructure code, not by hand — if it is missing, fix it in code rather than in the portal.
+
+---
+
+## Step 0 — Create a temporary onboarding secret
+
+In the Azure portal: **App registrations → `spn-azure-arc-onboarding-screening-<env>` → Certificates & secrets → New client secret**, with:
+
+- **Description**: identify the hospital (e.g. `Somerset-NHS-Foundation-Trust - Arc onboarding`)
+- **Expiry**: custom, **24 hours** from creation
+
+Then:
+
+- Copy the **secret value** immediately — it is shown only once.
+- Record the **Secret ID** for audit and revocation.
+- Store the value securely. Provide it to the hospital IT engineer **only during the onboarding call** — never in advance, never over unsecured channels.
 
 ---
 
@@ -117,6 +146,8 @@ Run the ADO pipeline **Deploy Arc Infrastructure - \<env\>** manually. Terraform
 
 **Verify**: In the Azure portal, navigate to `relay-manbrs-<env>` → Hybrid Connections → `hc-gw-hull-university-teaching-hospitals-nhs-trust-rwa-01` is present.
 
+> **If the relay namespace itself is missing** (new environment): it is provisioned by the **dtos-manage-breast-screening** repo (`enable_relay = true` in that environment's tfvars), into the Manage spoke subscription — **not** the hub. See [Create Environment](../infrastructure/create-environment.md).
+
 ## Step 5 — Deploy the gateway application
 
 Run the ADO pipeline **Deploy Gateway - \<env\>** with:
@@ -132,17 +163,53 @@ The pipeline:
 2. Sends an Arc Run Command to `gw-hull-university-teaching-hospitals-nhs-trust-rwa-01` that writes `.env` and runs `deploy.ps1`
 3. Polls for completion and reports success or failure
 
-## Step 6 — Smoke test
+## Step 6 — Verify (acceptance criteria)
 
-Run from the gateway VM or via Arc Run Command:
+Onboarding is complete when **all** of the following hold:
 
-```powershell
-Get-Service Gateway-PACS, Gateway-MWL, Gateway-Upload, Gateway-Relay | Select-Object Name, Status
+- [ ] The deployment pipeline completed with no errors or failed stages
+- [ ] The Arc machine shows **Connected** in the portal
+- [ ] The Hybrid Connection `hc-gw-<site>-<ods>-<instance>` shows **1 listener**
+- [ ] The validation command below succeeds (`executionState: Succeeded`, `exitCode: 0`) showing all four services **Running** and ports **104** and **11112** listening
+- [ ] The relay listener log ends with `Connected - waiting for worklist actions...`
+- [ ] No manual intervention was needed after the pipeline completed
+
+Validation command (or run the script block directly on the VM):
+
+```bash
+az connectedmachine run-command create \
+  --machine-name gw-<site>-<ods>-<instance> \
+  --resource-group rg-mbsgw-<env>-uks-arc-enabled-servers \
+  --location uksouth \
+  --name check-services-1 \
+  --script "Get-Service Gateway-PACS, Gateway-MWL, Gateway-Upload, Gateway-Relay | Format-Table Name, Status; Get-NetTCPConnection -State Listen | Where-Object { \$_.LocalPort -in 104, 11112 } | Format-Table LocalAddress, LocalPort"
+```
+
+Expected output (in `instanceView.output`):
+
+```text
+Name           Status
+----           ------
+Gateway-MWL    Running
+Gateway-PACS   Running
+Gateway-Relay  Running
+Gateway-Upload Running
+
+LocalAddress LocalPort
+------------ ---------
+0.0.0.0          11112
+0.0.0.0            104
+```
+
+Relay listener check (same pattern, fresh `--name`):
+
+```bash
+  --script "Get-Content 'C:\Program Files\NHS\ManageBreastScreeningGateway\logs\Gateway-Relay.log' | Where-Object { \$_ -notmatch 'opentelemetry' } | Select-Object -Last 5"
 ```
 
-Expected: all four services **Running**.
+Expected: the last line is `Connected - waiting for worklist actions...`
 
-Check Log Analytics Workspace for an initial heartbeat within 5 minutes of service start.
+> A **Running** service that has not logged `Connected - waiting` is not healthy — the relay connection is the part that matters. Do not rely on the Log Analytics heartbeat as a liveness signal: telemetry export from gateway VMs is a known issue at the time of writing.
 
 ---
 
@@ -195,6 +262,14 @@ If the script is still blocked after `Unblock-File` (e.g. due to a stricter mach
 Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process
 ```
 
+### Onboarding fails: service principal not authorised to onboard machines
+
+The onboarding SPN needs the **Azure Connected Machine Onboarding** role on the spoke subscription. Check and assign via: Subscription → **Access control (IAM)** → Add role assignment → role `Azure Connected Machine Onboarding` → member `spn-azure-arc-onboarding-screening-<env>`. Allow a few minutes for propagation, then rerun the script. (This blocked the first production onboarding.)
+
+### Onboarding fails: resource providers not registered
+
+The spoke subscription must have the **`Microsoft.HybridCompute`** and **`Microsoft.HybridConnectivity`** resource providers registered. Check via: Subscription → **Resource providers** → search each → Register. Then rerun the script.
+
 ### Arc machine shows as Disconnected after onboarding
 
 The agent may have lost connectivity. Check: