From 30463aab5ba66113bec4afa365dbb2af6d319568 Mon Sep 17 00:00:00 2001 From: Carlos Martinez Date: Tue, 30 Jun 2026 11:51:56 +0100 Subject: [PATCH] Fold first-production lessons into onboarding and promotion runbooks Onboarding: add the permissions the engineer actually needs (PIM for hub and spoke, enterprise app ownership), the temporary-secret procedure and its 24-hour trap, the two subscription misconfigurations that blocked the first prod onboarding (missing Azure Connected Machine Onboarding role, unregistered resource providers), where the relay namespace is provisioned, and a proper acceptance-criteria verification step - including that a Running service which never logs "Connected - waiting" is not healthy. Promotion: the cleanup verify now has commands and a warning (pre-prod data survived the first promotion onto the prod gateway and was found a month later during a live clinic); re-onboarding creates a new managed identity so Gateway.oid must be updated or every image upload 403s; the end-to-end check goes through the Gateway actions admin; the gotchas table gains the failure modes we actually hit. Also, add live-clinic debugging runbook and diagnostic tooling. Co-authored-by: Ife Ayelabola --- docs/deployment/deployment-pipeline.md | 2 +- .../runbooks/live-clinic-debugging.md | 229 ++++++++++++++++++ .../runbooks/onboard-hospital-vm.md | 89 ++++++- .../runbooks/promote-vm-preprod-to-prod.md | 159 ++++++++++++ scripts/diagnostics/relay_echo_probe.py | 154 ++++++++++++ scripts/powershell/debug_toolkit.ps1 | 179 ++++++++++++++ 6 files changed, 804 insertions(+), 8 deletions(-) create mode 100644 docs/deployment/runbooks/live-clinic-debugging.md create mode 100644 docs/deployment/runbooks/promote-vm-preprod-to-prod.md create mode 100644 scripts/diagnostics/relay_echo_probe.py create mode 100644 scripts/powershell/debug_toolkit.ps1 diff --git a/docs/deployment/deployment-pipeline.md b/docs/deployment/deployment-pipeline.md index 501e113d..de7d7dca 100644 --- a/docs/deployment/deployment-pipeline.md +++ b/docs/deployment/deployment-pipeline.md @@ -1,7 +1,7 @@ # Deployment Pipeline > **Scope**: End-to-end flow from a GitHub tag or main-branch push through Azure DevOps pipelines to an Arc Run Command executing on hospital gateway VMs. -> **Related docs**: [Windows Service Deploy](./windows-service-deploy.md) | [Onboard Hospital VM](./runbooks/onboard-hospital-vm.md) | [Rollback Runbook](./runbooks/rollback.md) +> **Related docs**: [Windows Service Deploy](./windows-service-deploy.md) | [Onboard Hospital VM](./runbooks/onboard-hospital-vm.md) | [Promote VM Pre-Prod → Prod](./runbooks/promote-vm-preprod-to-prod.md) | [Rollback Runbook](./runbooks/rollback.md) --- diff --git a/docs/deployment/runbooks/live-clinic-debugging.md b/docs/deployment/runbooks/live-clinic-debugging.md new file mode 100644 index 00000000..dfd3e56d --- /dev/null +++ b/docs/deployment/runbooks/live-clinic-debugging.md @@ -0,0 +1,229 @@ +# Live clinic debugging + +> **When to use**: On-the-day support for a live clinic — checking worklist delivery, +> image receipt and upload from the gateway VM, and tracing problems through the +> Azure logs for both the gateway and Manage. + +Two vantage points: + +- **The gateway VM** (RDP) — direct access to the MWL/PACS SQLite databases, DICOM + storage and service logs. Use [`scripts/powershell/debug_toolkit.ps1`](../../../scripts/powershell/debug_toolkit.ps1). +- **Your laptop** (Azure CLI / portal) — Application Insights for the gateway + services, container app logs for Manage, and Azure Relay connection health. + +## The pipeline at a glance + +``` +Manage (container app) + │ worklist item ─── Azure Relay (relay-manbrs-prod / hc-) ──► Gateway-Relay + │ │ + │ writes data\worklist.db + │ │ +Modality ── C-FIND (port 104) ──► Gateway-MWL ◄── reads worklist.db ──────────┘ +Modality ── C-STORE (port 11112) ──► Gateway-PACS ──► data\storage + data\pacs.db + │ +Manage /api/v1/dicom ◄── HTTPS upload (managed identity) ── Gateway-Upload +``` + +A failure at any hop has a distinct signature — the sections below work through them +in pipeline order. + +## On the gateway VM + +Copy `debug_toolkit.ps1` to the VM (or paste it into an editor there), then: + +```powershell +. .\debug_toolkit.ps1 +Get-GwHealth # start here: services, ports, relay connection, disk +``` + +Reference points: + +| Thing | Where | +|---|---| +| Install root | `C:\Program Files\NHS\ManageBreastScreeningGateway` | +| Services (NSSM) | `Gateway-Relay`, `Gateway-MWL`, `Gateway-PACS`, `Gateway-Upload` | +| Worklist DB (SQLite) | `data\worklist.db` — table `worklist_items` | +| PACS DB (SQLite) | `data\pacs.db` — table `stored_instances` | +| DICOM files | `data\storage\` (hash-based directory layout; use `Get-GwDicomFiles`) | +| Service logs | `logs\Gateway-.log` (NSSM stdout+stderr) | +| Deployment logs | `logs\deployments\deploy-*` | +| Config | `.env` in the install root | +| MWL | AET `RUBIE_MWL_PROD`, port 104 | +| PACS | AET `RUBIE_PACS_PROD`, port 11112 | + +### Symptom → first command + +| Symptom | Check | +|---|---| +| Appointment checked in on Manage but not on the modality worklist | `Get-GwWorklist` — is the item in the DB? If **no**: relay hop failed → `Watch-GwLog Relay`, then the Azure Relay + Manage sections below. If **yes**: modality⇄MWL hop → `Watch-GwLog MWL` while the modality refreshes; confirm port 104 listening; scheduled_date is stored in **UTC** `YYYYMMDD` | +| Exposure taken but images not on the VM | `Watch-GwLog PACS` while the modality sends; `Get-GwImages`; confirm port 11112 and that the modality is configured with AE title `RUBIE_PACS_PROD`, correct IP and port | +| Images on VM but not appearing in Manage | `Get-GwUploadFailures` — look at `upload_error` and `upload_attempt_count`; `Watch-GwLog Upload`. Auth errors here usually mean the Arc managed identity / `Gateway.Access` app role or `Gateway.oid` in Manage is wrong | +| Item status stuck at SCHEDULED after procedure started | `Get-GwWorklistItem ` — MPPS updates set `status` / `mpps_instance_uid`; check `Watch-GwLog MWL` for MPPS N-CREATE/N-SET messages | +| Anything crashing / restart loops | `Search-GwLogs`; `Get-Service Gateway-*` (NSSM restarts failed services automatically — repeated banner lines in the log mean crash-looping) | + +## From your laptop — Azure + +```bash +az account set --subscription "Breast Screening - Manage Breast Screening - Prod" +``` + +### Gateway service telemetry (Application Insights) + +All four VM services send OpenTelemetry to `ai-mbsgw-prod-arc-uks` +(resource group `rg-mbsgw-prod-uks-arc-enabled-servers`). `cloud_RoleName` values: +`relay-listener`, `mwl-server`, `pacs-server`, `upload-listener`. + +```bash +# Errors and warnings in the last hour, all services +az monitor app-insights query \ + --app ai-mbsgw-prod-arc-uks -g rg-mbsgw-prod-uks-arc-enabled-servers \ + --analytics-query "traces + | where timestamp > ago(1h) and severityLevel >= 2 + | project timestamp, cloud_RoleName, message + | order by timestamp desc" -o table + +# Exceptions with stack traces +az monitor app-insights query \ + --app ai-mbsgw-prod-arc-uks -g rg-mbsgw-prod-uks-arc-enabled-servers \ + --analytics-query "exceptions + | where timestamp > ago(4h) + | project timestamp, cloud_RoleName, type, outerMessage + | order by timestamp desc" -o table + +# Everything one service said in a window (e.g. relay listener around a failure) +az monitor app-insights query \ + --app ai-mbsgw-prod-arc-uks -g rg-mbsgw-prod-uks-arc-enabled-servers \ + --analytics-query "traces + | where timestamp > ago(1h) and cloud_RoleName == 'relay-listener' + | project timestamp, message | order by timestamp desc" -o table +``` + +Portal equivalent: the App Insights resource → **Logs**, or **Live Metrics** for +real-time output during the clinic. + +> Telemetry is a no-op if `APPLICATIONINSIGHTS_CONNECTION_STRING` wasn't set at +> deploy time — if queries return nothing at all, fall back to the VM logs and +> check the deploy log for the "Application Insights resource not found" warning. + +### Azure Relay connection health + +The namespace is `relay-manbrs-prod` in `rg-manbrs-prod-uks`; the Hull hybrid +connection is `hc-`. + +```bash +# Does the HC exist / listener count (listenerCount > 0 means the VM is connected) +az relay hyco show --namespace-name relay-manbrs-prod -g rg-manbrs-prod-uks \ + -n hc-gw-hull-university-teaching-hospitals-nhs-trust-rwa-01 \ + --query "{name:name, listenerCount:listenerCount}" + +# Namespace metrics: listeners and sender connections over the morning +az monitor metrics list \ + --resource "$(az relay namespace show -g rg-manbrs-prod-uks -n relay-manbrs-prod --query id -o tsv)" \ + --metric "ListenerConnections-Success" "SenderConnections-Success" "ListenerConnections-ClientError" \ + --interval PT5M --offset 4h -o table +``` + +`listenerCount: 0` means `Gateway-Relay` on the VM is not connected — check the +service on the VM (`Watch-GwLog Relay`; look for "Connected - waiting for worklist +actions"). Sender client errors while listeners are healthy point at the Manage side +(its managed identity / `AZURE_RELAY_CLIENT_ID`). + +> **Beware the zombie listener**: `listenerCount: 1` does not prove the listener is +> *functional* — a connection that died without a clean close can stay registered +> while rendezvous dispatches go nowhere. The echo probe below is the truth test. + +### The echo probe — test the relay → gateway path without Manage + +[`scripts/diagnostics/relay_echo_probe.py`](../../../scripts/diagnostics/relay_echo_probe.py) +connects to the hybrid connection **as a sender** (exactly like Manage does), sends an +`echo` action — which the relay listener answers without touching its database — and +waits for the reply. It needs only Python, `pip install websockets`, and a credential; +no Azure login, runnable from any laptop. + +```bash +python scripts/diagnostics/relay_echo_probe.py \ + --namespace relay-manbrs-prod.servicebus.windows.net \ + --hc hc-gw-hull-university-teaching-hospitals-nhs-trust-rwa-01 \ + --key '' # namespace-level key, NOT the HC's listen-only key +``` + +| Result | Meaning | +|---|---| +| `RESPONSE` in <1s | Relay and gateway listener fully healthy → the fault is on Manage's side | +| `TIMEOUT` waiting for reply | Sender accepted but listener never answered → zombie listener or gateway-side fault; restart `Gateway-Relay` and retry | +| `CONNECT FAILED: HTTP 401` | Bad key — the per-HC `listen` policy key cannot send; use the namespace root key | +| `CONNECT FAILED: HTTP 404` | Wrong HC name, or no listener registered at all | + +There is also `--bearer-token` mode (see the script's docstring) to test the AAD +bearer path Manage uses in production. To run the probe from *inside* the Manage +container (testing Manage's actual network/DNS/identity), see the container-side +snippet in the script's docstring — during the July 2026 incident that variant is +what exposed the relay hostname resolving to a private endpoint IP inside the VNet. + +Rotate the namespace key after a debugging session — nothing in a deployed +environment uses SAS, so rotation is free. + +### Manage application logs + +Discover the names once (they come from the shared naming module, so don't guess): + +```bash +az containerapp list -g rg-manbrs-prod-uks --query "[].name" -o tsv +az monitor log-analytics workspace list -g rg-manbrs-prod-uks --query "[].{name:name, id:customerId}" -o table +``` + +Then either live-stream: + +```bash +az containerapp logs show -n -g rg-manbrs-prod-uks --type console --follow --tail 50 +``` + +or query Log Analytics (better for searching): + +```bash +# Worklist-send activity and errors from the gateway app code +az monitor log-analytics query --workspace \ + --analytics-query "ContainerAppConsoleLogs_CL + | where TimeGenerated > ago(2h) + | where Log_s has_any ('relay', 'GatewayAction', 'worklist', 'ERROR', 'Traceback') + | project TimeGenerated, Log_s | order by TimeGenerated desc" -o table + +# Incoming DICOM uploads from the gateway (the /api/v1/dicom endpoint) +az monitor log-analytics query --workspace \ + --analytics-query "ContainerAppConsoleLogs_CL + | where TimeGenerated > ago(2h) + | where Log_s has 'dicom' + | project TimeGenerated, Log_s | order by TimeGenerated desc" -o table +``` + +Manage-side data checks (worklist item actually created and sent?) are quickest in +the Django admin: **Gateway actions** — a healthy send is `status: confirmed`; +`failed` rows carry `last_error` and the retry schedule. + +## Pre-flight checklist (run before the clinic starts) + +On the VM: + +1. `Get-GwHealth` — all four services **Running**, ports 104 and 11112 listening, + relay log shows "Connected - waiting for worklist actions", disk has headroom. +2. `Get-GwWorklist` — after the first participant is checked in on Manage, the item + appears here within seconds. That one check proves Manage → Relay → VM end to end. +3. On the modality: query the worklist and confirm the participant is listed. + +From the laptop: + +4. `az relay hyco show ... --query listenerCount` returns ≥ 1. +5. App Insights errors query over `ago(12h)` is quiet (no crash loops overnight). + +## During the clinic — a useful passive setup + +On the VM, keep two windows open: + +```powershell +Watch-GwLog PACS # window 1 — see each C-STORE arrive +Watch-GwLog Upload # window 2 — see each upload to Manage +``` + +After each participant: `Get-GwImages` should show the expected image count with +`uploaded` equal to `images` within a minute or two. diff --git a/docs/deployment/runbooks/onboard-hospital-vm.md b/docs/deployment/runbooks/onboard-hospital-vm.md index 856f6e25..31b0efe2 100644 --- a/docs/deployment/runbooks/onboard-hospital-vm.md +++ b/docs/deployment/runbooks/onboard-hospital-vm.md @@ -15,7 +15,36 @@ - [ ] Trust ODS code confirmed via the [ODS portal](https://odsportal.nhsbsa.nhs.uk/) - [ ] NHS region confirmed (`nw` | `neyh` | `mids` | `eoe` | `lon` | `se` | `sw`) - [ ] Deployment ring agreed with the programme team -- [ ] `arc-onboarding-spn-client-id` retrieved from Entra ID; new `arc-onboarding-spn-client-secret` generated with 1-day expiry and shared with the hospital trust IT team before the onboarding call +- [ ] Onboarding engineer has the permissions listed below +- [ ] Temporary onboarding secret created (Step 0) ready to share on the onboarding call + +## Who can run this + +The engineer driving the onboarding needs, for the target environment: + +| Requirement | Why | +| --- | --- | +| Membership of `screening_mbsgw_` (PIM-activated) | Access to the gateway spoke subscription | +| PIM roles activated for **both** the Core Services hub and the Manage Breast Screening spoke subscriptions | Deployment tooling runs against the hub subscription and writes to the spoke, so both are needed during onboarding | +| **Owner** of the `spn-manbrs-web-api-` enterprise application *and* its app registration | Required to run `assign-arc-app-roles` (Step 3) | +| Rights to create client secrets on the `spn-azure-arc-onboarding-screening-` app registration | Step 0 | + +> The pipeline's managed identity additionally requires the **Monitoring Contributor** role. This is assigned by the environment's infrastructure code, not by hand — if it is missing, fix it in code rather than in the portal. + +--- + +## Step 0 — Create a temporary onboarding secret + +In the Azure portal: **App registrations → `spn-azure-arc-onboarding-screening-` → Certificates & secrets → New client secret**, with: + +- **Description**: identify the hospital (e.g. `Somerset-NHS-Foundation-Trust - Arc onboarding`) +- **Expiry**: custom, **24 hours** from creation + +Then: + +- Copy the **secret value** immediately — it is shown only once. +- Record the **Secret ID** for audit and revocation. +- Store the value securely. Provide it to the hospital IT engineer **only during the onboarding call** — never in advance, never over unsecured channels. --- @@ -117,6 +146,8 @@ Run the ADO pipeline **Deploy Arc Infrastructure - \** manually. Terraform **Verify**: In the Azure portal, navigate to `relay-manbrs-` → Hybrid Connections → `hc-gw-hull-university-teaching-hospitals-nhs-trust-rwa-01` is present. +> **If the relay namespace itself is missing** (new environment): it is provisioned by the **dtos-manage-breast-screening** repo (`enable_relay = true` in that environment's tfvars), into the Manage spoke subscription — **not** the hub. See [Create Environment](../infrastructure/create-environment.md). + ## Step 5 — Deploy the gateway application Run the ADO pipeline **Deploy Gateway - \** with: @@ -132,17 +163,53 @@ The pipeline: 2. Sends an Arc Run Command to `gw-hull-university-teaching-hospitals-nhs-trust-rwa-01` that writes `.env` and runs `deploy.ps1` 3. Polls for completion and reports success or failure -## Step 6 — Smoke test +## Step 6 — Verify (acceptance criteria) -Run from the gateway VM or via Arc Run Command: +Onboarding is complete when **all** of the following hold: -```powershell -Get-Service Gateway-PACS, Gateway-MWL, Gateway-Upload, Gateway-Relay | Select-Object Name, Status +- [ ] The deployment pipeline completed with no errors or failed stages +- [ ] The Arc machine shows **Connected** in the portal +- [ ] The Hybrid Connection `hc-gw---` shows **1 listener** +- [ ] The validation command below succeeds (`executionState: Succeeded`, `exitCode: 0`) showing all four services **Running** and ports **104** and **11112** listening +- [ ] The relay listener log ends with `Connected - waiting for worklist actions...` +- [ ] No manual intervention was needed after the pipeline completed + +Validation command (or run the script block directly on the VM): + +```bash +az connectedmachine run-command create \ + --machine-name gw--- \ + --resource-group rg-mbsgw--uks-arc-enabled-servers \ + --location uksouth \ + --name check-services-1 \ + --script "Get-Service Gateway-PACS, Gateway-MWL, Gateway-Upload, Gateway-Relay | Format-Table Name, Status; Get-NetTCPConnection -State Listen | Where-Object { \$_.LocalPort -in 104, 11112 } | Format-Table LocalAddress, LocalPort" +``` + +Expected output (in `instanceView.output`): + +```text +Name Status +---- ------ +Gateway-MWL Running +Gateway-PACS Running +Gateway-Relay Running +Gateway-Upload Running + +LocalAddress LocalPort +------------ --------- +0.0.0.0 11112 +0.0.0.0 104 +``` + +Relay listener check (same pattern, fresh `--name`): + +```bash + --script "Get-Content 'C:\Program Files\NHS\ManageBreastScreeningGateway\logs\Gateway-Relay.log' | Where-Object { \$_ -notmatch 'opentelemetry' } | Select-Object -Last 5" ``` -Expected: all four services **Running**. +Expected: the last line is `Connected - waiting for worklist actions...` -Check Log Analytics Workspace for an initial heartbeat within 5 minutes of service start. +> A **Running** service that has not logged `Connected - waiting` is not healthy — the relay connection is the part that matters. Do not rely on the Log Analytics heartbeat as a liveness signal: telemetry export from gateway VMs is a known issue at the time of writing. --- @@ -195,6 +262,14 @@ If the script is still blocked after `Unblock-File` (e.g. due to a stricter mach Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process ``` +### Onboarding fails: service principal not authorised to onboard machines + +The onboarding SPN needs the **Azure Connected Machine Onboarding** role on the spoke subscription. Check and assign via: Subscription → **Access control (IAM)** → Add role assignment → role `Azure Connected Machine Onboarding` → member `spn-azure-arc-onboarding-screening-`. Allow a few minutes for propagation, then rerun the script. (This blocked the first production onboarding.) + +### Onboarding fails: resource providers not registered + +The spoke subscription must have the **`Microsoft.HybridCompute`** and **`Microsoft.HybridConnectivity`** resource providers registered. Check via: Subscription → **Resource providers** → search each → Register. Then rerun the script. + ### Arc machine shows as Disconnected after onboarding The agent may have lost connectivity. Check: diff --git a/docs/deployment/runbooks/promote-vm-preprod-to-prod.md b/docs/deployment/runbooks/promote-vm-preprod-to-prod.md new file mode 100644 index 00000000..44cd6d38 --- /dev/null +++ b/docs/deployment/runbooks/promote-vm-preprod-to-prod.md @@ -0,0 +1,159 @@ +# Promote a Gateway VM from Pre-Prod to Prod + +> **When to use**: A hospital VM currently running the **pre-prod** gateway needs to become the **prod** gateway (same infrastructure, different environment). +> **Related**: [Onboard Hospital VM](./onboard-hospital-vm.md) | [Cleanup](./cleanup.md) | [Deployment Pipeline](../deployment-pipeline.md) + +--- + +## Why this is not just "re-point the gateway" + +Pre-prod and prod are separate Azure subscriptions, with separate Arc resource groups, relay namespaces (`relay-manbrs-`), web-API service principals (`spn-manbrs-web-api-`) and ADO pipelines. An Arc-connected machine belongs to **exactly one** subscription, and the gateway services authenticate using that machine's managed identity. A pre-prod machine identity holds the `Gateway.Access` role on the **pre-prod** API and cannot talk to prod. + +So you cannot promote by editing `.env`. The machine must be **disconnected from the pre-prod subscription and re-onboarded into the prod subscription** — effectively the [onboarding runbook](./onboard-hospital-vm.md) run for prod, preceded by a clean decommission of the pre-prod gateway. + +> One box hosts one gateway. After this procedure there is **no pre-prod gateway at this site**. If pre-prod is still needed at the site, use a different VM. + +--- + +## Before the day (do these in advance) + +- [ ] **You can actually run this.** Check [Onboarding — Who can run this](./onboard-hospital-vm.md#who-can-run-this): PIM for the prod hub and spoke, Owner on `spn-manbrs-web-api-prod`, secret rights on the onboarding SPN. Discovering a missing permission on the day cost the first promotion most of a week. +- [ ] **`GATEWAY_RINGS` is set for prod.** `infrastructure/environments/prod/variables.sh` must set `GATEWAY_RINGS` to include the ring the site is tagged with (for initial production installs this will typically be `ring1`). If it is unset it defaults to `ring0` and the prod deploy pipeline **silently skips** the machine. +- [ ] **Prod infrastructure exists.** Confirm the prod relay namespace, `spn-manbrs-web-api-prod`, Log Analytics workspace and the prod ADO pipelines are provisioned and have had a successful infra deploy. +- [ ] **Prod Manage (Rubie) is seeded.** The prod Rubie instance has the site's clinic/setting, a `Relay` record pointing at `relay-manbrs-prod` and the prod gateway, the `gateway_images` feature flag enabled, and a test appointment for the end-to-end check. +- [ ] **DHCP reservation confirmed** for the VM, so the IP the modality targets cannot change between decommission and go-live. +- [ ] **Modality engineer briefed** that the gateway **AE titles change** (see [Step 5](#step-5--reconfigure-the-modality-on-site)) and the modality must be reconfigured and re-tested on the day. +- [ ] **Fresh prod Arc onboarding SPN secret** created per [Onboarding — Step 0](./onboard-hospital-vm.md#step-0--create-a-temporary-onboarding-secret) — on the day (24h expiry), shared only during the call. + +--- + +## Step 1 — Decommission the pre-prod gateway (on the VM) + +Run from an **elevated PowerShell session** on the VM. First confirm you are on the right machine: + +```powershell +hostname +Get-Service Gateway-* | Format-Table Name, Status +``` + +Then run the cleanup script, which stops and removes the services and **removes the installation directory including all pre-prod data** (`worklist.db`, `pacs.db`, `data\storage`): + +```powershell +.\scripts\powershell\cleanup.ps1 +``` + +**Verify**: no `Gateway-*` services remain and `C:\Program Files\NHS\ManageBreastScreeningGateway` is gone (see [Cleanup runbook — Verify](./cleanup.md)): + +```powershell +Get-Service Gateway-* -ErrorAction SilentlyContinue +Test-Path 'C:\Program Files\NHS\ManageBreastScreeningGateway' # must be False +``` + +## Step 2 — Disconnect from the pre-prod Arc subscription + +Still on the VM, elevated: + +```powershell +azcmagent disconnect +``` + +This removes the machine's registration from the pre-prod resource group. The Arc **agent stays installed** — only the registration is removed, so re-onboarding in Step 3 skips the agent install. + +**Verify**: `azcmagent show` reports the agent as **Disconnected**. + +## Step 3 — Re-onboard into prod + +This is [Onboarding runbook Step 2](./onboard-hospital-vm.md#step-2--run-arc-onboarding-script-on-the-gateway-vm) with **prod parameters**. The site parameters (`SiteName`, `ODSCode`, `Instance`) are unchanged, so the Arc resource name is identical — but it is now created in the prod resource group. + +```powershell +.\arc-setup.ps1 ` + -SubscriptionId "" ` + -TenantId "" ` + -ResourceGroup "rg-mbsgw-prod-uks-arc-enabled-servers" ` + -Location "uksouth" ` + -ServicePrincipalId "" ` + -ServicePrincipalSecret "" ` + -SiteName "Hull-University-Teaching-Hospitals-NHS-Trust" ` + -ODSCode "RWA" ` + -Instance "01" ` + -NHSRegion "neyh" ` + -SiteType "static" ` + -DeploymentRing "ring1" +``` + +**Verify**: in the Azure portal, `rg-mbsgw-prod-uks-arc-enabled-servers` → Azure Arc machines → `gw-<...>-rwa-01` is **Connected**. + +## Step 4 — Provision prod and deploy the prod gateway + +From here, follow the onboarding runbook against **prod**: + +1. **Grant API access** — `make prod assign-arc-app-roles` ([Step 3](./onboard-hospital-vm.md#step-3--grant-api-access)). This assigns `Gateway.Access` on `spn-manbrs-web-api-prod` to the machine's new prod managed identity. The old pre-prod assignment is irrelevant (different identity) and is cleaned up in Step 6. +2. **Provision the Hybrid Connection** — run **Deploy Arc Infrastructure - prod** ([Step 4](./onboard-hospital-vm.md#step-4--trigger-terraform-to-provision-the-hybrid-connection)). Creates `hc-gw-<...>-rwa-01` in `relay-manbrs-prod`. +3. **Deploy the application** — run **Deploy Gateway - prod** with a **released** `releaseTag` (not a pre-prod build) ([Step 5](./onboard-hospital-vm.md#step-5--deploy-the-gateway-application)). This writes a fresh prod `.env` (prod relay namespace, `CLOUD_API_HOSTNAME=manage-breast-screening.nhs.uk`, prod AE titles), fully replacing the pre-prod `.env`. +4. **Update prod Rubie's `Gateway` record with the machine's new identity.** Re-onboarding created a **new managed identity** — the pre-prod OID is dead. Image uploads are authorised against `Gateway.oid`, so until this is updated every upload will be rejected with a 403. Get the new principal ID and set it on the Gateway record in the prod Django admin: + + ```bash + az connectedmachine show \ + -g rg-mbsgw-prod-uks-arc-enabled-servers \ + -n gw--- \ + --query identity.principalId -o tsv + ``` + + While in the admin, confirm the `Relay` record's **Setting** matches the site's clinics' Setting — a mismatch silently routes every appointment to the manual-images flow. + +**Verify**: the [onboarding acceptance criteria](./onboard-hospital-vm.md#step-6--verify-acceptance-criteria) — all four services Running, ports listening, **and the relay log ends with `Connected - waiting for worklist actions...`**. A Running service that never logs `Connected` is not healthy. Do not rely on the Log Analytics heartbeat (VM telemetry export is a known issue at the time of writing). Also confirm the databases start empty: + +```powershell +& 'C:\Program Files\NHS\ManageBreastScreeningGateway\current\.venv\Scripts\python.exe' -c "import sqlite3; print('worklist rows:', sqlite3.connect(r'C:\Program Files\NHS\ManageBreastScreeningGateway\data\worklist.db').execute('SELECT count(*) FROM worklist_items').fetchone()[0])" +``` + +## Step 5 — Reconfigure the modality + +The gateway AE titles are environment-specific: the `.env` builder sets `MWL_AET=RUBIE_MWL_` / `PACS_AET=RUBIE_PACS_`, where the environment is uppercased and truncated to 3 characters if longer than 4. + +| Environment | MWL AE title | PACS AE title | +|-------------|--------------|---------------| +| pre-prod (`PREPROD` → `PRE`) | `RUBIE_MWL_PRE` | `RUBIE_PACS_PRE` | +| prod (`PROD`) | `RUBIE_MWL_PROD` | `RUBIE_PACS_PROD` | + +The modality is currently configured to send to the **pre-prod** AE titles. The modality's MWL and PACS destinations must be updated to the **prod** AE titles. The VM IP and ports (`104` / `11112`) will typically be unchanged. + +**Verify**: a C-ECHO from the modality to each prod AE title succeeds. (A C-ECHO to the old `..._PRE` titles will now fail — expected.) + +## Step 6 — Clean up the pre-prod side (Azure) + +Disconnecting the machine in Step 2 leaves orphaned pre-prod resources: a disconnected Arc machine, a dangling Hybrid Connection (`hc-gw-<...>-rwa-01` in `relay-manbrs-preprod`), and a stale `Gateway.Access` role assignment. + +- [ ] Run **Deploy Arc Infrastructure - preprod** (Terraform) so it destroys the now-orphaned Hybrid Connection. +- [ ] Remove the disconnected Arc machine resource from `rg-mbsgw-preprod-uks-arc-enabled-servers` if Terraform/Arc has not already. +- [ ] Confirm the stale pre-prod app-role assignment is removed (re-running `make preprod assign-arc-app-roles` reconciles, or remove it via the portal). + +--- + +## End-to-end check + +With the modality reconfigured and prod Rubie seeded: + +1. Restart the relay listener first (`Restart-Service Gateway-Relay`, wait for `Connected - waiting`) so the check runs against a fresh relay connection. +2. Start the seeded test appointment in **prod** Rubie → the Gateway action in the Django admin goes **`confirmed`** within seconds, and the item is in the gateway's `worklist.db`. +3. Query the worklist from the modality → the item appears. +4. Acquire and send images → they arrive in the prod gateway PACS and are forwarded to prod Rubie. +5. Confirm the images appear against the appointment in prod Rubie. + +If step 2 fails, the Gateway action's `last_error` names the failing phase — see the [live clinic debugging runbook](./live-clinic-debugging.md) for the symptom → check mapping. + +--- + +## Gotchas + +| Symptom | Cause | Fix | +|---------|-------|-----| +| **Deploy Gateway - prod** reports "No machines found for ring1 — skipping" | `GATEWAY_RINGS` not set for prod, defaulting to `ring0` | Set `GATEWAY_RINGS="ring1"` in `prod/variables.sh` and redeploy | +| Services start but fail to authenticate against the cloud API | `Gateway.Access` not assigned to the **prod** managed identity | Run `make prod assign-arc-app-roles` | +| Modality C-ECHO fails after promotion | Modality still targeting `..._PRE` AE titles | Reconfigure modality to `..._PROD` (Step 5) | +| Hybrid Connection not created by Terraform | Arc machine not yet **Connected** in the prod RG | Confirm Step 3 verify, then re-run the infra pipeline | +| Pre-prod relay still shows connection attempts | Orphaned pre-prod HC / registration | Complete Step 6 | +| Rubie check-ins time out with `phase: connecting` while the listener is healthy | Rubie-side network/DNS — historically, the relay hostname resolving to a private endpoint IP inside the Rubie VNet | See [live clinic debugging](./live-clinic-debugging.md); verify the relay FQDN resolves to a **public** IP from the Rubie container | +| Image uploads rejected with 403 after promotion | `Gateway.oid` still holds the old pre-prod identity | Step 4.4 — update to the new principal ID | +| Rubie shows the manual-images flow instead of awaiting images | `Relay` record's Setting doesn't match the clinic's Setting, or the appointment already has a manual study | Fix the Setting FK in the admin; retest with a fresh appointment | +| Sends fail after the gateway has been idle overnight | Relay listener connection died silently (known issue) | `Restart-Service Gateway-Relay`; restart before every clinic until the listener liveness fix ships | diff --git a/scripts/diagnostics/relay_echo_probe.py b/scripts/diagnostics/relay_echo_probe.py new file mode 100644 index 00000000..983d2acb --- /dev/null +++ b/scripts/diagnostics/relay_echo_probe.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python3 +"""Standalone Azure Relay sender probe. + +Connects to a Hybrid Connection as a sender (exactly like Manage does), +sends an `echo` action, and waits for the gateway relay listener's reply. +The listener answers echo actions without touching its database, so this +is a safe end-to-end test of the relay -> gateway dispatch path that +bypasses Manage entirely. + +Usage: + pip install websockets + python relay_echo_probe.py \ + --namespace relay-manbrs-prod.servicebus.windows.net \ + --hc hc-gw-hull-university-teaching-hospitals-nhs-trust-rwa-01 \ + --key-name RootManageSharedAccessKey \ + --key ' Shared access policies>' + +Interpreting the output: + CONNECTED + RESPONSE -> relay dispatch to the gateway works; the fault + is specific to Manage's sender connection. + CONNECTED + TIMEOUT -> same failure as Manage: the relay accepts the + sender but the listener never receives the + rendezvous. Fault is relay<->listener side. + CONNECT FAILED -> note the HTTP status: 404 = no listener / + wrong HC name; 401 = bad token (the per-HC + "listen" policy key cannot send - use the + namespace-level RootManageSharedAccessKey). + +Running the equivalent probe from INSIDE the Manage container tests Manage's +actual network path, DNS resolution and managed identity (a laptop probe can +succeed while the container is broken - e.g. private DNS hijacking the relay +hostname inside the VNet, which is how the July 2026 outage was found): + + az containerapp exec -n -g --command sh + + # 1. What does the relay hostname resolve to? Must be a PUBLIC IP. + /app/.venv/bin/python -c "import socket; print(socket.getaddrinfo( + 'relay-manbrs-prod.servicebus.windows.net', 443)[0][4])" + + # 2. Echo through the app's own Relay record, token and code: + /app/.venv/bin/python /app/manage.py shell -c " + import json + from manage_breast_screening.gateway.models import Relay + from manage_breast_screening.gateway.relay_service import RelayURI + from websockets.sync.client import connect + url = RelayURI(Relay.objects.first()).connection_url() + with connect(url, compression=None, open_timeout=30) as ws: + ws.send(json.dumps({'action_type': 'echo', 'timestamp': 'probe'})) + print('RESPONSE:', ws.recv(timeout=35)) + " + +Note: manage.py shell is NOT gevent-patched, so a shell probe passing does not +prove the deployed request path works (see ADR-009 in the Manage repo). The +definitive Manage-side test is a real check-in from the web UI. +""" + +import argparse +import base64 +import hashlib +import hmac +import json +import sys +import time +import urllib.parse +from datetime import datetime, timezone + +try: + from websockets.sync.client import connect +except ImportError: + sys.exit("Please `pip install websockets` (needs version 12+ for the sync client).") + +RECEIVE_TIMEOUT_SECONDS = 35 # a little over Manage's 30s so we mirror its failure mode + + +def create_sas_token(namespace: str, hc_name: str, key_name: str, key: str) -> str: + # Mirrors Manage's RelayURI._create_sas_token + uri = f"https://{namespace}/{hc_name}" + encoded_uri = urllib.parse.quote_plus(uri) + expiry = str(int(time.time() + 3600)) + signature = base64.b64encode(hmac.new(key.encode(), f"{encoded_uri}\n{expiry}".encode(), hashlib.sha256).digest()) + return f"SharedAccessSignature sr={encoded_uri}&sig={urllib.parse.quote_plus(signature)}&se={expiry}&skn={key_name}" + + +def log(message: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {message}", flush=True) + + +def main() -> int: + parser = argparse.ArgumentParser(description="Azure Relay sender echo probe") + parser.add_argument("--namespace", required=True, help="e.g. relay-manbrs-prod.servicebus.windows.net") + parser.add_argument("--hc", required=True, help="hybrid connection name, e.g. hc-gw-...-rwa-01") + parser.add_argument("--key-name", default="RootManageSharedAccessKey") + group = parser.add_mutually_exclusive_group(required=True) + group.add_argument("--key", help="shared access key value") + group.add_argument( + "--bearer-token", + help=( + "AAD access token for the relay, to test the bearer path Manage uses " + "in production. Requires the caller to hold Azure Relay Sender. " + "Get one with: az account get-access-token --resource https://relay.azure.net --query accessToken -o tsv" + ), + ) + args = parser.parse_args() + + if args.bearer_token: + token = f"Bearer {args.bearer_token}" + else: + token = create_sas_token(args.namespace, args.hc, args.key_name, args.key) + url = f"wss://{args.namespace}/$hc/{args.hc}?sb-hc-action=connect&sb-hc-token={urllib.parse.quote_plus(token)}" + + payload = { + "action_type": "echo", + "timestamp": datetime.now(timezone.utc).isoformat(), + "source": "relay_echo_probe", + } + + log(f"Connecting as sender to wss://{args.namespace}/$hc/{args.hc} ...") + started = time.monotonic() + try: + with connect(url, compression=None, open_timeout=30) as ws: + log(f"CONNECTED in {time.monotonic() - started:.1f}s - sending echo action...") + ws.send(json.dumps(payload)) + log(f"Sent. Waiting up to {RECEIVE_TIMEOUT_SECONDS}s for the gateway's reply...") + try: + response = ws.recv(timeout=RECEIVE_TIMEOUT_SECONDS) + except TimeoutError: + log("TIMEOUT - no response from the gateway listener.") + log("=> Same failure as Manage: relay accepted the sender but the") + log(" listener never answered. Fault is on the relay<->listener side.") + return 2 + log(f"RESPONSE received in {time.monotonic() - started:.1f}s total:") + print(json.dumps(json.loads(response), indent=2)) + log("=> Relay dispatch to the gateway WORKS. The fault is specific to") + log(" Manage's sender connection.") + return 0 + except Exception as e: + log(f"CONNECT FAILED after {time.monotonic() - started:.1f}s: {type(e).__name__}: {e}") + response = getattr(e, "response", None) + if response is not None: + for header in ("x-ms-error-code", "www-authenticate", "trackingid"): + value = response.headers.get(header) + if value: + log(f" {header}: {value}") + body = getattr(response, "body", b"") + if body: + log(f" body: {body.decode('utf-8', errors='replace')[:500]}") + log("=> 404 means no listener / wrong HC name; 401 means bad token/key") + log(" (note: each HC also has a listen-only policy - its key cannot send;") + log(" use the namespace-level RootManageSharedAccessKey).") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/powershell/debug_toolkit.ps1 b/scripts/powershell/debug_toolkit.ps1 new file mode 100644 index 00000000..50219b29 --- /dev/null +++ b/scripts/powershell/debug_toolkit.ps1 @@ -0,0 +1,179 @@ +# Live-clinic debugging toolkit for the gateway VM. +# +# Usage: RDP to the gateway VM, open PowerShell, then dot-source this file: +# +# . .\debug_toolkit.ps1 +# +# All Gw* query functions are READ-ONLY (SQLite opened with mode=ro). +# The only mutating helper is Restart-GwService, which prompts first. + +$script:GwBase = "C:\Program Files\NHS\ManageBreastScreeningGateway" +$script:GwPython = Join-Path $GwBase "current\.venv\Scripts\python.exe" +$script:GwLogs = Join-Path $GwBase "logs" +$script:GwStorage = Join-Path $GwBase "data\storage" +$script:GwMwlDb = Join-Path $GwBase "data\worklist.db" +$script:GwPacsDb = Join-Path $GwBase "data\pacs.db" +$script:GwServices = @("Gateway-Relay", "Gateway-MWL", "Gateway-PACS", "Gateway-Upload") + +# ── SQLite (via the app's venv Python — no sqlite3.exe needed) ──────────────── + +function Invoke-GwSql { + <# Run an arbitrary read-only SQL query. Usage: + Invoke-GwSql -Db mwl -Query "SELECT * FROM worklist_items LIMIT 5" + Invoke-GwSql -Db pacs -Query "SELECT count(*) AS n FROM stored_instances" #> + param( + [Parameter(Mandatory)][ValidateSet("mwl", "pacs")][string]$Db, + [Parameter(Mandatory)][string]$Query + ) + $dbPath = if ($Db -eq "mwl") { $GwMwlDb } else { $GwPacsDb } + $dbUri = "file:///" + ($dbPath -replace '\\', '/') + "?mode=ro" + $py = @" +import sqlite3, json, sys +conn = sqlite3.connect(r'$dbUri', uri=True, timeout=10) +conn.row_factory = sqlite3.Row +try: + rows = [dict(r) for r in conn.execute(sys.argv[1])] + print(json.dumps(rows, indent=2, default=str)) +finally: + conn.close() +"@ + & $GwPython -c $py $Query | ConvertFrom-Json +} + +function Get-GwWorklist { + <# Today's worklist items (or all with -All). The MWL serves these to the modality. #> + param([switch]$All) + $where = if ($All) { "1=1" } else { "scheduled_date = strftime('%Y%m%d','now')" } + Invoke-GwSql -Db mwl -Query @" +SELECT accession_number, patient_name, patient_id, scheduled_date, scheduled_time, + status, study_instance_uid, updated_at +FROM worklist_items WHERE $where +ORDER BY scheduled_time +"@ | Format-Table accession_number, patient_name, scheduled_time, status, updated_at -AutoSize +} + +function Get-GwWorklistItem { + <# Full detail for one worklist item. Usage: Get-GwWorklistItem ACC20260707XXXX #> + param([Parameter(Mandatory)][string]$AccessionNumber) + Invoke-GwSql -Db mwl -Query "SELECT * FROM worklist_items WHERE accession_number = '$AccessionNumber'" +} + +function Get-GwImages { + <# Images received today, grouped per accession, with upload progress. #> + Invoke-GwSql -Db pacs -Query @" +SELECT accession_number, + count(*) AS images, + sum(CASE WHEN upload_status = 'COMPLETE' THEN 1 ELSE 0 END) AS uploaded, + sum(CASE WHEN upload_status = 'FAILED' THEN 1 ELSE 0 END) AS failed, + printf('%.1f MB', sum(file_size) / 1048576.0) AS size, + max(created_at) AS last_received +FROM stored_instances +WHERE date(created_at) = date('now') +GROUP BY accession_number +ORDER BY last_received DESC +"@ | Format-Table -AutoSize +} + +function Get-GwImageDetail { + <# Per-image rows for one accession: file path, upload status, errors. #> + param([Parameter(Mandatory)][string]$AccessionNumber) + Invoke-GwSql -Db pacs -Query @" +SELECT sop_instance_uid, storage_path, file_size, source_aet, created_at, + upload_status, upload_attempt_count, last_upload_attempt, upload_error +FROM stored_instances WHERE accession_number = '$AccessionNumber' +ORDER BY created_at +"@ +} + +function Get-GwUploadFailures { + <# All instances not yet uploaded to Manage (PENDING/UPLOADING/FAILED), with errors. #> + Invoke-GwSql -Db pacs -Query @" +SELECT accession_number, sop_instance_uid, upload_status, upload_attempt_count, + last_upload_attempt, substr(upload_error, 1, 200) AS upload_error +FROM stored_instances +WHERE upload_status != 'COMPLETE' +ORDER BY created_at +"@ | Format-Table -AutoSize +} + +# ── File system ─────────────────────────────────────────────────────────────── + +function Get-GwDicomFiles { + <# DICOM files written to storage in the last N hours (default 8). #> + param([int]$Hours = 8) + Get-ChildItem -Path $GwStorage -Recurse -File -ErrorAction SilentlyContinue | + Where-Object { $_.LastWriteTime -gt (Get-Date).AddHours(-$Hours) } | + Sort-Object LastWriteTime | + Format-Table LastWriteTime, @{n = "MB"; e = { "{0:N1}" -f ($_.Length / 1MB) } }, FullName -AutoSize +} + +function Get-GwDiskSpace { + Get-PSDrive C | Format-Table Name, + @{n = "UsedGB"; e = { "{0:N1}" -f ($_.Used / 1GB) } }, + @{n = "FreeGB"; e = { "{0:N1}" -f ($_.Free / 1GB) } } -AutoSize +} + +# ── Logs ────────────────────────────────────────────────────────────────────── + +function Watch-GwLog { + <# Live-tail one service log. Usage: Watch-GwLog PACS (Relay|MWL|PACS|Upload) #> + param([Parameter(Mandatory)][ValidateSet("Relay", "MWL", "PACS", "Upload")][string]$Service) + Get-Content -Path (Join-Path $GwLogs "Gateway-$Service.log") -Tail 50 -Wait +} + +function Search-GwLogs { + <# Search all four service logs. Usage: Search-GwLogs "ERROR|Traceback" -Last 200 #> + param([string]$Pattern = "ERROR|Traceback|Exception", [int]$Last = 100) + Get-ChildItem -Path (Join-Path $GwLogs "Gateway-*.log") | ForEach-Object { + Select-String -Path $_.FullName -Pattern $Pattern | Select-Object -Last $Last + } +} + +function Get-GwDeployLog { + <# Most recent deployment log (what version went out, and when). #> + Get-ChildItem -Path (Join-Path $GwLogs "deployments") -Filter "deploy-*" | + Sort-Object LastWriteTime -Descending | Select-Object -First 1 | Get-Content +} + +# ── Service and network health ─────────────────────────────────────────────── + +function Get-GwHealth { + <# One-screen health check: services, ports, relay connection, disk, version. #> + Write-Host "`n── Services ──" -ForegroundColor Cyan + Get-Service -Name "Gateway-*" | Format-Table Name, Status, StartType -AutoSize + + Write-Host "── Listening ports (MWL 104, PACS 11112) ──" -ForegroundColor Cyan + Get-NetTCPConnection -State Listen -ErrorAction SilentlyContinue | + Where-Object { $_.LocalPort -in 104, 11112 } | + Format-Table LocalAddress, LocalPort, OwningProcess -AutoSize + + Write-Host "── Relay listener (last connection lines) ──" -ForegroundColor Cyan + Select-String -Path (Join-Path $GwLogs "Gateway-Relay.log") -Pattern "Connecting to Azure Relay|Connected - waiting|credentials verified|ERROR" | + Select-Object -Last 5 | ForEach-Object { $_.Line } + + Write-Host "`n── Current version ──" -ForegroundColor Cyan + (Get-Item (Join-Path $GwBase "current")).Target + + Write-Host "`n── Disk ──" -ForegroundColor Cyan + Get-GwDiskSpace +} + +function Restart-GwService { + <# Restart one gateway service (prompts for confirmation). + Usage: Restart-GwService Gateway-Relay #> + param([Parameter(Mandatory)][ValidateSet("Gateway-Relay", "Gateway-MWL", "Gateway-PACS", "Gateway-Upload")][string]$Name) + Restart-Service -Name $Name -Confirm + Get-Service -Name $Name +} + +Write-Host "Gateway debug toolkit loaded. Key commands:" -ForegroundColor Green +Write-Host " Get-GwHealth - services / ports / relay / disk, one screen" +Write-Host " Get-GwWorklist - today's MWL items" +Write-Host " Get-GwWorklistItem ACC - one item in full" +Write-Host " Get-GwImages - today's received images + upload progress" +Write-Host " Get-GwImageDetail ACC - per-image detail for one accession" +Write-Host " Get-GwUploadFailures - anything not uploaded to Manage" +Write-Host " Get-GwDicomFiles - files on disk, last 8h" +Write-Host " Watch-GwLog PACS - live tail (Relay|MWL|PACS|Upload)" +Write-Host " Search-GwLogs - grep ERROR/Traceback across all logs" +Write-Host " Invoke-GwSql - arbitrary read-only SQL (mwl|pacs)"