Skip to content

Commit 5b21fed

Browse files
carlosmartinezcarlosmartinez
committed
Runbook for promoting a Gateway VM from pre-prod to prod
1 parent 809c8f6 commit 5b21fed

2 files changed

Lines changed: 137 additions & 1 deletion

File tree

‎docs/deployment/deployment-pipeline.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Deployment Pipeline
22

33
> **Scope**: End-to-end flow from a GitHub tag or main-branch push through Azure DevOps pipelines to an Arc Run Command executing on hospital gateway VMs.
4-
> **Related docs**: [Windows Service Deploy](./windows-service-deploy.md) | [Onboard Hospital VM](./runbooks/onboard-hospital-vm.md) | [Rollback Runbook](./runbooks/rollback.md)
4+
> **Related docs**: [Windows Service Deploy](./windows-service-deploy.md) | [Onboard Hospital VM](./runbooks/onboard-hospital-vm.md) | [Promote VM Pre-Prod → Prod](./runbooks/promote-vm-preprod-to-prod.md) | [Rollback Runbook](./runbooks/rollback.md)
55
66
---
77

‎docs/deployment/runbooks/promote-vm-preprod-to-prod.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# Promote a Gateway VM from Pre-Prod to Prod
2+
3+
> **When to use**: A hospital VM currently running the **pre-prod** gateway needs to become the **prod** gateway (same infrastructure, different environment).
4+
> **Related**: [Onboard Hospital VM](./onboard-hospital-vm.md) | [Cleanup](./cleanup.md) | [Deployment Pipeline](../deployment-pipeline.md)
5+
6+
---
7+
8+
## Why this is not just "re-point the gateway"
9+
10+
Pre-prod and prod are separate Azure subscriptions, with separate Arc resource groups, relay namespaces (`relay-manbrs-<env>`), web-API service principals (`spn-manbrs-web-api-<env>`) and ADO pipelines. An Arc-connected machine belongs to **exactly one** subscription, and the gateway services authenticate using that machine's managed identity. A pre-prod machine identity holds the `Gateway.Access` role on the **pre-prod** API and cannot talk to prod.
11+
12+
So you cannot promote by editing `.env`. The machine must be **disconnected from the pre-prod subscription and re-onboarded into the prod subscription** — effectively the [onboarding runbook](./onboard-hospital-vm.md) run for prod, preceded by a clean decommission of the pre-prod gateway.
13+
14+
> One box hosts one gateway. After this procedure there is **no pre-prod gateway at this site**. If pre-prod is still needed at the site, use a different VM.
15+
16+
---
17+
18+
## Before the day (do these in advance)
19+
20+
- [ ] **`GATEWAY_RINGS` is set for prod.** `infrastructure/environments/prod/variables.sh` must set `GATEWAY_RINGS` to include the ring the site is tagged with (for initial production installs this will typically be `ring1`). If it is unset it defaults to `ring0` and the prod deploy pipeline **silently skips** the machine.
21+
- [ ] **Prod infrastructure exists.** Confirm the prod relay namespace, `spn-manbrs-web-api-prod`, Log Analytics workspace and the prod ADO pipelines are provisioned and have had a successful infra deploy.
22+
- [ ] **Prod Manage (Rubie) is seeded.** The prod Rubie instance has the site's clinic/setting, a `Relay` record pointing at `relay-manbrs-prod` and the prod gateway, the `gateway_images` feature flag enabled, and a test appointment for the end-to-end check.
23+
- [ ] **DHCP reservation confirmed** for the VM, so the IP the modality targets cannot change between decommission and go-live.
24+
- [ ] **Modality engineer briefed** that the gateway **AE titles change** (see [Step 5](#step-5--reconfigure-the-modality-on-site)) and the modality must be reconfigured and re-tested on the day.
25+
- [ ] **Fresh prod Arc onboarding SPN secret** generated with 1-day expiry (`arc-onboarding-spn-client-secret`), shared with hospital IT before the onboarding call.
26+
27+
---
28+
29+
## Step 1 — Decommission the pre-prod gateway (on the VM)
30+
31+
Run from an **elevated PowerShell session** on the VM. First confirm you are on the right machine:
32+
33+
```powershell
34+
hostname
35+
Get-Service Gateway-* | Format-Table Name, Status
36+
```
37+
38+
Then run the cleanup script, which stops and removes the services and **removes the installation directory including all pre-prod data** (`worklist.db`, `pacs.db`, `data\storage`):
39+
40+
```powershell
41+
.\scripts\powershell\cleanup.ps1
42+
```
43+
44+
**Verify**: no `Gateway-*` services remain and `C:\Program Files\NHS\ManageBreastScreeningGateway` is gone (see [Cleanup runbook — Verify](./cleanup.md)).
45+
46+
## Step 2 — Disconnect from the pre-prod Arc subscription
47+
48+
Still on the VM, elevated:
49+
50+
```powershell
51+
azcmagent disconnect
52+
```
53+
54+
This removes the machine's registration from the pre-prod resource group. The Arc **agent stays installed** — only the registration is removed, so re-onboarding in Step 3 skips the agent install.
55+
56+
**Verify**: `azcmagent show` reports the agent as **Disconnected**.
57+
58+
## Step 3 — Re-onboard into prod
59+
60+
This is [Onboarding runbook Step 2](./onboard-hospital-vm.md#step-2--run-arc-onboarding-script-on-the-gateway-vm) with **prod parameters**. The site parameters (`SiteName`, `ODSCode`, `Instance`) are unchanged, so the Arc resource name is identical — but it is now created in the prod resource group.
61+
62+
```powershell
63+
.\arc-setup.ps1 `
64+
-SubscriptionId "<prod-spoke-subscription-id>" `
65+
-TenantId "<tenant-id>" `
66+
-ResourceGroup "rg-mbsgw-prod-uks-arc-enabled-servers" `
67+
-Location "uksouth" `
68+
-ServicePrincipalId "<arc-onboarding-spn-client-id>" `
69+
-ServicePrincipalSecret "<prod-arc-onboarding-spn-client-secret>" `
70+
-SiteName "Hull-University-Teaching-Hospitals-NHS-Trust" `
71+
-ODSCode "RWA" `
72+
-Instance "01" `
73+
-NHSRegion "neyh" `
74+
-SiteType "static" `
75+
-DeploymentRing "ring1"
76+
```
77+
78+
**Verify**: in the Azure portal, `rg-mbsgw-prod-uks-arc-enabled-servers` → Azure Arc machines → `gw-<...>-rwa-01` is **Connected**.
79+
80+
## Step 4 — Provision prod and deploy the prod gateway
81+
82+
From here, follow the onboarding runbook against **prod**:
83+
84+
1. **Grant API access** — `make prod assign-arc-app-roles` ([Step 3](./onboard-hospital-vm.md#step-3--grant-api-access)). This assigns `Gateway.Access` on `spn-manbrs-web-api-prod` to the machine's new prod managed identity. The old pre-prod assignment is irrelevant (different identity) and is cleaned up in Step 6.
85+
2. **Provision the Hybrid Connection** — run **Deploy Arc Infrastructure - prod** ([Step 4](./onboard-hospital-vm.md#step-4--trigger-terraform-to-provision-the-hybrid-connection)). Creates `hc-gw-<...>-rwa-01` in `relay-manbrs-prod`.
86+
3. **Deploy the application** — run **Deploy Gateway - prod** with a **released** `releaseTag` (not a pre-prod build) ([Step 5](./onboard-hospital-vm.md#step-5--deploy-the-gateway-application)). This writes a fresh prod `.env` (prod relay namespace, `CLOUD_API_HOSTNAME=manage-breast-screening.nhs.uk`, prod AE titles), fully replacing the pre-prod `.env`.
87+
88+
**Verify** (smoke test): all four services Running, and an initial heartbeat in the prod Log Analytics workspace within 5 minutes.
89+
90+
```powershell
91+
Get-Service Gateway-PACS, Gateway-MWL, Gateway-Upload, Gateway-Relay | Select-Object Name, Status
92+
```
93+
94+
## Step 5 — Reconfigure the modality
95+
96+
The gateway AE titles are environment-specific: the `.env` builder sets `MWL_AET=RUBIE_MWL_<ENV>` / `PACS_AET=RUBIE_PACS_<ENV>`, where the environment is uppercased and truncated to 3 characters if longer than 4.
97+
98+
| Environment | MWL AE title | PACS AE title |
99+
|-------------|--------------|---------------|
100+
| pre-prod (`PREPROD` → `PRE`) | `RUBIE_MWL_PRE` | `RUBIE_PACS_PRE` |
101+
| prod (`PROD`) | `RUBIE_MWL_PROD` | `RUBIE_PACS_PROD` |
102+
103+
The modality is currently configured to send to the **pre-prod** AE titles. The modality's MWL and PACS destinations must be updated to the **prod** AE titles. The VM IP and ports (`104` / `11112`) will typically be unchanged.
104+
105+
**Verify**: a C-ECHO from the modality to each prod AE title succeeds. (A C-ECHO to the old `..._PRE` titles will now fail — expected.)
106+
107+
## Step 6 — Clean up the pre-prod side (Azure)
108+
109+
Disconnecting the machine in Step 2 leaves orphaned pre-prod resources: a disconnected Arc machine, a dangling Hybrid Connection (`hc-gw-<...>-rwa-01` in `relay-manbrs-preprod`), and a stale `Gateway.Access` role assignment.
110+
111+
- [ ] Run **Deploy Arc Infrastructure - preprod** (Terraform) so it destroys the now-orphaned Hybrid Connection.
112+
- [ ] Remove the disconnected Arc machine resource from `rg-mbsgw-preprod-uks-arc-enabled-servers` if Terraform/Arc has not already.
113+
- [ ] Confirm the stale pre-prod app-role assignment is removed (re-running `make preprod assign-arc-app-roles` reconciles, or remove it via the portal).
114+
115+
---
116+
117+
## End-to-end check
118+
119+
With the modality reconfigured and prod Rubie seeded:
120+
121+
1. Start the seeded test appointment in **prod** Rubie → a worklist item reaches the prod gateway.
122+
2. Query the worklist from the modality → the item appears.
123+
3. Acquire and send images → they arrive in the prod gateway PACS and are forwarded to prod Rubie.
124+
4. Confirm the images appear against the appointment in prod Rubie.
125+
126+
---
127+
128+
## Gotchas
129+
130+
| Symptom | Cause | Fix |
131+
|---------|-------|-----|
132+
| **Deploy Gateway - prod** reports "No machines found for ring1 — skipping" | `GATEWAY_RINGS` not set for prod, defaulting to `ring0` | Set `GATEWAY_RINGS="ring1"` in `prod/variables.sh` and redeploy |
133+
| Services start but fail to authenticate against the cloud API | `Gateway.Access` not assigned to the **prod** managed identity | Run `make prod assign-arc-app-roles` |
134+
| Modality C-ECHO fails after promotion | Modality still targeting `..._PRE` AE titles | Reconfigure modality to `..._PROD` (Step 5) |
135+
| Hybrid Connection not created by Terraform | Arc machine not yet **Connected** in the prod RG | Confirm Step 3 verify, then re-run the infra pipeline |
136+
| Pre-prod relay still shows connection attempts | Orphaned pre-prod HC / registration | Complete Step 6 |

0 commit comments

Comments
 (0)