Skip to content

Commit 3c3bdd1

Browse files
authored
Merge pull request #112 from NHSDigital/DTOSS-12685-emulate-modality-workflow
[DTOSS 12685] Emulate modality workflow
2 parents 468a169 + 759a5ce commit 3c3bdd1

14 files changed

Lines changed: 533 additions & 6 deletions

File tree

Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ RUN uv sync --frozen --no-dev
2424
# Copy source code
2525
COPY src/ ./src/
2626
COPY scripts/ ./scripts/
27+
COPY sample_images/ ./sample_images/
2728

2829
# Set environment variables with defaults
2930
ENV PACS_AET=SCREENING_PACS \

compose.yml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,29 @@ services:
9292
pacs:
9393
condition: service_healthy
9494
restart: unless-stopped
95+
network_mode: host
96+
97+
modality_emulator:
98+
build:
99+
context: .
100+
dockerfile: Dockerfile
101+
container_name: modality-emulator
102+
command: ["uv", "run", "python", "-m", "src.modality_emulator"]
103+
volumes:
104+
- pacs-storage:/var/lib/pacs/storage
105+
- pacs-db:/var/lib/pacs
106+
environment:
107+
- PACS_DB_PATH=/var/lib/pacs/pacs.db
108+
- PACS_STORAGE_PATH=/var/lib/pacs/storage
109+
- MWL_DB_PATH=/var/lib/pacs/worklist.db
110+
- LOG_LEVEL=INFO
111+
depends_on:
112+
mwl:
113+
condition: service_healthy
114+
pacs:
115+
condition: service_healthy
116+
restart: unless-stopped
117+
network_mode: host
95118

96119
volumes:
97120
pacs-storage:
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# ADR-006: Use a modality emulator for non-production environments
2+
3+
Date: 2026-06-04
4+
5+
Status: Accepted
6+
7+
## Context
8+
9+
The Manage Breast Screening service sends worklist items corresponding to screening appointments as the appointment starts. These are received by the gateway via Azure Relay and processed to trigger DICOM API uploads of the screening study data. The worklist items are queried via DICOM C-FIND requests to the gateway MWL server and stored on the gateway PACS server using the DICOM C-STORE protocol.
10+
11+
In production this query is performed by the modality, which is a physical machine in the hospital network that runs software to query the MWL and display results to radiographers. The modality then sends the resulting study DICOM files to the Gateway PACS server using C-STORE. The PACS server then compresses the image data in the DICOM files and uploads it to the DICOM API hosted on Manage.
12+
13+
In non-production environments, there is no modality available to perform this query. This means that testing the full flow of worklist items through the gateway and into the DICOM API is difficult without deploying to production or using complex stubs/mocks.
14+
15+
## Decision
16+
17+
To facilitate end-to-end testing of the worklist flow in non-production environments, we will implement a simple modality emulator. This will be a lightweight service that simulates the behavior of a real modality by performing queries against the MWL server and storing the emulated study data on the Gateway PACS server, mimicking the behavior of a real modality.
18+
19+
The emulator runs on the same host as the MWL and PACS servers in non-production environments. It runs as a Python process that makes a C-FIND request at a configurable polling interval, checking for new worklist items. When it finds a new item, it creates a DICOM dataset from the worklist details, includes some sample image data and sends them to the PACS server via C-STORE.
20+
21+
The sample image data corresponds to the 4 main laterality and view position combinations (LCC, LMLO, RCC, RMLO) to allow testing of the laterality and view position logic in the gateway images screens as the DICOM files are uploaded to Manage.
22+
23+
## Consequences
24+
25+
We no longer have to use a third party emulator or do any UI work to test the happy path of worklist items flowing through the gateway and into the DICOM API in non-production environments. This also allows us to make more comprehensive end-to-end tests that cover the full flow, including the C-FIND and C-STORE interactions with the MWL and PACS servers.
26+
27+
### Positive Consequences
28+
29+
- **End-to-end testing:** Enables testing of the full worklist flow in non-production environments without needing a physical modality or complex stubs/mocks.
30+
- **Simplified test setup:** Developers can run the emulator locally or in test environments to simulate modality behavior without additional infrastructure.
31+
- **Improved test coverage:** Allows for more comprehensive tests that cover the interactions with the MWL and PACS servers, as well as the gateway's processing logic for worklist items.
32+
- **Configurable behavior:** The emulator can be configured to simulate different scenarios, such as varying polling intervals or different worklist item details, to test edge cases and error handling in the gateway.
33+
34+
### Negative Consequences
35+
36+
- The emulator and sample images are packaged with every release. This increases the size of the release artifact and may have implications for storage and distribution.

sample_images/LCC.jpg

512 KB
Loading

sample_images/LMLO.jpg

568 KB
Loading

scripts/bash/deploy_arc_ring.sh

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,8 @@ PACS_AET=SCREENING_PACS
8282
PACS_PORT=4244
8383
PACS_STORAGE_PATH=${BASE_PATH}/data/storage
8484
PACS_DB_PATH=${BASE_PATH}/data/pacs.db
85-
LOG_LEVEL=INFO"
85+
LOG_LEVEL=INFO
86+
SAMPLE_IMAGES_PATH=${BASE_PATH}/current/sample_images"
8687

8788
# Cross-platform base64 encoding (works on macOS and Linux)
8889
ENV_CONTENT_B64=$(printf '%s' "$ENV_CONTENT" | base64 | tr -d '\n')
@@ -105,6 +106,7 @@ LOG_LEVEL=INFO"
105106
--arg pyver "$PYTHON_VERSION" \
106107
--arg envb64 "$ENV_CONTENT_B64" \
107108
--arg token "$GITHUB_TOKEN" \
109+
--arg env "$ENVIRONMENT" \
108110
'{
109111
location: $loc,
110112
properties: {
@@ -113,7 +115,8 @@ LOG_LEVEL=INFO"
113115
{ name: "ReleaseTag", value: $tag },
114116
{ name: "PythonVersion", value: $pyver },
115117
{ name: "EnvContentB64", value: $envb64 },
116-
{ name: "GitHubToken", value: $token }
118+
{ name: "GitHubToken", value: $token },
119+
{ name: "Environment", value: $env }
117120
],
118121
runAsSystem: true,
119122
timeoutInSeconds: 1800

scripts/bash/package_release.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ echo ""
6161

6262
# ── Validate required files ───────────────────────────────────────────────────
6363

64-
REQUIRED_FILES=("src/" "scripts/python/" "pyproject.toml" "uv.lock" "README.md")
64+
REQUIRED_FILES=("src/" "sample_images/" "scripts/python/" "pyproject.toml" "uv.lock" "README.md" "LICENCE.md")
6565

6666
for item in "${REQUIRED_FILES[@]}"; do
6767
if [[ ! -e "${REPO_ROOT}/${item}" ]]; then

scripts/powershell/deploy.ps1

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,10 @@ param(
4343
[string]$GitHubToken,
4444

4545
[Parameter()]
46-
[string]$EnvContentB64 = ""
46+
[string]$EnvContentB64 = "",
47+
48+
[Parameter()]
49+
[string]$Environment = "prod"
4750
)
4851

4952
Set-StrictMode -Version Latest
@@ -337,6 +340,9 @@ $services = @(
337340
@{ Name = "Gateway-MWL"; Script = "mwl_main.py" },
338341
@{ Name = "Gateway-Upload"; Script = "upload_main.py" }
339342
)
343+
if (($Environment -ne "prod") -and ($Environment -ne "preprod")) {
344+
$services += @{ Name = "Gateway-Emulator"; Script = "modality_emulator.py" }
345+
}
340346

341347
# -- Extraction ---------------------------------------------------------------
342348

scripts/powershell/rollback.ps1

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,8 @@ $services = @(
6161
@{ Name = "Gateway-Relay"; Script = "relay_listener.py" },
6262
@{ Name = "Gateway-PACS"; Script = "pacs_main.py" },
6363
@{ Name = "Gateway-MWL"; Script = "mwl_main.py" },
64-
@{ Name = "Gateway-Upload"; Script = "upload_main.py" }
64+
@{ Name = "Gateway-Upload"; Script = "upload_main.py" },
65+
@{ Name = "Gateway-Emulator"; Script = "modality_emulator.py" }
6566
)
6667

6768
# -- Resolve current version --------------------------------------------------

src/modality_emulator.py

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
import datetime
2+
import logging
3+
import os
4+
import time
5+
6+
import numpy as np
7+
from PIL import Image
8+
from pydicom.dataset import Dataset, FileMetaDataset
9+
from pydicom.uid import ExplicitVRLittleEndian, generate_uid
10+
from pynetdicom import AE
11+
from pynetdicom.sop_class import (
12+
DigitalMammographyXRayImageStorageForPresentation, # type: ignore
13+
ModalityWorklistInformationFind, # type: ignore
14+
)
15+
16+
from environment import Environment
17+
from services.dicom import PENDING, PENDING_WARNING, SUCCESS
18+
from services.mwl import MWLStatus
19+
from services.storage import MWLStorage
20+
21+
logging.basicConfig(
22+
level=os.getenv("LOG_LEVEL", "INFO").upper(),
23+
format=os.getenv("LOG_FORMAT", "%(asctime)s - %(name)s - %(levelname)s - %(message)s"),
24+
)
25+
26+
logger = logging.getLogger(__name__)
27+
28+
29+
DICOM_LATERALITIES = ["L", "R"]
30+
DICOM_VIEWS = ["CC", "MLO"]
31+
EMULATED_PROCEDURE_DURATION_SECONDS = int(os.getenv("EMULATED_PROCEDURE_DURATION_SECONDS", "5"))
32+
MODALITY = "MG"
33+
MWL_AET = os.getenv("MWL_AET", "SCREENING_MWL")
34+
MWL_DB_PATH = os.getenv("MWL_DB_PATH", "/var/lib/pacs/worklist.db")
35+
MWL_HOST = os.getenv("MWL_HOST", "localhost")
36+
MWL_PORT = int(os.getenv("MWL_PORT", "4243"))
37+
PACS_AET = os.getenv("PACS_AET", "SCREENING_PACS")
38+
PACS_DB_PATH = os.getenv("PACS_DB_PATH", "/var/lib/pacs/pacs.db")
39+
PACS_HOST = os.getenv("PACS_HOST", "localhost")
40+
PACS_PORT = int(os.getenv("PACS_PORT", "4244"))
41+
SAMPLE_IMAGES_PATH = os.getenv("SAMPLE_IMAGES_PATH", "sample_images")
42+
43+
44+
class DicomExample:
45+
def __init__(
46+
self, dataset: Dataset | None, laterality: str, view: str, study_instance_uid: str, series_number: int
47+
):
48+
self.dataset = dataset
49+
self.laterality = laterality
50+
self.view = view
51+
self.study_instance_uid = study_instance_uid
52+
self.series_number = series_number
53+
self.data = self.generate_dicom()
54+
55+
def generate_dicom(self) -> Dataset:
56+
ds = Dataset()
57+
if not self.dataset:
58+
logger.error("No dataset provided for DICOM generation")
59+
return ds
60+
61+
img_path = f"{SAMPLE_IMAGES_PATH}/L{self.view}.jpg"
62+
img = Image.open(img_path).convert("L")
63+
if self.laterality == "R":
64+
img = img.transpose(Image.Transpose.FLIP_LEFT_RIGHT)
65+
columns, rows = img.size
66+
pixel_array = np.array(img, dtype=np.uint8)
67+
pixel_bytes = pixel_array.tobytes()
68+
if len(pixel_bytes) % 2 != 0:
69+
pixel_bytes += b"\x00"
70+
71+
file_meta = FileMetaDataset()
72+
file_meta.MediaStorageSOPClassUID = DigitalMammographyXRayImageStorageForPresentation
73+
file_meta.MediaStorageSOPInstanceUID = generate_uid()
74+
file_meta.ImplementationClassUID = generate_uid()
75+
file_meta.TransferSyntaxUID = ExplicitVRLittleEndian
76+
77+
ds.SOPClassUID = file_meta.MediaStorageSOPClassUID
78+
ds.SOPInstanceUID = file_meta.MediaStorageSOPInstanceUID
79+
ds.SamplesPerPixel = 1
80+
ds.PhotometricInterpretation = "MONOCHROME2"
81+
ds.Rows = rows
82+
ds.Columns = columns
83+
ds.BitsAllocated = 8
84+
ds.BitsStored = 8
85+
ds.HighBit = 7
86+
ds.PixelRepresentation = 0
87+
ds.PixelData = pixel_bytes
88+
ds.ImageLaterality = self.laterality
89+
ds.ViewPosition = self.view
90+
91+
ds.AccessionNumber = self.dataset.AccessionNumber
92+
ds.PatientID = self.dataset.PatientID
93+
ds.PatientName = self.dataset.PatientName
94+
ds.PatientBirthDate = self.dataset.PatientBirthDate
95+
ds.PatientSex = self.dataset.PatientSex
96+
scheduled_step = self.dataset.ScheduledProcedureStepSequence[0]
97+
ds.StudyDate = scheduled_step.ScheduledProcedureStepStartDate
98+
ds.StudyTime = scheduled_step.ScheduledProcedureStepStartTime
99+
ds.StudyInstanceUID = self.study_instance_uid
100+
ds.StudyID = f"STUDY{self.study_instance_uid[-8:]}"
101+
ds.SeriesInstanceUID = generate_uid()
102+
ds.SeriesNumber = self.series_number
103+
ds.Modality = MODALITY
104+
ds.InstanceNumber = 1
105+
ds.file_meta = file_meta
106+
107+
logger.debug(f"Generated DICOM for worklist item {self.dataset.AccessionNumber} - {self.laterality}{self.view}")
108+
logger.debug(f"{ds}")
109+
110+
return ds
111+
112+
113+
class ModalityEmulator:
114+
def __init__(self, mwl_storage: MWLStorage):
115+
self.mwl_storage = mwl_storage
116+
self.processed_items = set()
117+
118+
def process_worklist_items(self, ae: AE):
119+
"""
120+
Queries the MWL for items scheduled for today and sends generated DICOM files to the PACS server for each item.
121+
"""
122+
mwl_assoc = ae.associate(MWL_HOST, MWL_PORT, ae_title=MWL_AET)
123+
pacs_assoc = ae.associate(PACS_HOST, PACS_PORT, ae_title=PACS_AET)
124+
125+
if mwl_assoc.is_established and pacs_assoc.is_established:
126+
logger.info(f"Connected to MWL server {MWL_HOST}:{MWL_PORT} ({MWL_AET})")
127+
logger.info(f"Connected to PACS server {PACS_HOST}:{PACS_PORT} ({PACS_AET})")
128+
129+
logger.info("Querying MWL for scheduled items...")
130+
responses = mwl_assoc.send_c_find(self.c_find_dataset, query_model=ModalityWorklistInformationFind)
131+
for status, ds in responses:
132+
status_code = getattr(status, "Status", SUCCESS)
133+
134+
if status_code in (PENDING, PENDING_WARNING):
135+
accession_number = getattr(ds, "AccessionNumber", "UNKNOWN")
136+
137+
if accession_number in self.processed_items:
138+
logger.info(f"Skipping already processed worklist item {accession_number}")
139+
continue
140+
141+
self.processed_items.add(accession_number)
142+
study_instance_uid = generate_uid()
143+
series_number = 1
144+
for laterality in DICOM_LATERALITIES:
145+
for view in DICOM_VIEWS:
146+
logger.info(
147+
f"Processing worklist item {accession_number} - generating DICOM for {laterality}{view}"
148+
)
149+
dicom_example = DicomExample(ds, laterality, view, study_instance_uid, series_number)
150+
dataset = dicom_example.data
151+
152+
if getattr(dataset, "SOPInstanceUID", None) is None:
153+
logger.error(
154+
f"Skipping DICOM generation for {laterality}{view} of worklist item {accession_number}"
155+
)
156+
continue
157+
158+
pacs_assoc.send_c_store(dataset)
159+
logger.info(
160+
f"Sent DICOM for {laterality}{view} of worklist item {accession_number}. Series# {series_number}"
161+
)
162+
series_number += 1
163+
164+
time.sleep(1) # Allow C-STORE operations to complete before updating status
165+
166+
self.mwl_storage.update_status(accession_number, MWLStatus.COMPLETED.value)
167+
logger.info(f"Completed processing for worklist item {accession_number}")
168+
elif status_code == SUCCESS:
169+
logger.info("C-FIND query completed successfully")
170+
else:
171+
logger.error(f"C-FIND query failed with status: 0x{status_code:04X}")
172+
173+
else:
174+
logger.error("Failed to make MWL and PACS associations")
175+
176+
mwl_assoc.release()
177+
pacs_assoc.release()
178+
179+
@property
180+
def c_find_dataset(self) -> Dataset:
181+
date_today = datetime.date.today()
182+
ds = Dataset()
183+
sps_dataset = Dataset()
184+
sps_dataset.Modality = MODALITY
185+
sps_dataset.ScheduledProcedureStepStartDate = date_today.strftime("%Y%m%d")
186+
sps_dataset.ScheduledProcedureStepStartTime = "000000-"
187+
ds.ScheduledProcedureStepSequence = [sps_dataset]
188+
return ds
189+
190+
191+
def main():
192+
if Environment().production:
193+
raise RuntimeError("Modality Emulator should not be run in production environment")
194+
195+
logger.info("Modality Emulator Starting...")
196+
mwl_storage = MWLStorage(db_path=MWL_DB_PATH)
197+
emulator = ModalityEmulator(mwl_storage)
198+
ae = AE(ae_title="ModalityEmulator")
199+
ae.add_requested_context(DigitalMammographyXRayImageStorageForPresentation)
200+
ae.add_requested_context(ModalityWorklistInformationFind)
201+
202+
while True:
203+
try:
204+
time.sleep(EMULATED_PROCEDURE_DURATION_SECONDS)
205+
emulator.process_worklist_items(ae)
206+
except KeyboardInterrupt:
207+
logger.warning("\n Modality Emulator shutting down...")
208+
break
209+
except Exception:
210+
logger.exception("Error in Modality Emulator")
211+
time.sleep(5) # Wait before retrying to avoid tight error loop
212+
213+
214+
if __name__ == "__main__":
215+
main()

0 commit comments

Comments
 (0)