Backup and reset MWL on a schedule (#76)

The worklist is inherently ephemeral. Stale worklist items from a previous day are not meaningful to the modality and could cause confusion if they appeared in a C-FIND response.

Therefore we backup and reset the MWL database on a configurable schedule (default: daily at 02:00 UTC).

Author: carlosmartinez

.env.development

@@ -11,6 +11,8 @@ MWL_AET=SCREENING_MWL
 MWL_PORT=4243
 MWL_DB_PATH=/var/lib/pacs/worklist.db
 
+BACKUP_PATH=/var/lib/pacs/backups
+
 # PACS Server Configuration
 PACS_AET=SCREENING_PACS
 PACS_PORT=4244

Makefile

@@ -69,7 +69,7 @@ _install-uv:
 #  Database backup
 # ---------------------------------------------------------------------------
 backup-db: # Backup sqlite databases @Operations
-	PYTHONPATH=scripts/python uv run python -m backup_database
+	PYTHONPATH=src uv run python -m backup_main
 
 # ---------------------------------------------------------------------------
 # Testing

docs/adr/ADR-004_MWL_Daily_Backup_And_Reset.md

@@ -0,0 +1,46 @@
+# ADR-004: Daily backup and reset of the MWL database
+
+Date: 2026-04-02
+
+Status: Accepted
+
+## Context
+
+The gateway MWL database holds scheduled appointment data that is consumed by mammography modality via DICOM C-FIND. This data originates from Manage Breast Screening and is written to the gateway by the relay listener.
+
+The worklist is inherently ephemeral: appointments are scheduled per clinic, and a clinic session does not span calendar days. Stale worklist items from a previous day are not meaningful to the modality and could cause confusion if they appeared in a C-FIND response.
+
+The gateway MWL is not intended to be a canonical or long-term data store. Retaining worklist items indefinitely would cause the database to grow to unwieldy proportions and would give a false impression of data durability.
+
+The gateway runs as a Python process on a Windows VM managed by Azure Arc. Scheduling therefore belongs in the infrastructure layer (Windows Task Scheduler, configurable via Arc).
+
+## Decision
+
+Reset the MWL database on a schedule managed by Windows Task Scheduler by:
+
+1. Backing up the database before clearing it, using SQLite's native `conn.backup()` API
+2. Deleting all rows from `worklist_items`
+
+`reset_main.py` is performs these two steps and exits. Windows Task Scheduler (registered via `scripts/bat/schtasks.bat`) invokes it on whatever schedule is configured in infrastructure.
+
+**Alternatives considered:**
+
+- **In-process Python scheduler** (`croniter` sleep loop) — puts scheduling logic in application code; duplicates a facility the OS already provides; requires a long-running process that serves no other purpose
+- **Reset on startup only** — would not handle long-running deployments where the process is not restarted between clinics
+- **No reset** — leads to unbounded growth and stale data visible to the modality
+
+## Consequences
+
+### Positive Consequences
+
+- Worklist is clean at the start of each clinic with no manual intervention
+- Database size remains bounded
+- Backup before clear means data is recoverable if needed
+- Schedule is owned by infrastructure (Arc / Task Scheduler), not application code — no code change needed to adjust timing
+- `reset_main.py` is a simple, testable, one-shot script with no scheduler machinery
+
+### Negative Consequences
+
+- Schedule configuration lives outside the codebase; requires infrastructure access to change
+- Any worklist items written very close to the reset time could be cleared before the modality queries them. This is mitigated by choosing a reset time well outside clinic hours
+- Backups accumulate on disk and will need periodic pruning; this is not currently automated

docs/mwl/README.md

@@ -9,7 +9,8 @@ The MWL server is a lightweight, production-ready DICOM worklist solution that:
 - Provides scheduled procedure information via [DICOM C-FIND](https://dicom.nema.org/medical/dicom/current/output/html/part04.html#chapter_C) protocol
 - Stores worklist items in SQLite database
 - Supports filtering by modality, date, and patient ID
-- Runs in a separate container alongside the [PACS Server](../pacs/README.md)
+- Resets the worklist daily via a scheduled `reset_main.py` script invoked by Windows Task Scheduler
+- Runs as a Python process on a Windows VM managed by Azure Arc
 
 ## Architecture
 
@@ -21,54 +22,53 @@ The MWL server is a lightweight, production-ready DICOM worklist solution that:
 ├─────────────────────────────────────────────────────────────┤
 │                                                             │
 │  ┌──────────────┐      ┌──────────────┐      ┌──────────┐   │
-│  │   C-FIND     │─────▶│   Storage    │─────▶│ SQLite   │   │
-│  │   Handler    │      │   Layer      │      │ Database │   │
-│  └──────────────┘      └──────────────┘      └──────────┘   │
-│         │                      ▲                            │
-│         │                      │                            │
-│         └──────────────────────┘                            │
-│         Query & Response                                    │
-└─────────────────────────────────────────────────────────────┘
-           ▲                                     ▲
-           │                                     │
-    ┌──────┴──────┐                      ┌───────┴────────┐
-    │  Modality   │                      │ Relay Listener │
-    │  (SCU)      │                      │ (Populates DB) │
-    └─────────────┘                      └────────────────┘
+│  │   C-FIND     │─────▶│   Storage    │─────▶│ SQLite   │◀──┼──────────────────┐
+│  │   Handler    │      │   Layer      │      │ Database │   │                  │
+│  └──────────────┘      └──────────────┘      └──────────┘   │                  │
+│         │                      ▲                            │                  │
+│         │                      │                            │        backup + clear
+│         └──────────────────────┘                            │      (Task Scheduler)
+│         Query & Response                                    │                  │
+└─────────────────────────────────────────────────────────────┘                  │
+           ▲                         ▲                                    ┌───────┴────────┐
+           │                         │                                    │  reset_main.py │
+    ┌──────┴──────┐           ┌───────┴────────┐                          └────────────────┘
+    │  Modality   │           │ Relay Listener │
+    │  (SCU)      │           │ (Populates DB) │
+    └─────────────┘           └────────────────┘
 ```
 
 ### Workflow
 
-1. **Worklist Creation**: Relay listener receives appointments from web app and creates worklist items (NB not yet implemented; worklist items must be created programmatically via `scripts/add_worklist_item.py`)
+1. **Worklist Creation**: Relay listener receives appointments from Manage Breast Screening and creates worklist items
 2. **Worklist Query**: Modality sends C-FIND request to MWL server
 3. **Filtering**: MWL server filters by modality, date, patient ID, status
 4. **Response**: Server returns matching worklist items to modality
-5. **Status Updates**: [MPPS (Modality Performed Procedure Step)](https://dicom.nema.org/medical/dicom/current/output/html/part04.html#chapter_F) updates procedure status (NB not yet implemented)
+5. **Status Updates**: C-STORE receipt transitions items from `SCHEDULED` to `IN PROGRESS`; [MPPS](https://dicom.nema.org/medical/dicom/current/output/html/part04.html#chapter_F) transitions items to `COMPLETED` or `DISCONTINUED`
+6. **Daily Reset**: Windows Task Scheduler invokes `reset_main.py`, which backs up and clears the database
 
 ## Running the MWL Server
 
-The MWL server runs in a separate container:
+The gateway runs as a Python process on a Windows VM:
 
 ```bash
-# Start both PACS and MWL servers
-docker compose up -d
+# Start the MWL server
+uv run python -m mwl_main
 
-# Start only MWL server
-docker compose up -d mwl
-
-# View logs
-docker compose logs -f mwl
+# Run a one-shot backup and reset (normally invoked by Task Scheduler)
+uv run python -m reset_main
+```
 
-# Stop servers
-docker compose down
+For local development, Docker Compose is available:
 
-# Reset databases
-docker compose down -v
+```bash
+docker compose up -d mwl
+docker compose logs -f mwl
 ```
 
 ## Configuration
 
-Environment variables:
+### MWL server
 
 | Variable | Default | Description |
 |----------|---------|-------------|
@@ -77,6 +77,26 @@ Environment variables:
 | `MWL_DB_PATH` | `/var/lib/pacs/worklist.db` | SQLite database path |
 | `LOG_LEVEL` | `INFO` | Logging level |
 
+### Reset script (`reset_main.py`)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MWL_DB_PATH` | `/var/lib/pacs/worklist.db` | SQLite database path |
+| `BACKUP_PATH` | `/var/lib/pacs/backups` | Directory for database backups |
+| `LOG_LEVEL` | `INFO` | Logging level |
+
+The reset schedule is configured in Windows Task Scheduler (registered via `scripts/bat/schtasks.bat`), not in the application.
+
+## Scheduling the daily reset (Windows)
+
+Register the scheduled task (run once, on the gateway VM):
+
+```bat
+scripts\bat\schtasks.bat
+```
+
+This creates a Task Scheduler task that runs `reset_main.py` daily at midnight. The schedule can be adjusted in Task Scheduler or via Azure Arc without any code change.
+
 ## Example query
 
 ```python
@@ -112,14 +132,14 @@ assoc.release()
 Check worklist items:
 
 ```bash
-docker compose exec gateway sqlite3 /var/lib/pacs/worklist.db \
+sqlite3 /var/lib/pacs/worklist.db \
   "SELECT accession_number, patient_name, scheduled_date, status FROM worklist_items;"
 ```
 
 Add test worklist item:
 
 ```bash
-docker compose exec gateway sqlite3 /var/lib/pacs/worklist.db <<EOF
+sqlite3 /var/lib/pacs/worklist.db <<EOF
 INSERT INTO worklist_items (
     accession_number, patient_id, patient_name, patient_birth_date,
     scheduled_date, scheduled_time, modality, study_description
@@ -139,31 +159,12 @@ uv run pytest tests/integration/test_c_find_returns_worklist_items.py -v
 uv run pytest tests/integration/test_request_cfind_on_worklist.py -v
 ```
 
-## Multi-container architecture
-
-The PACS and MWL servers run in separate containers. See [ADR-003: Separate containers for PACS and MWL](../adr/ADR-003_Separate_containers_for_PACS_and_MWL.md) for the architectural decision and trade-offs.
-
-**Docker Compose services:**
+## Worklist item status transitions
 
-```yaml
-services:
-  pacs:
-    container_name: pacs-server
-    command: ["uv", "run", "python", "-m", "pacs_main"]
-    ports:
-      - "4244:4244"
-
-  mwl:
-    container_name: mwl-server
-    command: ["uv", "run", "python", "-m", "mwl_main"]
-    ports:
-      - "4243:4243"
+```
+SCHEDULED ──(first C-STORE)──▶ IN PROGRESS ──(MPPS N-SET)──▶ COMPLETED
+                                     │
+                                     └────────(MPPS N-SET)──▶ DISCONTINUED
 ```
 
-Each server:
-
-- Runs in its own container
-- Has its own Application Entity (AE)
-- Uses a separate SQLite database
-- Can be scaled and deployed independently
-- Handles different DICOM operations (C-STORE vs C-FIND)
+See [ADR-003: Separate containers for PACS and MWL](../adr/ADR-003_Separate_containers_for_PACS_and_MWL.md) and [ADR-004: Daily backup and reset of the MWL database](../adr/ADR-004_MWL_Daily_Backup_And_Reset.md) for architectural decisions.

scripts/bat/backup_database.bat

@@ -1,4 +1,6 @@
-:: Batch file to run the backup_database.py script
+:: Backs up and resets the MWL worklist database. Intended to be run via Windows Task Scheduler.
+:: Assumes the project venv is at .venv\ relative to the root directory (uv default).
+:: Adjust the Python path below if the deployment creates the venv elsewhere.
 @echo off
-set %rootdir=%1
-python "%rootdir%\scripts\python\backup_database.py"
+set rootdir=%~1
+"%rootdir%\.venv\Scripts\python.exe" -m mwl_reset

scripts/bat/schtasks.bat

@@ -1,5 +1,7 @@
-:: This batch file creates a scheduled task to run the backup_database.bat script daily at midnight.
+:: Registers a daily scheduled task to back up and reset the MWL worklist database.
+:: Default schedule: daily at 02:00. Adjust /st to change the time.
+:: /f overwrites the task if it already exists.
 @echo off
 set thisdir=%~dp0
 set rootdir="%thisdir%..\.."
-schtasks /create /tn BackupDatabaseTask /tr "%thisdir%backup_database.bat %rootdir%" /sc daily /st 00:00
+schtasks /create /tn MWLDailyReset /tr "%thisdir%backup_database.bat %rootdir%" /sc daily /st 02:00 /f

scripts/python/backup_database.py

@@ -0,0 +1,58 @@
+"""Entry point for backing up PACS and MWL databases."""
+
+import logging
+import os
+import sys
+
+from dotenv import load_dotenv
+
+from db_backup import backup_database
+from telemetry import configure_telemetry
+
+load_dotenv()
+configure_telemetry(service_name="backup")
+
+
+def main():
+    """
+    Main entry point for database backup.
+
+    Environment variables:
+    PACS_DB_PATH:  Path to the PACS SQLite database
+    MWL_DB_PATH:   Path to the MWL SQLite database
+    BACKUP_PATH:   Directory for backups (default: ./backups)
+    """
+    logging.basicConfig(
+        level=os.getenv("LOG_LEVEL", "INFO").upper(),
+        format=os.getenv("LOG_FORMAT", "%(asctime)s - %(name)s - %(levelname)s - %(message)s"),
+    )
+
+    pacs_db_path = os.getenv("PACS_DB_PATH")
+    mwl_db_path = os.getenv("MWL_DB_PATH")
+    backup_path = os.getenv("BACKUP_PATH", "./backups")
+
+    success = True
+
+    if pacs_db_path:
+        try:
+            backup_database(pacs_db_path, backup_path)
+        except Exception as e:
+            logging.error(f"PACS backup failed: {e}")
+            success = False
+    else:
+        logging.info("PACS_DB_PATH not set, skipping PACS backup")
+
+    if mwl_db_path:
+        try:
+            backup_database(mwl_db_path, backup_path)
+        except Exception as e:
+            logging.error(f"MWL backup failed: {e}")
+            success = False
+    else:
+        logging.info("MWL_DB_PATH not set, skipping MWL backup")
+
+    sys.exit(0 if success else 1)
+
+
+if __name__ == "__main__":
+    main()