Add testing statement and publish test report (#139)

Add docs/testing-statement.md for the internal assurance team, describing
the testing approach (per ADR-002), what automated tests cover, the CI
quality gates, where evidence lives, and what is not covered by automated
testing and how each gap is otherwise assured.

Publish a human-readable test report to GitHub Pages on every main build
so the statement can cite a stable, per-build link:

- add pytest-html; the test stage emits a self-contained reports/index.html
  and uploads it as an artifact
- add a publish-test-report job to the main-branch workflow that deploys
  the artifact to Pages

Author: carlosmartinez

.github/workflows/cicd-2-main-branch.yaml

@@ -33,6 +33,39 @@ jobs:
     uses: ./.github/workflows/stage-2-test.yaml
     secrets: inherit
 
+  # Publish the test report (produced by the test stage) to GitHub Pages so
+  # assurance reviewers have a stable, per-build link to the latest results.
+  # Runs even if tests fail, so a failing report is still published.
+  publish-test-report:
+    name: Publish test report
+    needs: test-stage
+    if: always() && needs.test-stage.result != 'skipped'
+    runs-on: ubuntu-latest
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Download test report
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          name: test-report
+          path: reports
+
+      - name: Configure Pages
+        uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
+
+      - name: Upload Pages artifact
+        uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
+        with:
+          path: reports
+
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0
+
   # Resolve the latest release tag once so downstream jobs can consume it.
   resolve:
     name: Resolve deployment targets

.github/workflows/stage-2-test.yaml

@@ -36,7 +36,7 @@ jobs:
         run: uv sync --all-extras
 
       - name: Run tests
-        run: uv run pytest
+        run: uv run pytest --html=reports/index.html --self-contained-html
 
       - name: Upload coverage reports
         uses: codecov/codecov-action@57e3a136b779b570ffcdbf80b3bdc90e7fab3de2 # v6.0.0
@@ -47,6 +47,14 @@ jobs:
           name: codecov-umbrella
           fail_ci_if_error: false
 
+      - name: Upload test report
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always() # capture the report even when tests fail
+        with:
+          name: test-report
+          path: reports/
+          if-no-files-found: error
+
   test-lint:
     name: Lint and type checking
     runs-on: ubuntu-latest

.gitignore

@@ -41,6 +41,7 @@ build/
 .coverage.*
 htmlcov/
 coverage.xml
+reports/
 .tox/
 .hypothesis/
 

docs/testing-statement.md

@@ -0,0 +1,77 @@
+# Rubie Gateway — Testing Statement
+
+**Purpose:** Describe how the Rubie Gateway is tested, what is and is not covered by automated testing, and where test evidence can be found.
+**Status:** Living document — figures reflect the state of `main` and should be refreshed when materially changed.
+
+---
+
+## 1. What the gateway does (context for scope)
+
+The Rubie Gateway is a Python application that runs on a VM inside an NHS Trust network and bridges the on-site imaging modality and the cloud-based Rubie service. Its responsibilities are:
+
+- **MWL (Modality Worklist) SCP** — receives worklist create/update commands from Rubie (via Azure Relay), and serves them to the modality over DICOM C-FIND.
+- **PACS SCP** — receives acquired images from the modality over DICOM C-STORE, and answers DICOM C-ECHO connectivity checks.
+- **Upload service** — forwards stored images to the Rubie cloud API.
+- **Relay listener** — maintains the Azure Relay connection over which Rubie sends commands.
+
+Testing is scoped to the behaviour of this application. The wider distributed system (Rubie cloud, NBSS, the Trust PACS, the physical modality) is out of scope for automated testing here.
+
+## 2. Testing approach
+
+The strategy is recorded in [ADR-002: Testing strategy](./adr/ADR-002_Testing_strategy.md). In summary, the suite is layered:
+
+| Layer | What it exercises | Dependencies |
+| --- | --- | --- |
+| **Unit** | Individual functions and classes in isolation | Mocked |
+| **Integration** | Components working together, including the gateway server running in a background thread | Real SQLite database, real filesystem |
+| **End-to-end (in-process)** | A full relay-command → MWL → C-STORE → upload flow | Real SQLite + filesystem; external network boundaries emulated |
+
+Because it is impractical to stand up the entire Rubie + PACS + modality ecosystem for automated testing, the suite uses **emulation** at the system boundaries: crafted DICOM datasets and a modality emulator stand in for real imaging hardware, and the Azure Relay transport is mocked so command-processing logic can be tested deterministically.
+
+## 3. What is covered
+
+Automated tests cover the core DICOM and messaging behaviour, including:
+
+- **MWL**: C-FIND worklist queries (including patient-name search), worklist item creation, and status transitions.
+- **PACS**: C-STORE image receipt and metadata persistence, handling of already-stored SOP instances, and C-ECHO verification (both PACS and MWL application entities).
+- **Relay listener**: processing of inbound action commands and dispatch to the correct handler.
+- **Upload**: forwarding of stored instances to the Rubie API.
+- **Modality emulator**: worklist querying and image generation used to drive end-to-end tests.
+
+**Current figures (refresh on material change):**
+
+- **Over 200 automated tests** (194 unit, 19 integration), all passing on `main`.
+- **Approximately 90% line coverage** of application code (`src/`), measured by `pytest-cov` and reported to Codecov on every run.
+
+## 4. Automated quality gates (CI)
+
+Every pull request and every merge to `main` runs the following in GitHub Actions; all must pass before code is deployable:
+
+| Stage | Check | Tool |
+| --- | --- | --- |
+| Commit | Secret scanning across branch history | gitleaks |
+| Test | Full unit + integration suite | pytest |
+| Test | Line-coverage measurement and reporting | pytest-cov / Codecov |
+| Test | Linting and formatting | ruff |
+| Test | Static type checking | pyright |
+| Scheduled | Static application security testing (SAST) | CodeQL (weekly) |
+
+## 5. Test evidence
+
+- **Per-run evidence:** the GitHub Actions run for each PR and each `main` build records the full test output, pass/fail status, and coverage upload. These are retained against the commit and pull request.
+- **Coverage trend:** reported to Codecov, giving per-file coverage and change-over-time.
+- **Local reproduction:** the full suite is `make test`; subsets are `make test-unit` and `make test-integration`. Coverage HTML is written to `htmlcov/`.
+- **Published report:** a human-readable test report is published to GitHub Pages on every `main` build and is available at <https://nhsdigital.github.io/manage-breast-screening-gateway/>. This gives assurance reviewers a stable link to the latest results without navigating CI logs, and is refreshed automatically per build.
+
+## 6. What is NOT covered by automated testing
+
+| Not automatically tested | Why | Compensating assurance |
+| --- | --- | --- |
+| **Real modality hardware** | No physical modality in CI | Emulated with crafted DICOM and the modality emulator; validated against real Hologic kit during site dry runs. |
+| **Live Azure Relay / Managed Identity auth** | No Azure resources in CI; the transport is mocked | Credential and connection logic is exercised by deployments to dev/pre-prod and by post-deployment smoke tests. |
+| **PowerShell deployment scripts** (`deploy.ps1`, `rollback.ps1`) | Windows blue/green cutover, NSSM service install and rollback are not unit-testable in this suite | Exercised by real deployments to non-production environments with smoke tests; Bash helper scripts are covered by ShellCheck. |
+| **The end-to-end distributed system** (Rubie cloud, NBSS, Trust PACS) | Impractical to assemble for automated testing (ADR-002) | Emulation at boundaries; manual/clinical validation via site dry runs. |
+
+## 7. Maintaining this statement
+
+This statement should be reviewed when the testing strategy changes materially — for example a new test layer, a significant coverage change, a new CI gate, or a change to what is emulated versus tested against real dependencies. The figures in §3 are point-in-time and can be regenerated with `make test`.

pyproject.toml

@@ -23,6 +23,7 @@ dependencies = [
 dev = [
   "pytest>=9.0.3,<10",
   "pytest-cov>=7.0.0,<8",
+  "pytest-html>=4.1,<5",
   "ruff>=0.14.1,<0.16",
   "pyright>=1.1.390",
   "ipdb>=0.13.13,<0.14",