Merge pull request #2808 from NHSDigital/develop

APIM Cut 1.54

Author: EdwardWills-nhs

.github/CODEOWNERS

@@ -9,11 +9,11 @@
 
 # For version files that are updated by dependabot we have a cut down list
 
-/sandbox/Dockerfile @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
-/sandbox/package.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
-/sandbox/package-lock.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
-/package.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
-/package-lock.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
-/poetry.lock @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
-/poetry.toml @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
-/pyproject.toml @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs
+/sandbox/Dockerfile @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2
+/sandbox/package.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2
+/sandbox/package-lock.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2
+/package.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2
+/package-lock.json @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2
+/poetry.lock @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2
+/poetry.toml @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2
+/pyproject.toml @daniel-mcadam-nhs @kevinmason-nhs @EdwardWills-nhs @georgeCraftReferrals @csaw-nhs @petkopetkov2

.github/copilot-instructions.md

@@ -0,0 +1,106 @@
+# e-Referrals Service API — Copilot Context
+
+## What is this repo?
+
+This is the **e-Referral Service (e-RS) FHIR API** maintained by NHS Digital. It provides RESTful endpoints for creating paperless referrals from primary to secondary care. The repo contains the OpenAPI specification, an Apigee API proxy layer, a Node.js sandbox (mock server), and Python-based test suites.
+
+This repo is the **API layer** — it does NOT contain the e-RS core application. The Apigee proxy forwards authenticated requests to the real e-RS backend at `/ers-api`. The sandbox returns fixture data for development and never touches the real backend.
+
+The live API documentation is published to the [NHS Developer Hub](https://digital.nhs.uk/developer/api-catalogue/e-referral-service-fhir).
+
+## Sibling repository
+
+This repo has a sibling: **`e-referrals-service-patient-care-api`**. Both repos follow the same structure and procedures (environment setup, `make publish`, release process). If you know how to work in one, you can work in the other.
+
+## Repository structure
+
+| Directory | Purpose |
+|---|---|
+| `specification/` | OpenAPI 3.0 spec with components split across `components/r4/` (FHIR R4) and `components/stu3/` (FHIR STU3) |
+| `sandbox/` | Node.js (Hapi) mock server. Entry: `sandbox/src/app.js`. Routes: `sandbox/src/routes/{stu3,r4}/`. Fixtures: `sandbox/src/mocks/` |
+| `proxies/live/` | Production Apigee proxy (~60 policies: OAuth, ASID, ODS allowlist, rate limiting, header swapping) |
+| `proxies/sandbox/` | Lightweight Apigee proxy (~11 policies, no auth) forwarding to the mock server |
+| `tests/` | Python pytest suites: `sandbox/`, `integration/`, `smoke/` |
+| `terraform/` | Apigee infrastructure-as-code using `api-platform-service-module` |
+| `scripts/` | Build & dev utilities |
+| `azure/` | Azure DevOps CI/CD pipeline definitions |
+| `build/` | Generated output — bundled single-file OAS JSON |
+
+## FHIR versions
+
+- **STU3** — the original version, most endpoints live here
+- **R4** — newer endpoints (business functions, healthcare services, service requests, attachments)
+
+Routes, schemas, examples, and tests all mirror this split.
+
+## Package management
+
+| Ecosystem | Tool | Config file | Install |
+|---|---|---|---|
+| Python | **Poetry** (package-mode=false) | `pyproject.toml` / `poetry.lock` | `poetry install` |
+| Node (root) | **npm** | `package.json` | `npm install` |
+| Node (sandbox) | **npm** | `sandbox/package.json` | `cd sandbox && npm install` |
+
+- Python ≥ 3.13 required
+- Root `package.json` is only `@redocly/cli` for OAS linting/bundling
+- `make install` installs all three plus git hooks
+
+## Key make targets
+
+```
+make clean-environment    # Delete the pyenv 'apigee' virtual environment
+make setup-environment    # Bootstrap dev environment (pyenv, Python 3.13, dependencies) — run in a NEW terminal after
+make install              # Install all deps (node + poetry + git hooks)
+make lint                 # Lint OAS spec, sandbox JS, XML proxies, Python
+make publish              # Bundle OAS spec → build/e-referrals-service-api.json
+make serve                # Preview spec docs on port 5000
+make sandbox              # Build & run sandbox Docker container (port 9100→9000)
+make sandbox-tests        # Run sandbox pytest suite
+make release              # Full release build
+```
+
+## Environment setup (first time / reset)
+
+1. `make clean-environment` — removes the pyenv `apigee` virtual environment (skip on first setup)
+2. Open a **new terminal** (so shell profile changes take effect)
+3. `make setup-environment` — installs pyenv, Python 3.13, creates the `apigee` venv, installs Poetry
+4. Open a **new terminal** again
+5. `pyenv version` — should show `apigee`. If not, repeat steps 1–4
+6. `make install` — installs Node deps (root + sandbox) and Poetry deps + git hooks
+
+The `.python-version` file auto-activates the `apigee` venv when you `cd` into the repo.
+
+## Release process
+
+1. `make publish` — bundles the OAS spec into a single JSON file:
+   - Redocly CLI reads `specification/e-referrals-service-api.yaml`, resolves all `$ref`s, dereferences, and removes unused components
+   - Piped through `scripts/set_version.py` — calculates the version from git commit messages (using `+major`, `+minor`, `+setstatus` commands in commit messages) and injects it into `info.version`
+   - Piped through `scripts/populate_placeholders.py` — replaces `[[HYPERLINK_*]]` placeholders with actual markdown links for the Developer Hub
+   - Output: `build/e-referrals-service-api.json` — this is the OAS file that goes into Apigee
+2. `make release` — runs `clean` → `publish` → `build_proxy` → packages everything into `dist/`
+3. Deployment is handled by the release management process (not individual developers) — the Azure DevOps release pipeline is triggered by a `v*` tag push, which deploys to Apigee environments
+
+## CI/CD
+
+- **Azure DevOps** pipelines in `azure/` — build, PR validation, release
+- Extends shared templates from `NHSDigital/api-management-utils`
+- Release pipeline packages spec + proxies + tests into `dist/`
+- GitHub has `dependabot.yml` for dependency updates and PR/issue templates
+
+## Branching
+
+- **develop** — default working branch; create PRs from here
+- **master** — release branch; merges to master are part of the release process
+
+## Licensing
+
+Dual licensed MIT + OGL (Open Government License). **No GPL or AGPL dependencies allowed** — this would violate the terms of those libraries' licenses. Check before adding any new package.
+
+## Deeper context
+
+Detailed context for specific areas is split into separate instruction files that load automatically when you're working in the relevant part of the codebase:
+
+- `proxies/**` → proxy architecture, request flow, policies, shared flows
+- `sandbox/**` → sandbox architecture, route handler patterns, mock response provider
+- `tests/**` → test structure, Actor model, activity codes, SandboxTest base class
+- `specification/**` → OAS workflow, examples pipeline, Redocly config

.github/instructions/proxy.instructions.md

@@ -0,0 +1,140 @@
+---
+applyTo: "proxies/**"
+description: "Apigee proxy architecture, request flow, policies, shared flows, and how to modify proxy configuration"
+---
+
+# Apigee Proxy Architecture
+
+The `proxies/` directory contains two independent Apigee API proxy bundles that share the same base path (`{{ SERVICE_BASE_PATH }}` → `/referrals`).
+
+## `proxies/live/` — production proxy
+
+Forwards requests to the real e-RS backend. This is the most policy-heavy part of the repo.
+
+### Layout
+
+| Path | Purpose |
+|---|---|
+| `apiproxy/ers.xml` | Root proxy descriptor |
+| `apiproxy/proxies/default.xml` | ProxyEndpoint — inbound routing (`_ping`, `_status`, catch-all to target) |
+| `apiproxy/targets/ers-target.xml` | TargetEndpoint — outbound to backend via `{{ ERS_TARGET_SERVER }}`, plus all auth/error handling |
+| `apiproxy/policies/` | ~60 XML policy files (see breakdown below) |
+| `apiproxy/resources/jsc/` | Inline JavaScript used by policies (`IsFhirR4Path.js`, `SetCurrentTimestamp.js`, `SetStatusResponse.js`) |
+
+### Request flow (PreFlow on TargetEndpoint)
+
+1. `javascript.IsFhirR4Path` — sets `isFhirR4Path` boolean by matching `/FHIR/R4/` in the path suffix; this flag drives R4-vs-STU3 branching throughout
+2. `OauthV2.VerifyAccessToken` — validates OAuth2 token; accepted scopes: `app:level3`, `user-nhs-id:aal3`, `user-nhs-id:aal2`
+3. `FlowCallout.ExtendedAttributes` + `FlowCallout.EUOAllowlistVerify` — (user-restricted only, excluding `/FHIR/R4/PractitionerRole`) validates the end-user organisation ODS code against an allowlist
+4. ASID validation — `RaiseFault.MissingAsid` if `app.asid` is empty; then `AssignMessage.PopulateAsidFromApp` + `AssignMessage.SetAsidHeader` copy the ASID onto the request
+5. `AssignMessage.AddBaseUrlHeader` — adds the base URL header for the backend
+6. `FlowCallout.ApplyRateLimiting` — spike arrest + quota enforcement (delegates to external shared flow — rate/quota values are NOT in this repo)
+
+### Shared flows (external dependencies)
+
+Several `FlowCallout.*` policies delegate to shared flows that are NOT defined in this repo:
+- `ApplyRateLimiting` — spike arrest and quota enforcement (rate/quota values configured externally)
+- `ExtendedAttributes` — retrieves extended app attributes
+- `EUOAllowlistVerify` — validates ODS code against the end-user organisation allowlist
+- `LogToSplunk` — audit logging
+
+These shared flows are deployed separately to the Apigee environment. This repo only controls *when* they are called and *how* their errors are handled.
+
+### Conditional flows (TargetEndpoint)
+
+- **`user-restricted-flow`** (`accesstoken.auth_type == "user"`) — rejects app-only business functions (`AUTHORISED_APPLICATION`), swaps NHSD headers from external to internal naming, sets AAL/IAL/AMR headers, and enforces IAL ≥ 3
+- **`app-restricted-flow`** (`accesstoken.auth_type == "app"`) — rejects any manually-set `x-ers-*` headers (403), then sets fixed app-restricted values for ODS, business function, user-id, and access-mode
+- **`undefined-flow`** — catch-all that returns 403 (should never trigger)
+
+Both flows finish with `AssignMessage.Swap.CorrelationHeader` which converts `X-Correlation-ID` → `NHSD-Correlation-ID` for the backend.
+
+### Header transformation mapping
+
+The proxy swaps external consumer-facing headers to internal backend headers:
+
+| External header (consumer sends) | Internal header (backend receives) |
+|---|---|
+| `X-Correlation-ID` | `NHSD-Correlation-ID` (appended with `.{messageid}`) |
+| `nhsd-end-user-organisation-ods` | `x-ers-ods-code` |
+| `nhsd-ers-business-function` | `x-ers-business-function` |
+| `nhsd-ers-comm-rule-org` | `x-ers-comm-rule-org` |
+| `nhsd-ers-file-name` | `x-ers-file-name` |
+| `nhsd-ers-referral-id` | `x-ers-referral-id` |
+| `NHSD-eRS-On-Behalf-Of-User-ID` | `x-ers-on-behalf-of-user-id` |
+
+Additional headers set by the proxy (not from consumer input):
+- `x-ers-access-mode` — `user` or `app`
+- `x-ers-user-id` — from OAuth token (user) or app config (app)
+- `x-ers-authentication-assurance-level` — from token
+- `x-ers-id-assurance-level` — from token
+- `x-ers-amr` — authentication method reference from token
+
+### Response flow
+
+- Sets `X-Request-ID` flag, swaps `x_ers_transaction_id` to the response, removes `nhsd-correlation-id` from the response
+
+### FaultRules (TargetEndpoint)
+
+Error responses are FHIR-version-aware — `isFhirR4Path` selects between R4 and pre-R4 `OperationOutcome` response shapes. Handled faults:
+- OAuth token failures (scope errors → AAL insufficient)
+- Spike arrest / quota exceeded (rate limiting)
+- Insufficient IAL (identity assurance level < 3)
+- Missing ASID
+- Invalid business function
+- ODS header missing / not in partner allowlist
+- EUO allowlist internal errors (500)
+
+### Backend connection
+
+- Target server: `{{ ERS_TARGET_SERVER }}` (defaults to `e-referrals-service-api`)
+- Backend path: `/ers-api`
+- TLS enabled; conditional truststore for feature-test (`--ft-`) environments
+- I/O timeout: 180 seconds
+- Jinja2 templating (`{{ }}` placeholders) is resolved at build time by `scripts/build_proxy.sh`
+
+### Policy naming conventions
+
+- `AssignMessage.Set.*` — set a header/variable to a fixed value
+- `AssignMessage.Swap.*` — rename/transform a header between external and internal naming
+- `AssignMessage.Remove.*` — strip a header
+- `AssignMessage.SetOperationOutcome*` — prepare FHIR OperationOutcome error variables
+- `RaiseFault.*` — return an HTTP error status
+- `FlowCallout.*` — delegate to a shared flow
+- `KeyValueMapOperations.*` — read from Apigee KVM stores
+
+## `proxies/sandbox/` — sandbox proxy
+
+Lightweight proxy that forwards to the sandbox container (Hapi.js mock server) via Apigee hosted targets.
+
+### Key differences from live
+
+- Only ~11 policies (vs ~60 in live) — no OAuth, no ASID, no rate limiting, no header swapping
+- Adds CORS preflight handling (`AssignMessage.AddCors` on OPTIONS)
+- Uses `DecodeJWT.FromJWTHeader` to decode (but not validate) the JWT for inspection
+- Target is `{{ HOSTED_TARGET_CONNECTION }}` — the sandbox container deployed alongside the proxy
+- No fault rules or auth enforcement
+
+### Shared structure
+
+Both proxies share `_ping` and `_status` flows — `_ping` returns a canned response directly (no backend call), `_status` uses an API key from KVM + `ServiceCallout.CallHealthcheckEndpoint`.
+
+## Proxy build process
+
+`scripts/build_proxy.sh` copies both proxy bundles into `build/proxies/`, and for sandbox, rsyncs the entire `sandbox/` app into `build/proxies/sandbox/apiproxy/resources/hosted/` so Apigee can run it as a hosted target.
+
+Jinja2 template variables (e.g. `{{ SERVICE_BASE_PATH }}`, `{{ ERS_TARGET_SERVER }}`, `{{ HOSTED_TARGET_CONNECTION }}`) are resolved during the CI/CD pipeline deployment step.
+
+## Infrastructure
+
+- **Apigee** API gateway managed via Terraform (`terraform/main.tf`)
+- Uses the `api-platform-service-module` from NHSDigital
+- Backend: Azure (`azurerm` state backend)
+- Proxy type auto-selects `sandbox` or `live` based on Apigee environment name
+
+## Adding or modifying a proxy policy
+
+1. Create/edit the XML file in `proxies/live/apiproxy/policies/` (or `sandbox/`)
+2. Reference it by `<Name>` in the appropriate flow in `proxies/default.xml` or `ers-target.xml`
+3. If it uses JavaScript, add the `.js` file to `apiproxy/resources/jsc/` and reference via `<ResourceURL>jsc://filename.js</ResourceURL>`
+4. Run `make lint` — validates all proxy XML via `scripts/xml_validator.py`
+5. Remember to handle both R4 and pre-R4 paths if the policy produces FHIR error responses