Skip to content

Commit 5af7eff

Browse files
EdwardWills-nhsEdwardWills-nhs
authored
Merge pull request #2790 from NHSDigital/feature/RAA-7531
Feature/raa 7531
2 parents 0e0c107 + 9db13dc commit 5af7eff

14 files changed

Lines changed: 1445 additions & 2809 deletions

‎.github/copilot-instructions.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# e-Referrals Service API — Copilot Context
2+
3+
## What is this repo?
4+
5+
This is the **e-Referral Service (e-RS) FHIR API** maintained by NHS Digital. It provides RESTful endpoints for creating paperless referrals from primary to secondary care. The repo contains the OpenAPI specification, an Apigee API proxy layer, a Node.js sandbox (mock server), and Python-based test suites.
6+
7+
This repo is the **API layer** — it does NOT contain the e-RS core application. The Apigee proxy forwards authenticated requests to the real e-RS backend at `/ers-api`. The sandbox returns fixture data for development and never touches the real backend.
8+
9+
The live API documentation is published to the [NHS Developer Hub](https://digital.nhs.uk/developer/api-catalogue/e-referral-service-fhir).
10+
11+
## Sibling repository
12+
13+
This repo has a sibling: **`e-referrals-service-patient-care-api`**. Both repos follow the same structure and procedures (environment setup, `make publish`, release process). If you know how to work in one, you can work in the other.
14+
15+
## Repository structure
16+
17+
| Directory | Purpose |
18+
|---|---|
19+
| `specification/` | OpenAPI 3.0 spec with components split across `components/r4/` (FHIR R4) and `components/stu3/` (FHIR STU3) |
20+
| `sandbox/` | Node.js (Hapi) mock server. Entry: `sandbox/src/app.js`. Routes: `sandbox/src/routes/{stu3,r4}/`. Fixtures: `sandbox/src/mocks/` |
21+
| `proxies/live/` | Production Apigee proxy (~60 policies: OAuth, ASID, ODS allowlist, rate limiting, header swapping) |
22+
| `proxies/sandbox/` | Lightweight Apigee proxy (~11 policies, no auth) forwarding to the mock server |
23+
| `tests/` | Python pytest suites: `sandbox/`, `integration/`, `smoke/` |
24+
| `terraform/` | Apigee infrastructure-as-code using `api-platform-service-module` |
25+
| `scripts/` | Build & dev utilities |
26+
| `azure/` | Azure DevOps CI/CD pipeline definitions |
27+
| `build/` | Generated output — bundled single-file OAS JSON |
28+
29+
## FHIR versions
30+
31+
- **STU3** — the original version, most endpoints live here
32+
- **R4** — newer endpoints (business functions, healthcare services, service requests, attachments)
33+
34+
Routes, schemas, examples, and tests all mirror this split.
35+
36+
## Package management
37+
38+
| Ecosystem | Tool | Config file | Install |
39+
|---|---|---|---|
40+
| Python | **Poetry** (package-mode=false) | `pyproject.toml` / `poetry.lock` | `poetry install` |
41+
| Node (root) | **npm** | `package.json` | `npm install` |
42+
| Node (sandbox) | **npm** | `sandbox/package.json` | `cd sandbox && npm install` |
43+
44+
- Python ≥ 3.13 required
45+
- Root `package.json` is only `@redocly/cli` for OAS linting/bundling
46+
- `make install` installs all three plus git hooks
47+
48+
## Key make targets
49+
50+
```
51+
make clean-environment # Delete the pyenv 'apigee' virtual environment
52+
make setup-environment # Bootstrap dev environment (pyenv, Python 3.13, dependencies) — run in a NEW terminal after
53+
make install # Install all deps (node + poetry + git hooks)
54+
make lint # Lint OAS spec, sandbox JS, XML proxies, Python
55+
make publish # Bundle OAS spec → build/e-referrals-service-api.json
56+
make serve # Preview spec docs on port 5000
57+
make sandbox # Build & run sandbox Docker container (port 9100→9000)
58+
make sandbox-tests # Run sandbox pytest suite
59+
make release # Full release build
60+
```
61+
62+
## Environment setup (first time / reset)
63+
64+
1. `make clean-environment` — removes the pyenv `apigee` virtual environment (skip on first setup)
65+
2. Open a **new terminal** (so shell profile changes take effect)
66+
3. `make setup-environment` — installs pyenv, Python 3.13, creates the `apigee` venv, installs Poetry
67+
4. Open a **new terminal** again
68+
5. `pyenv version` — should show `apigee`. If not, repeat steps 1–4
69+
6. `make install` — installs Node deps (root + sandbox) and Poetry deps + git hooks
70+
71+
The `.python-version` file auto-activates the `apigee` venv when you `cd` into the repo.
72+
73+
## Release process
74+
75+
1. `make publish` — bundles the OAS spec into a single JSON file:
76+
- Redocly CLI reads `specification/e-referrals-service-api.yaml`, resolves all `$ref`s, dereferences, and removes unused components
77+
- Piped through `scripts/set_version.py` — calculates the version from git commit messages (using `+major`, `+minor`, `+setstatus` commands in commit messages) and injects it into `info.version`
78+
- Piped through `scripts/populate_placeholders.py` — replaces `[[HYPERLINK_*]]` placeholders with actual markdown links for the Developer Hub
79+
- Output: `build/e-referrals-service-api.json` — this is the OAS file that goes into Apigee
80+
2. `make release` — runs `clean` → `publish` → `build_proxy` → packages everything into `dist/`
81+
3. Deployment is handled by the release management process (not individual developers) — the Azure DevOps release pipeline is triggered by a `v*` tag push, which deploys to Apigee environments
82+
83+
## CI/CD
84+
85+
- **Azure DevOps** pipelines in `azure/` — build, PR validation, release
86+
- Extends shared templates from `NHSDigital/api-management-utils`
87+
- Release pipeline packages spec + proxies + tests into `dist/`
88+
- GitHub has `dependabot.yml` for dependency updates and PR/issue templates
89+
90+
## Branching
91+
92+
- **develop** — default working branch; create PRs from here
93+
- **master** — release branch; merges to master are part of the release process
94+
95+
## Licensing
96+
97+
Dual licensed MIT + OGL (Open Government License). **No GPL or AGPL dependencies allowed** — this would violate the terms of those libraries' licenses. Check before adding any new package.
98+
99+
## Deeper context
100+
101+
Detailed context for specific areas is split into separate instruction files that load automatically when you're working in the relevant part of the codebase:
102+
103+
- `proxies/**` → proxy architecture, request flow, policies, shared flows
104+
- `sandbox/**` → sandbox architecture, route handler patterns, mock response provider
105+
- `tests/**` → test structure, Actor model, activity codes, SandboxTest base class
106+
- `specification/**` → OAS workflow, examples pipeline, Redocly config

‎.github/instructions/proxy.instructions.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
applyTo: "proxies/**"
3+
description: "Apigee proxy architecture, request flow, policies, shared flows, and how to modify proxy configuration"
4+
---
5+
6+
# Apigee Proxy Architecture
7+
8+
The `proxies/` directory contains two independent Apigee API proxy bundles that share the same base path (`{{ SERVICE_BASE_PATH }}` → `/referrals`).
9+
10+
## `proxies/live/` — production proxy
11+
12+
Forwards requests to the real e-RS backend. This is the most policy-heavy part of the repo.
13+
14+
### Layout
15+
16+
| Path | Purpose |
17+
|---|---|
18+
| `apiproxy/ers.xml` | Root proxy descriptor |
19+
| `apiproxy/proxies/default.xml` | ProxyEndpoint — inbound routing (`_ping`, `_status`, catch-all to target) |
20+
| `apiproxy/targets/ers-target.xml` | TargetEndpoint — outbound to backend via `{{ ERS_TARGET_SERVER }}`, plus all auth/error handling |
21+
| `apiproxy/policies/` | ~60 XML policy files (see breakdown below) |
22+
| `apiproxy/resources/jsc/` | Inline JavaScript used by policies (`IsFhirR4Path.js`, `SetCurrentTimestamp.js`, `SetStatusResponse.js`) |
23+
24+
### Request flow (PreFlow on TargetEndpoint)
25+
26+
1. `javascript.IsFhirR4Path` — sets `isFhirR4Path` boolean by matching `/FHIR/R4/` in the path suffix; this flag drives R4-vs-STU3 branching throughout
27+
2. `OauthV2.VerifyAccessToken` — validates OAuth2 token; accepted scopes: `app:level3`, `user-nhs-id:aal3`, `user-nhs-id:aal2`
28+
3. `FlowCallout.ExtendedAttributes` + `FlowCallout.EUOAllowlistVerify` — (user-restricted only, excluding `/FHIR/R4/PractitionerRole`) validates the end-user organisation ODS code against an allowlist
29+
4. ASID validation — `RaiseFault.MissingAsid` if `app.asid` is empty; then `AssignMessage.PopulateAsidFromApp` + `AssignMessage.SetAsidHeader` copy the ASID onto the request
30+
5. `AssignMessage.AddBaseUrlHeader` — adds the base URL header for the backend
31+
6. `FlowCallout.ApplyRateLimiting` — spike arrest + quota enforcement (delegates to external shared flow — rate/quota values are NOT in this repo)
32+
33+
### Shared flows (external dependencies)
34+
35+
Several `FlowCallout.*` policies delegate to shared flows that are NOT defined in this repo:
36+
- `ApplyRateLimiting` — spike arrest and quota enforcement (rate/quota values configured externally)
37+
- `ExtendedAttributes` — retrieves extended app attributes
38+
- `EUOAllowlistVerify` — validates ODS code against the end-user organisation allowlist
39+
- `LogToSplunk` — audit logging
40+
41+
These shared flows are deployed separately to the Apigee environment. This repo only controls *when* they are called and *how* their errors are handled.
42+
43+
### Conditional flows (TargetEndpoint)
44+
45+
- **`user-restricted-flow`** (`accesstoken.auth_type == "user"`) — rejects app-only business functions (`AUTHORISED_APPLICATION`), swaps NHSD headers from external to internal naming, sets AAL/IAL/AMR headers, and enforces IAL ≥ 3
46+
- **`app-restricted-flow`** (`accesstoken.auth_type == "app"`) — rejects any manually-set `x-ers-*` headers (403), then sets fixed app-restricted values for ODS, business function, user-id, and access-mode
47+
- **`undefined-flow`** — catch-all that returns 403 (should never trigger)
48+
49+
Both flows finish with `AssignMessage.Swap.CorrelationHeader` which converts `X-Correlation-ID` → `NHSD-Correlation-ID` for the backend.
50+
51+
### Header transformation mapping
52+
53+
The proxy swaps external consumer-facing headers to internal backend headers:
54+
55+
| External header (consumer sends) | Internal header (backend receives) |
56+
|---|---|
57+
| `X-Correlation-ID` | `NHSD-Correlation-ID` (appended with `.{messageid}`) |
58+
| `nhsd-end-user-organisation-ods` | `x-ers-ods-code` |
59+
| `nhsd-ers-business-function` | `x-ers-business-function` |
60+
| `nhsd-ers-comm-rule-org` | `x-ers-comm-rule-org` |
61+
| `nhsd-ers-file-name` | `x-ers-file-name` |
62+
| `nhsd-ers-referral-id` | `x-ers-referral-id` |
63+
| `NHSD-eRS-On-Behalf-Of-User-ID` | `x-ers-on-behalf-of-user-id` |
64+
65+
Additional headers set by the proxy (not from consumer input):
66+
- `x-ers-access-mode` — `user` or `app`
67+
- `x-ers-user-id` — from OAuth token (user) or app config (app)
68+
- `x-ers-authentication-assurance-level` — from token
69+
- `x-ers-id-assurance-level` — from token
70+
- `x-ers-amr` — authentication method reference from token
71+
72+
### Response flow
73+
74+
- Sets `X-Request-ID` flag, swaps `x_ers_transaction_id` to the response, removes `nhsd-correlation-id` from the response
75+
76+
### FaultRules (TargetEndpoint)
77+
78+
Error responses are FHIR-version-aware — `isFhirR4Path` selects between R4 and pre-R4 `OperationOutcome` response shapes. Handled faults:
79+
- OAuth token failures (scope errors → AAL insufficient)
80+
- Spike arrest / quota exceeded (rate limiting)
81+
- Insufficient IAL (identity assurance level < 3)
82+
- Missing ASID
83+
- Invalid business function
84+
- ODS header missing / not in partner allowlist
85+
- EUO allowlist internal errors (500)
86+
87+
### Backend connection
88+
89+
- Target server: `{{ ERS_TARGET_SERVER }}` (defaults to `e-referrals-service-api`)
90+
- Backend path: `/ers-api`
91+
- TLS enabled; conditional truststore for feature-test (`--ft-`) environments
92+
- I/O timeout: 180 seconds
93+
- Jinja2 templating (`{{ }}` placeholders) is resolved at build time by `scripts/build_proxy.sh`
94+
95+
### Policy naming conventions
96+
97+
- `AssignMessage.Set.*` — set a header/variable to a fixed value
98+
- `AssignMessage.Swap.*` — rename/transform a header between external and internal naming
99+
- `AssignMessage.Remove.*` — strip a header
100+
- `AssignMessage.SetOperationOutcome*` — prepare FHIR OperationOutcome error variables
101+
- `RaiseFault.*` — return an HTTP error status
102+
- `FlowCallout.*` — delegate to a shared flow
103+
- `KeyValueMapOperations.*` — read from Apigee KVM stores
104+
105+
## `proxies/sandbox/` — sandbox proxy
106+
107+
Lightweight proxy that forwards to the sandbox container (Hapi.js mock server) via Apigee hosted targets.
108+
109+
### Key differences from live
110+
111+
- Only ~11 policies (vs ~60 in live) — no OAuth, no ASID, no rate limiting, no header swapping
112+
- Adds CORS preflight handling (`AssignMessage.AddCors` on OPTIONS)
113+
- Uses `DecodeJWT.FromJWTHeader` to decode (but not validate) the JWT for inspection
114+
- Target is `{{ HOSTED_TARGET_CONNECTION }}` — the sandbox container deployed alongside the proxy
115+
- No fault rules or auth enforcement
116+
117+
### Shared structure
118+
119+
Both proxies share `_ping` and `_status` flows — `_ping` returns a canned response directly (no backend call), `_status` uses an API key from KVM + `ServiceCallout.CallHealthcheckEndpoint`.
120+
121+
## Proxy build process
122+
123+
`scripts/build_proxy.sh` copies both proxy bundles into `build/proxies/`, and for sandbox, rsyncs the entire `sandbox/` app into `build/proxies/sandbox/apiproxy/resources/hosted/` so Apigee can run it as a hosted target.
124+
125+
Jinja2 template variables (e.g. `{{ SERVICE_BASE_PATH }}`, `{{ ERS_TARGET_SERVER }}`, `{{ HOSTED_TARGET_CONNECTION }}`) are resolved during the CI/CD pipeline deployment step.
126+
127+
## Infrastructure
128+
129+
- **Apigee** API gateway managed via Terraform (`terraform/main.tf`)
130+
- Uses the `api-platform-service-module` from NHSDigital
131+
- Backend: Azure (`azurerm` state backend)
132+
- Proxy type auto-selects `sandbox` or `live` based on Apigee environment name
133+
134+
## Adding or modifying a proxy policy
135+
136+
1. Create/edit the XML file in `proxies/live/apiproxy/policies/` (or `sandbox/`)
137+
2. Reference it by `<Name>` in the appropriate flow in `proxies/default.xml` or `ers-target.xml`
138+
3. If it uses JavaScript, add the `.js` file to `apiproxy/resources/jsc/` and reference via `<ResourceURL>jsc://filename.js</ResourceURL>`
139+
4. Run `make lint` — validates all proxy XML via `scripts/xml_validator.py`
140+
5. Remember to handle both R4 and pre-R4 paths if the policy produces FHIR error responses

‎.github/instructions/sandbox.instructions.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
---
2+
applyTo: "sandbox/**"
3+
description: "Sandbox mock server architecture, route handler patterns, mock response provider, and how to add new endpoints"
4+
---
5+
6+
# Sandbox Architecture
7+
8+
The sandbox is a **Hapi.js (v21)** mock server that simulates the e-RS FHIR API. It returns static fixture data and never connects to the real backend.
9+
10+
## Structure
11+
12+
| Path | Purpose |
13+
|---|---|
14+
| `sandbox/src/app.js` | Server entry point — port 9000, CORS config, common response headers |
15+
| `sandbox/src/routes/index.js` | Aggregates all route handlers from `stu3/` and `r4/` subdirectories |
16+
| `sandbox/src/routes/stu3/` | STU3 FHIR endpoint handlers (one file per endpoint) |
17+
| `sandbox/src/routes/r4/` | R4 FHIR endpoint handlers |
18+
| `sandbox/src/routes/common/validationUtils.js` | Shared validation utilities (business function checks, UUID validation) |
19+
| `sandbox/src/routes/stu3/services/mockResponseProvider.js` | Maps request bodies/parameters to fixture file paths for STU3 |
20+
| `sandbox/src/mocks/` | Static JSON/text fixtures organised by FHIR version and endpoint |
21+
| `sandbox/src/routes/objectStore.js` | Handles `/ObjectStore` routes for file upload/download |
22+
| `sandbox/Dockerfile` | Container build — runs on port 9000 |
23+
24+
## Common response headers
25+
26+
`app.js` adds these to every response via an `onPreResponse` extension:
27+
- `X-Correlation-ID` — echoed from the request's `x-correlation-id` header
28+
- `X-Request-ID` — fixed value `58621d65-d5ad-4c3a-959f-0438e355990e-1` (except ObjectStore routes)
29+
30+
## Route handler pattern
31+
32+
Every route handler follows the same structure:
33+
34+
```javascript
35+
const mockResponseProvider = require('./services/mockResponseProvider')
36+
const validationUtils = require('../common/validationUtils')
37+
38+
function handleEndpoint(request, h) {
39+
// 1. Define allowed business functions for this endpoint
40+
const allowedBusinessFunctions = ["REFERRING_CLINICIAN", "SERVICE_PROVIDER_CLINICIAN"]
41+
42+
// 2. Validate business function header — returns 403 if not in allowed list
43+
const validationResult = validationUtils.validateBusinessFunction(request, h, allowedBusinessFunctions)
44+
if (validationResult) return validationResult
45+
46+
// 3. Use mockResponseProvider to map request → fixture file path
47+
const { responsePath } = mockResponseProvider.getExampleResponseForX(request)
48+
if (responsePath != null) {
49+
return h.file(responsePath, { etagMethod: false }).code(200).type("application/fhir+json")
50+
}
51+
52+
// 4. Fall back to error fixture if no match
53+
return h.file('stu3/STU3-SandboxErrorOutcome.json').code(422)
54+
}
55+
56+
// 5. Export as Hapi route array
57+
module.exports = [
58+
{ method: 'GET', path: '/FHIR/STU3/Resource/{id}', handler: (request, h) => handleEndpoint(request, h) }
59+
]
60+
```
61+
62+
## Mock response provider (`mockResponseProvider.js`)
63+
64+
This module maps request inputs to response fixtures. Two main patterns:
65+
66+
### POST endpoints (request body matching)
67+
```javascript
68+
mapExampleResponse(request, {
69+
'src/mocks/stu3/endpoint/requests/Request.json': 'stu3/endpoint/responses/Response.json'
70+
})
71+
```
72+
Uses `lodash.isEqual` to deep-compare the incoming request body against stored example request bodies. If a match is found, returns the corresponding fixture path.
73+
74+
### GET endpoints (parameter matching)
75+
```javascript
76+
mapExampleGetResponse(parameterValue, {
77+
'paramValue': 'stu3/endpoint/responses/Response.json'
78+
})
79+
```
80+
Simple key-value lookup by a request parameter or header value.
81+
82+
### Business function branching
83+
Many handlers select different response maps based on the `nhsd-ers-business-function` header, returning different fixtures for different user roles.
84+
85+
## Business function validation
86+
87+
`validationUtils.validateBusinessFunction()` checks:
88+
1. The `nhsd-ers-business-function` header is in the endpoint's allowed list → 403 if not
89+
2. If `SERVICE_PROVIDER_CLINICIAN_ADMIN`, an `nhsd-ers-on-behalf-of-user-id` header must be present → 403 if missing
90+
3. For all other roles, `nhsd-ers-on-behalf-of-user-id` must NOT be present → 403 if provided
91+
92+
## Mock fixture organisation
93+
94+
```
95+
sandbox/src/mocks/
96+
├── stu3/
97+
│ ├── createReferral/
98+
│ │ ├── requests/ # Example request bodies for matching
99+
│ │ └── responses/ # Response fixtures
100+
│ ├── retrieveWorklist/
101+
│ └── ...
102+
├── r4/
103+
│ ├── retrieveBusinessFunctions/
104+
│ └── ...
105+
└── NotFoundOutcome.txt # Generic 404 response
106+
```
107+
108+
Fixtures serve as the **source of truth** for both the sandbox and the OAS spec examples — `make copy-examples` copies them into `specification/components/*/examples/`.
109+
110+
## Running the sandbox
111+
112+
- **Docker**: `make sandbox` (builds and runs on port 9100→9000)
113+
- **Direct**: `cd sandbox && npm start` (runs on port 9000)
114+
- **Debug**: `cd sandbox && npm run debug` (with Node.js inspector on port 9229)
115+
116+
## Adding a new sandbox endpoint
117+
118+
1. Create a response fixture in `sandbox/src/mocks/{stu3,r4}/endpointName/responses/`
119+
2. If a POST endpoint, create example request bodies in `.../requests/`
120+
3. Add a response mapping function in `mockResponseProvider.js` (or equivalent for R4)
121+
4. Create a route handler file in `sandbox/src/routes/{stu3,r4}/endpointName.js` following the pattern above
122+
5. Import and add the route to the array in `sandbox/src/routes/index.js`
123+
6. Run `make copy-examples` then `make lint` to validate

0 commit comments

Comments
 (0)