forms-submission-api

API to track form submissions. Currently tracks file submissions only.

See docs/ for documentation.

• Requirements
  • Node.js
• Local development
  • Setup
  • Development
  • Production
  • Npm scripts
  • Database Migrations
• Integration Tests
  • Prerequisites
  • Running Integration Tests
    • Quick Start
    • Manual Step-by-Step Execution
  • What the Integration Tests Cover
  • Test Environment Details
  • Test Reports
  • Troubleshooting
    • Common Issues:
    • Viewing Logs:
• API endpoints
• Docker
  • Development Image
  • Production Image
• Swagger
• Licence
  • About the licence

Requirements

Node.js

Please install the Node.js version in .nvmrc using Node Version Manager nvm via:

cd forms-submission-api
nvm use

Local development

Setup

1. Install Docker

2. Bring up runtime dependencies

docker compose up

3. Create a .env file with the following mandatory environment variables populated at root level:

MONGO_URI=""
MONGO_DATABASE=""
OIDC_JWKS_URI=""
OIDC_VERIFY_AUD=""
OIDC_VERIFY_ISS=""
COGNITO_JWKS_URI=""
COGNITO_CLIENT_ID=""
COGNITO_VERIFY_ISS=""
HTTP_PROXY=
HTTPS_PROXY=
NO_PROXY=

For proxy options, see https://www.npmjs.com/package/proxy-from-env which is used by https://github.com/TooTallNate/proxy-agents/tree/main/packages/proxy-agent. It's currently supports Hapi Wreck only, e.g. in the JWKS lookup.

4. Database setup: See Database Migrations for information on how database migrations work in this project.

Development

To run the application in development mode run:

npm run dev

Production

To mimic the application running in production mode locally run:

npm start

Npm scripts

All available Npm scripts can be seen in package.json To view them in your command line run:

npm run

Integration Tests

The integration tests use Docker Compose to create a complete testing environment with MongoDB, LocalStack (for S3 emulation), OIDC mock server, and Newman (Postman CLI) for API testing.

Prerequisites

• Docker and Docker Compose installed
• Node.js (version as specified in .nvmrc)

Running Integration Tests

Quick Start

To run the full integration test suite:

npm run test:integration

This command will:

1. Start all required services (MongoDB, OIDC mock, app)
2. Wait for services to be ready
3. Run the Postman collection tests
4. Clean up all services

Manual Step-by-Step Execution

For debugging or development purposes, you can run the integration tests step by step:

1. Start infrastructure services:

   npm run test:integration:setup

2. Start the application:

   npm run test:integration:start

3. Wait for services to be ready:

   npm run test:integration:wait

4. Run the tests:

   npm run test:integration:run

5. Clean up:

   npm run test:integration:stop

What the Integration Tests Cover

The integration tests verify all the key functionality of the forms-submission-api:

1. Health Check - Verifies the API is running
2. File Ingestion - Tests the CDP callback endpoint (POST /file)
3. File Status Check - Tests file existence verification (GET /file/{fileId})
4. Presigned Link Generation - Tests secure file access (POST /file/link)
5. File Persistence - Tests moving files from 7-day to 30-day storage (POST /files/persist)
6. Form Submission - Tests complete form submission flow (POST /submit)
7. Error Handling - Tests validation failures and invalid requests
8. Batch Operations - Tests multiple file handling

Test Environment Details

The integration test environment includes:

• MongoDB: Replica set configuration for transaction support
• OIDC Mock Server: Authentication provider for secure endpoints
• Newman: Postman CLI for running API test collections

Test Reports

When running on the main branch, HTML reports are generated:

• Location: ./newman-reports/newman-report.html
• Includes detailed request/response logs and test results
• Available as CI artifacts in GitHub Actions

Troubleshooting

Common Issues:

1. Port conflicts: The tests use ports 3002 (API), 5556, and 27018. Make sure these are available.

2. Docker resources: The integration tests require sufficient Docker resources. Increase Docker memory if needed.

3. Services not ready: If tests fail due to services not being ready, increase the wait time in the test:integration:wait script.

4. Clean up: If services are left running, use:

   npm run test:integration:stop

Viewing Logs:

To view service logs:

docker compose -f docker-compose.integration-test.yml logs [service_name]

Available services: mongo_test, oidc, app_test, newman

Database Migrations

This project uses migrate-mongo to manage database migrations.

Production

In production, migrations run automatically when the Docker container starts via the scripts/run-migrations-and-start.sh shell script. This script:

1. Runs all pending migrations (migrate-mongo up)
2. Starts the application server
3. Logs migration progress to the container output

No manual intervention is required - migrations execute automatically on container startup.

Local Development

For local development, you have two options:

Option 1: Using Docker (Recommended)

Migrations run automatically when using Docker:

docker compose up --build forms-submission-api

This mimics the production environment and runs migrations via the same shell script.

Option 2: Manual Migration Commands

To work with migrations manually, you can install migrate-mongo globally:

npm install -g migrate-mongo

Available migration commands:

# Check migration status
npm run migrate:status

# Run all pending migrations
npm run migrate:up

# Rollback the last migration
npm run migrate:down

# Create a new migration
npx migrate-mongo create <migration-name> -f migrate-mongo-config.js

Important: When running migrations manually, ensure your .env file contains the correct MONGO_URI and MONGO_DATABASE values that match your local MongoDB instance.

API endpoints

Endpoint	Description
GET: /health	Health
POST: /file	Ingests a file with a 7 day expiry. Called by the CDP uploader as a callback (upon file upload)
GET: /file/{fileId}	Checks that a file has been ingested.
POST: /file/link	Creates a link to a file which can be accessed by a user. Valid for 60 minutes.
POST: /files/persist	Extends the expiry to 30 days. Called upon form submission.

Docker

Development image

Build:

docker build --target development --no-cache --tag forms-submission-api:development .

Run:

docker run -e GITHUB_API_TOKEN -p 3008:3008 forms-submission-api:development

Production image

Build:

docker build --no-cache --tag forms-submission-api .

Run:

docker run -e GITHUB_API_TOKEN -p 3002:3002 forms-submission-api

Swagger

When running locally, you can view swagger documentation on http://localhost:3002/documentation this is built automatically from the joi schema and Hapi endpoints using Swagger Hapi.

Licence

THIS INFORMATION IS LICENSED UNDER THE CONDITIONS OF THE OPEN GOVERNMENT LICENCE found at:

http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3

The following attribution statement MUST be cited in your products and applications when using this information.

Contains public sector information licensed under the Open Government license v3

About the licence

The Open Government Licence (OGL) was developed by the Controller of Her Majesty's Stationery Office (HMSO) to enable information providers in the public sector to license the use and re-use of their information under a common open licence.

It is designed to encourage use and re-use of information freely and flexibly, with only a few conditions.