Chore: add database migration tooling and configuration (#330)

* chore: add database migration tooling and configuration

* lint: lint fixes

* Resolve merge conflict in README.md table of contents

* Update integration test configurations: reduce retry counts and adjust wait logic for app and LocalStack readiness

* refavtor: integration test wait logic: replace inline wait command with dedicated script

* refactor: add attempt logging and container log output for debugging during startup

* chore: remove completed migration script

The file-upload-status to files migration has already been applied.
Remove the script and keep the empty migrations directory for future use.

* fix: remove unused @ts-expect-error directives in repository tests

* chore: remove temporary collection fallback documentation

* fix: update health check URL in wait-for-app script from port 3001 to 3002

* fix(integration): align docker compose and Postman with port 3002

Match integration stack to default PORT and wait-for-app health check.

Author: mokhld

Dockerfile

@@ -33,11 +33,15 @@ USER node
 
 COPY --from=development /home/node/package*.json ./
 COPY --from=development /home/node/.server ./.server/
+COPY --from=development /home/node/migrate-mongo-config.cjs ./
+COPY --from=development /home/node/migrations ./migrations/
+COPY --from=development /home/node/scripts ./scripts/
 
-RUN npm ci --omit=dev
+RUN npm ci --omit=dev && \
+    chmod +x scripts/run-migrations-and-start.sh
 
 ARG PORT
 ENV PORT ${PORT}
 EXPOSE ${PORT}
 
-CMD [ "npm", "start", "--ignore-scripts" ]
+CMD [ "./scripts/run-migrations-and-start.sh" ]

README.md

@@ -4,32 +4,32 @@ API to track form submissions. Currently tracks file submissions only.
 
 See [docs/](docs/) for documentation.
 
-- [forms-submission-api](#forms-submission-api)
-  - [Requirements](#requirements)
-    - [Node.js](#nodejs)
-  - [Local development](#local-development)
-    - [Setup](#setup)
-    - [Development](#development)
-    - [Production](#production)
-    - [Npm scripts](#npm-scripts)
-  - [Integration Tests](#integration-tests)
-    - [Prerequisites](#prerequisites)
-    - [Running Integration Tests](#running-integration-tests)
-      - [Quick Start](#quick-start)
-      - [Manual Step-by-Step Execution](#manual-step-by-step-execution)
-    - [What the Integration Tests Cover](#what-the-integration-tests-cover)
-    - [Test Environment Details](#test-environment-details)
-    - [Test Reports](#test-reports)
-    - [Troubleshooting](#troubleshooting)
-      - [Common Issues:](#common-issues)
-      - [Viewing Logs:](#viewing-logs)
-  - [API endpoints](#api-endpoints)
-  - [Docker](#docker)
-    - [Development image](#development-image)
-    - [Production image](#production-image)
-  - [Swagger](#swagger)
-  - [Licence](#licence)
-    - [About the licence](#about-the-licence)
+- [Requirements](#requirements)
+  - [Node.js](#nodejs)
+- [Local development](#local-development)
+  - [Setup](#setup)
+  - [Development](#development)
+  - [Production](#production)
+  - [Npm scripts](#npm-scripts)
+  - [Database Migrations](#database-migrations)
+- [Integration Tests](#integration-tests)
+  - [Prerequisites](#prerequisites)
+  - [Running Integration Tests](#running-integration-tests)
+    - [Quick Start](#quick-start)
+    - [Manual Step-by-Step Execution](#manual-step-by-step-execution)
+  - [What the Integration Tests Cover](#what-the-integration-tests-cover)
+  - [Test Environment Details](#test-environment-details)
+  - [Test Reports](#test-reports)
+  - [Troubleshooting](#troubleshooting)
+    - [Common Issues:](#common-issues)
+    - [Viewing Logs:](#viewing-logs)
+- [API endpoints](#api-endpoints)
+- [Docker](#docker)
+  - [Development Image](#development-image)
+  - [Production Image](#production-image)
+- [Swagger](#swagger)
+- [Licence](#licence)
+  - [About the licence](#about-the-licence)
 
 ## Requirements
 
@@ -72,6 +72,8 @@ NO_PROXY=
 
 For proxy options, see https://www.npmjs.com/package/proxy-from-env which is used by https://github.com/TooTallNate/proxy-agents/tree/main/packages/proxy-agent. It's currently supports Hapi Wreck only, e.g. in the JWKS lookup.
 
+4. **Database setup**: See [Database Migrations](#database-migrations) for information on how database migrations work in this project.
+
 ### Development
 
 To run the application in `development` mode run:
@@ -189,7 +191,7 @@ When running on the main branch, HTML reports are generated:
 
 #### Common Issues:
 
-1. **Port conflicts**: The tests use ports 3001, 5556, and 27018. Make sure these are available.
+1. **Port conflicts**: The tests use ports 3002 (API), 5556, and 27018. Make sure these are available.
 
 2. **Docker resources**: The integration tests require sufficient Docker resources. Increase Docker memory if needed.
 
@@ -210,6 +212,60 @@ docker compose -f docker-compose.integration-test.yml logs [service_name]
 
 Available services: `mongo_test`, `oidc`, `app_test`, `newman`
 
+### Database Migrations
+
+This project uses [migrate-mongo](https://www.npmjs.com/package/migrate-mongo) to manage database migrations.
+
+#### Production
+
+In production, migrations run automatically when the Docker container starts via the `scripts/run-migrations-and-start.sh` shell script. This script:
+
+1. Runs all pending migrations (`migrate-mongo up`)
+2. Starts the application server
+3. Logs migration progress to the container output
+
+**No manual intervention is required** - migrations execute automatically on container startup.
+
+#### Local Development
+
+For local development, you have two options:
+
+##### Option 1: Using Docker (Recommended)
+
+Migrations run automatically when using Docker:
+
+```bash
+docker compose up --build forms-submission-api
+```
+
+This mimics the production environment and runs migrations via the same shell script.
+
+##### Option 2: Manual Migration Commands
+
+To work with migrations manually, you can install migrate-mongo globally:
+
+```bash
+npm install -g migrate-mongo
+```
+
+Available migration commands:
+
+```bash
+# Check migration status
+npm run migrate:status
+
+# Run all pending migrations
+npm run migrate:up
+
+# Rollback the last migration
+npm run migrate:down
+
+# Create a new migration
+npx migrate-mongo create <migration-name> -f migrate-mongo-config.cjs
+```
+
+**Important**: When running migrations manually, ensure your `.env` file contains the correct `MONGO_URI` and `MONGO_DATABASE` values that match your local MongoDB instance.
+
 ## API endpoints
 
 | Endpoint               | Description                                                                                     |
@@ -247,7 +303,7 @@ docker build --no-cache --tag forms-submission-api .
 Run:
 
 ```bash
-docker run -e GITHUB_API_TOKEN -p 3001:3001 forms-submission-api
+docker run -e GITHUB_API_TOKEN -p 3002:3002 forms-submission-api
 ```
 
 ## Swagger

docker-compose.integration-test.yml

@@ -12,7 +12,7 @@ services:
       interval: 5s
       timeout: 30s
       start_period: 10s
-      retries: 30
+      retries: 12
     environment:
       MONGO_INITDB_DATABASE: forms-submission-api-test
     networks:
@@ -80,19 +80,19 @@ services:
         ]
       interval: 10s
       timeout: 10s
-      retries: 18
-      start_period: 90s
+      retries: 12
+      start_period: 60s
 
   app_test:
     build:
       context: .
       target: production
     container_name: forms-submission-api-app-test
     ports:
-      - '3001:3001'
+      - '3002:3002'
     environment:
       MONGO_URI: mongodb://mongo_test:27017/forms-submission-api-test?replicaSet=rs0&directConnection=true
-      PORT: 3001
+      PORT: 3002
       NODE_ENV: production
       OIDC_JWKS_URI: 'http://oidc:80/.well-known/openid-configuration/jwks'
       OIDC_VERIFY_AUD: 'newman-test-client'
@@ -131,7 +131,7 @@ services:
       NEWMAN_CLIENT_SECRET: 'newman-mock-secret'
       NEWMAN_SCOPE: 'openid profile email newman-test-client'
       NEWMAN_TEST_EMAIL: 'test.email@defra.gov.uk'
-      API_URL: 'http://app_test:3001'
+      API_URL: 'http://app_test:3002'
     volumes:
       - ./test/integration/postman:/etc/newman
       - ./newman-reports:/etc/newman/reports

docs/temporary-collection-fallback.md

@@ -0,0 +1,42 @@
+/* eslint-disable */
+// In this file you can configure migrate-mongo
+require('dotenv/config')
+
+const config = {
+  mongodb: {
+    // Use environment variables to match the app's configuration
+    url: process.env.MONGO_URI || 'mongodb://127.0.0.1:27017/',
+
+    // Use the same database name as the app
+    databaseName: process.env.MONGO_DATABASE || 'forms-submission-api',
+
+    options: {
+      //   connectTimeoutMS: 3600000, // increase connection timeout to 1 hour
+      //   socketTimeoutMS: 3600000, // increase socket timeout to 1 hour
+    }
+  },
+
+  // The migrations dir, can be an relative or absolute path. Only edit this when really necessary.
+  migrationsDir: 'migrations',
+
+  // The mongodb collection where the applied changes are stored. Only edit this when really necessary.
+  changelogCollectionName: 'changelog',
+
+  // The mongodb collection where the lock will be created.
+  lockCollectionName: 'changelog_lock',
+
+  // The value in seconds for the TTL index that will be used for the lock. Value of 0 will disable the feature.
+  lockTtl: 0,
+
+  // The file extension to create migrations and search for in migration dir
+  migrationFileExtension: '.cjs',
+
+  // Enable the algorithm to create a checksum of the file contents and use that in the comparison to determine
+  // if the file should be run.  Requires that scripts are coded to be run multiple times.
+  useFileHash: false,
+
+  // Don't change this, unless you know what you're doing
+  moduleSystem: 'commonjs'
+}
+
+module.exports = config