Backport: runner-v2 (events and templates) (#745)

* Add version to release workflow

* Chore: Backport forms runner V2 (#728)

* PLATFORM: Add support for page events, custom views and injectable controllers (#13)

* Add support for page onLoad event and providing a custom view

* Inject controllers to plugin registration

* Temporarily supress custom controller TS error

* Pass controller through to FormModel (#16)

* PLATFORM: Feature/page events payload (#22)

* Add support for TerminalPageController

* Update start page logic when using page conditions engine

* Resolve page condition in controller constructor

* Add page condition routing logic

* Bump @defra/forms-model dep to 3.0.395

* Make engine optional on FormModel class

* Add TerminalPageController test

* Add QuetionPageController V2 tests

* Refactor getFormContext to reduce cognitive complexity

* Change logic for when the form Continue button is displayed for V2

* Support for injectable forms services

* Use injected forms services in page controllers

* Add defaultServices file

* Simplify type where value is not used

* fix: prevent 500s from malformed requests in crumb generation (#676)

* fix: prevent 500s from malformed requests in crumb generation

* test: added test for crumb state checks

* refactor: split the check into helper function with tests

* refactor: made helper function clearer

* Structured emails for machine ingestion (#679)

* extract notification service to a abstract service

* machine output

* Move v1 formatter into formatters folder

* Move output formatters into separate folder

* add form def to JSON output

* switch outputFormatter based on condition

* Refactor tests to support new formatters and injectable service

* fix import path

* Add tests for new macine output

* simplify functions in machine formatter

* Allow unused vars if they begin with underscore

* Resolve jest-extended tsc errors

* #504353 Update expiry in the notify email since it's been increased (#685)

* Update expiry in the notify email since it's been increased

* Update test

* Support machine emails via form definition flag (#684)

* use output format from form def and encode Notify JSON as base64

* Categorise outputs into main and repeaters

* Add files to machine v1 output

* Use 3.0.398 of model

* use forms-designer path from config

* Fix typo

* Replace POST payload with structured data

* Update page event payload to use the machine output format

* Remove test page event from example grant

* Temporarily suppress TS error

---------

Co-authored-by: Alex Luckett <alexgluckett@gmail.com>
Co-authored-by: Alex Luckett <alexluckett@users.noreply.github.com>
Co-authored-by: Mohammed Khalid <emailmokhld@gmail.com>
Co-authored-by: Jignesh Nayi <47786869+nayij@users.noreply.github.com>

* PLATFORM: feature/templates (#33)

* Remove toLowerCase from component schema message titles as they could contain interpolated variables

* Add convenience maps onto FormContext

* Add liquid templates

* Add convenience maps onto FormContext type

* Add liquid helpers to Nunjucks environment

* Use liquid helpers in Nunjucks templates

* Add types

* Fix tests

* Add templating documentation

* Add Liquid helper tests

* Add Liquid helper test definition file

* Add TS comment exceptions

* Remove page and field liquid filters

* Add test showing liquid evaluation can use values from `context.data`

* Update template README

* Evaluate templates in check answers summary rows

* Update nunjucks liquid filter

* Add template journey test

* Include summary rows title checks in template journey tests

* Update Templates README

* Add missing types

* Enable tracing (#36)

* Remove ts incremental build cache file and add to gitignore (#37)

* PLATFORM: Add href, page, field and answer filters to nunjucks (#46)

* added href and answer as nunjucks filters

Signed-off-by: David Barker <david.barker@equalexperts.com>

* added filters

Signed-off-by: David Barker <david.barker@equalexperts.com>

* modified index to include filters

Signed-off-by: David Barker <david.barker@equalexperts.com>

* Update nunjucks filters

* Update documentation

---------

Signed-off-by: David Barker <david.barker@equalexperts.com>
Co-authored-by: davidjamesstone <davidjamesstone@gmail.com>

* test: tests for answer filter

* test: tests enviroment.js checkErrorTemplates

* test: field.js test suite

* test: href filter

* refactor(test)

* test: PageController coverage

* test: test cases for when pageDef.view is present

* test: test suite for page filter

* test: test suite for logger-options

* PLATFORM: Change tracing so it doesn't overwrite existing headers (#54)

* Change tracing so it doesn't overwrite existing headers

* Add tests to ensure additional headers are preserved

* Merge headers inside applyTraceHeaders utility function

* Replace merge with assign

* Remove unused JSDoc declaration

---------

Co-authored-by: Andrew Folga <andrew.folga@equalexperts.com>

* test: evaluate and href filters

* test: add coverage for falsy tracing header config

* refactor: improve testability of applyTraceHeaders

* test: add coverage for HTML component string content check

* test: add coverage for Nunjucks evaluate function and component label handling

* refactor(sonar)

* fix: Add type assertion for custom page controllers

---------

Signed-off-by: David Barker <david.barker@equalexperts.com>
Co-authored-by: David Stone <davidjamesstone@gmail.com>
Co-authored-by: Alex Luckett <alexgluckett@gmail.com>
Co-authored-by: Alex Luckett <alexluckett@users.noreply.github.com>
Co-authored-by: Jignesh Nayi <47786869+nayij@users.noreply.github.com>
Co-authored-by: Andrew-Folga-EE <173707519+Andrew-Folga@users.noreply.github.com>
Co-authored-by: David Barker <david.barker@equalexperts.com>
Co-authored-by: Andrew Folga <andrew.folga@equalexperts.com>

---------

Signed-off-by: David Barker <david.barker@equalexperts.com>
Co-authored-by: Mohammed Khalid <emailmokhld@gmail.com>
Co-authored-by: David Stone <davidjamesstone@gmail.com>
Co-authored-by: Jignesh Nayi <47786869+nayij@users.noreply.github.com>
Co-authored-by: Andrew-Folga-EE <173707519+Andrew-Folga@users.noreply.github.com>
Co-authored-by: David Barker <david.barker@equalexperts.com>
Co-authored-by: Andrew Folga <andrew.folga@equalexperts.com>

Author: 6 people

.github/workflows/publish.yml

@@ -15,6 +15,10 @@ on:
       - '!**/*.test.*'
 
   workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version number manual override'
+        required: false
 
 concurrency:
   group: publish
@@ -41,3 +45,4 @@ jobs:
         uses: DEFRA/cdp-build-action/build@main
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
+          version: ${{ inputs.version }}

.gitignore

@@ -9,3 +9,4 @@ coverage
 .server
 .cache
 .env
+tsconfig.tsbuildinfo

jest.config.cjs

@@ -45,7 +45,8 @@ module.exports = {
     `node_modules/(?!${[
       '@defra/forms-model/.*',
       'nanoid', // Supports ESM only
-      'slug' // Supports ESM only
+      'slug', // Supports ESM only
+      '@defra/hapi-tracing' // Supports ESM only|
     ].join('|')}/)`
   ],
   testTimeout: 10000,

package-lock.json

@@ -41,6 +41,7 @@
   "license": "SEE LICENSE IN LICENSE",
   "dependencies": {
     "@defra/forms-model": "^3.0.409",
+    "@defra/hapi-tracing": "^1.0.0",
     "@elastic/ecs-pino-format": "^1.5.0",
     "@hapi/boom": "^10.0.1",
     "@hapi/catbox": "^12.1.1",
@@ -74,6 +75,7 @@
     "humanize-duration": "^3.32.1",
     "ioredis": "^5.4.2",
     "joi": "^17.13.3",
+    "liquidjs": "^10.20.3",
     "lodash": "^4.17.21",
     "marked": "^15.0.6",
     "nunjucks": "^3.2.3",

package.json

@@ -152,6 +152,14 @@ export const config = convict({
       env: 'REDIS_KEY_PREFIX'
     } as SchemaObj<string>
   },
+  tracing: {
+    header: {
+      doc: 'Tracing header name',
+      format: String,
+      default: 'x-cdp-request-id',
+      env: 'TRACING_HEADER'
+    } as SchemaObj<string>
+  },
 
   /**
    * Email outputs

src/config/index.ts

@@ -0,0 +1,50 @@
+import { getTraceId } from '@defra/hapi-tracing'
+
+import { loggerOptions } from '~/src/server/common/helpers/logging/logger-options.js'
+
+jest.mock('@defra/hapi-tracing', () => ({
+  getTraceId: jest.fn()
+}))
+
+describe('logger-options', () => {
+  beforeEach(() => {
+    jest.clearAllMocks()
+  })
+
+  describe('configuration', () => {
+    it('has the expected properties', () => {
+      expect(loggerOptions).toHaveProperty('enabled')
+      expect(loggerOptions).toHaveProperty('ignorePaths')
+      expect(loggerOptions).toHaveProperty('redact')
+      expect(loggerOptions).toHaveProperty('level')
+    })
+
+    it('ignores health endpoint', () => {
+      expect(loggerOptions.ignorePaths).toContain('/health')
+    })
+
+    it('has redaction configuration', () => {
+      expect(loggerOptions.redact).toBeDefined()
+    })
+  })
+
+  describe('mixin function', () => {
+    it('returns an empty object when no trace ID is available', () => {
+      jest.mocked(getTraceId).mockReturnValue(null)
+
+      const result = loggerOptions.mixin()
+      expect(result).toEqual({})
+    })
+
+    it('includes trace ID when available', () => {
+      jest.mocked(getTraceId).mockReturnValue('some-trace-id')
+
+      const result = loggerOptions.mixin()
+      expect(result).toEqual({
+        trace: {
+          id: 'some-trace-id'
+        }
+      })
+    })
+  })
+})

src/server/common/helpers/logging/logger-options.test.ts

@@ -1,3 +1,4 @@
+import { getTraceId } from '@defra/hapi-tracing'
 import { ecsFormat } from '@elastic/ecs-pino-format'
 import { type Options } from 'hapi-pino'
 import { type LoggerOptions, type TransportSingleOptions } from 'pino'
@@ -33,5 +34,13 @@ export const loggerOptions = {
     remove: true
   },
   level: logConfig.level,
-  ...formatters[logConfig.format]
+  ...formatters[logConfig.format],
+  mixin() {
+    const mixinValues: { trace?: { id: string } } = {}
+    const traceId = getTraceId()
+    if (traceId) {
+      mixinValues.trace = { id: traceId }
+    }
+    return mixinValues
+  }
 } satisfies Options

src/server/common/helpers/logging/logger-options.ts

@@ -0,0 +1,10 @@
+import { tracing } from '@defra/hapi-tracing'
+
+import { config } from '~/src/config/index.js'
+
+// Explicitly declare the expected type
+
+export const requestTracing = {
+  plugin: tracing,
+  options: { tracingHeader: config.get('tracing').header }
+}

src/server/common/helpers/logging/request-tracing.js

@@ -14,6 +14,7 @@ import { ProxyAgent } from 'proxy-agent'
 
 import { config } from '~/src/config/index.js'
 import { requestLogger } from '~/src/server/common/helpers/logging/request-logger.js'
+import { requestTracing } from '~/src/server/common/helpers/logging/request-tracing.js'
 import { buildRedisClient } from '~/src/server/common/helpers/redis-client.js'
 import { configureBlankiePlugin } from '~/src/server/plugins/blankie.js'
 import { configureCrumbPlugin } from '~/src/server/plugins/crumb.js'
@@ -124,6 +125,7 @@ export async function createServer(routeConfig?: RouteConfig) {
   await server.register(pluginRouter)
   await server.register(pluginErrorPages)
   await server.register(blipp)
+  await server.register(requestTracing)
 
   server.state('cookieConsent', {
     ttl: 365 * 24 * 60 * 60 * 1000, // 1 year in ms