Improve 404 recovery with related page suggestions

[theletterf]

Summary

• redesign the 404 page around recovery actions: related pages, docs home, and a link to full search
• derive a bounded search query from the missing URL and use forced hybrid lexical/semantic search to suggest up to five pages
• preserve a true 404 response in both production-style and local assembled-site flows
• add frontend, API, query/service, Razor rendering, and synthetic coverage
• gate the related-pages UI behind the RELATED_PAGES docs-builder feature flag; it is enabled for preview and edge only, and remains disabled in production

Preview testing

The shared preview CloudFront distribution has a static error target at /docs/404/index.html. It cannot select the generated 404 page beneath an individual PR path prefix, so an arbitrary missing preview URL currently renders the generic distribution-level error page.

As a temporary review aid, the assembler-preview workflow publishes this generated 404 page at a known route:

https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3655/opentelemetry-collector

This route validates the branch's server-rendered 404 layout and frontend bundle without relying on CloudFront error handling. It intentionally returns HTTP 200 and therefore does not validate the final CloudFront status mapping.

It is not currently an end-to-end test of related-page embeddings. The preview distribution does not expose the docs search API at the preview path, and the API is currently protected by the searchOrAskAi feature cookie in WAF. The related-pages component hides itself when that API request fails.

The 404 intentionally does not include an inline search bar in this PR. Search remains available through the Open full search link; adding an embedded search experience can be discussed separately with the wider team.

The route-copy workaround is isolated in the workflow step named Publish temporary 404 experience preview. Delete that single step after the infrastructure fallback is available.

Required docs-infra follow-up

Before enabling RELATED_PAGES in production, docs-infra needs to:

1. make the docs search API available independently of the Search or Ask AI UI experiment and remove the searchOrAskAi cookie gate from WAF;
2. configure preview error handling to load the generated 404 page from the current PR path prefix while preserving the original missing URL and a 404 response; and
3. enable RELATED_PAGES for production after the API and error routing are ready.

The preview distribution currently uses the static /docs/404/index.html object. Relevant configuration:
https://github.com/elastic/docs-infra/blob/main/modules/elastic-docs-v3/cloudfront-distribution.tf

Local testing

The regular docs development server renders the repository 404.md for missing documentation routes. The assembled static host also serves the generated 404 page with status 404, providing local coverage of the layout and response status.

The related-pages component can be rendered locally by enabling RELATED_PAGES, but real suggestions still require a reachable docs search API with embeddings. Local layout testing therefore does not by itself validate semantic results.

Validation

• ./build.sh lint
• dotnet test tests/Elastic.Documentation.Api.Infrastructure.Tests/ --no-restore
• dotnet test tests/Elastic.Documentation.Api.Tests/ --no-restore
• dotnet test tests/Elastic.Markdown.Tests/ --no-restore --filter FullyQualifiedName~NotFoundPageTests
• npm run test -- RelatedPages
• npm run compile:check
• npm run lint
• npm run fmt:check
• npm run build

Addresses https://github.com/elastic/docs-eng-team/issues/663

[reakaleek]

Since this is hard to preview, can you attach a video or screenshots?

[reakaleek]

We shouldn't show anything at all if the feature is not enabled.

[reakaleek]

Also this link doesn't officially exist.