diff --git a/docs/enhancements/branded-error-pages/README.md b/docs/enhancements/branded-error-pages/README.md new file mode 100644 index 00000000..02e93b9a --- /dev/null +++ b/docs/enhancements/branded-error-pages/README.md @@ -0,0 +1,265 @@ +--- +status: provisional +stage: alpha +--- + +# Branded Data-Plane Error Pages + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories](#user-stories) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Where the page is served](#where-the-page-is-served) + - [Which responses get the page](#which-responses-get-the-page) + - [Content ownership: a ConfigMap, with a baked-in fallback](#content-ownership-a-configmap-with-a-baked-in-fallback) + - [Updating the page](#updating-the-page) + - [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message) + - [Verifying the experience](#verifying-the-experience) + - [Rollout](#rollout) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + +## Summary + +When a visitor reaches a customer's site through Datum's edge — a Connector-backed +proxy, or any Gateway on the downstream data plane — and something goes wrong +(the tunnel is offline, the origin is down, a request times out), the response is +generated by Envoy, not by the customer's application. By default that response +is a **raw, unstyled error**: an HTTP 503 with a bare body like +`no healthy upstream`, with no indication of where it came from or what to do +next. + +This enhancement delivers a **Datum-branded HTML error page** for those +edge-generated errors on the downstream / Connector data plane. The page is +rendered by the **extension server** — the component that already customizes this +data plane during Envoy Gateway translation — and its **content is sourced from a +mounted `ConfigMap`**, with a copy compiled into the operator as an +always-valid fallback. That makes the page editable through GitOps without an +operator release, while guaranteeing a valid page even if the override is absent. + +## Motivation + +Datum's network services route a visitor's request through edge Envoy to a +customer's backend — either directly, or, for the **Connector**, through an +outbound tunnel to a private origin. When that path can't be completed, Envoy +itself returns the response. Common cases on the Connector data plane: + +- The Connector tunnel is **offline** (agent not running / not yet connected). +- The origin behind the tunnel or backend is **down or unreachable**. +- The request **times out** or trips a circuit breaker. + +In all of these, the visitor never reaches the customer's app, so the customer's +own error page can't be shown. The edge is the only thing that can speak — and +what it says, and how it looks, is Datum's responsibility. A blank `503` with +`no healthy upstream` reads like a broken platform; a branded page that says +"this site is temporarily unavailable" with Datum's identity reads like a +platform that's working as intended and tells you what's happening. + +This is a **trust and brand-perception** surface. It is the page a customer's +*own* end-users see when the customer is mid-incident — precisely when the +experience matters most. + +### Goals + +- Serve a **branded HTML error page** for edge-generated 5xx responses on the + downstream / Connector data plane. +- Cover **all** customer-facing listeners on that data plane, including HTTP/3 / + QUIC, so the experience is consistent regardless of how the browser connects. +- Let the page's **content be updated independently of code** — a content edit + should be a GitOps change, not an operator image build and release. +- **Fail safe**: a missing or malformed page must never take the data plane down + or block configuration updates. + +### Non-Goals + +- Changing **which** status codes are considered errors, or the HTTP status code + returned. We brand the body; we do not alter routing or response codes. +- Per-customer or per-Gateway custom error pages. This proposal delivers one + Datum-branded page across the data plane. Per-tenant theming is possible future + work and is called out as a non-goal here. +- The **standard** (non-downstream) gateways, which already serve a branded page + and are out of scope. +- Replacing application-level error pages. When the customer's origin is + reachable, its own responses are returned unchanged. + +## Proposal + +Deliver the branded error page through the extension server's existing +per-listener mutation pass. During Envoy Gateway's translation, the extension +server already rewrites listeners (for the WAF) and routes and clusters (for the +Connector); we add one more listener mutation that attaches a **local reply +configuration** carrying the branded HTML. The mechanics — which responses are +branded, where the content lives, and how updates propagate — are covered under +[Design Details](#design-details). + +### User Stories + +- **As a visitor to a customer's site that is temporarily unavailable**, I see a + clear, branded "temporarily unavailable" page instead of a blank or cryptic + error, so I understand the site (not my browser) has a problem and that it's + likely transient. +- **As a Datum customer whose Connector is offline**, when my users hit my proxy + they get a polished, on-brand page, so an outage on my side doesn't look like a + broken platform. +- **As Datum's brand/design owner**, I can update the wording and styling of the + edge error page by changing a single content artifact in Git, and have it roll + out across the edge without waiting on an operator release. +- **As a Datum operator**, I can confirm the branded page is actually being + served, and I'm confident that a bad content edit can't break traffic — the + edge falls back to a known-good page. + +### Risks and Mitigations + +Each risk is mitigated by a design choice detailed in the linked section below. + +| Risk | Mitigation | +| --- | --- | +| A missing or malformed page removes the experience or, worse, blocks config updates (the downstream extension hook is fail-closed). | Embedded fallback + `optional` mount; content is an override only. See [Content ownership](#content-ownership-a-configmap-with-a-baked-in-fallback). | +| A content edit appears not to take effect. | Content changes roll the extension server to force a fresh translation. See [Updating the page](#updating-the-page). | +| The branded page overrides the Connector's terse tunnel-control signal. | Visitor traffic and tunnel-control traffic are distinct; handling is a deliberate decision. See [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message). | + +## Design Details + +> [!NOTE] +> This section stays at the product/architecture level. The translation hook, +> security, and re-translation machinery are described in detail in the +> [Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md) +> enhancement; this proposal reuses them. + +### Where the page is served + +The extension server already participates in every Envoy Gateway translation for +the downstream data plane and already mutates listeners. We add a per-listener +step that attaches Envoy's **local reply** configuration to each customer-facing +HTTP connection manager. "Local reply" is Envoy's term for a response Envoy +generates itself (an upstream failure, a timeout, a direct response) — exactly +the responses a customer's app can't answer. Attaching the branded body there +means every such response on that listener renders the page. + +Because the step runs over **all** listeners the translation produces, it +covers both the TLS (HTTP/1.1 + HTTP/2) and the QUIC / HTTP/3 listeners +uniformly, with no per-listener targeting to maintain and no class of "we forgot +a listener" bug. + +Internal, non-customer listeners (for example Envoy Gateway's own readiness +endpoint) are **excluded** by construction, so health checks and infrastructure +endpoints are never branded. + +### Which responses get the page + +**Any edge-generated response with status ≥ 500** is rendered as the branded page +(configurable threshold). The original HTTP status code is preserved — a 503 +stays a 503 — only the body and content-type are replaced. A runtime switch is +retained so the behavior can be disabled in an emergency without a redeploy. + +### Content ownership: a ConfigMap, with a baked-in fallback + +The page is **content**, owned by brand/design, not by operator code. So: + +- The HTML lives in a **`ConfigMap`** managed in the infrastructure (GitOps) + repository. +- The extension server **mounts** that `ConfigMap` to disk (an `optional` + volume) and reads the file at startup. +- The operator also carries a **compiled-in default** page. If the mounted + override is missing or unreadable, the default is used. + +This split gives us the best of both: content is editable by GitOps without an +operator release, and there is always a valid page even if the override is +absent — which is essential because the downstream extension hook is fail-closed +(a hard error would otherwise stall configuration for the whole data plane). + +A minimal operator configuration block selects the behavior; the bytes come from +the mount or the embedded fallback: + +```yaml +gateway: + errorPage: + enabled: true + bodyPath: /etc/datum/error-pages/error-5xx.html # optional; falls back to embedded default + minStatusCode: 500 +``` + +### Updating the page + +A `ConfigMap` edit lands on disk but does **not** automatically reach Envoy: the +extension server only influences Envoy during a translation, and Envoy Gateway +does not watch the extension server's mounted files. So a content change is only +applied on the **next** translation. + +To make updates deterministic, a content change **rolls the extension server** +(via a content checksum on the Deployment or a reload controller). The restart +re-reads the file and the extension server's reconnection drives a fresh +translation, so the new page propagates predictably across the edge. This mirrors +the broader re-translation considerations covered in the extension-server +enhancement. + +### Interaction with the Connector "tunnel offline" message + +The Connector path has a second, separate signal: a terse `503 "Tunnel not +online"` returned on the tunnel-control (HTTP `CONNECT`) path, used by tunnel +clients rather than browsers. Normal end-users never traverse that path — their +browser requests are ordinary `GET`s — so the **visitor-facing** experience is +governed entirely by the branded page described here. + +Because the branded page matches any response ≥ 500, it would, if applied +indiscriminately, also rewrite that tunnel-control 503. Two acceptable options: + +1. **Brand everything ≥ 500** (simplest; one consistent rule). Tunnel clients are + robust to body changes and key on status, not body. +2. **Preserve the terse tunnel-control body** by scoping the page to + visitor-facing responses. + +We default to (1) for simplicity and revisit only if a tunnel client needs the +terse body. Either way, the visitor experience — the thing this proposal is +about — is the branded page. + +### Verifying the experience + +- A browser request to a proxy whose Connector is **offline** returns a `503` + whose body is the **branded HTML** (correct `content-type: text/html`), not + `no healthy upstream`. +- The same holds for an **origin-down** backend and for a **timeout**. +- Envoy Gateway's own **readiness** endpoint and other internal listeners are + **not** branded. +- Editing the `ConfigMap` and letting the rollout complete changes the served + page; removing the `ConfigMap` falls back to the embedded default with no + traffic impact. +- HTTP/3 / QUIC clients get the same branded page as HTTP/2 clients. + +### Rollout + +1. Ship the extension-server delivery and embedded fallback — the page renders + from the baked-in default even before any `ConfigMap` exists. +2. Add the `ConfigMap` with the page content and wire the mount and the + content-change rollout. +3. Verify the branded page on the downstream data plane in staging, then + production, using the checks above. + +## Drawbacks + +- It adds one more responsibility to the extension server, which is on the + critical translation path for the data plane — though the work is small and + isolated. +- Content updates are not instantaneous: they require a translation, which we + trigger via a controlled rollout. This is inherent to the extension-server + model and is an acceptable trade for not coupling content to code releases. +- Two copies of the page exist (the live `ConfigMap` and the embedded fallback). + The fallback is intentionally minimal to keep this cost low. + +## Alternatives + +- **Bake the page only into the operator image (no `ConfigMap`).** Simpler, and a + reasonable starting point, but every wording or styling tweak becomes an + operator release. The `ConfigMap` keeps content a GitOps concern; the embedded + copy is retained as a fallback, not as the primary source. +- **Serve the error page from a separate service** the edge routes to on failure. + More moving parts, a new failure domain on the unhappy path, and added latency + exactly when the system is already degraded. The local-reply approach keeps the + page entirely within the edge proxy. +- **Do nothing.** Visitors continue to see raw `no healthy upstream` bodies on + customer outages, undercutting the trust and brand experience this proposal + delivers.