docs: branded data-plane error pages enhancement proposal

docs/enhancements/branded-error-pages/README.md

@@ -0,0 +1,265 @@
+---
+status: provisional
+stage: alpha
+---
+
+# Branded Data-Plane Error Pages
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [Where the page is served](#where-the-page-is-served)
+  - [Which responses get the page](#which-responses-get-the-page)
+  - [Content ownership: a ConfigMap, with a baked-in fallback](#content-ownership-a-configmap-with-a-baked-in-fallback)
+  - [Updating the page](#updating-the-page)
+  - [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message)
+  - [Verifying the experience](#verifying-the-experience)
+  - [Rollout](#rollout)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+
+## Summary
+
+When a visitor reaches a customer's site through Datum's edge — a Connector-backed
+proxy, or any Gateway on the downstream data plane — and something goes wrong
+(the tunnel is offline, the origin is down, a request times out), the response is
+generated by Envoy, not by the customer's application. By default that response
+is a **raw, unstyled error**: an HTTP 503 with a bare body like
+`no healthy upstream`, with no indication of where it came from or what to do
+next.
+
+This enhancement delivers a **Datum-branded HTML error page** for those
+edge-generated errors on the downstream / Connector data plane. The page is
+rendered by the **extension server** — the component that already customizes this
+data plane during Envoy Gateway translation — and its **content is sourced from a
+mounted `ConfigMap`**, with a copy compiled into the operator as an
+always-valid fallback. That makes the page editable through GitOps without an
+operator release, while guaranteeing a valid page even if the override is absent.
+
+## Motivation
+
+Datum's network services route a visitor's request through edge Envoy to a
+customer's backend — either directly, or, for the **Connector**, through an
+outbound tunnel to a private origin. When that path can't be completed, Envoy
+itself returns the response. Common cases on the Connector data plane:
+
+- The Connector tunnel is **offline** (agent not running / not yet connected).
+- The origin behind the tunnel or backend is **down or unreachable**.
+- The request **times out** or trips a circuit breaker.
+
+In all of these, the visitor never reaches the customer's app, so the customer's
+own error page can't be shown. The edge is the only thing that can speak — and
+what it says, and how it looks, is Datum's responsibility. A blank `503` with
+`no healthy upstream` reads like a broken platform; a branded page that says
+"this site is temporarily unavailable" with Datum's identity reads like a
+platform that's working as intended and tells you what's happening.
+
+This is a **trust and brand-perception** surface. It is the page a customer's
+*own* end-users see when the customer is mid-incident — precisely when the
+experience matters most.
+
+### Goals
+
+- Serve a **branded HTML error page** for edge-generated 5xx responses on the
+  downstream / Connector data plane.
+- Cover **all** customer-facing listeners on that data plane, including HTTP/3 /
+  QUIC, so the experience is consistent regardless of how the browser connects.
+- Let the page's **content be updated independently of code** — a content edit
+  should be a GitOps change, not an operator image build and release.
+- **Fail safe**: a missing or malformed page must never take the data plane down
+  or block configuration updates.
+
+### Non-Goals
+
+- Changing **which** status codes are considered errors, or the HTTP status code
+  returned. We brand the body; we do not alter routing or response codes.
+- Per-customer or per-Gateway custom error pages. This proposal delivers one
+  Datum-branded page across the data plane. Per-tenant theming is possible future
+  work and is called out as a non-goal here.
+- The **standard** (non-downstream) gateways, which already serve a branded page
+  and are out of scope.
+- Replacing application-level error pages. When the customer's origin is
+  reachable, its own responses are returned unchanged.
+
+## Proposal
+
+Deliver the branded error page through the extension server's existing
+per-listener mutation pass. During Envoy Gateway's translation, the extension
+server already rewrites listeners (for the WAF) and routes and clusters (for the
+Connector); we add one more listener mutation that attaches a **local reply
+configuration** carrying the branded HTML. The mechanics — which responses are
+branded, where the content lives, and how updates propagate — are covered under
+[Design Details](#design-details).
+
+### User Stories
+
+- **As a visitor to a customer's site that is temporarily unavailable**, I see a
+  clear, branded "temporarily unavailable" page instead of a blank or cryptic
+  error, so I understand the site (not my browser) has a problem and that it's
+  likely transient.
+- **As a Datum customer whose Connector is offline**, when my users hit my proxy
+  they get a polished, on-brand page, so an outage on my side doesn't look like a
+  broken platform.
+- **As Datum's brand/design owner**, I can update the wording and styling of the
+  edge error page by changing a single content artifact in Git, and have it roll
+  out across the edge without waiting on an operator release.
+- **As a Datum operator**, I can confirm the branded page is actually being
+  served, and I'm confident that a bad content edit can't break traffic — the
+  edge falls back to a known-good page.
+
+### Risks and Mitigations
+
+Each risk is mitigated by a design choice detailed in the linked section below.
+
+| Risk | Mitigation |
+| --- | --- |
+| A missing or malformed page removes the experience or, worse, blocks config updates (the downstream extension hook is fail-closed). | Embedded fallback + `optional` mount; content is an override only. See [Content ownership](#content-ownership-a-configmap-with-a-baked-in-fallback). |
+| A content edit appears not to take effect. | Content changes roll the extension server to force a fresh translation. See [Updating the page](#updating-the-page). |
+| The branded page overrides the Connector's terse tunnel-control signal. | Visitor traffic and tunnel-control traffic are distinct; handling is a deliberate decision. See [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message). |
+
+## Design Details
+
+> [!NOTE]
+> This section stays at the product/architecture level. The translation hook,
+> security, and re-translation machinery are described in detail in the
+> [Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md)
+> enhancement; this proposal reuses them.
+
+### Where the page is served
+
+The extension server already participates in every Envoy Gateway translation for
+the downstream data plane and already mutates listeners. We add a per-listener
+step that attaches Envoy's **local reply** configuration to each customer-facing
+HTTP connection manager. "Local reply" is Envoy's term for a response Envoy
+generates itself (an upstream failure, a timeout, a direct response) — exactly
+the responses a customer's app can't answer. Attaching the branded body there
+means every such response on that listener renders the page.
+
+Because the step runs over **all** listeners the translation produces, it
+covers both the TLS (HTTP/1.1 + HTTP/2) and the QUIC / HTTP/3 listeners
+uniformly, with no per-listener targeting to maintain and no class of "we forgot
+a listener" bug.
+
+Internal, non-customer listeners (for example Envoy Gateway's own readiness
+endpoint) are **excluded** by construction, so health checks and infrastructure
+endpoints are never branded.
+
+### Which responses get the page
+
+**Any edge-generated response with status ≥ 500** is rendered as the branded page
+(configurable threshold). The original HTTP status code is preserved — a 503
+stays a 503 — only the body and content-type are replaced. A runtime switch is
+retained so the behavior can be disabled in an emergency without a redeploy.
+
+### Content ownership: a ConfigMap, with a baked-in fallback
+
+The page is **content**, owned by brand/design, not by operator code. So:
+
+- The HTML lives in a **`ConfigMap`** managed in the infrastructure (GitOps)
+  repository.
+- The extension server **mounts** that `ConfigMap` to disk (an `optional`
+  volume) and reads the file at startup.
+- The operator also carries a **compiled-in default** page. If the mounted
+  override is missing or unreadable, the default is used.
+
+This split gives us the best of both: content is editable by GitOps without an
+operator release, and there is always a valid page even if the override is
+absent — which is essential because the downstream extension hook is fail-closed
+(a hard error would otherwise stall configuration for the whole data plane).
+
+A minimal operator configuration block selects the behavior; the bytes come from
+the mount or the embedded fallback:
+
+```yaml
+gateway:
+  errorPage:
+    enabled: true
+    bodyPath: /etc/datum/error-pages/error-5xx.html  # optional; falls back to embedded default
+    minStatusCode: 500
+```
+
+### Updating the page
+
+A `ConfigMap` edit lands on disk but does **not** automatically reach Envoy: the
+extension server only influences Envoy during a translation, and Envoy Gateway
+does not watch the extension server's mounted files. So a content change is only
+applied on the **next** translation.
+
+To make updates deterministic, a content change **rolls the extension server**
+(via a content checksum on the Deployment or a reload controller). The restart
+re-reads the file and the extension server's reconnection drives a fresh
+translation, so the new page propagates predictably across the edge. This mirrors
+the broader re-translation considerations covered in the extension-server
+enhancement.
+
+### Interaction with the Connector "tunnel offline" message
+
+The Connector path has a second, separate signal: a terse `503 "Tunnel not
+online"` returned on the tunnel-control (HTTP `CONNECT`) path, used by tunnel
+clients rather than browsers. Normal end-users never traverse that path — their
+browser requests are ordinary `GET`s — so the **visitor-facing** experience is
+governed entirely by the branded page described here.
+
+Because the branded page matches any response ≥ 500, it would, if applied
+indiscriminately, also rewrite that tunnel-control 503. Two acceptable options:
+
+1. **Brand everything ≥ 500** (simplest; one consistent rule). Tunnel clients are
+   robust to body changes and key on status, not body.
+2. **Preserve the terse tunnel-control body** by scoping the page to
+   visitor-facing responses.
+
+We default to (1) for simplicity and revisit only if a tunnel client needs the
+terse body. Either way, the visitor experience — the thing this proposal is
+about — is the branded page.
+
+### Verifying the experience
+
+- A browser request to a proxy whose Connector is **offline** returns a `503`
+  whose body is the **branded HTML** (correct `content-type: text/html`), not
+  `no healthy upstream`.
+- The same holds for an **origin-down** backend and for a **timeout**.
+- Envoy Gateway's own **readiness** endpoint and other internal listeners are
+  **not** branded.
+- Editing the `ConfigMap` and letting the rollout complete changes the served
+  page; removing the `ConfigMap` falls back to the embedded default with no
+  traffic impact.
+- HTTP/3 / QUIC clients get the same branded page as HTTP/2 clients.
+
+### Rollout
+
+1. Ship the extension-server delivery and embedded fallback — the page renders
+   from the baked-in default even before any `ConfigMap` exists.
+2. Add the `ConfigMap` with the page content and wire the mount and the
+   content-change rollout.
+3. Verify the branded page on the downstream data plane in staging, then
+   production, using the checks above.
+
+## Drawbacks
+
+- It adds one more responsibility to the extension server, which is on the
+  critical translation path for the data plane — though the work is small and
+  isolated.
+- Content updates are not instantaneous: they require a translation, which we
+  trigger via a controlled rollout. This is inherent to the extension-server
+  model and is an acceptable trade for not coupling content to code releases.
+- Two copies of the page exist (the live `ConfigMap` and the embedded fallback).
+  The fallback is intentionally minimal to keep this cost low.
+
+## Alternatives
+
+- **Bake the page only into the operator image (no `ConfigMap`).** Simpler, and a
+  reasonable starting point, but every wording or styling tweak becomes an
+  operator release. The `ConfigMap` keeps content a GitOps concern; the embedded
+  copy is retained as a fallback, not as the primary source.
+- **Serve the error page from a separate service** the edge routes to on failure.
+  More moving parts, a new failure domain on the unhappy path, and added latency
+  exactly when the system is already degraded. The local-reply approach keeps the
+  page entirely within the edge proxy.
+- **Do nothing.** Visitors continue to see raw `no healthy upstream` bodies on
+  customer outages, undercutting the trust and brand experience this proposal
+  delivers.