From 3b2913accf9829e2f2817e3e70f8202a748b7391 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 19 Jun 2026 08:09:10 -0500 Subject: [PATCH 1/5] docs: add branded data-plane error pages enhancement proposal Product-focused proposal to restore the branded HTML error page on the downstream/Connector data plane. The page was delivered via an EnvoyPatchPolicy (local-reply-5xx) that went inert when enableEnvoyPatchPolicy=false was set during the extension-server migration, so customer outages (e.g. offline Connector) now render raw Envoy bodies like "no healthy upstream" instead of the branded page. Proposes delivering the page through the extension server's per-listener mutation (the supported path now that EnvoyPatchPolicy is disabled there), sourcing content from a mounted ConfigMap with a compiled-in fallback so content edits are a GitOps change and a missing/bad page can never block the fail-closed data plane. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../branded-error-pages/README.md | 302 ++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100644 docs/enhancements/branded-error-pages/README.md diff --git a/docs/enhancements/branded-error-pages/README.md b/docs/enhancements/branded-error-pages/README.md new file mode 100644 index 00000000..020adb18 --- /dev/null +++ b/docs/enhancements/branded-error-pages/README.md @@ -0,0 +1,302 @@ +--- +status: provisional +stage: alpha +--- + +# Branded Data-Plane Error Pages + +- [Summary](#summary) +- [Motivation](#motivation) + - [Background: what a customer sees when something breaks](#background-what-a-customer-sees-when-something-breaks) + - [The regression: the branded page went silently dark](#the-regression-the-branded-page-went-silently-dark) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories](#user-stories) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Where the page is served](#where-the-page-is-served) + - [Which responses get the page](#which-responses-get-the-page) + - [Content ownership: a ConfigMap, with a baked-in fallback](#content-ownership-a-configmap-with-a-baked-in-fallback) + - [Updating the page](#updating-the-page) + - [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message) + - [Verifying the experience](#verifying-the-experience) + - [Transition Plan](#transition-plan) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + +## Summary + +When a customer's site is reached through Datum's edge — a Connector-backed +proxy, or any Gateway on the downstream data plane — and something goes wrong +(the tunnel is offline, the origin is down, a request times out), the visitor's +browser gets a response generated by Envoy, not by the customer's application. +Today, on the downstream/Connector data plane, that response is a **raw, +unstyled error**: an HTTP 503 with a bare body like `no healthy upstream` and no +indication of where it came from or what to do next. + +Datum already authored a **branded HTML error page** for exactly this moment. It +was delivered through an `EnvoyPatchPolicy`, but that mechanism was switched off +on the downstream data plane as part of the +[Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md) +migration. As a result the branded page has been **inert for months** on the +Connector data plane, and visitors have been seeing the raw Envoy output instead. + +This enhancement restores the branded error experience by moving its delivery +into the extension server (the supported path now that `EnvoyPatchPolicy` is +disabled there), and makes the page's **content** a configuration artifact — a +mounted `ConfigMap` — so it can be updated by GitOps without a code release. + +## Motivation + +### Background: what a customer sees when something breaks + +Datum's network services route a visitor's request through edge Envoy to a +customer's backend — either directly, or, for the **Connector**, through an +outbound tunnel to a private origin. When that path can't be completed, Envoy +itself returns the response. Common cases on the Connector data plane: + +- The Connector tunnel is **offline** (agent not running / not yet connected). +- The origin behind the tunnel or backend is **down or unreachable**. +- The request **times out** or trips a circuit breaker. + +In all of these, the visitor never reaches the customer's app, so the customer's +own error page can't be shown. The edge is the only thing that can speak — and +what it says, and how it looks, is Datum's responsibility. A blank `503` with +`no healthy upstream` reads like a broken platform; a branded page that says +"this site is temporarily unavailable" with Datum's identity reads like a +platform that's working as intended and tells you what's happening. + +This is a **trust and brand-perception** surface. It is the page a customer's +*own* end-users see when the customer is mid-incident — precisely when the +experience matters most. + +### The regression: the branded page went silently dark + +Datum built the branded page (an `` document, served as +`text/html`) and shipped it to the downstream gateway as an `EnvoyPatchPolicy` +named `local-reply-5xx`, matching any response with status ≥ 500. + +The [extension-server migration](../envoy-gateway-extension-server/README.md) +intentionally set `enableEnvoyPatchPolicy: false` on the downstream Envoy Gateway +— that was the whole point of moving WAF and Connector logic off +`EnvoyPatchPolicy`. But the branded error page was **never migrated** to the +extension server. Disabling `EnvoyPatchPolicy` disabled *every* such policy on +that data plane, including the error page, with no signal to a customer or +operator that the experience had degraded. The policy still exists in Git and on +the cluster; it is simply `Accepted=False` and never reaches Envoy. + +Net effect: on the Connector / downstream data plane, every edge-generated 5xx — +including the very common "Connector is offline" case — is served as a bare +Envoy body. The branded experience that product intended is not being delivered. + +### Goals + +- Restore the **branded HTML error page** for edge-generated 5xx responses on the + downstream / Connector data plane, at parity with what the `EnvoyPatchPolicy` + delivered. +- Make it work **without `EnvoyPatchPolicy`**, using the extension server, which + is the supported customization path on that data plane. +- Cover **all** customer-facing listeners on that data plane, including HTTP/3 / + QUIC, so the experience is consistent regardless of how the browser connects. +- Let the page's **content be updated independently of code** — a content edit + should be a GitOps change, not an operator image build and release. +- **Fail safe**: a missing or malformed page must never take the data plane down + or block configuration updates. + +### Non-Goals + +- Changing **which** status codes are considered errors, or the HTTP status code + returned. We brand the body; we do not alter routing or response codes. +- Per-customer or per-Gateway custom error pages. This proposal delivers one + Datum-branded page across the data plane. Per-tenant theming is possible future + work and is called out as a non-goal here. +- The **standard** (non-downstream) gateways, which still run `EnvoyPatchPolicy` + and already serve the branded page. They are out of scope and unchanged. +- Replacing application-level error pages. When the customer's origin is + reachable, its own responses are returned unchanged. + +## Proposal + +Deliver the branded error page through the extension server's existing +per-listener mutation pass. During Envoy Gateway's translation, the extension +server already rewrites listeners (for the WAF) and routes/clusters (for the +Connector). We add one more listener mutation: attach a **local reply +configuration** that renders the branded HTML for any response with status +≥ 500. The page's bytes come from a mounted `ConfigMap`, with a copy compiled +into the operator as an always-valid fallback. + +### User Stories + +- **As a visitor to a customer's site that is temporarily unavailable**, I see a + clear, branded "temporarily unavailable" page instead of a blank or cryptic + error, so I understand the site (not my browser) has a problem and that it's + likely transient. +- **As a Datum customer whose Connector is offline**, when my users hit my proxy + they get a polished, on-brand page rather than `no healthy upstream`, so an + outage on my side doesn't look like a broken platform. +- **As Datum's brand/design owner**, I can update the wording and styling of the + edge error page by changing a single content artifact in Git, and have it roll + out across the edge without waiting on an operator release. +- **As a Datum operator**, I can confirm the branded page is actually being + served, and I'm confident that a bad content edit can't break traffic — the + edge falls back to a known-good page. + +### Risks and Mitigations + +| Risk | Mitigation | +| --- | --- | +| A missing/empty/typo'd `ConfigMap` removes the page or, worse, blocks config updates (the downstream extension hook is fail-closed). | The operator ships an **embedded default** page and treats the `ConfigMap` purely as an *override*. The mount is `optional`. If the override is absent or unreadable, the embedded page is used; the hook never errors on content. | +| A content edit appears not to take effect. | Editing a mounted `ConfigMap` does not by itself reach Envoy; the new content only applies on the next Envoy Gateway translation. We make content changes **roll the extension server** (which forces a fresh translation) so propagation is deterministic. See [Updating the page](#updating-the-page). | +| The branded page overrides the Connector's terse "tunnel offline" signal used by tunnel clients. | The page is scoped to browser-facing responses; the tunnel-control response is preserved or treated as a deliberate, documented decision. See [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message). | +| Drift between the embedded fallback and the live `ConfigMap`. | The `ConfigMap` is the single source of truth in infra; the embedded copy is a minimal, rarely-changed safety net, not a second design to maintain. | + +## Design Details + +> This section stays at the product/architecture level. The translation hook, +> security, and re-translation machinery are described in detail in the +> [Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md) +> enhancement; this proposal reuses them. + +### Where the page is served + +The extension server already participates in every Envoy Gateway translation for +the downstream data plane and already mutates listeners. We add a per-listener +step that attaches Envoy's **local reply** configuration to each customer-facing +HTTP connection manager. "Local reply" is Envoy's term for a response Envoy +generates itself (an upstream failure, a timeout, a direct response) — exactly +the responses a customer's app can't answer. Attaching the branded body there +means every such response on that listener renders the page. + +Because the step runs over **all** listeners the translation produces, it +naturally covers both the TLS (HTTP/1.1 + HTTP/2) and the QUIC / HTTP/3 +listeners. The previous `EnvoyPatchPolicy` had to hand-target each listener by +name and was extended specifically to cover HTTP/3; that maintenance burden and +class of "we forgot a listener" bug goes away. + +Internal, non-customer listeners (for example Envoy Gateway's own readiness +endpoint) are **excluded** by construction, so health checks and infrastructure +endpoints are never branded. + +### Which responses get the page + +Parity with the original policy: **any edge-generated response with status +≥ 500** is rendered as the branded page (configurable threshold). The original +HTTP status code is preserved — a 503 stays a 503 — only the body and +content-type are replaced. A runtime switch is retained so the behavior can be +disabled in an emergency without a redeploy. + +### Content ownership: a ConfigMap, with a baked-in fallback + +The page is **content**, owned by brand/design, not by operator code. So: + +- The HTML lives in a **`ConfigMap`** managed in the infrastructure (GitOps) + repository — the same place the page lives today, expressed as configuration + instead of an `EnvoyPatchPolicy`. +- The extension server **mounts** that `ConfigMap` to disk (an `optional` + volume) and reads the file at startup. +- The operator also carries a **compiled-in default** page. If the mounted + override is missing or unreadable, the default is used. + +This split gives us the best of both: content is editable by GitOps without an +operator release, and there is always a valid page even if the override is +absent — which is essential because the downstream extension hook is fail-closed +(a hard error would otherwise stall configuration for the whole data plane). + +A minimal operator configuration block selects the behavior; the bytes come from +the mount or the embedded fallback: + +```yaml +gateway: + errorPage: + enabled: true + bodyPath: /etc/datum/error-pages/error-5xx.html # optional; falls back to embedded default + minStatusCode: 500 +``` + +### Updating the page + +A `ConfigMap` edit lands on disk but does **not** automatically reach Envoy: the +extension server only influences Envoy during a translation, and Envoy Gateway +does not watch the extension server's mounted files. So a content change is only +applied on the **next** translation. + +To make updates deterministic, a content change **rolls the extension server** +(via a content checksum on the Deployment or a reload controller). The restart +re-reads the file and the extension server's reconnection drives a fresh +translation, so the new page propagates predictably across the edge. This mirrors +the broader re-translation considerations covered in the extension-server +enhancement. + +### Interaction with the Connector "tunnel offline" message + +The Connector path has a second, separate signal: a terse `503 "Tunnel not +online"` returned on the tunnel-control (HTTP `CONNECT`) path, used by tunnel +clients rather than browsers. Normal end-users never traverse that path — their +browser requests are ordinary `GET`s — so the **visitor-facing** experience is +governed entirely by the branded page described here. + +Because the branded page matches any response ≥ 500, it would, if applied +indiscriminately, also rewrite that tunnel-control 503. Two acceptable options: + +1. **Brand everything ≥ 500** (simplest; matches the original "brand any 5xx" + intent). Tunnel clients are robust to body changes and key on status, not body. +2. **Preserve the terse tunnel-control body** by scoping the page to + visitor-facing responses. + +We default to (1) for simplicity and revisit only if a tunnel client needs the +terse body. Either way, the visitor experience — the thing this proposal is +about — is the branded page. + +### Verifying the experience + +- A browser request to a proxy whose Connector is **offline** returns a `503` + whose body is the **branded HTML** (correct `content-type: text/html`), not + `no healthy upstream`. +- The same holds for an **origin-down** backend and for a **timeout**. +- Envoy Gateway's own **readiness** endpoint and other internal listeners are + **not** branded. +- Editing the `ConfigMap` and letting the rollout complete changes the served + page; removing the `ConfigMap` falls back to the embedded default with no + traffic impact. +- HTTP/3 / QUIC clients get the same branded page as HTTP/2 clients. + +### Transition Plan + +1. Ship the extension-server delivery and embedded fallback (page renders from + the baked-in default even before any `ConfigMap` exists). +2. Move the existing HTML from the `local-reply-5xx` `EnvoyPatchPolicy` into the + `ConfigMap`; wire the mount and the content-change rollout. +3. Verify the branded page on the downstream data plane in staging, then + production, using the checks above. +4. Decommission the now-redundant downstream `local-reply-5xx` `EnvoyPatchPolicy` + app. The standard-gateway error page (still `EnvoyPatchPolicy`-based) is left + untouched. + +## Drawbacks + +- It adds one more responsibility to the extension server, which is on the + critical translation path for the data plane. This is mitigated by the + embedded fallback and fail-safe handling, and the work is small and isolated. +- Content updates are not instantaneous: they require a translation, which we + trigger via a controlled rollout. This is inherent to the extension-server + model and is an acceptable trade for not coupling content to code releases. +- Two copies of the page exist (the live `ConfigMap` and the embedded fallback). + The fallback is intentionally minimal to keep this cost low. + +## Alternatives + +- **Re-enable `EnvoyPatchPolicy` just for the error page.** Rejected: it + reintroduces the exact mechanism the data plane deliberately moved off, and the + scaling/operability reasons for disabling it still apply. +- **Bake the page only into the operator image (no `ConfigMap`).** Simpler, and a + reasonable starting point, but every wording or styling tweak becomes an + operator release. The `ConfigMap` keeps content a GitOps concern; the embedded + copy is retained as a fallback, not as the primary source. +- **Serve the error page from a separate service** the edge routes to on failure. + More moving parts, a new failure domain on the unhappy path, and added latency + exactly when the system is already degraded. The local-reply approach keeps the + page entirely within the edge proxy. +- **Do nothing.** Visitors continue to see raw `no healthy upstream` bodies on + customer outages, undercutting the trust and brand experience the page was + built to provide. From 603dde44cedb52fa32ac8d9a436ac036b4955595 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 19 Jun 2026 08:15:18 -0500 Subject: [PATCH 2/5] docs: drop duration framing and de-duplicate error-pages proposal - Remove the (incorrect) 'inert for months' claim; how long EnvoyPatchPolicy has been disabled is unknown and not relevant to the proposal. - Consolidate content: Risks table now indexes the detailed design sections instead of restating them; Proposal intro defers mechanics to Design Details. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../branded-error-pages/README.md | 27 ++++++++++--------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/docs/enhancements/branded-error-pages/README.md b/docs/enhancements/branded-error-pages/README.md index 020adb18..69bb432b 100644 --- a/docs/enhancements/branded-error-pages/README.md +++ b/docs/enhancements/branded-error-pages/README.md @@ -39,8 +39,8 @@ Datum already authored a **branded HTML error page** for exactly this moment. It was delivered through an `EnvoyPatchPolicy`, but that mechanism was switched off on the downstream data plane as part of the [Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md) -migration. As a result the branded page has been **inert for months** on the -Connector data plane, and visitors have been seeing the raw Envoy output instead. +migration. As a result the branded page is not being delivered on the Connector +data plane, and visitors see the raw Envoy output instead. This enhancement restores the branded error experience by moving its delivery into the extension server (the supported path now that `EnvoyPatchPolicy` is @@ -120,11 +120,11 @@ Envoy body. The branded experience that product intended is not being delivered. Deliver the branded error page through the extension server's existing per-listener mutation pass. During Envoy Gateway's translation, the extension -server already rewrites listeners (for the WAF) and routes/clusters (for the -Connector). We add one more listener mutation: attach a **local reply -configuration** that renders the branded HTML for any response with status -≥ 500. The page's bytes come from a mounted `ConfigMap`, with a copy compiled -into the operator as an always-valid fallback. +server already rewrites listeners (for the WAF) and routes and clusters (for the +Connector); we add one more listener mutation that attaches a **local reply +configuration** carrying the branded HTML. The mechanics — which responses are +branded, where the content lives, and how updates propagate — are covered under +[Design Details](#design-details). ### User Stories @@ -144,12 +144,13 @@ into the operator as an always-valid fallback. ### Risks and Mitigations +Each risk is mitigated by a design choice detailed in the linked section below. + | Risk | Mitigation | | --- | --- | -| A missing/empty/typo'd `ConfigMap` removes the page or, worse, blocks config updates (the downstream extension hook is fail-closed). | The operator ships an **embedded default** page and treats the `ConfigMap` purely as an *override*. The mount is `optional`. If the override is absent or unreadable, the embedded page is used; the hook never errors on content. | -| A content edit appears not to take effect. | Editing a mounted `ConfigMap` does not by itself reach Envoy; the new content only applies on the next Envoy Gateway translation. We make content changes **roll the extension server** (which forces a fresh translation) so propagation is deterministic. See [Updating the page](#updating-the-page). | -| The branded page overrides the Connector's terse "tunnel offline" signal used by tunnel clients. | The page is scoped to browser-facing responses; the tunnel-control response is preserved or treated as a deliberate, documented decision. See [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message). | -| Drift between the embedded fallback and the live `ConfigMap`. | The `ConfigMap` is the single source of truth in infra; the embedded copy is a minimal, rarely-changed safety net, not a second design to maintain. | +| A missing or malformed page removes the experience or, worse, blocks config updates (the downstream extension hook is fail-closed). | Embedded fallback + `optional` mount; content is an override only. See [Content ownership](#content-ownership-a-configmap-with-a-baked-in-fallback). | +| A content edit appears not to take effect. | Content changes roll the extension server to force a fresh translation. See [Updating the page](#updating-the-page). | +| The branded page overrides the Connector's terse tunnel-control signal. | Visitor traffic and tunnel-control traffic are distinct; handling is a deliberate decision. See [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message). | ## Design Details @@ -276,8 +277,8 @@ about — is the branded page. ## Drawbacks - It adds one more responsibility to the extension server, which is on the - critical translation path for the data plane. This is mitigated by the - embedded fallback and fail-safe handling, and the work is small and isolated. + critical translation path for the data plane — though the work is small and + isolated. - Content updates are not instantaneous: they require a translation, which we trigger via a controlled rollout. This is inherent to the extension-server model and is an acceptable trade for not coupling content to code releases. From 1baec1e50838d571e3c70ec1083dca5313c0dca5 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 19 Jun 2026 08:19:26 -0500 Subject: [PATCH 3/5] docs: reframe error-pages proposal around the capability Lead with the functionality the branded data-plane error page provides rather than the migration history. Drop the regression narrative and the EnvoyPatchPolicy-comparison framing; describe the extension-server delivery as the design, not a restoration. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../branded-error-pages/README.md | 112 ++++++------------ 1 file changed, 38 insertions(+), 74 deletions(-) diff --git a/docs/enhancements/branded-error-pages/README.md b/docs/enhancements/branded-error-pages/README.md index 69bb432b..d20ddd81 100644 --- a/docs/enhancements/branded-error-pages/README.md +++ b/docs/enhancements/branded-error-pages/README.md @@ -8,7 +8,6 @@ stage: alpha - [Summary](#summary) - [Motivation](#motivation) - [Background: what a customer sees when something breaks](#background-what-a-customer-sees-when-something-breaks) - - [The regression: the branded page went silently dark](#the-regression-the-branded-page-went-silently-dark) - [Goals](#goals) - [Non-Goals](#non-goals) - [Proposal](#proposal) @@ -21,31 +20,27 @@ stage: alpha - [Updating the page](#updating-the-page) - [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message) - [Verifying the experience](#verifying-the-experience) - - [Transition Plan](#transition-plan) + - [Rollout](#rollout) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) ## Summary -When a customer's site is reached through Datum's edge — a Connector-backed +When a visitor reaches a customer's site through Datum's edge — a Connector-backed proxy, or any Gateway on the downstream data plane — and something goes wrong -(the tunnel is offline, the origin is down, a request times out), the visitor's -browser gets a response generated by Envoy, not by the customer's application. -Today, on the downstream/Connector data plane, that response is a **raw, -unstyled error**: an HTTP 503 with a bare body like `no healthy upstream` and no -indication of where it came from or what to do next. - -Datum already authored a **branded HTML error page** for exactly this moment. It -was delivered through an `EnvoyPatchPolicy`, but that mechanism was switched off -on the downstream data plane as part of the -[Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md) -migration. As a result the branded page is not being delivered on the Connector -data plane, and visitors see the raw Envoy output instead. - -This enhancement restores the branded error experience by moving its delivery -into the extension server (the supported path now that `EnvoyPatchPolicy` is -disabled there), and makes the page's **content** a configuration artifact — a -mounted `ConfigMap` — so it can be updated by GitOps without a code release. +(the tunnel is offline, the origin is down, a request times out), the response is +generated by Envoy, not by the customer's application. By default that response +is a **raw, unstyled error**: an HTTP 503 with a bare body like +`no healthy upstream`, with no indication of where it came from or what to do +next. + +This enhancement delivers a **Datum-branded HTML error page** for those +edge-generated errors on the downstream / Connector data plane. The page is +rendered by the **extension server** — the component that already customizes this +data plane during Envoy Gateway translation — and its **content is sourced from a +mounted `ConfigMap`**, with a copy compiled into the operator as an +always-valid fallback. That makes the page editable through GitOps without an +operator release, while guaranteeing a valid page even if the override is absent. ## Motivation @@ -71,32 +66,10 @@ This is a **trust and brand-perception** surface. It is the page a customer's *own* end-users see when the customer is mid-incident — precisely when the experience matters most. -### The regression: the branded page went silently dark - -Datum built the branded page (an `` document, served as -`text/html`) and shipped it to the downstream gateway as an `EnvoyPatchPolicy` -named `local-reply-5xx`, matching any response with status ≥ 500. - -The [extension-server migration](../envoy-gateway-extension-server/README.md) -intentionally set `enableEnvoyPatchPolicy: false` on the downstream Envoy Gateway -— that was the whole point of moving WAF and Connector logic off -`EnvoyPatchPolicy`. But the branded error page was **never migrated** to the -extension server. Disabling `EnvoyPatchPolicy` disabled *every* such policy on -that data plane, including the error page, with no signal to a customer or -operator that the experience had degraded. The policy still exists in Git and on -the cluster; it is simply `Accepted=False` and never reaches Envoy. - -Net effect: on the Connector / downstream data plane, every edge-generated 5xx — -including the very common "Connector is offline" case — is served as a bare -Envoy body. The branded experience that product intended is not being delivered. - ### Goals -- Restore the **branded HTML error page** for edge-generated 5xx responses on the - downstream / Connector data plane, at parity with what the `EnvoyPatchPolicy` - delivered. -- Make it work **without `EnvoyPatchPolicy`**, using the extension server, which - is the supported customization path on that data plane. +- Serve a **branded HTML error page** for edge-generated 5xx responses on the + downstream / Connector data plane. - Cover **all** customer-facing listeners on that data plane, including HTTP/3 / QUIC, so the experience is consistent regardless of how the browser connects. - Let the page's **content be updated independently of code** — a content edit @@ -111,8 +84,8 @@ Envoy body. The branded experience that product intended is not being delivered. - Per-customer or per-Gateway custom error pages. This proposal delivers one Datum-branded page across the data plane. Per-tenant theming is possible future work and is called out as a non-goal here. -- The **standard** (non-downstream) gateways, which still run `EnvoyPatchPolicy` - and already serve the branded page. They are out of scope and unchanged. +- The **standard** (non-downstream) gateways, which already serve a branded page + and are out of scope. - Replacing application-level error pages. When the customer's origin is reachable, its own responses are returned unchanged. @@ -133,8 +106,8 @@ branded, where the content lives, and how updates propagate — are covered unde error, so I understand the site (not my browser) has a problem and that it's likely transient. - **As a Datum customer whose Connector is offline**, when my users hit my proxy - they get a polished, on-brand page rather than `no healthy upstream`, so an - outage on my side doesn't look like a broken platform. + they get a polished, on-brand page, so an outage on my side doesn't look like a + broken platform. - **As Datum's brand/design owner**, I can update the wording and styling of the edge error page by changing a single content artifact in Git, and have it roll out across the edge without waiting on an operator release. @@ -170,10 +143,9 @@ the responses a customer's app can't answer. Attaching the branded body there means every such response on that listener renders the page. Because the step runs over **all** listeners the translation produces, it -naturally covers both the TLS (HTTP/1.1 + HTTP/2) and the QUIC / HTTP/3 -listeners. The previous `EnvoyPatchPolicy` had to hand-target each listener by -name and was extended specifically to cover HTTP/3; that maintenance burden and -class of "we forgot a listener" bug goes away. +covers both the TLS (HTTP/1.1 + HTTP/2) and the QUIC / HTTP/3 listeners +uniformly, with no per-listener targeting to maintain and no class of "we forgot +a listener" bug. Internal, non-customer listeners (for example Envoy Gateway's own readiness endpoint) are **excluded** by construction, so health checks and infrastructure @@ -181,19 +153,17 @@ endpoints are never branded. ### Which responses get the page -Parity with the original policy: **any edge-generated response with status -≥ 500** is rendered as the branded page (configurable threshold). The original -HTTP status code is preserved — a 503 stays a 503 — only the body and -content-type are replaced. A runtime switch is retained so the behavior can be -disabled in an emergency without a redeploy. +**Any edge-generated response with status ≥ 500** is rendered as the branded page +(configurable threshold). The original HTTP status code is preserved — a 503 +stays a 503 — only the body and content-type are replaced. A runtime switch is +retained so the behavior can be disabled in an emergency without a redeploy. ### Content ownership: a ConfigMap, with a baked-in fallback The page is **content**, owned by brand/design, not by operator code. So: - The HTML lives in a **`ConfigMap`** managed in the infrastructure (GitOps) - repository — the same place the page lives today, expressed as configuration - instead of an `EnvoyPatchPolicy`. + repository. - The extension server **mounts** that `ConfigMap` to disk (an `optional` volume) and reads the file at startup. - The operator also carries a **compiled-in default** page. If the mounted @@ -240,8 +210,8 @@ governed entirely by the branded page described here. Because the branded page matches any response ≥ 500, it would, if applied indiscriminately, also rewrite that tunnel-control 503. Two acceptable options: -1. **Brand everything ≥ 500** (simplest; matches the original "brand any 5xx" - intent). Tunnel clients are robust to body changes and key on status, not body. +1. **Brand everything ≥ 500** (simplest; one consistent rule). Tunnel clients are + robust to body changes and key on status, not body. 2. **Preserve the terse tunnel-control body** by scoping the page to visitor-facing responses. @@ -262,17 +232,14 @@ about — is the branded page. traffic impact. - HTTP/3 / QUIC clients get the same branded page as HTTP/2 clients. -### Transition Plan +### Rollout -1. Ship the extension-server delivery and embedded fallback (page renders from - the baked-in default even before any `ConfigMap` exists). -2. Move the existing HTML from the `local-reply-5xx` `EnvoyPatchPolicy` into the - `ConfigMap`; wire the mount and the content-change rollout. +1. Ship the extension-server delivery and embedded fallback — the page renders + from the baked-in default even before any `ConfigMap` exists. +2. Add the `ConfigMap` with the page content and wire the mount and the + content-change rollout. 3. Verify the branded page on the downstream data plane in staging, then production, using the checks above. -4. Decommission the now-redundant downstream `local-reply-5xx` `EnvoyPatchPolicy` - app. The standard-gateway error page (still `EnvoyPatchPolicy`-based) is left - untouched. ## Drawbacks @@ -287,9 +254,6 @@ about — is the branded page. ## Alternatives -- **Re-enable `EnvoyPatchPolicy` just for the error page.** Rejected: it - reintroduces the exact mechanism the data plane deliberately moved off, and the - scaling/operability reasons for disabling it still apply. - **Bake the page only into the operator image (no `ConfigMap`).** Simpler, and a reasonable starting point, but every wording or styling tweak becomes an operator release. The `ConfigMap` keeps content a GitOps concern; the embedded @@ -299,5 +263,5 @@ about — is the branded page. exactly when the system is already degraded. The local-reply approach keeps the page entirely within the edge proxy. - **Do nothing.** Visitors continue to see raw `no healthy upstream` bodies on - customer outages, undercutting the trust and brand experience the page was - built to provide. + customer outages, undercutting the trust and brand experience this proposal + delivers. From f1c2251fbeabb2e55fe53c32a1e74413432dc61c Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 19 Jun 2026 08:21:24 -0500 Subject: [PATCH 4/5] docs: drop redundant Background subheader in Motivation Motivation content reads directly under the section header. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/enhancements/branded-error-pages/README.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/enhancements/branded-error-pages/README.md b/docs/enhancements/branded-error-pages/README.md index d20ddd81..d23ee6a1 100644 --- a/docs/enhancements/branded-error-pages/README.md +++ b/docs/enhancements/branded-error-pages/README.md @@ -7,7 +7,6 @@ stage: alpha - [Summary](#summary) - [Motivation](#motivation) - - [Background: what a customer sees when something breaks](#background-what-a-customer-sees-when-something-breaks) - [Goals](#goals) - [Non-Goals](#non-goals) - [Proposal](#proposal) @@ -44,8 +43,6 @@ operator release, while guaranteeing a valid page even if the override is absent ## Motivation -### Background: what a customer sees when something breaks - Datum's network services route a visitor's request through edge Envoy to a customer's backend — either directly, or, for the **Connector**, through an outbound tunnel to a private origin. When that path can't be completed, Envoy From a02507e8a33801f2c5104764639bc1c9188bac58 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 19 Jun 2026 08:28:18 -0500 Subject: [PATCH 5/5] docs: use GitHub callout syntax for the Design Details note Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/enhancements/branded-error-pages/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/enhancements/branded-error-pages/README.md b/docs/enhancements/branded-error-pages/README.md index d23ee6a1..02e93b9a 100644 --- a/docs/enhancements/branded-error-pages/README.md +++ b/docs/enhancements/branded-error-pages/README.md @@ -124,6 +124,7 @@ Each risk is mitigated by a design choice detailed in the linked section below. ## Design Details +> [!NOTE] > This section stays at the product/architecture level. The translation hook, > security, and re-translation machinery are described in detail in the > [Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md)