Skip to content

Commit 3b2913a

Browse files
scotwellsclaude
scotwells
and
claude
committed
docs: add branded data-plane error pages enhancement proposal
Product-focused proposal to restore the branded HTML error page on the downstream/Connector data plane. The page was delivered via an EnvoyPatchPolicy (local-reply-5xx) that went inert when enableEnvoyPatchPolicy=false was set during the extension-server migration, so customer outages (e.g. offline Connector) now render raw Envoy bodies like "no healthy upstream" instead of the branded page. Proposes delivering the page through the extension server's per-listener mutation (the supported path now that EnvoyPatchPolicy is disabled there), sourcing content from a mounted ConfigMap with a compiled-in fallback so content edits are a GitOps change and a missing/bad page can never block the fail-closed data plane. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent de24148 commit 3b2913a

1 file changed

Lines changed: 302 additions & 0 deletions

File tree

  • docs/enhancements/branded-error-pages

‎docs/enhancements/branded-error-pages/README.md‎

Lines changed: 302 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,302 @@
1+
---
2+
status: provisional
3+
stage: alpha
4+
---
5+
6+
# Branded Data-Plane Error Pages
7+
8+
- [Summary](#summary)
9+
- [Motivation](#motivation)
10+
- [Background: what a customer sees when something breaks](#background-what-a-customer-sees-when-something-breaks)
11+
- [The regression: the branded page went silently dark](#the-regression-the-branded-page-went-silently-dark)
12+
- [Goals](#goals)
13+
- [Non-Goals](#non-goals)
14+
- [Proposal](#proposal)
15+
- [User Stories](#user-stories)
16+
- [Risks and Mitigations](#risks-and-mitigations)
17+
- [Design Details](#design-details)
18+
- [Where the page is served](#where-the-page-is-served)
19+
- [Which responses get the page](#which-responses-get-the-page)
20+
- [Content ownership: a ConfigMap, with a baked-in fallback](#content-ownership-a-configmap-with-a-baked-in-fallback)
21+
- [Updating the page](#updating-the-page)
22+
- [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message)
23+
- [Verifying the experience](#verifying-the-experience)
24+
- [Transition Plan](#transition-plan)
25+
- [Drawbacks](#drawbacks)
26+
- [Alternatives](#alternatives)
27+
28+
## Summary
29+
30+
When a customer's site is reached through Datum's edge — a Connector-backed
31+
proxy, or any Gateway on the downstream data plane — and something goes wrong
32+
(the tunnel is offline, the origin is down, a request times out), the visitor's
33+
browser gets a response generated by Envoy, not by the customer's application.
34+
Today, on the downstream/Connector data plane, that response is a **raw,
35+
unstyled error**: an HTTP 503 with a bare body like `no healthy upstream` and no
36+
indication of where it came from or what to do next.
37+
38+
Datum already authored a **branded HTML error page** for exactly this moment. It
39+
was delivered through an `EnvoyPatchPolicy`, but that mechanism was switched off
40+
on the downstream data plane as part of the
41+
[Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md)
42+
migration. As a result the branded page has been **inert for months** on the
43+
Connector data plane, and visitors have been seeing the raw Envoy output instead.
44+
45+
This enhancement restores the branded error experience by moving its delivery
46+
into the extension server (the supported path now that `EnvoyPatchPolicy` is
47+
disabled there), and makes the page's **content** a configuration artifact — a
48+
mounted `ConfigMap` — so it can be updated by GitOps without a code release.
49+
50+
## Motivation
51+
52+
### Background: what a customer sees when something breaks
53+
54+
Datum's network services route a visitor's request through edge Envoy to a
55+
customer's backend — either directly, or, for the **Connector**, through an
56+
outbound tunnel to a private origin. When that path can't be completed, Envoy
57+
itself returns the response. Common cases on the Connector data plane:
58+
59+
- The Connector tunnel is **offline** (agent not running / not yet connected).
60+
- The origin behind the tunnel or backend is **down or unreachable**.
61+
- The request **times out** or trips a circuit breaker.
62+
63+
In all of these, the visitor never reaches the customer's app, so the customer's
64+
own error page can't be shown. The edge is the only thing that can speak — and
65+
what it says, and how it looks, is Datum's responsibility. A blank `503` with
66+
`no healthy upstream` reads like a broken platform; a branded page that says
67+
"this site is temporarily unavailable" with Datum's identity reads like a
68+
platform that's working as intended and tells you what's happening.
69+
70+
This is a **trust and brand-perception** surface. It is the page a customer's
71+
*own* end-users see when the customer is mid-incident — precisely when the
72+
experience matters most.
73+
74+
### The regression: the branded page went silently dark
75+
76+
Datum built the branded page (an `<!doctype html>` document, served as
77+
`text/html`) and shipped it to the downstream gateway as an `EnvoyPatchPolicy`
78+
named `local-reply-5xx`, matching any response with status ≥ 500.
79+
80+
The [extension-server migration](../envoy-gateway-extension-server/README.md)
81+
intentionally set `enableEnvoyPatchPolicy: false` on the downstream Envoy Gateway
82+
— that was the whole point of moving WAF and Connector logic off
83+
`EnvoyPatchPolicy`. But the branded error page was **never migrated** to the
84+
extension server. Disabling `EnvoyPatchPolicy` disabled *every* such policy on
85+
that data plane, including the error page, with no signal to a customer or
86+
operator that the experience had degraded. The policy still exists in Git and on
87+
the cluster; it is simply `Accepted=False` and never reaches Envoy.
88+
89+
Net effect: on the Connector / downstream data plane, every edge-generated 5xx —
90+
including the very common "Connector is offline" case — is served as a bare
91+
Envoy body. The branded experience that product intended is not being delivered.
92+
93+
### Goals
94+
95+
- Restore the **branded HTML error page** for edge-generated 5xx responses on the
96+
downstream / Connector data plane, at parity with what the `EnvoyPatchPolicy`
97+
delivered.
98+
- Make it work **without `EnvoyPatchPolicy`**, using the extension server, which
99+
is the supported customization path on that data plane.
100+
- Cover **all** customer-facing listeners on that data plane, including HTTP/3 /
101+
QUIC, so the experience is consistent regardless of how the browser connects.
102+
- Let the page's **content be updated independently of code** — a content edit
103+
should be a GitOps change, not an operator image build and release.
104+
- **Fail safe**: a missing or malformed page must never take the data plane down
105+
or block configuration updates.
106+
107+
### Non-Goals
108+
109+
- Changing **which** status codes are considered errors, or the HTTP status code
110+
returned. We brand the body; we do not alter routing or response codes.
111+
- Per-customer or per-Gateway custom error pages. This proposal delivers one
112+
Datum-branded page across the data plane. Per-tenant theming is possible future
113+
work and is called out as a non-goal here.
114+
- The **standard** (non-downstream) gateways, which still run `EnvoyPatchPolicy`
115+
and already serve the branded page. They are out of scope and unchanged.
116+
- Replacing application-level error pages. When the customer's origin is
117+
reachable, its own responses are returned unchanged.
118+
119+
## Proposal
120+
121+
Deliver the branded error page through the extension server's existing
122+
per-listener mutation pass. During Envoy Gateway's translation, the extension
123+
server already rewrites listeners (for the WAF) and routes/clusters (for the
124+
Connector). We add one more listener mutation: attach a **local reply
125+
configuration** that renders the branded HTML for any response with status
126+
≥ 500. The page's bytes come from a mounted `ConfigMap`, with a copy compiled
127+
into the operator as an always-valid fallback.
128+
129+
### User Stories
130+
131+
- **As a visitor to a customer's site that is temporarily unavailable**, I see a
132+
clear, branded "temporarily unavailable" page instead of a blank or cryptic
133+
error, so I understand the site (not my browser) has a problem and that it's
134+
likely transient.
135+
- **As a Datum customer whose Connector is offline**, when my users hit my proxy
136+
they get a polished, on-brand page rather than `no healthy upstream`, so an
137+
outage on my side doesn't look like a broken platform.
138+
- **As Datum's brand/design owner**, I can update the wording and styling of the
139+
edge error page by changing a single content artifact in Git, and have it roll
140+
out across the edge without waiting on an operator release.
141+
- **As a Datum operator**, I can confirm the branded page is actually being
142+
served, and I'm confident that a bad content edit can't break traffic — the
143+
edge falls back to a known-good page.
144+
145+
### Risks and Mitigations
146+
147+
| Risk | Mitigation |
148+
| --- | --- |
149+
| A missing/empty/typo'd `ConfigMap` removes the page or, worse, blocks config updates (the downstream extension hook is fail-closed). | The operator ships an **embedded default** page and treats the `ConfigMap` purely as an *override*. The mount is `optional`. If the override is absent or unreadable, the embedded page is used; the hook never errors on content. |
150+
| A content edit appears not to take effect. | Editing a mounted `ConfigMap` does not by itself reach Envoy; the new content only applies on the next Envoy Gateway translation. We make content changes **roll the extension server** (which forces a fresh translation) so propagation is deterministic. See [Updating the page](#updating-the-page). |
151+
| The branded page overrides the Connector's terse "tunnel offline" signal used by tunnel clients. | The page is scoped to browser-facing responses; the tunnel-control response is preserved or treated as a deliberate, documented decision. See [Interaction with the Connector "tunnel offline" message](#interaction-with-the-connector-tunnel-offline-message). |
152+
| Drift between the embedded fallback and the live `ConfigMap`. | The `ConfigMap` is the single source of truth in infra; the embedded copy is a minimal, rarely-changed safety net, not a second design to maintain. |
153+
154+
## Design Details
155+
156+
> This section stays at the product/architecture level. The translation hook,
157+
> security, and re-translation machinery are described in detail in the
158+
> [Envoy Gateway Extension Server](../envoy-gateway-extension-server/README.md)
159+
> enhancement; this proposal reuses them.
160+
161+
### Where the page is served
162+
163+
The extension server already participates in every Envoy Gateway translation for
164+
the downstream data plane and already mutates listeners. We add a per-listener
165+
step that attaches Envoy's **local reply** configuration to each customer-facing
166+
HTTP connection manager. "Local reply" is Envoy's term for a response Envoy
167+
generates itself (an upstream failure, a timeout, a direct response) — exactly
168+
the responses a customer's app can't answer. Attaching the branded body there
169+
means every such response on that listener renders the page.
170+
171+
Because the step runs over **all** listeners the translation produces, it
172+
naturally covers both the TLS (HTTP/1.1 + HTTP/2) and the QUIC / HTTP/3
173+
listeners. The previous `EnvoyPatchPolicy` had to hand-target each listener by
174+
name and was extended specifically to cover HTTP/3; that maintenance burden and
175+
class of "we forgot a listener" bug goes away.
176+
177+
Internal, non-customer listeners (for example Envoy Gateway's own readiness
178+
endpoint) are **excluded** by construction, so health checks and infrastructure
179+
endpoints are never branded.
180+
181+
### Which responses get the page
182+
183+
Parity with the original policy: **any edge-generated response with status
184+
≥ 500** is rendered as the branded page (configurable threshold). The original
185+
HTTP status code is preserved — a 503 stays a 503 — only the body and
186+
content-type are replaced. A runtime switch is retained so the behavior can be
187+
disabled in an emergency without a redeploy.
188+
189+
### Content ownership: a ConfigMap, with a baked-in fallback
190+
191+
The page is **content**, owned by brand/design, not by operator code. So:
192+
193+
- The HTML lives in a **`ConfigMap`** managed in the infrastructure (GitOps)
194+
repository — the same place the page lives today, expressed as configuration
195+
instead of an `EnvoyPatchPolicy`.
196+
- The extension server **mounts** that `ConfigMap` to disk (an `optional`
197+
volume) and reads the file at startup.
198+
- The operator also carries a **compiled-in default** page. If the mounted
199+
override is missing or unreadable, the default is used.
200+
201+
This split gives us the best of both: content is editable by GitOps without an
202+
operator release, and there is always a valid page even if the override is
203+
absent — which is essential because the downstream extension hook is fail-closed
204+
(a hard error would otherwise stall configuration for the whole data plane).
205+
206+
A minimal operator configuration block selects the behavior; the bytes come from
207+
the mount or the embedded fallback:
208+
209+
```yaml
210+
gateway:
211+
errorPage:
212+
enabled: true
213+
bodyPath: /etc/datum/error-pages/error-5xx.html # optional; falls back to embedded default
214+
minStatusCode: 500
215+
```
216+
217+
### Updating the page
218+
219+
A `ConfigMap` edit lands on disk but does **not** automatically reach Envoy: the
220+
extension server only influences Envoy during a translation, and Envoy Gateway
221+
does not watch the extension server's mounted files. So a content change is only
222+
applied on the **next** translation.
223+
224+
To make updates deterministic, a content change **rolls the extension server**
225+
(via a content checksum on the Deployment or a reload controller). The restart
226+
re-reads the file and the extension server's reconnection drives a fresh
227+
translation, so the new page propagates predictably across the edge. This mirrors
228+
the broader re-translation considerations covered in the extension-server
229+
enhancement.
230+
231+
### Interaction with the Connector "tunnel offline" message
232+
233+
The Connector path has a second, separate signal: a terse `503 "Tunnel not
234+
online"` returned on the tunnel-control (HTTP `CONNECT`) path, used by tunnel
235+
clients rather than browsers. Normal end-users never traverse that path — their
236+
browser requests are ordinary `GET`s — so the **visitor-facing** experience is
237+
governed entirely by the branded page described here.
238+
239+
Because the branded page matches any response ≥ 500, it would, if applied
240+
indiscriminately, also rewrite that tunnel-control 503. Two acceptable options:
241+
242+
1. **Brand everything ≥ 500** (simplest; matches the original "brand any 5xx"
243+
intent). Tunnel clients are robust to body changes and key on status, not body.
244+
2. **Preserve the terse tunnel-control body** by scoping the page to
245+
visitor-facing responses.
246+
247+
We default to (1) for simplicity and revisit only if a tunnel client needs the
248+
terse body. Either way, the visitor experience — the thing this proposal is
249+
about — is the branded page.
250+
251+
### Verifying the experience
252+
253+
- A browser request to a proxy whose Connector is **offline** returns a `503`
254+
whose body is the **branded HTML** (correct `content-type: text/html`), not
255+
`no healthy upstream`.
256+
- The same holds for an **origin-down** backend and for a **timeout**.
257+
- Envoy Gateway's own **readiness** endpoint and other internal listeners are
258+
**not** branded.
259+
- Editing the `ConfigMap` and letting the rollout complete changes the served
260+
page; removing the `ConfigMap` falls back to the embedded default with no
261+
traffic impact.
262+
- HTTP/3 / QUIC clients get the same branded page as HTTP/2 clients.
263+
264+
### Transition Plan
265+
266+
1. Ship the extension-server delivery and embedded fallback (page renders from
267+
the baked-in default even before any `ConfigMap` exists).
268+
2. Move the existing HTML from the `local-reply-5xx` `EnvoyPatchPolicy` into the
269+
`ConfigMap`; wire the mount and the content-change rollout.
270+
3. Verify the branded page on the downstream data plane in staging, then
271+
production, using the checks above.
272+
4. Decommission the now-redundant downstream `local-reply-5xx` `EnvoyPatchPolicy`
273+
app. The standard-gateway error page (still `EnvoyPatchPolicy`-based) is left
274+
untouched.
275+
276+
## Drawbacks
277+
278+
- It adds one more responsibility to the extension server, which is on the
279+
critical translation path for the data plane. This is mitigated by the
280+
embedded fallback and fail-safe handling, and the work is small and isolated.
281+
- Content updates are not instantaneous: they require a translation, which we
282+
trigger via a controlled rollout. This is inherent to the extension-server
283+
model and is an acceptable trade for not coupling content to code releases.
284+
- Two copies of the page exist (the live `ConfigMap` and the embedded fallback).
285+
The fallback is intentionally minimal to keep this cost low.
286+
287+
## Alternatives
288+
289+
- **Re-enable `EnvoyPatchPolicy` just for the error page.** Rejected: it
290+
reintroduces the exact mechanism the data plane deliberately moved off, and the
291+
scaling/operability reasons for disabling it still apply.
292+
- **Bake the page only into the operator image (no `ConfigMap`).** Simpler, and a
293+
reasonable starting point, but every wording or styling tweak becomes an
294+
operator release. The `ConfigMap` keeps content a GitOps concern; the embedded
295+
copy is retained as a fallback, not as the primary source.
296+
- **Serve the error page from a separate service** the edge routes to on failure.
297+
More moving parts, a new failure domain on the unhappy path, and added latency
298+
exactly when the system is already degraded. The local-reply approach keeps the
299+
page entirely within the edge proxy.
300+
- **Do nothing.** Visitors continue to see raw `no healthy upstream` bodies on
301+
customer outages, undercutting the trust and brand experience the page was
302+
built to provide.

0 commit comments

Comments
 (0)