|
| 1 | +--- |
| 2 | +title: "Webhook Forward Proxy" |
| 3 | +description: "Route outgoing webhook deliveries through an HTTP forward proxy for static-IP egress or network isolation." |
| 4 | +--- |
| 5 | + |
| 6 | +Outpost can route outgoing webhook traffic through an HTTP forward proxy. Common reasons: |
| 7 | + |
| 8 | +- **Static-IP egress** — destinations that allowlist a specific source IP |
| 9 | +- **Network isolation** — keep delivery workers off the public internet |
| 10 | +- **Centralized egress policy** — single chokepoint for outbound traffic |
| 11 | + |
| 12 | +Proxy support is operator-configured and applies to every webhook destination served by the deployment. |
| 13 | + |
| 14 | +## Configuration |
| 15 | + |
| 16 | +| Env var | Description | |
| 17 | +|---------|-------------| |
| 18 | +| `DESTINATIONS_WEBHOOK_PROXY_URL` | Proxy URL, e.g. `http://user:pass@proxy.example.com:8080`. Supports basic auth. | |
| 19 | + |
| 20 | +When `DESTINATIONS_WEBHOOK_PROXY_URL` is set, Outpost installs an HTTP proxy on the webhook publisher's transport. HTTPS destinations use the standard `CONNECT` tunneling flow; HTTP destinations are forwarded request-by-request. |
| 21 | + |
| 22 | +## Error handling |
| 23 | + |
| 24 | +Putting a forward proxy in the delivery path introduces a new failure surface that should not be charged to the destination. A proxy-auth misconfiguration or a proxy outage isn't the destination's fault, and recording it as a failed delivery attempt burns retry budget on a problem the destination cannot resolve. |
| 25 | + |
| 26 | +Outpost distinguishes **proxy infrastructure failures** from **destination failures** (including destination failures that the proxy merely *reports* on the destination's behalf) and applies the right behavior to each. The base behavior below applies to any forward proxy; Envoy-specific signals are picked up automatically when present (see [Envoy support](#envoy-support)). |
| 27 | + |
| 28 | +### General behavior |
| 29 | + |
| 30 | +| Scenario | Attribution | Behavior | |
| 31 | +|---|---|---| |
| 32 | +| Proxy returns 407 / 401 / 403 on `CONNECT` | Infra | **Nack** — operator misconfiguration of proxy credentials | |
| 33 | +| Proxy unreachable (TCP dial to proxy fails) | Infra | **Nack** — proxy infrastructure outage | |
| 34 | +| `CONNECT` succeeds, destination returns real 4xx/5xx (HTTPS) | Destination | Record attempt with the actual status code | |
| 35 | +| `CONNECT` succeeds, destination TLS handshake fails (HTTPS) | Destination | Record attempt (`tls_error`) | |
| 36 | +| `CONNECT` succeeds, destination times out | Destination | Record attempt (`timeout`) | |
| 37 | +| Proxy returns other 5xx on `CONNECT` (cannot reach destination) | Destination | Record attempt (`connection_refused`) | |
| 38 | +| Real upstream response passed through (plain-HTTP) | Destination | Record as today | |
| 39 | +| Proxy itself overloaded or misbehaving (rare) | Destination (conservative) | Record attempt (`network_error`) — never nack on speculation | |
| 40 | + |
| 41 | +**Key principle:** when the proxy reports a failure that originated at the destination (DNS, connect refused, upstream timeout), the customer still sees it as a destination failure. Outpost rewrites the message so the response data is destination-attributed, not proxy-attributed. Nacking is reserved for cases where the proxy itself is the proximate cause. |
| 42 | + |
| 43 | +**HTTPS responses are byte-transparent.** Once the `CONNECT` tunnel opens, TLS runs end-to-end between Outpost and the destination; the forward proxy can no longer read or modify response bytes. Outpost therefore does not inspect or sanitize HTTPS response payloads — they are recorded as the destination sent them. Proxy-originated HTTPS failures (auth, unreachable, can't connect upstream) all happen at `CONNECT` time and are handled before the tunnel exists. |
| 44 | + |
| 45 | +Response-body and response-header sanitization on the plain-HTTP forwarding path is best-effort and depends on the proxy implementation being recognized. For an arbitrary forward proxy, Outpost can rewrite error messages but cannot reliably strip proxy-identifying response content. Sanitization is currently complete only for Envoy — see [Envoy support](#envoy-support). |
| 46 | + |
| 47 | +## Envoy support |
| 48 | + |
| 49 | +When the proxy is [Envoy](https://www.envoyproxy.io/), Outpost picks up Envoy-specific signals automatically — no configuration toggle required. These are additive on top of the general behavior above. |
| 50 | + |
| 51 | +### Additional behaviors |
| 52 | + |
| 53 | +Envoy-specific handling fires on two surfaces: the `CONNECT` response (HTTPS, where the proxy's response is visible to Outpost before the tunnel opens), and proxied plain-HTTP responses (where the proxy is in the byte path on the way back). Responses that arrived through an established `CONNECT` tunnel are not inspected — see the byte-transparency note above. |
| 54 | + |
| 55 | +| Scenario | Signal | Attribution | Behavior | |
| 56 | +|---|---|---|---| |
| 57 | +| Envoy `CONNECT` failure with response-flag header | `x-envoy-response-flags` on the `CONNECT` response | Destination | Record attempt with code refined from the flag (e.g. `DF` → `dns_error`, `UT` → `timeout`) instead of the generic `connection_refused` | |
| 58 | +| Envoy synthesizes 5xx response (plain-HTTP path) | `x-envoy-response-flags: UF` / `UC` / etc. on the response | Destination | Record attempt with code mapped from the flag; response body is dropped | |
| 59 | +| Real upstream response passed through (plain-HTTP) | `x-envoy-response-flags: -` or empty | Destination | Record as today | |
| 60 | +| Successful plain-HTTP response served via Envoy | `x-envoy-*` and `server: envoy` headers present | Destination | Record; headers stripped before storage | |
| 61 | +| HTTPS response (any status, any headers, post-`CONNECT`) | — | Destination | Pass-through unchanged — bytes never touched the forward proxy | |
| 62 | + |
| 63 | +Two Envoy-specific behaviors are layered on top, both scoped to surfaces where the forward proxy actually contributed to the bytes: |
| 64 | + |
| 65 | +- **Response-flag mapping** — `x-envoy-response-flags`, when present and non-empty, refines the destination error code. Mapping aligns with Outpost's non-proxy error vocabulary (`ClassifyNetworkError`), so customers see the same codes whether or not a proxy is in path: |
| 66 | + |
| 67 | + | Envoy flag | Outpost code | Meaning | |
| 68 | + |---|---|---| |
| 69 | + | `UF`, `UH`, `LH` | `connection_refused` | TCP dial failed / no healthy upstream / failed health check | |
| 70 | + | `UC`, `UR`, `LR` | `connection_reset` | Established connection dropped / remote or local reset | |
| 71 | + | `UT`, `SI`, `DT`, `UMSDR` | `timeout` | Upstream / stream-idle / duration / max-stream timeout | |
| 72 | + | `DF` | `dns_error` | DNS resolution failure (`dynamic_forward_proxy`) | |
| 73 | + | `NR`, `NC` | `network_unreachable` | No route / no cluster | |
| 74 | + | `UPE`, `DPE` | `protocol_error` | Upstream / downstream protocol error | |
| 75 | + | (any other flag) | `network_error` | Unmapped — operator-visible signal to expand the table | |
| 76 | + |
| 77 | + Applies to `CONNECT` responses and plain-HTTP responses, not to bytes returned through an HTTPS tunnel. |
| 78 | + |
| 79 | +- **Operator diagnostics** — when a flag fires, the raw `x-envoy-response-flags` value and `x-envoy-response-code-details` (the `stage{reason}` string, e.g. `upstream_reset_before_response_started{connection_timeout}`) are written into a generic proxy-diagnostics map on the error as `envoy_flag` / `envoy_details`. These surface in the underlying error message and on the publish-attempt error payload, visible in `consumer handler error` logs — but never written to the customer-visible attempt `response_data`, which mirrors a normal network failure (no status, no body). The map is intentionally untyped so other proxies (Squid, HAProxy) can populate their own keys without colliding. To enable this, the Envoy ref config emits the details header alongside the flag header. |
| 80 | + |
| 81 | +- **Header and body sanitization** — `x-envoy-*` and `server: envoy` headers are stripped from plain-HTTP responses; Envoy-synthesized plain-HTTP response bodies are replaced with a normalized message that does not leak Envoy. HTTPS responses are not sanitized. |
| 82 | + |
| 83 | +Support for other forward proxies (Squid, HAProxy, nginx, ...) can be added the same way — by detecting proxy-specific response signals and mapping them to destination error codes. None are currently implemented. |
| 84 | + |
| 85 | +#### Limitation: plain-HTTP destinations that are themselves behind Envoy |
| 86 | + |
| 87 | +If a destination is reached over plain HTTP and the destination itself sits behind its own Envoy edge, the destination's `x-envoy-*` headers (e.g. `x-envoy-upstream-service-time`, `server: envoy`) pass through the forward proxy and Outpost strips them. The destination's `x-envoy-response-flags` is overwritten by the forward Envoy's value, so attribution is still correct — the customer never sees a destination failure misattributed to the proxy — but some destination-side observability headers are lost on this code path. |
| 88 | + |
| 89 | +This does not affect HTTPS destinations: HTTPS responses are byte-transparent (see above), so any `x-envoy-*` headers from the destination's Envoy reach Outpost untouched. |
| 90 | + |
| 91 | +### Required Envoy configuration |
| 92 | + |
| 93 | +For Outpost to reliably distinguish Envoy-synthesized responses from real upstream responses, Envoy must emit its response flags as a response header. The response-code-details header is optional but recommended — without it, Outpost still classifies via the flag, but operators lose the precise stage/reason in logs. Add to the route configuration (Envoy rejects these fields on the HTTP connection manager — they belong on `RouteConfiguration`): |
| 94 | + |
| 95 | +```yaml |
| 96 | +route_config: |
| 97 | + response_headers_to_add: |
| 98 | + - header: |
| 99 | + key: "x-envoy-response-flags" |
| 100 | + value: "%RESPONSE_FLAGS%" |
| 101 | + append_action: OVERWRITE_IF_EXISTS_OR_ADD |
| 102 | + - header: |
| 103 | + key: "x-envoy-response-code-details" |
| 104 | + value: "%RESPONSE_CODE_DETAILS%" |
| 105 | + append_action: OVERWRITE_IF_EXISTS_OR_ADD |
| 106 | + virtual_hosts: |
| 107 | + # ... |
| 108 | +``` |
| 109 | + |
| 110 | +`OVERWRITE_IF_EXISTS_OR_ADD` is important — it prevents a misbehaving destination from spoofing either header to confuse Outpost's classification or pollute operator diagnostics. |
| 111 | + |
| 112 | +### Minimal reference Envoy |
| 113 | + |
| 114 | +A minimal forward-proxy Envoy listener with response-flag reporting enabled: |
| 115 | + |
| 116 | +```yaml |
| 117 | +admin: |
| 118 | + address: |
| 119 | + socket_address: { address: 127.0.0.1, port_value: 9901 } |
| 120 | + |
| 121 | +static_resources: |
| 122 | + listeners: |
| 123 | + - name: forward_proxy |
| 124 | + address: |
| 125 | + socket_address: { address: 0.0.0.0, port_value: 10000 } |
| 126 | + filter_chains: |
| 127 | + - filters: |
| 128 | + - name: envoy.filters.network.http_connection_manager |
| 129 | + typed_config: |
| 130 | + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager |
| 131 | + stat_prefix: forward_proxy |
| 132 | + codec_type: AUTO |
| 133 | + http_filters: |
| 134 | + - name: envoy.filters.http.dynamic_forward_proxy |
| 135 | + typed_config: |
| 136 | + "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig |
| 137 | + dns_cache_config: |
| 138 | + name: dfp_cache |
| 139 | + dns_lookup_family: V4_ONLY |
| 140 | + - name: envoy.filters.http.router |
| 141 | + typed_config: |
| 142 | + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router |
| 143 | + upgrade_configs: |
| 144 | + - upgrade_type: CONNECT |
| 145 | + route_config: |
| 146 | + response_headers_to_add: |
| 147 | + - header: |
| 148 | + key: "x-envoy-response-flags" |
| 149 | + value: "%RESPONSE_FLAGS%" |
| 150 | + append_action: OVERWRITE_IF_EXISTS_OR_ADD |
| 151 | + - header: |
| 152 | + key: "x-envoy-response-code-details" |
| 153 | + value: "%RESPONSE_CODE_DETAILS%" |
| 154 | + append_action: OVERWRITE_IF_EXISTS_OR_ADD |
| 155 | + virtual_hosts: |
| 156 | + - name: proxy |
| 157 | + domains: ["*"] |
| 158 | + routes: |
| 159 | + - match: { connect_matcher: {} } |
| 160 | + route: |
| 161 | + cluster: dfp_cluster |
| 162 | + upgrade_configs: |
| 163 | + - upgrade_type: CONNECT |
| 164 | + connect_config: {} |
| 165 | + - match: { prefix: "/" } |
| 166 | + route: { cluster: dfp_cluster } |
| 167 | + |
| 168 | + clusters: |
| 169 | + - name: dfp_cluster |
| 170 | + lb_policy: CLUSTER_PROVIDED |
| 171 | + cluster_type: |
| 172 | + name: envoy.clusters.dynamic_forward_proxy |
| 173 | + typed_config: |
| 174 | + "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig |
| 175 | + dns_cache_config: |
| 176 | + name: dfp_cache |
| 177 | + dns_lookup_family: V4_ONLY |
| 178 | +``` |
| 179 | + |
| 180 | +The example above is a minimal listener. Whether you need proxy authentication and TLS depends on your network topology: if Outpost and the proxy share a private network, neither is strictly required; if the proxy is reachable over the public internet, both are strongly recommended to prevent the proxy being used as an open relay. |
| 181 | + |
| 182 | +## Queue retry behavior |
| 183 | + |
| 184 | +When a proxy infrastructure error is nacked, the underlying message queue redelivers the event. Because nacks only fire for true infra failures (proxy auth or proxy unreachable), the redelivery rate is bounded by the proxy outage duration, not by destination behavior. |
| 185 | + |
| 186 | +Outpost's default GCP Pub/Sub provisioning (`internal/mqinfra/gcppubsub.go`) uses `retryPolicy: {minimumBackoff: 10s, maximumBackoff: 120s}` with `maxDeliveryAttempts: 6` (default `RetryLimit` + 1). That gives roughly **5 minutes of redelivery runway** before a nacked message lands in the dead-letter topic. A proxy outage shorter than this window is transparent to the destination; longer outages require manual replay from the DLQ. |
| 187 | + |
| 188 | +If you expect longer proxy outages, raise `MinRetryBackoff` / `MaxRetryBackoff` (config) and `RetryLimit` (policy) so the redelivery window covers your worst-case outage duration. RabbitMQ / SQS / Kafka have equivalent knobs on their delivery queues. |
0 commit comments