Commit 044a8d5
fix(electric-telemetry): Fix process_type parsing for request processes that have a request id that is not the standard 20 bytes. (#4658)
## Summary
Fixes an unbounded `process_type` cardinality leak in telemetry for
request-handling processes.
## Problem
`process_type` for request-handling processes is derived from the
process label set by `Electric.Plug.LabelProcessPlug`, which looks like:
```
Request F-jPUudNHxbD8lIAABQG - GET /v1/shape?table=users&offset=-1
```
`parse_binary_label/1` is meant to strip the request id and query,
collapsing this to a bounded `GET /v1/shape`. But it did so with a
fixed-width binary match that assumes the request id is exactly 20
bytes:
```elixir
"Request " <> <<_req_id::binary-20, " - ", rest::binary>> -> ...
```
That holds for ids `Plug.RequestId` self-generates, but `Plug.RequestId`
reuses an incoming `x-request-id` header when a client/proxy supplies
one. A UUID (36 chars), an Envoy/nginx id, or an LB trace id is not 20
bytes, so the clause falls through to the generic branch that returns
the first 20 bytes of the whole label — i.e. `Request <first 12 chars of
the id>`. Every distinct incoming request id that GCs slowly then adds a
permanent new series to `prometheus_metrics_dist` for events like
`vm.monitor.long_gc` and `vm.monitor.long_schedule`. Request handlers
building large shape responses are exactly the processes that trip those
monitors, so the table grows without bound when a fronting proxy injects
`x-request-id`.
## Solution
Keep the fast fixed-width binary match for the common 20-byte case
(`Plug.RequestId`'s self-generated ids), and add a slow path that only
runs when the id is a different length: it splits on the `" - "`
delimiter regardless of request-id length, then strips the query string.
All requests to a route collapse to a single `process_type` (e.g. `GET
/v1/shape`) regardless of request-id source or length. A `"Request "`
label with no delimiter falls back to truncation.
---
## Also includes: unrelated CI fix
CI for this PR was failing on `Electric.ShapeCacheTest` (in
`sync-service`, unrelated to the telemetry change). The cause is a
semantic merge collision already on `main`: #4635 changed `wait_until`'s
signature to `wait_until(fun, timeout_ms)` (function first), while #4651
added a new call site at `shape_cache_test.exs:1582` using the old arg
order `wait_until(1000, fn -> ... end)`. The two merged cleanly in git
but not in meaning, so the slow suite raised a `FunctionClauseError` on
every run. This PR swaps the arguments to match the current signature,
unblocking both `main` and this PR.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent 0130c4a commit 044a8d5
4 files changed
Lines changed: 65 additions & 19 deletions
File tree
- .changeset
- packages
- electric-telemetry
- lib/electric/telemetry
- test/electric/telemetry
- sync-service/test/electric
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
Lines changed: 32 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
262 | 262 | | |
263 | 263 | | |
264 | 264 | | |
| 265 | + | |
265 | 266 | | |
266 | 267 | | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
267 | 278 | | |
268 | | - | |
269 | | - | |
270 | | - | |
271 | | - | |
272 | | - | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
273 | 283 | | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
278 | 290 | | |
279 | | - | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
280 | 295 | | |
281 | | - | |
282 | | - | |
283 | | - | |
284 | | - | |
285 | | - | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
286 | 303 | | |
287 | 304 | | |
Lines changed: 21 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
19 | 40 | | |
20 | 41 | | |
21 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1579 | 1579 | | |
1580 | 1580 | | |
1581 | 1581 | | |
1582 | | - | |
1583 | | - | |
1584 | | - | |
1585 | | - | |
| 1582 | + | |
| 1583 | + | |
| 1584 | + | |
| 1585 | + | |
| 1586 | + | |
| 1587 | + | |
| 1588 | + | |
1586 | 1589 | | |
1587 | 1590 | | |
1588 | 1591 | | |
| |||
0 commit comments