docs: warn that Prometheus /metrics must be scraped when enabled (#4660)

## Summary

If `ELECTRIC_PROMETHEUS_PORT` is set but nothing ever scrapes the
`/metrics` endpoint, Electric's memory grows without bound and can
eventually OOM-kill the service. This came out of debugging a customer
whose `prometheus_metrics_dist` ETS table had grown to ~8 GB on an
OTel-only deployment that had the Prometheus port enabled but unscraped.

### Why it happens

`telemetry_metrics_prometheus_core` stores **distribution (histogram)**
metrics in a `:duplicate_bag` and appends one row per observation
(`distribution.ex` `:ets.insert`), only aggregating into buckets — and
draining the bag via `:ets.take` — **at scrape time** (documented in the
library's moduledoc: *"aggregations for distributions (histogram) only
occur at scrape time"*). So if the endpoint is never scraped, the bag is
never drained.

Electric's default Prometheus metric set includes
`electric.postgres.replication.transaction_received.receive_lag`, a
distribution emitted roughly once per replication transaction, so the
table accretes continuously under load (~0.55 GB/day in the customer's
case). Counters/sums/last-values are unaffected — they use bounded
`:set` storage.

This is purely a docs change to make the "must be scraped" requirement
explicit everywhere we mention Prometheus.

## Changes

- **Telemetry reference** (`website/docs/sync/reference/telemetry.md`) —
warning callout in the Metrics section.
- **Config reference** (`website/docs/sync/api/config.md`) — note on
`ELECTRIC_PROMETHEUS_PORT`.
- **Deployment guide** (`website/docs/sync/guides/deployment.md`) —
warning in the Observability section.
- **`prometheus_port` docstring**
(`packages/sync-service/lib/electric.ex`).

## Follow-up (not in this PR)

Worth considering a code-level guard so an enabled-but-unscraped
endpoint can't OOM the service (e.g. move high-frequency distributions
like `receive_lag` out of the default Prometheus set, or run an internal
periodic drain).

I have raised a PR with the library maintainer for a potential fix:
beam-telemetry/telemetry_metrics_prometheus_core#77

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: robacourt

packages/sync-service/lib/electric.ex

@@ -103,7 +103,7 @@ defmodule Electric do
     enable tracking of instance usage metrics across restarts, otherwise will be
     randomly generated at boot (default: a randomly generated UUID).
   - `telemetry_statsd_host` (`t:String.t/0`) - If set, send telemetry data to the given StatsD reporting endpoint (default: `#{default.(:telemetry_statsd_host)}`)
-  - `prometheus_port` (`t:integer/0`) - If set, expose a prometheus reporter for telemetry data on the specified port (default: `#{default.(:prometheus_port)}`)
+  - `prometheus_port` (`t:integer/0`) - If set, expose a prometheus reporter for telemetry data on the specified port (default: `#{default.(:prometheus_port)}`). The `/metrics` endpoint must be scraped regularly when enabled: distribution metrics buffer in memory between scrapes, so an enabled-but-unscraped endpoint grows without bound and can eventually crash the service. Leave unset if nothing is scraping it.
   - `call_home_telemetry?` (`t:boolean/0`) - Allow [anonymous usage
     data](https://electric-sql.com/docs/reference/telemetry#anonymous-usage-data)
     about the instance being sent to a central checkpoint service (default: `true` for production)

website/docs/sync/api/config.md

@@ -615,6 +615,8 @@ Name of your Honeycomb Dataset.
 
 Expose a prometheus reporter for telemetry data on the specified port.
 
+The `/metrics` endpoint exposed on this port **must be scraped regularly** (for example by a Prometheus server). Distribution metrics accumulate in memory between scrapes, so enabling this port without a scraper actively collecting from it causes unbounded memory growth that can eventually crash the service. Leave it unset if nothing is scraping it — for example, if you only use OpenTelemetry.
+
 </EnvVarConfig>
 
 ### ELECTRIC_STATSD_HOST

website/docs/sync/guides/deployment.md

@@ -229,6 +229,9 @@ Electric supports [OpenTelemetry](https://opentelemetry.io/) for exporting trace
 
 See the [Telemetry reference](/docs/sync/reference/telemetry#opentelemetry) for configuration details.
 
+> [!Warning] Scrape the Prometheus endpoint, or leave it disabled
+> If you enable the Prometheus endpoint with `ELECTRIC_PROMETHEUS_PORT`, it **must be scraped regularly**. An enabled-but-unscraped endpoint accumulates metrics in memory without bound and can eventually crash the service. If you only use OpenTelemetry, leave `ELECTRIC_PROMETHEUS_PORT` unset. See the [Telemetry reference](/docs/sync/reference/telemetry#metrics) for details.
+
 ### Caching proxy
 
 Electric is designed to run behind a caching proxy, such as [Nginx](https://nginx.org/en), [Caddy](https://caddyserver.com), [Varnish](https://varnish-cache.org) or a CDN like [Cloudflare](https://www.cloudflare.com/en-gb/application-services/primitives/cdn) or [Fastly](https://www.fastly.com/primitives/cdn). You don't _have_ to run a proxy in front of Electric but you will benefit from radically better performance if you do.

website/docs/sync/reference/telemetry.md

@@ -20,6 +20,11 @@ Metrics are reported in StatsD and Prometheus formats. To configure Electric to
 
 You can get the current status of the service by calling the `http://electric-hostname:PROMETHUES_PORT/metrics` endpoint.
 
+> [!Warning] Only enable Prometheus if you are scraping it
+> When `ELECTRIC_PROMETHEUS_PORT` is set, Electric starts a Prometheus reporter that **must be scraped regularly** — for example by a Prometheus server polling the `/metrics` endpoint on an interval. Distribution (histogram) metrics buffer their observations in memory between scrapes and are only aggregated when the endpoint is scraped. If you enable the port but never scrape it, that buffer grows without bound and will eventually exhaust memory and crash the service.
+>
+> Only set `ELECTRIC_PROMETHEUS_PORT` if a Prometheus-compatible scraper is actively collecting from the endpoint. If you only use [OpenTelemetry](#opentelemetry), leave it unset.
+
 ## OpenTelemetry
 
 Traces are exported using the OpenTelemetry Protocol (OTLP). You can configure the OpenTelemetry Exporter for Electric using the following environment variables.