feat(v1): expose pool env-server stats on an optional Prometheus /metrics endpoint

[dumko2001]

Description

The v1 env-server pool (verifiers/v1/serve/pool.py) — the process that actually serves rollouts during training — has no observability surface today: no stats snapshot, no /metrics, no way to see how many rollouts are in flight or whether a worker is saturated. The only existing metrics work (#1415) targets the legacy v0 verifiers/serve/server/ architecture, so v1 runs blind.

This adds a small, dependency-free Prometheus endpoint for the v1 pool, addressing #1188.

What it does

• New verifiers/v1/serve/metrics.py: a hand-rendered Prometheus text renderer (PoolMetrics) plus a minimal asyncio.start_server-based MetricsServer exposing GET /metrics. No new dependencies.
• The broker loop in EnvServerPool.run() updates cheap, O(1) counters/gauges as it dispatches and reaps requests; the metrics server is started/stopped inside the existing try/finally so its lifecycle is coupled to the broker.
• Opt-in: metrics_port is unset by default (no port → no server). metrics_address defaults to 127.0.0.1 (loopback) rather than binding all interfaces. Both are threaded through ServeConfig and the serve CLI.
• Pool-only: single-server (non-pool) mode is unchanged and logs a warning if metrics are configured.

Metrics exposed

verifiers_v1_env_active_rollouts                      # aggregate in-flight slots
verifiers_v1_env_worker_active_rollouts{worker_index} # per-worker in-flight slots
verifiers_v1_env_workers                              # current worker processes
verifiers_v1_env_configured_workers                   # configured limit (0 = unbounded)
verifiers_v1_env_pending_requests                     # requests awaiting a worker reply
verifiers_v1_env_requests_total                       # dispatched requests (counter)
verifiers_v1_env_request_latency_seconds_{count,sum}  # dispatch->reply latency (summary)

Deliberately scoped to observability: no worker-restart or event-loop-lag gauges, since the pool has no restart controller or lag monitor yet (exposing those would be misleading, and lifecycle work is being handled separately in #1993/#1995). The reply path does no payload decoding, so the broker's zero-copy copy=False forward is preserved.

Type of Change

• New feature (non-breaking change which adds functionality)

Testing

• All existing tests pass when running uv run pytest locally (targeted: tests/v1/test_configs.py, tests/test_env_server.py — 26 passed).
• New tests have been added to cover the changes

Per AGENTS.md ("Do not add unit tests to your PRs"), no unit tests are added. Verified with a throwaway script that starts a real one-worker EnvServerPool, issues a request, and curls the endpoint:

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
verifiers_v1_env_active_rollouts 0
verifiers_v1_env_worker_active_rollouts{worker_index="0"} 0
verifiers_v1_env_workers 1
verifiers_v1_env_configured_workers 1
verifiers_v1_env_pending_requests 0
verifiers_v1_env_requests_total 1
verifiers_v1_env_request_latency_seconds_count 1
verifiers_v1_env_request_latency_seconds_sum 1.157...

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• My changes generate no new warnings

Additional Notes

No new dependencies. Endpoint is disabled unless metrics_port is set, and binds to loopback by default. Related: #1188 (request), #1415 (v0 equivalent).

---

Note

Low Risk
Opt-in metrics on loopback by default; broker hot path only adds O(1) counter updates and does not change rollout protocol or single-server behavior.

Overview
Adds opt-in observability for the v1 EnvServerPool broker via a dependency-free GET /metrics HTTP endpoint.

New metrics.py defines PoolMetrics (Prometheus text) and a minimal MetricsServer on asyncio.start_server. The pool broker updates gauges/counters on dispatch and reply (active rollouts, per-worker load, pending depth, request totals, dispatch→reply latency) without decoding response payloads, so zero-copy forwarding is unchanged. The metrics server starts only when metrics_port is set and shuts down in the existing finally block.

ServeConfig and the serve CLI gain metrics_address (default 127.0.0.1) and metrics_port (unset = disabled), threaded through serve_env → EnvServerPool. Pool-only: single in-process server mode ignores metrics and logs a warning if configured.

^{Reviewed by Cursor Bugbot for commit c942773. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Expose pool env-server stats on an optional Prometheus /metrics endpoint

• Adds MetricsServer, a minimal asyncio HTTP server serving Prometheus text format at /metrics, and PoolMetrics, a model tracking active rollouts, workers, pending depth, request counts, and latency summaries in verifiers/v1/serve/metrics.py.
• EnvServerPool optionally starts MetricsServer when metrics_port is configured, updating metrics on every dispatch and reply, and shuts it down cleanly on exit.
• ServeConfig gains two new fields: metrics_address (default "0.0.0.0") and metrics_port (default None; unset disables metrics entirely).
• Metrics are only available in multi-worker pool mode; single-server mode is unchanged.

^{Macroscope summarized 65836a6.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c942773634

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bound idle HTTP connections before awaiting a request line

When metrics are enabled and reachable by an untrusted process (for example, with metrics_address=0.0.0.0), each client can open a TCP connection and send no newline, leaving this readline() task and its socket alive indefinitely. Enough idle connections exhaust the broker process's file descriptors/memory and can disrupt worker scaling or new service connections; apply a short request-read timeout and/or a connection limit.

Useful? React with 👍 / 👎.

[dumko2001]

Heads up on sequencing: this touches pool.py, which #1993 (worker-exit recovery) and #1995 (subprocess reaping) are also reworking. Happy to rebase on top of whichever lands first — the metrics here are additive and opt-in, so they should slot in without touching the lifecycle changes.