starknet_transaction_prover: /health returns 503 when service is saturated

[avi-starkware]

Adds SaturationMonitor (shared by ProvingRpcServerImpl and
HealthLayer) that tracks whether the concurrency semaphore has been
continuously rejecting proving requests. Once that has held for the
configured window (health_max_saturated_ms, default 10s), /health
returns 503 with an opaque body so load balancers can drain the pod
before in-flight requests start failing.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

[cursor]

PR Summary

Medium Risk
Changes load-balancer health semantics and pod draining behavior; logic is localized to admission/health with tests, but mis-tuned thresholds could drain pods prematurely or leave saturated pods marked healthy.

Overview
Adds SaturationMonitor, shared between ProvingRpcServerImpl and HealthLayer, so sustained concurrency shedding (queue full or worker wait timeout) can mark the pod unhealthy for load balancers.

/health still short-circuits GET /health, but now returns 503 with an opaque {"status":"unhealthy","reason":"saturated"} body once rejections have continued for health_max_saturated_ms (default 10s, configurable via CLI/env). A successful worker-slot acquire clears the window so brief bursts do not drain the pod.

Admission path updates record rejected_queue_full / rejected_wait_timeout on the prove outcome counter, bump mark_rejected on those paths, and add queue depth and wait duration Prometheus metrics (with pre-registration at startup).

HealthLayer is no longer a zero-config default: main builds one monitor clone for RPC and one for health, and HTTP/HTTPS start_server / middleware wiring pass the configured layer through.

^{Reviewed by Cursor Bugbot for commit f7a4037. Bugbot is set up for automated code reviews on this repo. Configure here.}

[avi-starkware]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• starknet_transaction_prover: log resolved config at startup (secrets redacted) #14684
• starknet_transaction_prover: document observability surface in README #14203
• starknet_transaction_prover: origin-level breadcrumbs for validation errors #14172
• starknet_transaction_prover: /health returns 503 when service is saturated #14171 👈 (View in Graphite)
• starknet_transaction_prover: bump prover_panics_total on panic hook fire #14170
• starknet_transaction_prover: HTTP request count + latency + in-flight metrics #14169
• starknet_transaction_prover: proving-job duration + outcome metrics #14168
• starknet_transaction_prover: Prometheus /metrics endpoint with build_info #14167
• starknet_transaction_prover: global panic hook + graceful SIGTERM shutdown #14166
• starknet_transaction_prover,tower_ohttp: OHTTP-unlinkable request-id for decapsulated content #14222
• starknet_transaction_prover: per-request structured log with request-id #14165
• starknet_transaction_prover: unify HTTP middleware stack via macro #14286
• main-v0.14.3

This stack of pull requests is managed by Graphite. Learn more about stacking.

[reviewable-StarkWare]

This change is 

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0989241. Configure here.}