ref(server): Replace bandwidth EWMA with lock-free GCRA buckets (#554)

Bandwidth rate limiting now uses debt-based GCRA (Generic Cell Rate
Algorithm) buckets instead of EWMA estimation. Each bucket tracks a
single atomic timestamp — the point at which all recorded traffic is
"paid off." Admission checks compare this deadline against wall-clock
time plus a configurable burst tolerance, with no background estimation
task required.

This removes the 50ms EWMA tick and its associated state
(per-scope/per-usecase shadow EWMA maps, the `EwmaEstimator` struct),
replacing them with lock-free `AtomicU64::update` operations that are
both simpler and more responsive — debt is visible immediately after
bytes are recorded rather than after the next estimation tick. The
`burst_ms` config parameter (default: 1000ms) controls how much
transient overshoot is tolerated before rejection.

EWMA gauges are removed from the KEDA endpoint since the monotonic
counters with `irate()` are strictly more responsive. The throughput
background task is simplified to emit only the limit and map-size gauges
at 1s intervals. The bandwidth limit gauge is now emitted once at
construction.

Author: jan-auer

objectstore-server/docs/architecture.md

@@ -182,26 +182,27 @@ Limits can be set at multiple granularities:
 
 ### Bandwidth
 
-Bandwidth limiting uses an **exponentially weighted moving average** (EWMA) to
-estimate current throughput. Payload streams are wrapped in a
-`MeteredPayloadStream` that reports bytes consumed. When the estimated bandwidth
-exceeds the configured limit, new requests are rejected.
+Bandwidth limiting uses **debt-based GCRA** (Generic Cell Rate Algorithm)
+buckets that track a theoretical arrival time. Payload streams are wrapped in a
+`MeteredPayloadStream` that track consumed bytes. When accumulated debt exceeds
+the configured threshold, new requests are rejected.
+
+The `burst_ms` parameter controls how much transient overshoot is tolerated
+before rejection (in milliseconds). Defaults to `1000` (1 second).
 
 Like throughput, bandwidth limits can be set at multiple granularities:
 
 - **Global**: a maximum bytes-per-second across all traffic (`global_bps`)
 - **Per-usecase**: a percentage of the global limit for each usecase
 - **Per-scope**: a percentage of the global limit for each scope value
 
-Each granularity maintains its own EWMA estimator. The `MeteredPayloadStream`
-increments all applicable accumulators (global + per-usecase + per-scope) for
-every chunk polled. For non-streamed payloads (e.g., batch INSERT where the
-size is known upfront), bytes are recorded directly via
-[`record_bandwidth`](state::Services::record_bandwidth).
+Each granularity maintains its own bucket. The `MeteredPayloadStream` charges
+all applicable buckets (global + per-usecase + per-scope) for every chunk
+polled. For non-streamed payloads, bytes are recorded directly.
 
 Rate-limited requests receive HTTP 429. When `report_only` is enabled, all
-accounting and metrics (EWMA and limit gauges, KEDA metrics) remain active, but
-requests exceeding the limit are admitted instead of rejected.
+accounting and metrics remain active, but requests exceeding the limit are
+admitted instead of rejected.
 
 ### Web Concurrency Limit
 
@@ -244,20 +245,14 @@ metrics so that it remains available when the server is at capacity.
 
 ### Exposed Metrics
 
-#### EWMA Gauges
-
-Pre-smoothed rates, self-contained per scrape (no `irate()` arithmetic needed):
+#### Gauges
 
 | Resource | Utilization | Limit |
 |---|---|---|
-| Bandwidth | `objectstore_bandwidth_ewma` | `objectstore_bandwidth_limit` (only when `global_bps` is set) |
-| Throughput | `objectstore_throughput_ewma` | `objectstore_throughput_limit` (only when `global_rps` is set) |
 | HTTP concurrency | `objectstore_requests_in_flight` | `objectstore_requests_limit` |
 | Task concurrency | `objectstore_tasks_running` | `objectstore_tasks_limit` |
 
-Throughput uses an EWMA with a 50 ms tick and α = 0.2, matching the existing
-bandwidth estimator. The accumulator counts fully admitted requests (requests
-that pass all throughput checks).
+Limit gauges are emitted only when the corresponding limit is configured.
 
 #### Counters
 
@@ -269,30 +264,10 @@ KEDA queries for an unsmoothed, immediately responsive rate:
 | `objectstore_bytes_total` | Total bytes transferred since startup |
 | `objectstore_requests_total` | Total admitted requests since startup |
 
-### Example KEDA ScaledObject Triggers
-
-#### Using EWMA gauges (backward-compatible)
-
-Scale on the highest utilization across all four resources:
-
-```yaml
-triggers:
-  - type: prometheus
-    metadata:
-      serverAddress: http://prometheus:9090
-      query: |
-        max(
-          objectstore_bandwidth_ewma / objectstore_bandwidth_limit
-          or objectstore_throughput_ewma / objectstore_throughput_limit
-          or objectstore_requests_in_flight / objectstore_requests_limit
-          or objectstore_tasks_running / objectstore_tasks_limit
-        )
-      threshold: "0.7"
-```
-
-#### Using counters with `irate()` (more responsive)
+### Example KEDA ScaledObject Trigger
 
-Uses the last two scraped values for an instantaneous rate with no smoothing lag:
+Scale on the highest utilization across all resources using `irate()` for
+bandwidth/throughput rates:
 
 ```yaml
 triggers:

objectstore-server/src/config.rs

@@ -1040,6 +1040,7 @@ mod tests {
                         pct: 10
                   bandwidth:
                     global_bps: 1048576
+                    burst_ms: 2000
                     usecase_pct: 50
                     scope_pct: 25
                     report_only: true
@@ -1074,6 +1075,7 @@ mod tests {
                 },
                 bandwidth: BandwidthLimits {
                     global_bps: Some(1_048_576),
+                    burst_ms: 2000,
                     usecase_pct: Some(50),
                     scope_pct: Some(25),
                     report_only: true,

objectstore-server/src/endpoints/keda.rs

@@ -2,8 +2,6 @@
 //!
 //! Exposes a `/keda` endpoint in Prometheus text format (version 0.0.4) with:
 //!
-//! - **EWMA gauges** (`objectstore_bandwidth_ewma`, `objectstore_throughput_ewma`): pre-computed
-//!   rates, self-contained per scrape with no `irate()` arithmetic needed (kept for backward compat).
 //! - **Monotonic counters** (`objectstore_bytes_total`, `objectstore_requests_total`):
 //!   cumulative totals since startup; use `irate(counter[window])` in KEDA queries for an
 //!   unsmoothed, immediately responsive rate.
@@ -21,10 +19,8 @@ pub fn router() -> Router<ServiceState> {
 }
 
 async fn keda(State(state): State<ServiceState>) -> impl IntoResponse {
-    let bw_ewma = state.rate_limiter.bandwidth_ewma();
     let bw_limit = state.rate_limiter.bandwidth_limit();
     let bw_total = state.rate_limiter.bandwidth_total_bytes();
-    let tp_rps = state.rate_limiter.throughput_rps();
     let tp_limit = state.rate_limiter.throughput_limit();
     let tp_total = state.rate_limiter.throughput_total_admitted();
     let req_in_flight = state.request_counter.count();
@@ -36,15 +32,9 @@ async fn keda(State(state): State<ServiceState>) -> impl IntoResponse {
         "# HELP objectstore_bytes_total Total bytes transferred since startup\n\
          # TYPE objectstore_bytes_total counter\n\
          objectstore_bytes_total {bw_total}\n\
-         # HELP objectstore_bandwidth_ewma Current bandwidth in bytes/s (EWMA)\n\
-         # TYPE objectstore_bandwidth_ewma gauge\n\
-         objectstore_bandwidth_ewma {bw_ewma}\n\
          # HELP objectstore_requests_total Total admitted requests since startup\n\
          # TYPE objectstore_requests_total counter\n\
          objectstore_requests_total {tp_total}\n\
-         # HELP objectstore_throughput_ewma Current admitted request rate in requests/s (EWMA)\n\
-         # TYPE objectstore_throughput_ewma gauge\n\
-         objectstore_throughput_ewma {tp_rps}\n\
          # HELP objectstore_requests_in_flight Current in-flight HTTP requests\n\
          # TYPE objectstore_requests_in_flight gauge\n\
          objectstore_requests_in_flight {req_in_flight}\n\