fix: prevent gunicorn worker recycling from corrupting histogram aggregation

Gunicorn worker recycling causes in-memory Prometheus counters to reset.
The OTel aggregation pipeline strips worker.name and sums all workers
into a single cumulative counter via groupbyattrs. When a worker recycles,
its counter resets to 0, decreasing the aggregate.

This manifests as a "hidden counter reset" in Prometheus: if the recycled
worker's final le=+Inf value coincidentally equals the new worker's
starting value (e.g. both are 1 because the new worker immediately handled
a slow request), Prometheus does not detect the reset for le=+Inf. But
le=1000 resets visibly. This inflates rate(le=1000) relative to
rate(le=+Inf), producing SLI ratios greater than 1.

Fix: insert cumulativetodelta before worker aggregation so we sum
per-worker deltas (always non-negative) instead of cumulative totals.
Worker recycles produce a 0-delta rather than a negative value that
corrupts the aggregate. Add deltatorumulative after groupbyattrs to
convert the aggregate delta back to a cumulative counter for the
Prometheus exporter.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: dkliban

deploy/clowdapp.yaml

@@ -43,6 +43,14 @@ objects:
           keys:
           - api.request_duration
 
+        cumulativetodelta:
+          include:
+            metrics:
+              - api.request_duration
+            match_type: strict
+
+        deltatorumulative:
+
         batch:
         
       exporters:
@@ -61,12 +69,14 @@ objects:
         
           metrics/aggregation:
             receivers: [otlp]
-            processors: 
+            processors:
             - memory_limiter
             - filter/filter_pulp_api_request_duration
+            - cumulativetodelta
             - attributes/remove_worker_name
             - batch/api_aggregation
             - groupbyattrs/api_aggregation
+            - deltatorumulative
             exporters: [prometheus]
         
           metrics/main: