fix: cap pg_stat_statements to top 100 queries, increase VM concurrency

100K distinct queryids were creating a cardinality explosion in
VictoriaMetrics, causing 429 (Too Many Requests) on all dashboard
queries. The WHERE filter alone (calls >= 5 OR exec_time >= 1000ms)
was insufficient.

Changes:
- Add ORDER BY exec_time_total DESC LIMIT 100 to both v11 and v17
  pg_stat_statements SQL queries — only the top 100 queries by
  execution time are exported to Prometheus
- Increase search.maxConcurrentRequests from 4 to 16 in both
  docker-compose.yml and Helm values to prevent 429s when Grafana
  dashboards issue many parallel queries

https://claude.ai/code/session_01SzJxzZNQjDQphaHyaX3RU7

Author: claude

config/pgwatch-prometheus/metrics.yml

@@ -1036,6 +1036,8 @@ metrics:
           temp_bytes_written::int8 as temp_bytes_written
           from aggregated_statements
           where calls >= 5 or exec_time_total >= 1000
+          order by exec_time_total desc
+          limit 100
       17: |
         with aggregated_statements as ( /* pgwatch_generated */
           select
@@ -1083,6 +1085,8 @@ metrics:
           temp_bytes_written::int8 as temp_bytes_written
           from aggregated_statements
           where calls >= 5 or exec_time_total >= 1000
+          order by exec_time_total desc
+          limit 100
     gauges:
       - calls
       - plans_total

docker-compose.yml

@@ -3,7 +3,7 @@
 # CPU overcommitment is intentional - containers rarely hit limits simultaneously.
 #
 # pgwatch-prometheus collects metrics including pg_stat_statements
-# (filtered: calls >= 5 OR exec_time >= 1000ms to control cardinality).
+# (top 100 queries by exec_time, filtered: calls >= 5 OR exec_time >= 1000ms).
 
 services:
   # Config Initializer - Copies static configs from image to volume
@@ -103,7 +103,7 @@ services:
       - "-promscrape.config.strictParse=false"
       - "-promscrape.maxScrapeSize=128000000"
       - "-search.maxQueryDuration=30s"
-      - "-search.maxConcurrentRequests=4"
+      - "-search.maxConcurrentRequests=16"
     depends_on:
       config-init:
         condition: service_completed_successfully

postgres_ai_helm/values.yaml

@@ -44,7 +44,9 @@ victoriaMetrics:
   image: victoriametrics/victoria-metrics:v1.105.0
   retentionPeriod: 336h
   scrapeInterval: 15s
-  extraArgs: []
+  extraArgs:
+    - -search.maxQueryDuration=30s
+    - -search.maxConcurrentRequests=16
   service:
     type: ClusterIP
     port: 8428