feat: add cluster_name and cluster_peer labels to Prometheus metrics (#436)

## Summary
- Add `cluster_name` and `cluster_peer` as `external_labels` in
Prometheus config
- These labels are already used by Alloy for logs but were missing from
metrics sent via `remote_write`
- Pass `CLUSTER_NAME` and `CLUSTER_PEER` env vars to the Prometheus
container in docker-compose
- Render the new variables in `run.sh` via `sed` (same pattern as
existing vars)

## Changes
- `prometheus/prometheus.yml.example` — add `cluster_name` and
`cluster_peer` external labels
- `prometheus/run.sh` — render `$CLUSTER_NAME` and `$CLUSTER_PEER` via
sed
- `docker-compose.yml` — pass `CLUSTER_NAME` and `CLUSTER_PEER` env vars
to prometheus service

Author: apham0001

docker-compose.yml

@@ -177,6 +177,8 @@ services:
     environment:
       PROM_REMOTE_WRITE_TOKEN: ${PROM_REMOTE_WRITE_TOKEN}
       SERVICE_OWNER: ${SERVICE_OWNER:-"obol-cdvn"}
+      CLUSTER_NAME: ${CLUSTER_NAME:-}
+      CLUSTER_PEER: ${CLUSTER_PEER:-}
     volumes:
       - ./prometheus:/etc/prometheus
       - ./data/prometheus:/prometheus

prometheus/prometheus.yml.example

@@ -3,6 +3,8 @@ global:
   evaluation_interval: 30s # Evaluate rules every 30 seconds.
   external_labels:
     service_owner: $SERVICE_OWNER # replace this with your Operator name you want to be identified by, it helps us route alerts and metrics to your notification channels easily
+    cluster_name: $CLUSTER_NAME
+    cluster_peer: $CLUSTER_PEER
 
 remote_write:
   - url: https://vm.monitoring.gcp.obol.tech/write

prometheus/run.sh

@@ -14,6 +14,8 @@ fi
 
 sed -e "s|\$PROM_REMOTE_WRITE_TOKEN|${PROM_REMOTE_WRITE_TOKEN}|g" \
     -e "s|\$SERVICE_OWNER|${SERVICE_OWNER}|g" \
+    -e "s|\$CLUSTER_NAME|${CLUSTER_NAME}|g" \
+    -e "s|\$CLUSTER_PEER|${CLUSTER_PEER}|g" \
     -e "s|\$ALERT_DISCORD_IDS|${ALERT_DISCORD_IDS}|g" \
     /etc/prometheus/prometheus.yml.example > /etc/prometheus/prometheus.yml