cardinality of user metrics should be configurable

Author: reggeenr

metrics-collector/README.md

@@ -184,9 +184,88 @@ The following 3 metrics are used to monitor the collector itself:
 #### Metric Labels
 
 All container metrics include the following labels:
-- `ibm_codeengine_instance_name`: Name of the pod instance
 - `ibm_codeengine_component_type`: Type of component (`app`, `job`, or `build`)
 - `ibm_codeengine_component_name`: Name of the Code Engine component
+- `ibm_codeengine_instance_name`: Name of the pod instance (optional, see cardinality control below)
+- `ibm_codeengine_subcomponent_name`: Name of the app revision (optional, see cardinality control below)
+
+#### User Metrics Scraping
+
+The metrics collector can automatically discover and scrape custom Prometheus metrics from your Code Engine applications. To enable this feature, add the following annotations to your application:
+
+**Required annotation:**
+- `codeengine.cloud.ibm.com/userMetricsScrape: 'true'` - Enables metrics scraping for this application
+
+**Optional annotations:**
+- `codeengine.cloud.ibm.com/userMetricsPath: '/metrics'` - Custom metrics endpoint path (default: `/metrics`)
+- `codeengine.cloud.ibm.com/userMetricsPort: '2112'` - Custom metrics port
+
+**Example:**
+```bash
+kubectl patch ksvc myapp --type merge -p '{
+  "spec": {
+    "template": {
+      "metadata": {
+        "annotations": {
+          "codeengine.cloud.ibm.com/userMetricsScrape": "true",
+          "codeengine.cloud.ibm.com/userMetricsPath": "/metrics",
+          "codeengine.cloud.ibm.com/userMetricsPort": "2112"
+        }
+      }
+    }
+  }
+}'
+```
+
+#### Cardinality Control for User Metrics
+
+To manage metric cardinality and reduce costs, you can control which labels are included in scraped user metrics using the following annotations:
+
+**Cardinality control annotations:**
+- `codeengine.cloud.ibm.com/userMetricsInstance: 'true'` - Include the `ibm_codeengine_instance_name` label (pod name)
+- `codeengine.cloud.ibm.com/userMetricsSubcomponent: 'true'` - Include the `ibm_codeengine_subcomponent_name` label (app revision name)
+
+**Default behavior:** By default, both `ibm_codeengine_instance_name` and `ibm_codeengine_subcomponent_name` labels are **excluded** from user metrics to minimize cardinality. These labels can create high cardinality because:
+- Instance names change with each pod restart or scale event
+- Revision names change with each application update
+
+**When to enable these labels:**
+- Enable `userMetricsInstance` when you need to track metrics per individual pod instance
+- Enable `userMetricsSubcomponent` when you need to compare metrics across different application revisions
+
+**Example with cardinality control:**
+```bash
+# Enable user metrics scraping with instance-level granularity
+kubectl patch ksvc myapp --type merge -p '{
+  "spec": {
+    "template": {
+      "metadata": {
+        "annotations": {
+          "codeengine.cloud.ibm.com/userMetricsScrape": "true",
+          "codeengine.cloud.ibm.com/userMetricsInstance": "true"
+        }
+      }
+    }
+  }
+}'
+
+# Enable user metrics scraping with both instance and revision granularity
+kubectl patch ksvc myapp --type merge -p '{
+  "spec": {
+    "template": {
+      "metadata": {
+        "annotations": {
+          "codeengine.cloud.ibm.com/userMetricsScrape": "true",
+          "codeengine.cloud.ibm.com/userMetricsInstance": "true",
+          "codeengine.cloud.ibm.com/userMetricsSubcomponent": "true"
+        }
+      }
+    }
+  }
+}'
+```
+
+**Note:** Only set these annotations to `'true'` when you specifically need the additional label granularity. Keeping them disabled (default) helps reduce metric cardinality and associated monitoring costs.
 
 #### Example Metrics Output
 

metrics-collector/prometheus.yml.template

@@ -49,9 +49,12 @@ scrape_configs:
     - action: replace
       replacement: '${CE_PROJECT_NAME}'
       target_label: ibm_codeengine_project_name
-    - source_labels: [__meta_kubernetes_pod_name]
+    # only set ibm_codeengine_instance_name if annotation codeengine.cloud.ibm.com/userMetricsInstance is set to 'true'
+    - source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_annotation_codeengine_cloud_ibm_com_userMetricsInstance]
       action: replace
       target_label: ibm_codeengine_instance_name
+      regex: ([^;]+);true
+      replacement: ${1}
     - source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_service]
       action: replace
       target_label: ibm_codeengine_component_name
@@ -60,9 +63,12 @@ scrape_configs:
       regex: (.+)
       replacement: app
       target_label: ibm_codeengine_component_type
-    - source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision]
+    # only set ibm_codeengine_subcomponent_name if annotation codeengine.cloud.ibm.com/userMetricsSubcomponent is set to 'true'
+    - source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision, __meta_kubernetes_pod_annotation_codeengine_cloud_ibm_com_userMetricsSubcomponent]
       action: replace
       target_label: ibm_codeengine_subcomponent_name
+      regex: ([^;]+);true
+      replacement: ${1}
     - source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revisionUID]
       action: replace
       regex: (.+)
@@ -96,4 +102,4 @@ remote_write:
     # Dropping scrape metrics (e.g. scrape_duration_seconds)
     - source_labels: [__name__]
       regex: 'scrape_duration_seconds|scrape_samples_scraped|scrape_series_added|scrape_samples_post_metric_relabeling'
-      action: drop
+      action: drop

metrics-examples/run

@@ -46,7 +46,7 @@ for i in "${languages[@]}"; do
             --src "." \
             --memory 0.5G \
             --cpu 0.25 \
-            --env HTTPBIN_BASE_URL=https://httpbin.284uby30ujzw.ca-tor.codeengine.appdomain.cloud
+            --env "HTTPBIN_BASE_URL=$httbin_url"
 
         # Wait for the Kubernetes service resource to be ready before patching
         echo "Waiting for ksvc '$app_name' to be ready..."

metrics-examples/run-job

@@ -0,0 +1,64 @@
+#!/bin/bash
+set -eo pipefail
+
+PREFIX="${PREFIX:=metrics-example}"
+LANGUAGE="${LANGUAGE:=all}"
+
+# Create the Code Engine project if it does not exist, yet
+if ! ibmcloud ce project get --name "$PREFIX" >/dev/null 2>&1;then 
+    ibmcloud ce project create --name "$PREFIX"
+fi
+
+# Select the Code Engine project
+ibmcloud ce project select --name "$PREFIX" --kubecfg
+
+httbin_url=https://httpbin.org
+if ibmcloud ce app get --name httpbin >/dev/null 2>&1; then
+    httbin_url=`ibmcloud ce app get --name httpbin --output url`
+fi
+echo "Using HTTPBin backend URL: '$httbin_url'"
+
+languages=(
+    python
+)
+
+for i in "${languages[@]}"; do
+
+    if [[ "$LANGUAGE" == "all" || "$LANGUAGE" == "$i" ]];then 
+
+        # Create or update the job
+        echo "Deploying Code Engine job for $i ..."
+        job_name="${PREFIX}-job-$i"
+
+        create_or_update=update
+        if ! ibmcloud ce job get --name $job_name >/dev/null 2>&1; then
+            echo -e "\nCreating the job '$job_name' ..."
+            create_or_update=create
+        else
+            echo -e "\nUpdating the job '$job_name' ..."
+        fi
+
+        # Create or update the job
+        ibmcloud ce job $create_or_update \
+            --name $job_name \
+            --context-dir "$i/" \
+            --src "." \
+            --memory 0.5G \
+            --cpu 0.25 \
+            --env "HTTPBIN_BASE_URL=$httbin_url" \
+            --wait
+        
+        
+        kubectl patch jobdefinition "$job_name" --type=json -p='[
+             {"op":"add","path":"/metadata/labels/codeengine.cloud.ibm.com~1userMetricsScrape","value":"true"},
+             {"op":"add","path":"/metadata/labels/codeengine.cloud.ibm.com~1userMetricsPath","value":"/metrics"},
+             {"op":"add","path":"/metadata/labels/codeengine.cloud.ibm.com~1userMetricsPort","value":"2112"}
+        ]'
+
+        # Submit a jobrun
+    else 
+        continue;
+    fi
+done
+
+echo "Done"