metrics-collector

IBM Cloud Code Engine - Metrics Collector

Table of contents

• Overview
• Send metrics to IBM Cloud Monitoring
  • How It Works
  • Setup Instructions
  • Scraping Resource Metrics
  • Scraping User-defined Metrics
  • User-defined metrics examples
• IBM Cloud Logs setup
  • Custom dashboard
  • Logs view
• Troubleshooting & Configuration
  • Common Issues
  • Environment Variables

Overview

Code Engine job that demonstrates how to collect resource consumption metrics (vCPU and memory) of running Code Engine apps, jobs, and builds.

Resource consumption metrics can either be render in IBM Cloud Monitoring (see instructions)

or in IBM Cloud Logs (see instructions)

Furthermore, this asset can collect custom metrics of running apps and jobs (see instructions).

Send metrics to IBM Cloud Monitoring

How It Works

The metrics-collector is an as-is asset, that can be use to transmit resource consumption metrics (like vCPU and memory), and user-defined metrics (also known as custom metrics) to any target IBM Cloud Monitoring instance. To collect metrics through the metrics-collector, an instance needs to be running. It's recommended to run the metrics-collector as a daemon job.

Note: metrics-collector is project scoped, which means only metrics of the corresponding project in which the metrics-collector instance itself resides are collected. To scrape metrics of multiple projects, a metrics-collector instance needs to get deployed and started into each of the projects.

While resource consumption metrics are automatically collected of all running app, jobrun and buildrun instances, user-defined metrics are only scraped if enabled on the corresponding application or job.

At a high level, the metrics-collector has following characteristics

1. An instance comprises of mainly two processes
   • an HTTP server that exposes vCPU and memory information as Prometheus metrics on localhost:9100/metrics
   • a prometheus agent which scrapes metrics and sends them to IBM Cloud Monitoring
2. Metrics are scraped every 30 seconds
3. For user-defined metrics the agent discovers and scrapes instances that are annotated with codeengine.cloud.ibm.com/userMetricsScrape: 'true'
4. All metrics are forwarded to IBM Cloud Monitoring via remote write
5. If one of the aforementioned processes crashes, the container exits with a non-zero code to trigger a restart

Setup Instructions

The metrics-collector supports two authentication methods for accessing IBM Cloud Monitoring:

• Option 1: Trusted Profile Authentication (Recommended) - Automatically obtains monitoring API keys using IBM Cloud Trusted Profiles
• Option 2: Explicit API Key Secret - Manually create and mount a monitoring API key as a secret

Prerequisites (Both Options)

Step 1: You need an IBM Cloud Monitoring instance

REGION=<yourMonitoringInstanceRegion>
MONITORING_INSTANCE_NAME="<yourMonitoringInstanceName>"
MONITORING_INSTANCE_GUID=$(ibmcloud resource service-instance "$MONITORING_INSTANCE_NAME" -o JSON|jq -r '.[0].guid')
echo "MONITORING_INSTANCE_GUID: '$MONITORING_INSTANCE_GUID'"

Step 2: The collector must run in a Code Engine project

# Create new Code Engine project
ibmcloud ce project create --name <yourCodeEngineProjectName>

# Select an existing Code Engine project
ibmcloud ce project select --name <yourProjectName>

Option 1: Trusted Profile Authentication (Recommended)

This method automatically obtains the monitoring API key using IBM Cloud Trusted Profiles. No manual secret creation or rotation is required.

Step 3a: Create an IBM Cloud Trusted Profile

Create a Trusted Profile that allows Code Engine compute resources to authenticate:

# Create the Trusted Profile
ibmcloud iam trusted-profile-create metrics-collector-profile \
    --description "Trusted profile for Code Engine metrics collector"

# Add Code Engine compute resources as trusted entities
ibmcloud iam trusted-profile-rule-create metrics-collector-profile \
    --name code-engine-rule \
    --type Profile-CR \
    --cr-type CE \
    --conditions claim:project_name,operator:EQUALS,value:"$(ibmcloud ce proj current --output json|jq -r '.name')" \
    --conditions claim:component_type,operator:EQUALS,value:"job" \
    --conditions claim:component_name,operator:EQUALS,value:"metrics-collector"

# Grant the profile access to the Monitoring instance
ibmcloud iam trusted-profile-policy-create metrics-collector-profile \
    --roles Viewer,Writer \
    --service-name sysdig-monitor \
    --service-instance $MONITORING_INSTANCE_GUID

Note: Replace <your-ce-project-namespace> with your Code Engine project's Kubernetes namespace (typically in the format <project-guid>).

Step 4a: Create your metrics-collector job with Trusted Profile configuration

ibmcloud ce job create \
    --name metrics-collector \
    --src "." \
    --mode daemon \
    --cpu 0.25 \
    --memory 0.5G \
    --service-account reader \
    --build-size xlarge \
    --trusted-profiles-enabled \
    --env INTERVAL=30 \
    --env METRICS_ENABLED=true \
    --env METRICS_REMOTE_WRITE_FQDN=ingest.prws.private.${REGION}.monitoring.cloud.ibm.com \
    --env CE_PROJECT_NAME="$(ibmcloud ce proj current --output json|jq -r '.name')" \
    --env MONITORING_INSTANCE_GUID="$MONITORING_INSTANCE_GUID" \
    --env MONITORING_REGION="$REGION" \
    --env TRUSTED_PROFILE_NAME="metrics-collector-profile"

Step 5a: Submit a daemon job run

ibmcloud ce jobrun submit \
    --job metrics-collector

Option 2: Explicit API Key Secret

This method requires manually creating and mounting a monitoring API key. Use this option for local development, testing, or when Trusted Profiles are not available.

Step 3b: Create a secret with your IBM Cloud Monitoring API token

# Obtain the Monitoring API token of the IBM Cloud Monitoring instance
# using the IAM access token of the current IBM CLI Session
MONITORING_INSTANCE_MONITORING_API_KEY=$(curl --silent -X GET https://$REGION.monitoring.cloud.ibm.com/api/token -H "Authorization: $(ibmcloud iam oauth-tokens --output JSON|jq -r '.iam_token')" -H "IBMInstanceID: $MONITORING_INSTANCE_GUID" -H "content-type: application/json"|jq -r '.token.key')

# Create a Code Engine secret that stores the Monitoring API Key
ibmcloud ce secret create \
    --name monitoring-apikey \
    --from-literal monitoring-apikey=$MONITORING_INSTANCE_MONITORING_API_KEY

Step 4b: Create your metrics-collector job with the required configuration

ibmcloud ce job create \
    --name metrics-collector \
    --src "." \
    --mode daemon \
    --cpu 0.25 \
    --memory 0.5G \
    --service-account reader \
    --build-size xlarge \
    --env INTERVAL=30 \
    --env METRICS_ENABLED=true \
    --env METRICS_REMOTE_WRITE_FQDN=ingest.prws.private.${REGION}.monitoring.cloud.ibm.com \
    --env CE_PROJECT_NAME="$(ibmcloud ce proj current --output json|jq -r '.name')" \
    --mount-secret /etc/secrets=monitoring-apikey

Step 5b: Submit a daemon job run

ibmcloud ce jobrun submit \
    --job metrics-collector

Scraping Resource Metrics

The metrics collector scrapes CPU/Memory metrics from your Code Engine applications, jobs, and builds. Metrics of all running instances are scraped every 30 seconds.

Once, metrics are ingested into an IBM Cloud Monitoring instance, the pre-populated dashboard "IBM Cloud Code Engine - Component Resource Overview" appears in Dashboard Manager.

Exposed Metrics

The following Prometheus metrics are exposed as gauges:

Container Metrics:

• ibm_codeengine_instance_cpu_usage_millicores: Current CPU usage in millicores
• ibm_codeengine_instance_cpu_limit_millicores: Configured CPU limit in millicores
• ibm_codeengine_instance_memory_usage_bytes: Current memory usage in bytes
• ibm_codeengine_instance_memory_limit_bytes: Configured memory limit in bytes
• ibm_codeengine_instance_ephemeral_storage_usage_bytes: Current ephemeral storage usage in bytes (if COLLECT_DISKUSAGE=true)

The following 3 metrics are used to monitor the collector itself:

• ibm_codeengine_collector_collection_duration_seconds: Time taken to collect metrics in seconds (if METRICS_INTERNAL_STATS=true)
• ibm_codeengine_collector_last_collection_timestamp_seconds: Unix timestamp of last successful collection (if METRICS_INTERNAL_STATS=true)
• ibm_codeengine_collector_collection_errors_total: Total number of collection errors (counter) (if METRICS_INTERNAL_STATS=true)

Sample metrics content:

# HELP ibm_codeengine_instance_cpu_usage_millicores Current CPU usage in millicores
# TYPE ibm_codeengine_instance_cpu_usage_millicores gauge
ibm_codeengine_instance_cpu_usage_millicores{ibm_codeengine_instance_name="myapp-00001-deployment-abc123",ibm_codeengine_component_type="app",ibm_codeengine_component_name="myapp"} 250

# HELP ibm_codeengine_instance_memory_usage_bytes Current memory usage in bytes
# TYPE ibm_codeengine_instance_memory_usage_bytes gauge
ibm_codeengine_instance_memory_usage_bytes{ibm_codeengine_instance_name="myapp-00001-deployment-abc123",ibm_codeengine_component_type="app",ibm_codeengine_component_name="myapp"} 134217728

Metric Labels

All container metrics include the following labels:

• ibm_codeengine_component_type: Type of component (app, job, or build)
• ibm_codeengine_component_name: Name of the Code Engine component
• ibm_codeengine_instance_name: Name of the pod instance (optional, see cardinality control below)
• ibm_codeengine_subcomponent_name: Name of the app revision (optional, see cardinality control below)

Scraping User-defined Metrics

The metrics collector can discover and scrape custom Prometheus metrics from your Code Engine applications and jobs. To enable this feature, add the following annotations to your application:

Required annotation:

• codeengine.cloud.ibm.com/userMetricsScrape: 'true' - Enables metrics scraping for this application
• codeengine.cloud.ibm.com/userMetricsPath: '/metrics' - Custom metrics endpoint path (default: /metrics)
• codeengine.cloud.ibm.com/userMetricsPort: '2112' - Custom metrics port

How to enable scraping of custom metrics on an app:

kubectl patch ksvc myapp --type merge -p '{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "codeengine.cloud.ibm.com/userMetricsScrape": "true",
          "codeengine.cloud.ibm.com/userMetricsPath": "/metrics",
          "codeengine.cloud.ibm.com/userMetricsPort": "2112"
        }
      }
    }
  }
}'

How to enable scraping of custom metrics on a job:

kubectl annotate jobdefinition myjob codeengine.cloud.ibm.com/userMetricsScrape=true
kubectl annotate jobdefinition myjob codeengine.cloud.ibm.com/userMetricsPath='/metrics'
kubectl annotate jobdefinition myjob codeengine.cloud.ibm.com/userMetricsPort=2112

Cardinality Control for User-defined metrics

To manage metric cardinality and reduce costs, you can control which labels are included in scraped user metrics using the following annotations:

Cardinality control annotations:

• codeengine.cloud.ibm.com/userMetricsInstance: 'true' - Include the ibm_codeengine_instance_name label (pod name)
• codeengine.cloud.ibm.com/userMetricsSubcomponent: 'true' - Include the ibm_codeengine_subcomponent_name label (app revision name)

Default behavior: By default, both ibm_codeengine_instance_name and ibm_codeengine_subcomponent_name labels are excluded from user metrics to minimize cardinality. These labels can create high cardinality because:

• Instance names change with each pod restart or scale event
• Revision names change with each application update

When to enable these labels:

• Enable userMetricsInstance when you need to track metrics per individual pod instance
• Enable userMetricsSubcomponent when you need to compare metrics across different application revisions

Example with cardinality control:

# Enable user metrics scraping with instance-level granularity
kubectl patch ksvc myapp --type merge -p '{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "codeengine.cloud.ibm.com/userMetricsScrape": "true",
          "codeengine.cloud.ibm.com/userMetricsInstance": "true"
        }
      }
    }
  }
}'

# Enable user metrics scraping with both instance and revision granularity
kubectl patch ksvc myapp --type merge -p '{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "codeengine.cloud.ibm.com/userMetricsScrape": "true",
          "codeengine.cloud.ibm.com/userMetricsInstance": "true",
          "codeengine.cloud.ibm.com/userMetricsSubcomponent": "true"
        }
      }
    }
  }
}'

Note: Only set these annotations to "true" when you specifically need the additional label granularity. Keeping them disabled (default) helps reduce metric cardinality and associated monitoring costs.

User-defined metrics examples

See the metrics-example/README.MD for language specific examples on how to emit prometheus metrics.

IBM Cloud Logs setup

Once your IBM Cloud Code Engine project has detected a corresponding IBM Cloud Logs instance, which is configured to receive platform logs, you can consume the resource metrics in IBM Cloud Logs. Use the filter metric:instance-resources to filter for log lines that print resource metrics for each detected IBM Cloud Code Engine instance that is running in a project.

Custom dashboard

Follow the steps below to create a custom dashboard in your IBM Cloud Logs instance, to gain insights into resource consumption metrics.

Setup instructions:

• Navigate to the "Custom dashboards" view, hover of the "New" button, and click "Import dashboard"

• In the "Import" modal, select the file ./setup/ibm-cloud-logs/dashboard-code_engine_resource_consumption_metrics.json located in this repository, and click "Import"

• Confirm the import by clicking "Import" again

Logs view

Follow the steps below to create a Logs view in your IBM Cloud Logs instance, that allows you to drill into individual instance-resources log lines.

Setup instructions:

• Filter only log lines related collected istio-proxy logs, by filtering for the following query

app:"codeengine" AND message.metric:"instance-resources"

• In the left bar, click "Add Filter" and add the following filters
  • Application
  • App
  • Label.Project
  • Message.Component_name

• In the top-right corner, click on "Columns" and configure the following columns:
  • Timestamp
  • label.Project
  • message.component_type
  • message.component_name
  • message.message
  • Text

• Once applied adjust the column widths appropriately

• In the top-right corner, select 1-line as view mode

• In the graph title it says "Count all grouped by Severity". Click on Severity and select message.component_name instead. Furthermore, select Max as aggregation metric and choose message.memory.usage as aggregation field

• Save the view

• Utilize the custom logs view to drill into HTTP requests

Troubleshooting & Configuration

Common Issues

Trusted Profile Authentication Failures:

• Missing environment variables: Ensure MONITORING_INSTANCE_GUID, MONITORING_REGION, and TRUSTED_PROFILE_NAME are all set
• Container resource token not found: Verify the job is running in Code Engine with proper service account permissions
• IAM token request failed: Check that the Trusted Profile exists and has the correct trust relationship configured
• Monitoring API key retrieval failed: Verify the Trusted Profile has appropriate permissions (Viewer, Writer) for the Monitoring instance
• Invalid region: Ensure MONITORING_REGION matches your Monitoring instance region (e.g., us-south, eu-de, eu-gb)

Explicit Secret Authentication Failures:

• Missing /etc/secrets/monitoring-apikey file: Ensure the secret is created and properly mounted
• Invalid API key: Regenerate the monitoring API key and update the secret

General Issues:

• Missing or wrong METRICS_REMOTE_WRITE_FQDN: Verify the endpoint matches your region's ingestion endpoint
• Prometheus agent fails to start: Check the logs for configuration errors or network connectivity issues

Environment Variables

Core Configuration:

• INTERVAL (default: 30): Collection interval in seconds (minimum 30 seconds). Controls how frequently metrics are collected from the Kubernetes API endpoint in daemon mode.
• COLLECT_DISKUSAGE (default: false): Set to true to collect disk space usage. Note: The metrics collector calculates the overall file size stored in the pod's filesystem, which includes files from the container image, ephemeral storage, and mounted COS buckets. This metric cannot be used to calculate ephemeral storage usage alone.
• METRICS_ENABLED (default: false): Set to true to enable the HTTP metrics server. When disabled, the collector still runs and logs metrics to stdout but does not expose the HTTP endpoint.
• METRICS_REMOTE_WRITE_FQDN: IBM Cloud Monitoring ingestion endpoint FQDN (required when METRICS_ENABLED=true)
• METRICS_PORT (default: 9100): HTTP server port for the Prometheus metrics endpoint. Only used when METRICS_ENABLED=true in daemon mode.

Trusted Profile Authentication (Option 1):

• MONITORING_INSTANCE_GUID: The GUID of your IBM Cloud Monitoring instance (required for Trusted Profile authentication)
• MONITORING_REGION: The region where your Monitoring instance is deployed, e.g., us-south, eu-de, eu-gb (required for Trusted Profile authentication)
• TRUSTED_PROFILE_NAME: The name of the IBM Cloud Trusted Profile to use for authentication (required for Trusted Profile authentication)
• CR_TOKEN_FILENAME (optional): Override the default container resource token file path (default: /var/run/secrets/codeengine.cloud.ibm.com/compute-resource-token/token)

Note: When all three Trusted Profile environment variables are set, the collector will attempt Trusted Profile authentication first. If it fails or the variables are not set, it will fall back to using a mounted secret at /etc/secrets/monitoring-apikey.