Skip to content

Verify process scraper cardinality behavior in host metrics config#1617

Closed
Copilot wants to merge 1 commit intofeat/add-more-host-metricsfrom
copilot/sub-pr-1616
Closed

Verify process scraper cardinality behavior in host metrics config#1617
Copilot wants to merge 1 commit intofeat/add-more-host-metricsfrom
copilot/sub-pr-1616

Conversation

Copy link
Copy Markdown

Copilot AI commented Dec 15, 2025

Responding to review comment asking to verify whether the OpenTelemetry process scraper collects metrics for all processes by default, potentially creating high cardinality on orchestrator nodes.

Verification Results

The claim is partially correct:

  • The process scraper without include/exclude filters collects metrics from all processes on the host
  • Each process generates separate metric series with resource attributes (process.pid, process.command, process.executable.name)
  • On orchestrators running hundreds of Firecracker VMs, this would normally create thousands of metric series

Cardinality is mitigated by existing config:
The metricstransform/aggregate_process_metrics processor (lines 255-284) aggregates per-process metrics to per-node totals:

metricstransform/aggregate_process_metrics:
  transforms:
    - include: "process.open_file_descriptors"
      action: update
      new_name: "system.open_file_descriptors"
      operations:
        - action: aggregate_labels
          label_set: [node.id]
          aggregation_type: sum

This collapses process.open_file_descriptors, process.context_switches, and process.threads from per-process to single host-level metrics, which is the intended behavior for monitoring host resource usage.

Outstanding Consideration

The original concern about adding mute_process_*_error flags remains valid - without them, the collector will log errors when attempting to read metrics from processes it lacks permissions to access.


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI changed the title [WIP] Update PR to address feedback on host metrics enhancement Verify process scraper cardinality behavior in host metrics config Dec 15, 2025
Copilot AI requested a review from dobrac December 15, 2025 18:58
@dobrac dobrac closed this Dec 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants