Skip to content

[CASCL-623] feat(cluster-agent): enhance DatadogPodAutoscaler metrics with dedicate metrics store#46833

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
clamoriniere/dpa-metrics-improvement
Feb 27, 2026
Merged

[CASCL-623] feat(cluster-agent): enhance DatadogPodAutoscaler metrics with dedicate metrics store#46833
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
clamoriniere/dpa-metrics-improvement

Conversation

@clamoriniere
Copy link
Copy Markdown
Contributor

@clamoriniere clamoriniere commented Feb 23, 2026

Scope of This PR

This PR revisits the previous attempt:
#42547

The focus here is strictly on refactoring the existing metric generation logic, without introducing new metrics.

A follow-up PR will build on this foundation to introduce additional DPA metrics.

Motivation

Today, DatadogPodAutoscaler (DPA) resource metrics are exposed as OpenMetrics/Prometheus metrics and scraped by the datadog_cluster_agent check.

While functional, this approach has several drawbacks:

  • All scraped metrics automatically inherit the Cluster Agent pod context (pod_name, kube_namespace, kube_container_name, etc.).
  • This creates confusion around DPA metric tags, as metrics appear tied to Cluster Agent pods rather than the DPA resources themselves.
  • It increases metric cardinality due to multiple Cluster Agent instances (leader, followers, rollouts) contributing additional metric contexts.

Proposed Approach

This PR changes how DPA metrics are generated.

Instead of exposing them via OpenMetrics and relying on the datadog_cluster_agent check for collection, DPA metrics are now produced directly within the Cluster Agent autoscaling component — following the same pattern used for kubernetes_state metrics.

This provides:

  • Better control over which tags are attached to DPA metrics
  • Cleaner and more accurate metric context
  • Reduced unnecessary metric cardinality

Summary

Core changes

  • Refactor ObserverFunc signature from (string, string) to (string, interface{}) to pass the actual object to observers, enabling richer metric generation
  • Add new pkg/clusteragent/autoscaling/workload/metrics package with a PodAutoscalerMetricsStore that generates and periodically sends structured metrics (gauges/counts) for DatadogPodAutoscaler objects via sender.Sender
  • Replace old telemetry helpers (telemetry.go tag-based metrics) with leader-aware metric submission; metrics are only emitted by the leader

Action metrics consolidation

  • Move horizontal/vertical scaling action metrics from event-driven Submit* functions in counters.go into the state-based GeneratePodAutoscalerMetrics generator
  • Delete counters.go entirely (SubmitReceivedRecommendationsVersion was dead code; remaining Submit* functions replaced by the generator)
  • Remove sender and isLeader from horizontalController and verticalController since they were only used for the now-removed Submit* calls

Metrics emitted

Metric Type Notes
received_recommendations_version Gauge RC version of last received main scaling values; only emitted when > 0
horizontal_scaling_received_replicas Gauge Replicas recommended by the product recommender
vertical_scaling_received_requests Gauge Per-container requested resources from recommender
vertical_scaling_received_limits Gauge Per-container resource limits from recommender
horizontal_scaling_applied_replicas Gauge Replicas from the last applied horizontal action
horizontal_scaling_actions Count Cumulative count of horizontal scaling actions, tagged status:ok or status:error
vertical_rollout_triggered Count Cumulative count of vertical rollout actions, tagged status:ok or status:error
autoscaler_conditions Gauge 1.0/0.0 per condition type from CRD status
local_fallback_enabled Gauge 1.0 when horizontal active source is local fallback

Model changes

  • Add mainScalingValuesVersion uint64 to PodAutoscalerInternal to persist the remote config version of the last received main scaling values
  • Extend UpdateFromMainValues to accept and persist the RC version; RemoveMainValues resets it to 0
  • Add horizontalActionErrorCount/horizontalActionSuccessCount and verticalActionErrorCount/verticalActionSuccessCount counter fields, incremented on each action outcome
  • Add public getters: MainScalingValuesVersion(), HorizontalActionErrorCount(), HorizontalActionSuccessCount(), VerticalActionErrorCount(), VerticalActionSuccessCount()

Test plan

  • Unit tests added for metrics/store, metrics/generator, and metrics/writer
  • generator_test.go covers all 9 metrics including both status:ok and status:error count variants
  • config_retriever_values_test.go updated with expected MainScalingValuesVersion values
  • controller_horizontal_test.go updated to assert HorizontalActionSuccessCount/HorizontalActionErrorCount
  • Existing workload controller tests updated to remove now-deleted sender/isLeader fixtures
  • All tests in pkg/clusteragent/autoscaling/... pass locally

🤖 Generated with Claude Code

@clamoriniere clamoriniere requested a review from a team as a code owner February 23, 2026 21:24
@dd-octo-sts dd-octo-sts Bot added the internal Identify a non-fork PR label Feb 23, 2026
@github-actions github-actions Bot added the long review PR is complex, plan time to review it label Feb 23, 2026
@agent-platform-auto-pr
Copy link
Copy Markdown
Contributor

agent-platform-auto-pr Bot commented Feb 23, 2026

Go Package Import Differences

Baseline: d75c25a
Comparison: 5d0021b

binaryosarchchange
cluster-agentlinuxamd64
+2, -0
+github.com/DataDog/datadog-agent/pkg/clusteragent/autoscaling/workload/metrics
+github.com/DataDog/datadog-agent/pkg/clusteragent/metricsstore
cluster-agentlinuxarm64
+2, -0
+github.com/DataDog/datadog-agent/pkg/clusteragent/autoscaling/workload/metrics
+github.com/DataDog/datadog-agent/pkg/clusteragent/metricsstore

@agent-platform-auto-pr
Copy link
Copy Markdown
Contributor

agent-platform-auto-pr Bot commented Feb 23, 2026

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor d75c25a
📊 Static Quality Gates Dashboard
🔗 SQG Job

Successful checks

Info

Quality gate Change Size (prev → curr → max)
docker_cluster_agent_amd64 +15.97 KiB (0.01% increase) 203.427 → 203.442 → 204.270
30 successful checks with minimal change (< 2 KiB)
Quality gate Current Size
agent_deb_amd64 744.326 MiB
agent_deb_amd64_fips 702.553 MiB
agent_heroku_amd64 310.884 MiB
agent_msi 608.197 MiB
agent_rpm_amd64 744.310 MiB
agent_rpm_amd64_fips 702.536 MiB
agent_rpm_arm64 722.366 MiB
agent_rpm_arm64_fips 683.645 MiB
agent_suse_amd64 744.310 MiB
agent_suse_amd64_fips 702.536 MiB
agent_suse_arm64 722.366 MiB
agent_suse_arm64_fips 683.645 MiB
docker_agent_amd64 804.911 MiB
docker_agent_arm64 807.761 MiB
docker_agent_jmx_amd64 995.822 MiB
docker_agent_jmx_arm64 987.455 MiB
docker_cluster_agent_arm64 217.835 MiB
docker_cws_instrumentation_amd64 7.135 MiB
docker_cws_instrumentation_arm64 6.689 MiB
docker_dogstatsd_amd64 38.512 MiB
docker_dogstatsd_arm64 36.812 MiB
dogstatsd_deb_amd64 29.736 MiB
dogstatsd_deb_arm64 27.893 MiB
dogstatsd_rpm_amd64 29.736 MiB
dogstatsd_suse_amd64 29.736 MiB
iot_agent_deb_amd64 42.624 MiB
iot_agent_deb_arm64 39.726 MiB
iot_agent_deb_armhf 40.455 MiB
iot_agent_rpm_amd64 42.625 MiB
iot_agent_suse_amd64 42.625 MiB
On-wire sizes (compressed)
Quality gate Change Size (prev → curr → max)
agent_deb_amd64 +6.8 KiB (0.00% increase) 173.741 → 173.748 → 186.090
agent_deb_amd64_fips -12.46 KiB (0.01% reduction) 164.739 → 164.727 → 180.330
agent_heroku_amd64 neutral 74.971 MiB → 88.440
agent_msi -28.0 KiB (0.02% reduction) 137.801 → 137.773 → 154.470
agent_rpm_amd64 +45.52 KiB (0.03% increase) 175.691 → 175.735 → 189.170
agent_rpm_amd64_fips neutral 167.566 MiB → 181.060
agent_rpm_arm64 +20.98 KiB (0.01% increase) 158.012 → 158.032 → 170.020
agent_rpm_arm64_fips +17.66 KiB (0.01% increase) 150.736 → 150.753 → 164.130
agent_suse_amd64 +45.52 KiB (0.03% increase) 175.691 → 175.735 → 189.170
agent_suse_amd64_fips neutral 167.566 MiB → 181.060
agent_suse_arm64 +20.98 KiB (0.01% increase) 158.012 → 158.032 → 170.020
agent_suse_arm64_fips +17.66 KiB (0.01% increase) 150.736 → 150.753 → 164.130
docker_agent_amd64 +5.47 KiB (0.00% increase) 266.268 → 266.274 → 279.410
docker_agent_arm64 neutral 253.551 MiB → 267.960
docker_agent_jmx_amd64 +4.24 KiB (0.00% increase) 334.913 → 334.918 → 348.040
docker_agent_jmx_arm64 neutral 318.172 MiB → 332.560
docker_cluster_agent_amd64 +6.12 KiB (0.01% increase) 71.198 → 71.203 → 71.920
docker_cluster_agent_arm64 -7.23 KiB (0.01% reduction) 66.834 → 66.827 → 67.220
docker_cws_instrumentation_amd64 neutral 2.995 MiB → 3.330
docker_cws_instrumentation_arm64 neutral 2.726 MiB → 3.090
docker_dogstatsd_amd64 neutral 14.903 MiB → 15.820
docker_dogstatsd_arm64 neutral 14.238 MiB → 14.830
dogstatsd_deb_amd64 neutral 7.855 MiB → 8.790
dogstatsd_deb_arm64 neutral 6.743 MiB → 7.710
dogstatsd_rpm_amd64 neutral 7.866 MiB → 8.800
dogstatsd_suse_amd64 neutral 7.866 MiB → 8.800
iot_agent_deb_amd64 neutral 11.238 MiB → 12.040
iot_agent_deb_arm64 neutral 9.602 MiB → 10.450
iot_agent_deb_armhf neutral 9.799 MiB → 10.620
iot_agent_rpm_amd64 neutral 11.256 MiB → 12.060
iot_agent_suse_amd64 neutral 11.256 MiB → 12.060

@clamoriniere clamoriniere force-pushed the clamoriniere/dpa-metrics-improvement branch from db5daee to 0293cd7 Compare February 23, 2026 22:16
@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da Bot commented Feb 23, 2026

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 83659a5f-59d7-4748-b993-01c45306ef7a

Baseline: d75c25a
Comparison: 5d0021b
Diff

❌ Experiments with retried target crashes

This is a critical error. One or more replicates failed with a non-zero exit code. These replicates may have been retried. See Replicate Execution Details for more information.

  • uds_dogstatsd_20mb_12k_contexts_20_senders

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
docker_containers_cpu % cpu utilization +3.07 [-0.05, +6.19] 1 Logs

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
docker_containers_cpu % cpu utilization +3.07 [-0.05, +6.19] 1 Logs
quality_gate_logs % cpu utilization +1.11 [-0.37, +2.60] 1 Logs bounds checks dashboard
quality_gate_metrics_logs memory utilization +0.59 [+0.37, +0.81] 1 Logs bounds checks dashboard
ddot_metrics memory utilization +0.54 [+0.32, +0.75] 1 Logs
docker_containers_memory memory utilization +0.42 [+0.35, +0.49] 1 Logs
ddot_metrics_sum_delta memory utilization +0.34 [+0.14, +0.53] 1 Logs
otlp_ingest_metrics memory utilization +0.29 [+0.13, +0.45] 1 Logs
quality_gate_idle_all_features memory utilization +0.13 [+0.10, +0.16] 1 Logs bounds checks dashboard
file_to_blackhole_1000ms_latency egress throughput +0.05 [-0.37, +0.46] 1 Logs
uds_dogstatsd_to_api_v3 ingress throughput +0.02 [-0.11, +0.15] 1 Logs
ddot_metrics_sum_cumulative memory utilization +0.00 [-0.16, +0.17] 1 Logs
uds_dogstatsd_to_api ingress throughput +0.00 [-0.12, +0.12] 1 Logs
file_to_blackhole_0ms_latency egress throughput +0.00 [-0.47, +0.47] 1 Logs
file_to_blackhole_100ms_latency egress throughput -0.00 [-0.04, +0.04] 1 Logs
tcp_dd_logs_filter_exclude ingress throughput -0.01 [-0.10, +0.09] 1 Logs
file_to_blackhole_500ms_latency egress throughput -0.06 [-0.44, +0.31] 1 Logs
quality_gate_idle memory utilization -0.07 [-0.11, -0.02] 1 Logs bounds checks dashboard
file_tree memory utilization -0.13 [-0.18, -0.07] 1 Logs
ddot_logs memory utilization -0.16 [-0.22, -0.10] 1 Logs
ddot_metrics_sum_cumulativetodelta_exporter memory utilization -0.21 [-0.43, +0.02] 1 Logs
uds_dogstatsd_20mb_12k_contexts_20_senders memory utilization -0.26 [-0.31, -0.20] 1 Logs
otlp_ingest_logs memory utilization -0.42 [-0.51, -0.33] 1 Logs
tcp_syslog_to_blackhole ingress throughput -1.93 [-2.02, -1.85] 1 Logs

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed links
docker_containers_cpu simple_check_run 10/10
docker_containers_memory memory_usage 10/10
docker_containers_memory simple_check_run 10/10
file_to_blackhole_0ms_latency lost_bytes 10/10
file_to_blackhole_0ms_latency memory_usage 10/10
file_to_blackhole_1000ms_latency lost_bytes 10/10
file_to_blackhole_1000ms_latency memory_usage 10/10
file_to_blackhole_100ms_latency lost_bytes 10/10
file_to_blackhole_100ms_latency memory_usage 10/10
file_to_blackhole_500ms_latency lost_bytes 10/10
file_to_blackhole_500ms_latency memory_usage 10/10
quality_gate_idle intake_connections 10/10 bounds checks dashboard
quality_gate_idle memory_usage 10/10 bounds checks dashboard
quality_gate_idle_all_features intake_connections 10/10 bounds checks dashboard
quality_gate_idle_all_features memory_usage 10/10 bounds checks dashboard
quality_gate_logs intake_connections 10/10 bounds checks dashboard
quality_gate_logs lost_bytes 10/10 bounds checks dashboard
quality_gate_logs memory_usage 10/10 bounds checks dashboard
quality_gate_metrics_logs cpu_usage 10/10 bounds checks dashboard
quality_gate_metrics_logs intake_connections 10/10 bounds checks dashboard
quality_gate_metrics_logs lost_bytes 10/10 bounds checks dashboard
quality_gate_metrics_logs memory_usage 10/10 bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

Replicate Execution Details

We run multiple replicates for each experiment/variant. However, we allow replicates to be automatically retried if there are any failures, up to 8 times, at which point the replicate is marked dead and we are unable to run analysis for the entire experiment. We call each of these attempts at running replicates a replicate execution. This section lists all replicate executions that failed due to the target crashing or being oom killed.

Note: In the below tables we bucket failures by experiment, variant, and failure type. For each of these buckets we list out the replicate indexes that failed with an annotation signifying how many times said replicate failed with the given failure mode. In the below example the baseline variant of the experiment named experiment_with_failures had two replicates that failed by oom kills. Replicate 0, which failed 8 executions, and replicate 1 which failed 6 executions, all with the same failure mode.

Experiment Variant Replicates Failure Logs Debug Dashboard
experiment_with_failures baseline 0 (x8) 1 (x6) Oom killed Debug Dashboard

The debug dashboard links will take you to a debugging dashboard specifically designed to investigate replicate execution failures.

❌ Retried Normal Replicate Execution Failures (non-profiling)

Experiment Variant Replicates Failure Debug Dashboard
uds_dogstatsd_20mb_12k_contexts_20_senders baseline 1 Failed to shutdown when requested Debug Dashboard

CI Pass/Fail Decision

Passed. All Quality Gates passed.

  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.

@clamoriniere clamoriniere force-pushed the clamoriniere/dpa-metrics-improvement branch from bd21d28 to 6f09307 Compare February 23, 2026 23:22
@clamoriniere clamoriniere added this to the 7.78.0 milestone Feb 24, 2026
@clamoriniere clamoriniere added qa/done QA done before merge and regressions are covered by tests changelog/no-changelog No changelog entry needed labels Feb 24, 2026
@clamoriniere clamoriniere changed the title [CASCL-623] feat(cluster-agent): enhance DatadogPodAutoscaler metrics with leader checks [CASCL-623] feat(cluster-agent): enhance DatadogPodAutoscaler metrics with dedicate metrics store Feb 24, 2026
Copy link
Copy Markdown

@HemeryJu HemeryJu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not fully familiar with the agent so you might want another review from someone else. But otherwise looks good :)

Comment thread pkg/clusteragent/autoscaling/workload/metrics/generator.go
Comment thread pkg/clusteragent/autoscaling/workload/metrics/generator_test.go Outdated
Comment thread pkg/clusteragent/autoscaling/workload/metrics/store.go Outdated
@clamoriniere clamoriniere requested a review from a team as a code owner February 24, 2026 12:49
@dd-octo-sts dd-octo-sts Bot added the team/container-platform The Container Platform Team label Feb 24, 2026
Comment thread pkg/clusteragent/metricsstore/store.go Outdated
@clamoriniere clamoriniere force-pushed the clamoriniere/dpa-metrics-improvement branch 2 times, most recently from eb9b548 to 83f7d29 Compare February 25, 2026 22:38
@clamoriniere clamoriniere force-pushed the clamoriniere/dpa-metrics-improvement branch 2 times, most recently from 792b7d5 to dfa0cb4 Compare February 25, 2026 23:33
@agent-platform-auto-pr
Copy link
Copy Markdown
Contributor

agent-platform-auto-pr Bot commented Feb 25, 2026

Files inventory check summary

File checks results against ancestor d75c25ac:

Results for datadog-agent_7.78.0~devel.git.124.5d0021b.pipeline.99408206-1_amd64.deb:

No change detected

@clamoriniere clamoriniere force-pushed the clamoriniere/dpa-metrics-improvement branch 2 times, most recently from 78c9b12 to d0de528 Compare February 26, 2026 20:21
… checks and tests

  Improve the metrics generation and submission for DatadogPodAutoscaler:

  * **Observer pattern enhancement**: Changed autoscaling.ObserverFunc signature
    from func(string, string) to func(string, interface{}) to allow observers
    direct access to stored objects without additional lookups

  * **Centralized tag generation**: Refactored generator.go to use helper
    functions (baseAutoscalerTags, autoscalerTagsWithSource,
    autoscalerTagsWithContainer, conditionTags) making it easier to add or
    modify tags across all metrics

  * **Leader-only metric submission**: Added isLeader checks to
    SenderMetricsWriter, horizontalController, and verticalController to
    prevent non-leader instances from submitting duplicate metrics

  * **Backward compatibility**: Retained isLeader:true tag on all metrics
    via baseAutoscalerTags helper function
@clamoriniere clamoriniere force-pushed the clamoriniere/dpa-metrics-improvement branch from d0de528 to 5d0021b Compare February 27, 2026 13:06
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot merged commit 3561a7b into main Feb 27, 2026
287 checks passed
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d Bot deleted the clamoriniere/dpa-metrics-improvement branch February 27, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog No changelog entry needed internal Identify a non-fork PR long review PR is complex, plan time to review it qa/done QA done before merge and regressions are covered by tests team/container-autoscaling team/container-platform The Container Platform Team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants