Skip to content

awsentity/service/telegraf processor auto-injected into statsd pipeline silently drops all custom metrics in agent versions 1.300031+ #2180

Description

@tthoma24

Describe the bug
When configuring the CloudWatch agent with a statsd-only custom metrics configuration on EC2,
all metrics are silently dropped and never published to CloudWatch. The agent receives statsd
packets, scrapes them on schedule, but never calls PutMetricData. No error is logged even
in debug mode.

The root cause appears to be that the config translator automatically injects the
awsentity/service/telegraf processor into the statsd pipeline regardless of whether
Application Signals is configured. This processor has scrape_datapoint_attribute: true,
which requires entity metadata attributes on metric datapoints that standard statsd metrics
never carry, causing them to be silently dropped before reaching the CloudWatch exporter.

Steps to reproduce

  1. Launch an EC2 instance (Ubuntu 24.04, amd64)
  2. Install CloudWatch agent version 1.300069 via SSM AWS-ConfigureAWSPackage
  3. Create the following minimal configuration and load it with fetch-config:
{
  "agent": {
    "metrics_collection_interval": 60
  },
  "metrics": {
    "namespace": "MyNamespace",
    "metrics_collected": {
      "statsd": {
        "service_address": ":8125",
        "metrics_collection_interval": 60,
        "metrics_aggregation_interval": 60
      }
    }
  }
}
  1. Send a statsd gauge: echo -n "TestMetric:1|g" > /dev/udp/127.0.0.1/8125
  2. Wait 90 seconds
  3. Check CloudWatch — no metrics appear
  4. Check agent log — no publish attempt, no error

What did you expect to see?
TestMetric published to MyNamespace in CloudWatch every 60 seconds.

What did you see instead?
The agent receives and scrapes statsd packets (confirmed via tcpdump on lo:8125 and the
debug log line Begin scraping metrics with adapter receiver=statsd) but never publishes.
No error is logged even with debug: true.

The generated YAML pipeline shows the awsentity processor being injected automatically
despite Application Signals not being configured anywhere in the JSON config:

processors:
    awsentity/service/telegraf:
        entity_type: Service
        platform: ec2
        scrape_datapoint_attribute: true

pipelines:
    metrics/hostCustomMetrics:
        exporters:
            - awscloudwatch
        processors:
            - awsentity/service/telegraf
        receivers:
            - telegraf_statsd

The following warning appears in debug logs:

Failed to get service name from instance tags. This is likely because instance tag is
not enabled for IMDS but will not affect agent functionality.

Despite the claim that this "will not affect agent functionality", all statsd metrics are
silently dropped. Enabling instance tags in IMDS via:

aws ec2 modify-instance-metadata-options \
  --instance-id <instance-id> \
  --instance-metadata-tags enabled \
  --region us-east-1

...and restarting the agent does not resolve the issue. Metrics continue to be dropped.

What version did you use?
Version: v1.300069.0

What config did you use?

{
  "agent": {
    "metrics_collection_interval": 60
  },
  "metrics": {
    "namespace": "MyNamespace",
    "metrics_collected": {
      "statsd": {
        "service_address": ":8125",
        "metrics_collection_interval": 60,
        "metrics_aggregation_interval": 60
      }
    }
  }
}

Environment
OS: Ubuntu 24.04.4 LTS (Noble Numbat), amd64
EC2 instance type: t3.small
Region: us-east-1
Installation method: SSM AWS-ConfigureAWSPackage
IAM permissions: cloudwatch:PutMetricData confirmed working independently via awscli v2

Additional context

  • The config wizard (amazon-cloudwatch-agent-config-wizard) generates the same broken
    pipeline with awsentity injected — there is no wizard option to opt out
  • No error or warning is logged when metrics are dropped, making this extremely difficult
    to diagnose
  • The workaround is to bypass the agent entirely and call
    aws cloudwatch put-metric-data directly from a script
  • This is a regression — statsd custom metrics worked correctly in agent versions prior
    to the introduction of Application Signals pipeline injection (pre-1.300031)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions