metricbeat can't collect metrics - permission denied

[nskk9]

Problem description

Hello, for some reason I can't get metricbeat to collect metrics from the operating system and services running on the server. I get logs about permission denied despite running metricbeat as root user. Metrics from elk-modules and beats are displayed correctly in the Stack Monitoring panel.

Extra information

Stack configuration

Tested on 9.0.1 and 8.18.3

Docker setup

$ docker version

Client: Docker Engine - Community
 Version:           28.3.2
 API version:       1.51
 Go version:        go1.24.5
 Git commit:        578ccf6
 Built:             Wed Jul  9 16:13:55 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.3.2
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.5
  Git commit:       e77ff99
  Built:            Wed Jul  9 16:13:55 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker compose version

Docker Compose version v2.38.2

Container logs

$ docker compose logs

metricbeat-1  | {"log.level":"error","@timestamp":"2025-07-16T08:39:25.778Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError","file.name":"module/wrapper.go","file.line":324},"message":"Error fetching data for metricset system.process: non fatal error; reporting partial metrics: error fetching PID metrics for 106 processes, most likely a \"permission denied\" error. Enable debug logging to determine the exact cause.","service.name":"metricbeat","ecs.version":"1.6.0"}

$ docker compose logs

metricbeat-1  | {"log.level":"debug","@timestamp":"2025-07-16T09:21:20.316Z","log.logger":"processes","log.origin":{"function":"github.com/elastic/elastic-agent-system-metrics/metric/system/process.(*Stats).Get","file.name":"process/process.go","file.line":144},"message":"error fetching process metrics: non fatal error; reporting partial metrics: error for pid 1: non fatal error; reporting partial metrics: /io unavailable; if running inside a container, use SYS_PTRACE: error fetching IO metrics: read /hostfs/proc/1/io: permission denied\nerror for pid 2: non fatal error; reporting partial metrics: /io unavailable; if running inside a container, use SYS_PTRACE: error fetching IO metrics: read /hostfs/proc/2/io: permission denied\nerror for pid 3: non fatal error; reporting partial metrics: /io unavailable; if running inside a container, use SYS_PTRACE: error fetching IO metrics: read /hostfs/proc/3/io: permission denied\nerror for pid 4: non fatal error; reporting partial metrics: /io unavailable; if running inside a container, use SYS_PTRACE: error fetching IO metrics: read /hostfs/proc/4/io: permission denied\n

docker-compose.yml

  metricbeat:
    build:
      context: metricbeat/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    # Run as 'root' instead of 'metricbeat' (uid 1000) to allow reading
    # 'docker.sock' and the host's filesystem.
    user: root
    command:
      # Log to stderr.
      - -e
      # Disable config file permissions checks. Allows mounting
      # 'config/metricbeat.yml' even if it's not owned by root.
      # see: https://www.elastic.co/guide/en/beats/libbeat/8.18/config-file-permissions.html
      - --strict.perms=false
      # Mount point of the host’s filesystem. Required to monitor the host
      # from within a container.
      - --system.hostfs=/hostfs
    volumes:
      - ./metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro,Z
      - type: bind
        source: /
        target: /hostfs
        read_only: true
      - type: bind
        source: /sys/fs/cgroup
        target: /hostfs/sys/fs/cgroup
        read_only: true
      - type: bind
        source: /proc
        target: /hostfs/proc
        read_only: true
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true
    environment:
      METRICBEAT_INTERNAL_PASSWORD: ${METRICBEAT_INTERNAL_PASSWORD:-}
      MONITORING_INTERNAL_PASSWORD: ${MONITORING_INTERNAL_PASSWORD:-}
      BEATS_SYSTEM_PASSWORD: ${BEATS_SYSTEM_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch

metricbeat.yml

## Metricbeat configuration
## https://github.com/elastic/beats/blob/main/deploy/docker/metricbeat.docker.yml
#

name: metricbeat

metricbeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

metricbeat.modules:
- module: elasticsearch
  hosts: [ http://elasticsearch:9200 ]
  username: monitoring_internal
  password: ${MONITORING_INTERNAL_PASSWORD}
  xpack.enabled: true
  period: 10s
  enabled: true
- module: logstash
  hosts: [ http://logstash:9600 ]
  xpack.enabled: true
  period: 10s
  enabled: true
- module: kibana
  hosts: [ http://kibana:5601 ]
  username: monitoring_internal
  password: ${MONITORING_INTERNAL_PASSWORD}
  xpack.enabled: true
  period: 10s
  enabled: true
- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    #- image
    - memory
    - network
  hosts: [ unix:///var/run/docker.sock ]
  period: 10s
  enabled: true

processors:
  - add_cloud_metadata: ~

monitoring:
  enabled: true
  elasticsearch:
    username: beats_system
    password: ${BEATS_SYSTEM_PASSWORD}
  cluster_uuid: "Jc5qzYtsRnWqgJvelGkvFw"

output.elasticsearch:
  hosts: [ http://elasticsearch:9200 ]
  username: metricbeat_internal
  password: ${METRICBEAT_INTERNAL_PASSWORD}

## HTTP endpoint for health checking
## https://www.elastic.co/guide/en/beats/metricbeat/8.18/http-endpoint.html
#

http:
  enabled: true
  host: 0.0.0.0

[antoineco]

if running inside a container, use SYS_PTRACE

You're going to need an extra capability added to the Compose service despite running as root:

cap_add:
  - SYS_PTRACE

[antoineco]

I'm flagging this as a bug. We have diskio enabled by default so we should also enable that capability by default. Thanks for reporting!

[antoineco]

I pushed a fix, however please note that this capability may not be sufficient in all environments.
If AppArmor or SELinux are enabled on the host, some permissions may be denied unless the container is either running with escalated privileges or configured with the appropriate policy.

For instance, my host has AppArmor enabled and the docker-default profile does not allow reading from /proc/<pid>/io regardless of the added SYS_PTRACE capability:

audit: type=1400 audit(1753219199.297:115065): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=11339 comm="metricbeat" requested_mask="read" denied_mask="read" peer="unconfined"

[antoineco]

Note that the dashboards that come pre-loaded in Kibana are suitable for metrics originating from Elastic Agents, not from Metricbeat.

Metricbeat also includes dashboards, but these need to be loaded explicitly:

1. https://www.elastic.co/docs/reference/beats/metricbeat/setup-kibana-endpoint
2. https://www.elastic.co/docs/reference/beats/metricbeat/configuration-dashboards

Loading the dashboards is also possible using Metricbeat's setup command:

docker compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml \
  exec metricbeat \
  metricbeat setup --help