chore: replace control-plane promtail with alloy

[ma-hartma]

Description

Replaces promtail with alloy in the control-plane.
The partition replacement is treated separately in #592 and should go first.

• deploy alloy
  • role for helm chart in k8s cluster
• convert existing configs
  • pods
  • events
  • labels
  • metrics
• logs and metrics for alloy itself (https://grafana.com/docs/alloy/latest/collect/metamonitoring/=) to loki and prometheus
• WAL persistence to allow restarts
• check promtail helm charts, as they had lots of defaults that alloy doesn't: https://github.com/grafana/helm-charts/tree/main/charts/promtail

Question Marks:

• evaluate using https://github.com/grafana/k8s-monitoring-helm instead of https://github.com/prometheus-operator/kube-prometheus

After successful replacement

• update monitoring docs for the website
• migration guide
  • convert does not work with helm chart
• deprecate promtail roles

Manually tested in the mini-lab:

• metrics are pushed to thanos, accessible via Grafana
• labels work as documented and before
• cutover migration from promtail to alloy in the mini-lab including cleanup
• deprecation warnings show up
• Gardener Logging (do not know how to test this, maybe we can do it together @simcod?)

References:

• chore: replace partition promtail with alloy #592
• docs(monitoring): use alloy instead of promtail website#260 website docs update
• chore: add alloy helm chart releases#288
• Sflow integration #584 will have to replace promtail as well

Closes #552

Used AI-Tools ✨

• Claude Sonnet 4.6 for docs and migration guide generation and jinja templating

Release Notes

Required Actions

The `logging` and `gardener-logging` roles now support Grafana Alloy as the log collector. Promtail continues to run by default after upgrading — no breakage on upgrade — but **a deprecation warning will fire on every Ansible run** until you migrate. Promtail support will be removed in a future release, so plan your migration soon.

Follow the migration instructions:
- https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/logging/README.md#migration-from-promtail
- https://github.com/metal-stack/metal-roles/blob/master/control-plane/roles/gardener-logging/README.md#migration-from-promtail

Several migration scenarios are supported — from a hard cutover to a parallel run for verification before switching over. Note that running both simultaneously will produce duplicate log entries in Loki during the transition window.

Noteworthy

The `logging` and `gardener-logging` roles now use Grafana Alloy as the log collector instead of Promtail. Kubernetes events are collected natively without a separate event-exporter pod. Log positions are persisted across pod restarts to avoid duplicate log shipments. Alloy self-metrics are push-based via `prometheus.remote_write` — no ServiceMonitor required. New `*_promtail_migrate_cleanup` flags automate Helm release removal after cutover. Labels are identical to Promtail's defaults.

[ma-hartma]

The event-exporter used with promtail had the events labelled with job=monitoring/event-exporter.
I did actually re-label the logs from job=kubernetes-events to also match the legacy job name.
I haven't found any usage in metal-roles, but other users may rely on it.

[ma-hartma]

I'll leave the old username for now.

[ma-hartma]

In the shooted seeds, we do not have an in-cluster prometheus. Here we could actually use ServiceMonitors but I decided against it, because we would introduce a dependency on prometheus already existing before deploying the helm chart.