PVO11Y-5347 Add cardinality exporter labels to production writeRelabelConfigs

[kubasikus]

Summary

Add cardinality exporter labels (metric, label, label_pair, scraped_instance, sharded_instance, instance_namespace) to the production LabelKeep allowlist. These were added to staging but never promoted to production, causing the cardinality exporter metrics to reach RHOBS with their dimension labels stripped.

Risk Assessment

• Level: Low
• Description: Adds label names to the existing allowlist. Only affects series that already have these labels.
• Rollback: Revert the commit.

Validation

• Same labels already validated in staging since PR PVO11Y-5128 Allow cardinality exporter labels through remote-write #12819
• Production cardinality exporter deployed and running on stone-prod-p01 and kflux-prd-rh02

[github-actions]

Kustomize Render Diff

Comparing 54f4bb3ee → 2ec393e3e

Component	Environment	Changes
components/monitoring/prometheus/production/base	production	+1 -1
components/monitoring/prometheus/production/kflux-fedora-01	production	+1 -1
components/monitoring/prometheus/production/kflux-ocp-p01	production	+1 -1
components/monitoring/prometheus/production/kflux-osp-p01	production	+1 -1
components/monitoring/prometheus/production/kflux-prd-es01	production	+1 -1
components/monitoring/prometheus/production/kflux-prd-rh02	production	+1 -1
components/monitoring/prometheus/production/kflux-prd-rh03	production	+1 -1
components/monitoring/prometheus/production/kflux-rhel-p01	production	+1 -1
components/monitoring/prometheus/production/stone-prd-rh01	production	+1 -1
components/monitoring/prometheus/production/stone-prod-p01	production	+1 -1
components/monitoring/prometheus/production/stone-prod-p02	production	+1 -1

Total: 11 components, +11 -11 lines

📋 Full diff available in the workflow summary and as a downloadable artifact.

[kubasikus]

/rerun

[github-actions]

Re-triggered 1 of 1 failed GitHub Actions workflow run(s).

Prow/OpenShift CI E2E checks are not affected — comment /retest to re-trigger those.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.11%. Comparing base (54f4bb3) to head (7078822).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #13117   +/-   ##
=======================================
  Coverage   57.11%   57.11%           
=======================================
  Files          23       23           
  Lines        1455     1455           
=======================================
  Hits          831      831           
  Misses        548      548           
  Partials       76       76           

Flag	Coverage Δ
go	57.11% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[qodo-for-redhat-appstudio]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 3 rules

1. Allowlist duplication drift ✗ Dismissed 🐞 Bug ⚙ Maintainability

Description

Production and staging maintain separate, long LabelKeep allowlist regexes that already differ in
content, increasing the chance of future environment drift and repeated “missing label” incidents.
This PR updates only the production copy, reinforcing that changes must be manually kept in sync
across envs.

Code

components/monitoring/prometheus/production/base/federation/writeRelabelConfigs.yaml[R45-46]

+      condition|issuer_name|issuer_kind|issuer_group|\
+      metric|label|label_pair|scraped_instance|sharded_instance|instance_namespace"

Relevance

⭐⭐ Medium

No clear history accepting “dedupe prod/stage allowlist”; prior PRs keep separate env copies and
promote manually.

PR-#12162
PR-#12876
PR-#12920

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The repo contains two separate allowlist patch files for production and staging with very similar
(but not identical) long regex strings, demonstrating duplication and existing drift risk. A past
accepted bug in the same production allowlist area was also caused by a missing allowlisted label,
supporting this as a recurring pattern.

components/monitoring/prometheus/production/base/federation/writeRelabelConfigs.yaml[28-46]
components/monitoring/prometheus/staging/base/federation/writeRelabelConfigs.yaml[29-48]
components/monitoring/prometheus/staging/base/federation/writeRelabelConfigs.yaml[41-43]
PR-#12920

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Production and staging each carry their own copy of a large `LabelKeep` regex allowlist. The copies already differ (e.g., staging includes `normalized_uri` while production does not), which makes it easy to introduce accidental drift and to miss label promotions.

## Issue Context
This PR adds cardinality-exporter labels to production. Because the allowlist is duplicated across envs, changes require repeated manual edits and are harder to audit.

## Fix Focus Areas
- components/monitoring/prometheus/production/base/federation/writeRelabelConfigs.yaml[1-46]
- components/monitoring/prometheus/staging/base/federation/writeRelabelConfigs.yaml[1-48]
- components/monitoring/prometheus/production/base/federation/kustomization.yaml[1-26]
- components/monitoring/prometheus/staging/base/federation/kustomization.yaml[1-34]

## Suggested approach
- Create a shared/base allowlist fragment (or generate both env overlays from one canonical list) and have both production and staging reference it.
- If env-specific deltas are required, isolate them into small, clearly scoped additions so that the shared core remains identical.
- Optionally add a CI check (or a lint script) that flags unintended divergence between the two allowlists.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Production base affects all clusters 📘 Rule violation ☼ Reliability

Description

Updating the shared production writeRelabelConfigs allowlist in the production/base component
will apply to every production Prometheus overlay that includes ../base, resulting in a global
rollout instead of a ring-scoped change. This violates the requirement to roll out production
changes in rings to reduce blast radius.

Code

components/monitoring/prometheus/production/base/federation/writeRelabelConfigs.yaml[R45-46]

+      condition|issuer_name|issuer_kind|issuer_group|\
+      metric|label|label_pair|scraped_instance|sharded_instance|instance_namespace"

Relevance

⭐ Low

Ring-scoped rollout suggestions for production/base changes were rejected previously (same
file/pattern).

PR-#12920
PR-#13007
PR-#12898

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 491 requires production rollouts to be split into rings/subsets, not applied to all
production clusters at once. The changed allowlist is in the shared production/base federation
patch (writeRelabelConfigs.yaml), and multiple production cluster overlays include ../base,
demonstrating the change will roll out broadly across production clusters in a single PR.

Rule 491: Split production rollouts into rings, never all clusters at once
components/monitoring/prometheus/production/base/federation/writeRelabelConfigs.yaml[45-46]
components/monitoring/prometheus/production/base/kustomization.yaml[3-9]
components/monitoring/prometheus/production/stone-prod-p01/kustomization.yaml[1-6]
components/monitoring/prometheus/production/kflux-prd-rh02/kustomization.yaml[1-6]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A production-wide base change rolls out to all production clusters at once, instead of being constrained to a ring/subset.

## Issue Context
`components/monitoring/prometheus/production/*/kustomization.yaml` overlays include `../base`, and `production/base` includes `federation/` which applies the `writeRelabelConfigs.yaml` patch. Changing this file therefore impacts all production clusters that consume the base.

## Fix Focus Areas
- components/monitoring/prometheus/production/base/federation/writeRelabelConfigs.yaml[45-46]
- components/monitoring/prometheus/production/base/kustomization.yaml[3-9]
- components/monitoring/prometheus/production/stone-prod-p01/kustomization.yaml[1-6]
- components/monitoring/prometheus/production/kflux-prd-rh02/kustomization.yaml[1-6]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[kubasikus]

Response to Qodo:

Production base affects all clusters

Yes, but since this exporter will land on all clusters, it is fine to deploy as is.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kubasikus, pumahaka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• components/monitoring/OWNERS [kubasikus,pumahaka]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dgregor]

Production Approval Record

Field	Value
Action	APPROVED
Reviewer	@dgregor
Timestamp	2026-07-20T11:17:09.358Z
Risk Level	low