ARO-0000: Replace MDSD with OTEL for cluster logging by hlipsig · Pull Request #4881 · Azure/ARO-RP

hlipsig · 2026-06-10T04:14:12Z

Removes the MDSD controller and replaces it with OTEL. Also removes MDSD related tests, and install work on the ARO cluster side. Gateway side unchanged.

What this PR does / why we need it:

OTEL modernizes our observability stack. For this first implementation we are using it for log collection.

Test plan for issue:

Unit Tests, e2e tests follow in a different PR, staging testing and testing against a dev cluster.

Is there any documentation that needs to be updated for this PR?

Yes. A dedicated PR is being created for it.

How do you know this will function as expected in production?

E2E is green plus manual testing mentioned above.

openshift-ci · 2026-06-10T04:14:16Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copilot

Pull request overview

This PR replaces the cluster-side Geneva logging implementation (Fluent Bit + MDSD) with an OpenTelemetry Collector-based approach, removing MDSD-specific configuration/certificate handling and wiring the operator to deploy OTel collector daemonsets that export logs to the gateway.

Changes:

Add new operator flags for selecting OTel logging profiles (global + master/worker overrides) and default to a “minimal-logs” profile.
Replace the genevalogging controller’s Fluent Bit + MDSD daemonset with OTel collector daemonsets and an embedded OTel config template.
Remove MDSD certificate renewal/monitoring and add monitoring for OTel daemonset “cannot start” scenarios; add GatewayTelemetryDomain to the cluster CR spec.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pkg/operator/flags.go	Adds new Geneva logging OTel profile flags and defaults.
pkg/operator/deploy/deploy.go	Stops populating legacy Geneva logging spec fields; sets `GatewayTelemetryDomain` when gateway is enabled.
pkg/operator/deploy/deploy_test.go	Adds unit tests for `gatewayTelemetryDomain`.
pkg/operator/controllers/genevalogging/staticfiles/parsers.conf	Removes legacy Fluent Bit parser config.
pkg/operator/controllers/genevalogging/staticfiles/otel-config.yaml.tmpl	Introduces OTel Collector config template for journald/container/audit log ingestion and OTLP export.
pkg/operator/controllers/genevalogging/staticfiles/fluent.conf	Removes legacy Fluent Bit config.
pkg/operator/controllers/genevalogging/otel_config_template.go	Adds Go templating/rendering for the embedded OTel config.
pkg/operator/controllers/genevalogging/mode.go	Adds OTel profile parsing and master/worker profile selection logic.
pkg/operator/controllers/genevalogging/genevalogging.go	Reworks resources to create OTel configmaps + OTel daemonsets; adds gateway telemetry targeting logic.
pkg/operator/controllers/genevalogging/genevalogging_test.go	Replaces MDSD/Fluent Bit tests with OTel-focused tests and stale resource cleanup coverage.
pkg/operator/controllers/genevalogging/genevalogging_controller.go	Updates controller to remove secret dependency, clean up stale resources, and ensure daemonset node selectors aren’t used to “scale down”.
pkg/operator/controllers/genevalogging/const.go	Replaces Fluent Bit/MDSD constants + embedded files with OTel equivalents.
pkg/operator/apis/aro.openshift.io/v1alpha1/cluster_types.go	Adds `GatewayTelemetryDomain`; empties `GenevaLoggingSpec` now that MDSD fields are removed.
pkg/monitor/cluster/daemonsetstatuses.go	Emits a new gauge when OTel logging daemonsets have desired > 0 but available == 0.
pkg/monitor/cluster/daemonsetstatuses_test.go	Adds test for the new OTel “cannot start” metric emission.
pkg/monitor/cluster/cluster.go	Removes MDSD certificate expiry emission from monitor run list.
pkg/monitor/cluster/certificateexpirationstatuses.go	Removes MDSD cert expiry logic.
pkg/monitor/cluster/certificateexpirationstatuses_test.go	Removes MDSD cert expiry tests.
pkg/cluster/install.go	Removes MDSD certificate renewal from install/update step lists.
pkg/cluster/arooperator.go	Removes manager wrapper method for MDSD certificate renewal.
pkg/cluster/adminupdate_test.go	Updates expected admin update steps to exclude MDSD renewal.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 2 comments.

+func telemetryGatewayTarget(cluster *arov1alpha1.Cluster) (telemetryGatewayTargetSpec, bool, error) {
+	if cluster.Spec.GatewayPrivateEndpointIP == "" {
+		return telemetryGatewayTargetSpec{}, false, nil
+	}
+	gatewayPrivateEndpointIP := net.ParseIP(cluster.Spec.GatewayPrivateEndpointIP)
+	if gatewayPrivateEndpointIP == nil {
+		return telemetryGatewayTargetSpec{}, false, fmt.Errorf("invalid cluster spec field %q: %q is not a valid IP address", "gatewayPrivateEndpointIP", cluster.Spec.GatewayPrivateEndpointIP)
+	}
+


Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 27 out of 29 changed files in this pull request and generated 1 comment.

kimorris27 · 2026-06-18T20:44:56Z

+              gatewayTelemetryDomain:
+                type: string
              genevaLogging:
-                properties:
-                  configVersion:
-                    pattern: '[0-9]+.[0-9]+'
-                    type: string
-                  monitoringGCSAccount:
-                    enum:
-                    - AROClusterLogsINT
-                    - AROClusterLogsPROD
-                    - AROClusterLogs
-                    type: string
-                  monitoringGCSEnvironment:
-                    enum:
-                    - DiagnosticsProd
-                    - Test
-                    - CaFairfax
-                    type: string
-                  monitoringGCSNamespace:
-                    enum:
-                    - AROClusterLogsINT
-                    - AROClusterLogsPROD
-                    - AROClusterLogs
-                    type: string
                type: object
              infraId:


@hlipsig, what's the point of keeping around the object without the attributes?

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 27 out of 28 changed files in this pull request and generated 1 comment.

+.PHONY: publish-image-telemetrycollector-and-print-digest
+publish-image-telemetrycollector-and-print-digest: publish-image-telemetrycollector print-image-digest-telemetrycollector
+


kimorris27

Approved with the understanding that some naming, little follow-up nits, etc. will be addressed when we're under less of a time crunch.

hlipsig commented Jun 10, 2026

View reviewed changes

Comment thread pkg/operator/controllers/genevalogging/genevalogging_controller.go Outdated

hlipsig force-pushed the hlipsig/ARO-0000/aro-operator-otel branch from c3ff353 to b1b7e1c Compare June 10, 2026 18:01

cadenmarchese reviewed Jun 10, 2026

View reviewed changes

Comment thread pkg/monitor/cluster/daemonsetstatuses.go Outdated

Comment thread pkg/monitor/cluster/daemonsetstatuses.go

Comment thread pkg/operator/controllers/genevalogging/staticfiles/otel-config.yaml.tmpl

Comment thread pkg/operator/controllers/genevalogging/otel_config_template.go Outdated

ehvs reviewed Jun 11, 2026

View reviewed changes

Comment thread pkg/operator/controllers/genevalogging/staticfiles/otel-config.yaml.tmpl

hlipsig marked this pull request as ready for review June 11, 2026 14:25

Copilot AI review requested due to automatic review settings June 11, 2026 14:25

hlipsig requested review from alcasim, bennerv, hawkowl, kevinobriendotca, kimorris27, mociarain, mrWinston, rogbas, sankur-codes, tiguelu, tsatam, tuxerrante, ventifus, wanghaoran1988 and yjst2012 as code owners June 11, 2026 14:25

Copilot started reviewing on behalf of hlipsig June 11, 2026 14:26 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 11, 2026 14:45

Copilot started reviewing on behalf of hlipsig June 11, 2026 14:45 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

Comment thread pkg/operator/controllers/genevalogging/otel_config_template.go

Comment thread pkg/deploy/assets/env-development.json Outdated

Comment thread pkg/operator/controllers/genevalogging/staticfiles/otel-config.yaml.tmpl

Copilot started work on behalf of hlipsig June 11, 2026 14:57 View session

Copilot finished work on behalf of hlipsig June 11, 2026 15:04

Copilot started work on behalf of hlipsig June 11, 2026 16:00 View session

Copilot AI review requested due to automatic review settings June 18, 2026 17:24

Copilot started reviewing on behalf of hlipsig June 18, 2026 17:25 View session

Copilot started work on behalf of hlipsig June 18, 2026 17:27 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread pkg/deploy/generator/scripts/rpVMSS.sh Outdated

Comment thread pkg/operator/controllers/genevalogging/genevalogging.go Outdated

Copilot AI added 2 commits June 18, 2026 17:36

Harden validate-go tool bootstrap

d0964cf

Finalize validate-go fix

e1df288

Copilot AI review requested due to automatic review settings June 18, 2026 17:40

hlipsig removed the request for review from Copilot June 18, 2026 17:40

Copilot finished work on behalf of hlipsig June 18, 2026 17:40

Potential fix for pull request finding

1d7f06f

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 18, 2026 17:44

Copilot started reviewing on behalf of hlipsig June 18, 2026 17:45 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Copilot started work on behalf of hlipsig June 18, 2026 17:56 View session

Fix .bingo/go.mod Go version from 1.24.13 to 1.25.3

442b5c6

Copilot finished work on behalf of hlipsig June 18, 2026 18:04

Copilot started work on behalf of hlipsig June 18, 2026 19:35 View session

Fix generate-check by syncing bingo go.mod

20946a9

Copilot AI review requested due to automatic review settings June 18, 2026 19:48

hlipsig removed the request for review from Copilot June 18, 2026 19:48

Finalize generate-check fix

0227b52

Copilot AI review requested due to automatic review settings June 18, 2026 19:51

Restore Bingo fake go.mod format

7017ce5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread Makefile

kimorris27 reviewed Jun 18, 2026

View reviewed changes

Hilliary Lipsig (RED HAT INC) and others added 2 commits June 18, 2026 14:42

Remove validate-go prefetch bootstrap

628797b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Normalize .bingo/go.mod EOF formatting

b100c5c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread Makefile

Comment on lines +262 to +264

.PHONY: publish-image-telemetrycollector-and-print-digest

publish-image-telemetrycollector-and-print-digest: publish-image-telemetrycollector print-image-digest-telemetrycollector

tsatam approved these changes Jun 18, 2026

View reviewed changes

kimorris27 approved these changes Jun 18, 2026

View reviewed changes

		.PHONY: publish-image-telemetrycollector-and-print-digest
		publish-image-telemetrycollector-and-print-digest: publish-image-telemetrycollector print-image-digest-telemetrycollector

Uh oh!

Conversation

hlipsig commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

How do you know this will function as expected in production?

Uh oh!

openshift-ci Bot commented Jun 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kimorris27 Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

kimorris27 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hlipsig commented Jun 10, 2026 •

edited

Loading