[clusterobservability] fix config for bundled collectors

[jinja2]

Description: Follow-ups to the initial ClusterObservability framework so the bundled agent and cluster collectors generated from a ClusterObservability CR come up cleanly on stock Kubernetes and OpenShift.

Collector image

• Generate the bundled collectors with the k8s distribution of the collector (opentelemetry-collector-k8s) at the same version as the operator's default collector, exposed as a new Config.ClusterObservabilityCollectorImage and overridable via the --clusterobservability-collector-image flag. This keeps the version tied to the existing opentelemetry-collector entry in versions.txt.

Agent collector (DaemonSet)

• Inject K8S_NODE_NAME via the downward API so the bundled kubeletstats receiver's https://${env:K8S_NODE_NAME}:10250 endpoint resolves.
• Mount the whole host root at /hostfs (read-only, HostToContainer) instead of granular per-directory hostPath mounts, and set the hostmetric receiver's root_path to /hostfs, so the receiver can resolve symlinks and stat every host mount point.
• Exclude virtual / container-runtime filesystems from the hostmetrics filesystem scraper (exclude_mount_points + exclude_fs_types) to avoid /sysroot and /boot scrape errors on RHCOS.
• On OpenShift, run the agent as root (runAsUser: 0, with the spc_t SELinux type supplied by the generated SCC) so the filelog receiver can read the root-owned container logs under /var/log/pods. Capabilities stay dropped and privilege escalation disabled.

Cluster collector

• checkClusterCollectorStatus now reads the StatefulSet workload, so the CR no longer stays in Pending when the cluster collector is fully healthy (the cluster collector was moved to a StatefulSet in feat(clusterobservability): enable TargetAllocator support for cluster collectors #5139).

Auto-instrumentation

• Point the generated Instrumentation at the agent's OTLP HTTP port (:4318); most auto-instrumentations default to http/protobuf.

Refactor

• Move distro detection in the manifests builder to the shared config.DistroProvider, and extract agent security contexts / volumes / mounts into distro-gated helper functions.

Link to tracking Issue(s)

• Create an integration test checking the output of the managed CR application Create an integration test checking the output of the managed CR application #3821
• [Managed] Create a best practice managed experience with the operator [Managed] Create a best practice managed experience with the operator #3818

Testing

• New make e2e-clusterobservability chainsaw target. make prepare-e2e-clusterobservability runs add-rbac-permissions-to-operator (opt-in kustomize patches) and bakes in --feature-gates=+operator.clusterobservability for the e2e job.
• The chainsaw suite asserts telemetry flow end-to-end: kubeletstats/hostmetrics/filelog signals reach the sink, and a Node.js app with auto-instrumentation produces spans that arrive at the cluster collector (otelcol_receiver_accepted_spans).
• Reusable e2e workflow gets the new group, experimental: true with matching continue-on-error, so failures don't block other operator e2e jobs.
• Kind 1.25 / 1.35 enable serverTLSBootstrap; start-kind approves the kubelet-serving CSRs so the kubeletstats receiver can verify the kubelet against the cluster CA.
• Unit tests, including loader_test.go guarding the filesystem-exclude and filelog config.
• Manually validated on a ROSA (OpenShift) cluster: agent DaemonSet healthy, filelog/hostmetrics/kubeletstats reporting, auto-instrumented app traces flowing, and ClusterObservability status Available.

[github-actions]

E2E Test Results

 40 files  273 suites   2h 24m 20s ⏱️
116 tests 115 ✅ 1 💤 0 ❌
295 runs  293 ✅ 2 💤 0 ❌

Results for commit 92fc794.

♻️ This comment has been updated with latest results.

[pavolloffay]

I wonder if we could use the k8s distribution instead?

[frzifus]

Currently the default images are provided by setting via ldflags and reading the version from the version.txt file.
Right now the default is the core collector in version 0.152.1. During one of the SIG calls we discussed to maybe use the k8s distro as default. In that case, I think we dont need to change anything.

If we continue to use core, we may want to add a separate version entry for CollectorCRs generated by the ClusterObservability reconciler.

[jinja2]

Thanks! Switched to the k8s distribution. Instead of string-rewriting the image, the reconciler now reads a dedicated Config.ClusterObservabilityCollectorImage, which defaults to opentelemetry-collector-k8s:<version> using the existing opentelemetry-collector entry in versions.txt, with --clusterobservability-collector-image as an override.

I derived the version rather than adding a separate versions.txt entry for now, given the experimental maturity here. That said, I can see the case for an explicit opentelemetry-collector-k8s entry - it'd be self-documenting and let maintainers pin/patch the k8s distro independently if the release process ever diverges. Do you have a preference?

[frzifus]

Awesome, I will test it on talos and openshift during the week!

[frzifus]

Currently the default images are provided by setting via ldflags and reading the version from the version.txt file.
Right now the default is the core collector in version 0.152.1. During one of the SIG calls we discussed to maybe use the k8s distro as default. In that case, I think we dont need to change anything.

If we continue to use core, we may want to add a separate version entry for CollectorCRs generated by the ClusterObservability reconciler.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: jinja2 / name: Jina Jain (58415c9)