Incomplete RBAC validation for target allocator scrapeconfigs and probes permissions

[u-kai]

Component(s)

target allocator

What happened?

Description

The Target Allocator webhook validation only checks permissions defined in targetAllocatorCRPolicyRules, but the
actual runtime requires additional permissions for scrapeconfigs and probes resources. This causes the validation
to pass while runtime operations fail with permission errors.

Steps to Reproduce

1. Deploy OpenTelemetry Collector with Target Allocator enabled and Prometheus CR functionality
2. Webhook validation passes (only checks servicemonitors/podmonitors permissions)
3. Target Allocator starts and attempts to watch scrapeconfigs and probes resources
4. Runtime permission denied errors occur for these additional resources

Expected Result

• Webhook validation should check ALL permissions required at runtime
• Target Allocator should have complete RBAC permissions for all resources it accesses
• No permission errors should occur during operation

Actual Result

• Webhook validation passes with incomplete permission check
• Runtime fails with permission denied errors for scrapeconfigs and probes
• Gap between validation logic and actual runtime requirements

Kubernetes Version

1.33

Operator version

v0.137.0

Collector version

v0.137.0

Environment information

Environment

OS: (e.g., "MacOS")

Log output

kai@kainoMacBook-Pro opentelemetry-operator % kubectl logs -n opentelemetry-system otel-daemonset-targetallocator-8846786fd-m55zd
{"level":"info","ts":"2025-10-25T12:30:07Z","msg":"Starting the Target Allocator"}
{"level":"info","ts":"2025-10-25T12:30:07Z","logger":"allocator","msg":"Starting server..."}
{"level":"info","ts":"2025-10-25T12:30:07Z","msg":"Waiting for caches to sync for namespace"}
{"level":"info","ts":"2025-10-25T12:30:08Z","msg":"Caches are synced for namespace"}
{"level":"info","ts":"2025-10-25T12:30:08Z","msg":"Waiting for caches to sync for servicemonitors"}
{"level":"info","ts":"2025-10-25T12:30:08Z","msg":"Caches are synced for servicemonitors"}
{"level":"info","ts":"2025-10-25T12:30:08Z","msg":"Waiting for caches to sync for podmonitors"}
{"level":"info","ts":"2025-10-25T12:30:08Z","msg":"Caches are synced for podmonitors"}
{"level":"info","ts":"2025-10-25T12:30:08Z","msg":"Waiting for caches to sync for probes"}
{"level":"info","ts":"2025-10-25T12:30:08Z","msg":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope"}
{"level":"error","ts":"2025-10-25T12:30:08Z","msg":"Unhandled Error","logger":"UnhandledError","error":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Probe: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-10-25T12:30:09Z","msg":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope"}
{"level":"error","ts":"2025-10-25T12:30:09Z","msg":"Unhandled Error","logger":"UnhandledError","error":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Probe: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-10-25T12:30:11Z","msg":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope"}
{"level":"error","ts":"2025-10-25T12:30:11Z","msg":"Unhandled Error","logger":"UnhandledError","error":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Probe: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-10-25T12:30:12Z","logger":"allocator","msg":"Service Discovery watch event received","targets groups":1}
{"level":"info","ts":"2025-10-25T12:30:14Z","msg":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope"}
{"level":"error","ts":"2025-10-25T12:30:14Z","msg":"Unhandled Error","logger":"UnhandledError","error":"k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Probe: failed to list *v1.Probe: probes.monitoring.coreos.com is forbidden: User \"system:serviceaccount:opentelemetry-system:otel-collector-targetallocator\" cannot list resource \"probes\" in API group \"monitoring.coreos.com\" at the cluster scope","stacktrace":"k8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:166\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:316\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:226\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/backoff.go:227\nk8s.io/client-go/tools/cache.(*Reflector).Run\n\tk8s.io/client-go@v0.32.3/tools/cache/reflector.go:314\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:55\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\tk8s.io/apimachinery@v0.32.3/pkg/util/wait/wait.go:72"}

Additional context

I'm interested in contributing a fix for this issue if the proposed approach looks good to the maintainers.
Happy to submit a PR once we align on the solution.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[MikaelElkiaer]

#4608 might be related - was just hit by this after upgrading to 0.147.0.

[u-kai]

Thanks for the heads up!
I confirmed this is still present on the latest main.

The root cause is in apis/v1beta1/targetallocator_rbac.go:19 —
targetAllocatorCRPolicyRules only includes servicemonitors and podmonitors, missing probes and scrapeconfigs.
So the webhook validation passes without warning, but the TA fails at runtime with forbidden errors for those resources.

I reproduced this on a kind cluster built from main, webhook accepted the CR with no warnings, and the TA immediately started logging scrapeconfigs.monitoring.coreos.com is forbidden and probes.monitoring.coreos.com is forbidden.

PR #4608 fixes a different issue (Secret updatedetection), so it doesn't address this.

[swiatekm]

To begin with, this isn't validation, exactly. We don't reject CRs which don't pass this requirement. Rather, we emit a warning because users usually want this functionality. But not always - it's perfectly possible to use target allocator without any prometheus-operator CRs installed in the cluster at all.

The way we should fix this, if we want to change it at all, is to check which of these CRDs exist in the cluster, and then only include those in the RBAC check.

[u-kai]

@swiatekm
Thanks for the clarification! I created a draft PR following your suggestion — dynamically discovering which monitoring.coreos.com CRDs are installed and only including those in the RBAC check.

#5011

If the approach looks good to you, I'd like to mark it as ready for review.