ScrapeConfig targetAllocator bad password using basicAuth

[snahelou]

Component(s)

targetAllocator

What happened?

Hi ;)

It seems that the Target Allocator is picking the wrong password when using ScrapeConfig from Prometheus Operator (no issue with username in the same secret)

Description

The following ScrapeConfig is working fine using Prometheus Operator :

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: demo-scrapeconfig
spec:
  staticConfigs:
    - labels:
        job: demo
      targets:
        - demo-remote-service:9999
  metricsPath: /metrics
  scheme: HTTPS
  tlsConfig:
    insecureSkipVerify: true
  basicAuth:
    username:
      name: basic-auth
      key: user
    password:
      name: basic-auth
      key: password

But when using targetAllocator, I get 401 errors in OtelColector logs:

2025-02-25T18:41:37.945Z	debug	scrape/scrape.go:1347	Scrape failed	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "scrapeConfig/demo/demo-scrapeconfig", "target": "https://demo-remote-service:9999/metrics", "error": "server returned HTTP status 401 Unauthorized"}

Using curl 'localhost:8080/scrape_configs' | jq to get the scrape config, I can see that the password is not the one created in the k8s secret

  "scrapeConfig/demo/demo-scrapeconfig": {
    "basic_auth": {
      "password": "<secret>",
      "username": "demo-user"
    },

Using a remote webhook receiver to troubleshoot the final query, I can confirm that the secret value is not good

Steps to Reproduce

1. Create secret k create secret generic basic-auth-demo --from-literal='user=username-example' --from-literal='password=password-example'

2. Create a ScrapeConfig

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: demo-scrape-config
  labels:
    app: kube-prometheus-stack-prometheus
    release: prometheus-operator
    prometheus: system-monitoring-prometheus
spec:
  staticConfigs:
    - labels:
        job: demo-scrape-config
      targets:
        - webhook.site
  metricsPath: /3ad6310f-79ed-4816-b74c-5616ff68d2a1 #Change here with your endpoint
  scheme: HTTPS
  tlsConfig:
    insecureSkipVerify: true 
  basicAuth:
    username:
      name: basic-auth-demo
      key: user
    password:
      name: basic-auth-demo
      key: password

3. Use portforward / curl on TA to get config

curl 'localhost:8080/scrape_configs' | jq
... 
  "scrapeConfig/test-sna/demo-scrape-config": {
    "basic_auth": {
      "password": "<secret>",
      "username": "username-example"
    },

4. Check the webhook site

Expected Result

The right password:

    "basic_auth": {
      "password": "password-example",
      "username": "username-example"
    },

Actual Result

    "basic_auth": {
      "password": "<secret>",
      "username": "username-example"
    },

Kubernetes Version

v1.30.9

Operator version

otel/opentelemetry-collector-k8s:0.117.0

Collector version

otel/opentelemetry-collector-k8s:0.117.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

2025-02-25T18:41:37.945Z	debug	scrape/scrape.go:1347	Scrape failed	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "scrapeConfig/demo/demo-scrapeconfig", "target": "https://demo-remote-service:9999/metrics", "error": "server returned HTTP status 401 Unauthorized"}

Additional context

My OTEL Deployment:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: collector-with-ta
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
    prometheusCR:
      enabled: true
      serviceMonitorSelector: {}
      scrapeConfigSelector: {}
  config:
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 10s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]
            metric_relabel_configs:
            - action: labeldrop
              regex: (id|name)
            - action: labelmap
              regex: label_(.+)
              replacement: $$1

    exporters:
      debug: {}

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [debug]
      telemetry:
        logs:
          level: "debug"
        metrics:
          address: 0.0.0.0:8888

[CharlieTLe]

It has to do with how the scrape config response from the TA depends on whether the communication is happening via mTLS or not.

#2921

Enabling the mTLS feature flag on the operator should set up the collector and TA so that the scrape config can be readable by the collector.

It's defined here and used here.

[snahelou]

Does it mean that mTLS between TA & collector is mandatory to work properly with a basicAuth in scrapeConfig ?

[ChristianCiach]

I would like to have an option to expose the real secrets even when using the TA in http mode. Our service-mesh and Network-Policies already ensure a secure tunnel between the pods. There is no need to tunnel mTLS through mTLS.

[tillwulfram]

Heyhey,
im also interested in this feature and would like to propose two solutions which im currently seeing and would appreciate Feedback for that, based on the coming implementation from #4915 - #4971
Those implementations don't directly solve this issue because it still requires mtls enabled to retrieve secrets from endpoints with basicauth. It's nice that you can byo mtls certs, but inside of a service mesh it would require double mtls. And also just for test env to use this features it is to much overhead for this great feature (especially when requirements are running the ta with collector as standalone and without operator)
I would like to propose two approaches which would tackle in first place to disable the requirement of mtls for secrets when param is enabled like here
cmd/otel-allocator/internal/config/config.go and cmd/otel-allocator/internal/server/server.go to allow a new parameter to provide secret content via http and without client cert.
Also change https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/internal/server/server.go#L237

config param would be something like:
allowSecretsOverHTTP = true (disabled by default to align optin and backwards compatibility)

Additionally the ta could log something like WARN - secrets will be served via plain HTTP. Not recommended for production only use this if your network is secured by service mesh.

To enable this config parameter the suggested options would be:

1. enable it via container args.
   • TA / Operator CR - should already allow setting args which will be rendered to the container so no change
   • TA Standalone Helm chart - would require the allowance to define args but is no big change (I can add another issue then) and also add an similar key for retrieving secrets to (can be combined / set automatically the args flag) and add the get secret permission to the cluster role when enabled
2. enable it via an ConfigMap key
   • TA / Operator CR - config parameter like the mtls feature will do but has to add new fields, reconcile and more overhead
   • TA Standalone Helm Chart - the user can just add the parameter in the ConfigMap in helm oder byo cm so no change

I would choose option 1 bc its less invasive but im open for discussion and also other solutions :)

@swiatekm (and anyone else)

[swiatekm]

I'm fine with allowing auth secret exposure without mtls, as long as we make it very clear to users that it's on them to secure the target allocator endpoint in that case. The setting would need to be called something like allowInsecureAuthSecrets.

[tillwulfram]

Thanks for the Input! I would like to give it a try