Token/secret from file

[fopina]

Question

I need to load a secret from file (eg /var/run/secrets/kubernetes.io/serviceaccount/token) to use with prometheus_scrape

Right now, I've modified the helm chart command value to load it into an env variable and then I use it in token property, but is there a better way?

I'm thinking that maybe there could a generic approach to this (values from files) in vector config processing OR at least in the helm chart. I'd be happy to create a PR if there's some direction on the preferred approach.

Vector Config

command:
  - /bin/sh
  - -c
  - |
    export KUBE_TOKEN="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
    exec /usr/local/bin/vector --config-dir "/etc/vector/"

customConfig:
  sources:
... 
    pod_metrics:
      type: prometheus_scrape
      endpoints:
        - https://kubernetes.default.svc/api/v1/nodes/$VECTOR_SELF_NODE_NAME/proxy/metrics/cadvisor
      auth:
        strategy: bearer
        token: "$KUBE_TOKEN"
      tls:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
...

Vector Logs

No response

[thomasqueirozb]

Hi @fopina, you might want to look into the built in secret management which is likely what you're looking for.

[fopina]

Oh missed this, interesting!

I guess then _FILE env handling can be implemented with a simple “cat $xx_FILE” as secret backend

thanks, I’ll give it a go!

[emanuilov]

Hi,
I came across the same question when trying to setup Redpanda Operator prometheus scraping.

I noticed Kubernete service account tokens are valid for 1 hour, the Vector secrets backend loads the token/file on boot only doesn't refresh it thus would become invalid, unless I'm mistaken, checked with Codex on Vector v0.54.0 (latest as of today).

There's no way to set activeDeadlineSeconds in the vector dev helm chart, so either the Vector code has to change to refresh secret values, which is nicer imo, or the helm chart should have the option to set activeDeadlineSeconds so the pod can be recreated every hour.

A temporary fix possible now is to set these args to the Vector helm chart, on any changes to the serviceaccount dir it will reload the config which will repopulate the secrets:

    args = [
      "--config-dir",
      "/etc/vector/",
      "--config-dir",
      "/var/run/secrets/kubernetes.io/serviceaccount",
      "--watch-config",
    ]

Whole Codex code trace on Vector not refreshing secret backend secrets

Here is the full execution path proving why DirectoryBackend is not continuously refreshed on file changes by itself.

flowchart TD
    A[Vector startup or reload signal] --> B[load_from_paths_with_provider_and_secrets]
    B --> C[SecretBackendLoader::retrieve_secrets]
    C --> D[DirectoryBackend::retrieve]
    D --> E[Read files once per requested key]
    E --> F[Inject resolved values into config]
    F --> G[Runtime uses concrete token string]

    H[Token file rotates on disk] --> I{Any config reload event?}
    I -- No --> J[No call back into retrieve_secrets]
    I -- Yes --> B

Loading

Code path with references:

1. Backend itself (single call behavior)

• DirectoryBackend::retrieve() loops keys and reads files via tokio::fs::read_to_string().
• It returns a plain HashMap<String, String> once at the end.
• No watcher, timer, task, cache invalidation, or re-read loop exists in this file.

2. Where it is invoked

• Secret resolution happens in SecretBackendLoader::retrieve_secrets().
• That function calls each backend once per load via backend.retrieve(...).await.
• Results are inserted into a map as concrete values via secrets.insert(...).

3. When secret resolution runs

• Called by config loading entrypoint load_from_paths_with_provider_and_secrets(), specifically:
  • retrieve_secrets(signal_handler).await
  • then those values are baked into builder via secrets(secrets).
• Initial startup path calls that loader in load_configs(), at load_from_paths_with_provider_and_secrets(...).

4. What triggers a re-run later

• Reload paths in signal handler call the same loader again:
  • SignalTo::ReloadComponents branch reloads config at app.rs:364.
  • SignalTo::ReloadFromDisk branch reloads config at app.rs:387.
• SIGHUP maps to reload-from-disk in os_signals(), specifically SIGHUP -> SignalTo::ReloadFromDisk.
• The module docs explicitly state this lifecycle: secrets load at startup or reload on SIGHUP in secrets/mod.rs.

Why this means “no automatic refresh after boot”

• After startup, a rotating file under /var/run/secrets/... changes on disk.
• Nothing in directory.rs is subscribed to filesystem changes.
• The value only updates when the global config reload path runs and calls retrieve_secrets() again.

So the precise claim is:

• Not “never refreshes.”
• It refreshes on config reload events (startup reload paths/SIGHUP), not continuously or immediately on secret-file rotation.

@thomasqueirozb could you reopen this please

A side note secret backend name cannot contain dashes, if it contains dashes it will simply set the token to literraly SECRET[...] without substituting it, tried wrapping "SECRET[...]" in quotes, same issue

How I confirmed the SECRET name limitation

secret:
  prom-target-bearer-01:
    path: /var/run/secrets/kubernetes.io/serviceaccount
    remove_trailing_whitespace: true
    type: directory
sources:
  app_metrics_01:
    auth:
      strategy: bearer
      token: SECRET[prom-target-bearer-01.token]
    endpoint_tag: endpoint
    endpoints:
    - http://prom-echo.observability.svc.cluster.local:9102/metrics
    instance_tag: instance
    scrape_interval_secs: 30
    type: prometheus_scrape

Can be tested by creating a service with a pod that prints the token sent by Vector like so:

kubectl create namespace observability && kubectl -n observability apply -f - <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prom-echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prom-echo
  template:
    metadata:
      labels:
        app: prom-echo
    spec:
      containers:
        - name: prom-echo
          image: python:3.11-alpine
          ports:
            - containerPort: 9102
          command:
            - /bin/sh
            - -lc
          args:
            - |
              cat > /tmp/server.py <<'PYEOF'
              from http.server import BaseHTTPRequestHandler, HTTPServer
              import sys

              METRICS = b"# HELP test_up Always 1\n# TYPE test_up gauge\ntest_up 1\n"

              class H(BaseHTTPRequestHandler):
                  def do_GET(self):
                      auth = self.headers.get("Authorization")
                      sys.stdout.write(f"path={self.path} authorization={auth!r}\n")
                      sys.stdout.flush()

                      if self.path != "/metrics":
                          self.send_response(404)
                          self.end_headers()
                          self.wfile.write(b"not found\n")
                          return

                      self.send_response(200)
                      self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
                      self.send_header("Content-Length", str(len(METRICS)))
                      self.end_headers()
                      self.wfile.write(METRICS)

                  def log_message(self, fmt, *args):
                      return

              HTTPServer(("0.0.0.0", 9102), H).serve_forever()
              PYEOF
              python3 /tmp/server.py
---
apiVersion: v1
kind: Service
metadata:
  name: prom-echo
spec:
  selector:
    app: prom-echo
  ports:
    - name: http
      port: 9102
      targetPort: 9102
YAML

Details

[thomasqueirozb]

Hi @emanuilov please open a separate issue/discussion. I suspect that there should already exist an open issue about this