fix(agentex): bootstrap OTel auto-instrumentation in uvicorn spawn workers (#305)

## Summary

- Bootstrap OpenTelemetry auto-instrumentation at import time so uvicorn
**spawn** workers get HTTP/library instrumentation and custom metrics,
not just the parent process
- Assign a per-worker `service.instance.id` by patching
`OTEL_RESOURCE_ATTRIBUTES` before `initialize()`, fixing shared
timeseries when multiple workers inherit the same pod-level resource
attrs
([opentelemetry-python#4390](open-telemetry/opentelemetry-python#4390))
- Move `otel_metrics` to the first import in `app.py` so instrumentors
patch FastAPI/httpx/SQLAlchemy at import time; `init_otel_metrics()` at
lifespan startup attaches custom instruments to the existing global
`MeterProvider`

## Problem

The OTel Operator injects auto-instrumentation via `sitecustomize`,
which runs `initialize()` in the **parent** process and then strips
auto-instrumentation from `PYTHONPATH`. Uvicorn **spawn** workers are
fresh Python processes without `sitecustomize`, so they previously
served plain `FastAPI` with no OTel HTTP middleware or metrics.

Separately, spawn workers share the pod-level `OTEL_RESOURCE_ATTRIBUTES`
env var. Auto-instrumentation builds provider resources from env at
`initialize()` time via `Resource.create()` — so all workers would emit
on the same `service.instance.id` without a per-process suffix
([opentelemetry-python#4390](open-telemetry/opentelemetry-python#4390)).

## Solution

1. **`bootstrap_auto_instrumentation()`** — runs on `otel_metrics`
import; syncs `service.instance.id.<pid>` into env, then calls
`initialize()` to create global `TracerProvider`/`MeterProvider` and
load instrumentors ([auto-instrumentation
reference](https://opentelemetry.io/docs/languages/python/automatic/))
2. **`init_otel_metrics()`** — unchanged coexistence model: attaches
custom app metrics (`auth_cache_*`, `db_*`) to the bootstrap provider
when present; standalone OTLP pipeline only when no global provider
exists
3. **Import order** — `app.py` imports `otel_metrics` first, before
FastAPI and other auto-instrumented libraries

## References

- [OTel Python
auto-instrumentation](https://opentelemetry.io/docs/languages/python/automatic/)
- [OTel Python manual SDK
configuration](https://opentelemetry.io/docs/languages/python/getting-started/)
— `initialize()` creates providers via `Resource.create()` from env, not
patching-only
-
[opentelemetry-python#4390](open-telemetry/opentelemetry-python#4390)
— per-worker `service.instance.id` for multi-worker processes
- [Uvicorn deployment / workers](https://www.uvicorn.org/deployment/) —
spawn workers are separate processes

## Notes

- **Helm (single worker):** bootstrap runs in the worker process;
operator k8s resource labels are preserved; duplicate `initialize()` on
`--workers 1` only produces set-once warnings
- **Dockerfile (`--workers 4`):** each spawn worker bootstraps
independently with a distinct pid-suffixed `service.instance.id`
- **ddtrace coexistence:** documented in module docstring; Helm uses
`ddtrace-run` only when `datadog.env` is set
- Removed ineffective `NoOpMeterProvider` reset on shutdown — OTel
global `MeterProvider` is set-once and cannot be replaced

## Test plan

- [x] `pytest agentex/tests/unit/utils/test_otel_metrics.py` (24 tests)
- [ ] Deploy to cluster; confirm `_InstrumentedFastAPI` in spawn workers
- [ ] Confirm distinct `service.instance.id` per worker when `--workers
> 1`
- [ ] Confirm FastAPI HTTP metrics include `http_route` and k8s resource
labels from operator env
- [ ] Confirm custom metrics (`auth_cache_*`, `db_*`) export on the same
provider resource

Made with [Cursor](https://cursor.com)

<!-- greptile_comment -->

<h3>Greptile Summary</h3>

This PR bootstraps OTel auto-instrumentation in uvicorn spawn workers by
importing `otel_metrics` first in `app.py` and calling
`bootstrap_auto_instrumentation()` before any auto-instrumented library
is imported. It also assigns a per-worker `service.instance.id` by
appending the process PID to avoid shared timeseries across workers.

- **Bootstrap flow**: `bootstrap_auto_instrumentation()` syncs a
PID-suffixed `service.instance.id` into `OTEL_RESOURCE_ATTRIBUTES`, then
calls `initialize()` to install global `TracerProvider`/`MeterProvider`
and patch libraries; failures are caught and logged without crashing the
service.
- **Custom metrics coexistence**: `init_otel_metrics()` attaches to the
bootstrap provider when present, or creates a standalone OTLP pipeline
when no global provider exists; the `NoOpMeterProvider` reset on
shutdown is removed because OTel globals are set-once.
- **Test coverage**: 24 unit tests added for bootstrap, idempotency,
failure recovery, and resource attribute construction.

<details><summary><h3>Confidence Score: 5/5</h3></summary>

Safe to merge — the bootstrap path is well-guarded with try/except and
idempotency checks; the import-order change in app.py achieves the
intended patching window.

Bootstrap failures are caught and logged without crashing the service.
The per-worker service.instance.id logic correctly handles the
idempotency case and the env-mutation is scoped to resource attribute
construction before initialize(). The two observations noted are edge
cases with no current practical impact given how init_otel_metrics is
called today.

agentex/src/utils/otel_metrics.py — _sync_instance_id_to_env comma
splitting and the resource merge precedence in init_otel_metrics
standalone mode.
</details>

<h3>Important Files Changed</h3>

| Filename | Overview |
|----------|----------|
| agentex/src/utils/otel_metrics.py | Adds
bootstrap_auto_instrumentation() with proper exception handling and
idempotency; introduces _sync_instance_id_to_env that splits
OTEL_RESOURCE_ATTRIBUTES by bare comma without handling quoted values,
and a resource merge pattern where env-detected attributes in the
pid_resource can silently override explicit
service_name/service_version/environment args passed to
init_otel_metrics in standalone mode. |
| agentex/src/api/app.py | otel_metrics moved to first import with
bootstrap_auto_instrumentation() call before FastAPI/httpx/SQLAlchemy;
ruff E402 suppressed with a clear comment; no functional regressions. |
| agentex/tests/unit/utils/test_otel_metrics.py | Comprehensive
bootstrap test coverage: import blocking, failure/retry semantics,
idempotency, env mutation, and PID-suffixed resource attributes; autouse
fixture correctly resets bootstrap flag between tests. |

</details>

<details><summary><h3>Sequence Diagram</h3></summary>

```mermaid
sequenceDiagram
    participant UV as uvicorn spawn worker
    participant APP as app.py
    participant OTEL as otel_metrics.py
    participant SDK as OTel SDK
    participant LS as FastAPI lifespan

    UV->>APP: import app
    APP->>OTEL: import otel_metrics
    APP->>OTEL: bootstrap_auto_instrumentation()
    OTEL->>OTEL: _sync_instance_id_to_env with pid suffix
    OTEL->>SDK: initialize()
    SDK-->>OTEL: TracerProvider + MeterProvider set globally
    Note over SDK: FastAPI, httpx, SQLAlchemy patched
    OTEL-->>APP: True
    APP->>APP: import FastAPI and other libraries

    UV->>LS: startup lifespan
    LS->>OTEL: init_otel_metrics()
    alt Bootstrap provider exists
        OTEL-->>LS: attach custom instruments to bootstrap provider
    else No global provider standalone mode
        OTEL->>SDK: MeterProvider with pid-suffixed resource
        OTEL-->>LS: new standalone MeterProvider
    end

    UV->>LS: shutdown lifespan
    LS->>OTEL: shutdown_otel_metrics()
    Note over OTEL: Only shuts down module-created provider
```
</details>

<a
href="https://app.greptile.com/api/ide/cursor?prompt=Fix%20the%20following%202%20code%20review%20issues.%20Work%20through%20them%20one%20at%20a%20time%2C%20proposing%20concise%20fixes.%0A%0A---%0A%0A%23%23%23%20Issue%201%20of%202%0Aagentex%2Fsrc%2Futils%2Fotel_metrics.py%3A110-114%0AThe%20comma-split%20on%20%60OTEL_RESOURCE_ATTRIBUTES%60%20doesn't%20account%20for%20quoted%20values%20%28e.g.%2C%20%60some.key%3D%22hello%2Cworld%22%60%29.%20The%20OTel%20spec%20allows%20commas%20inside%20quoted%20values%2C%20and%20the%20%60OTELResourceDetector%60%20handles%20them%20correctly.%20This%20bare%20%60split%28%22%2C%22%29%60%20would%20shatter%20a%20quoted%20entry%20into%20fragments%20and%20reassemble%20a%20corrupted%20string.%20If%20the%20operator%20ever%20injects%20a%20label%20whose%20value%20contains%20a%20comma%2C%20the%20attribute%20will%20be%20silently%20mangled%20in%20the%20environment%20for%20every%20subsequent%20call%20that%20reads%20it%20%28including%20%60initialize%28%29%60%20itself%29.%0A%0A%60%60%60suggestion%0A%20%20%20%20import%20re%0A%0A%20%20%20%20%23%20Split%20only%20on%20commas%20that%20are%20NOT%20inside%20double-quoted%20values.%0A%20%20%20%20raw%20%3D%20os.environ.get%28%22OTEL_RESOURCE_ATTRIBUTES%22%2C%20%22%22%29%0A%20%20%20%20parts%20%3D%20%5B%0A%20%20%20%20%20%20%20%20part.strip%28%29%0A%20%20%20%20%20%20%20%20for%20part%20in%20re.split%28r'%2C%28%3F%3D%28%3F%3A%5B%5E%22%5D*%22%5B%5E%22%5D*%22%29*%5B%5E%22%5D*%24%29'%2C%20raw%29%0A%20%20%20%20%20%20%20%20if%20part.strip%28%29%20and%20not%20part.strip%28%29.startswith%28f%22%7Bkey%7D%3D%22%29%0A%20%20%20%20%5D%0A%60%60%60%0A%0A%23%23%23%20Issue%202%20of%202%0Aagentex%2Fsrc%2Futils%2Fotel_metrics.py%3A273-277%0A**Resource%20merge%20silently%20drops%20explicit%20constructor%20args**%0A%0A%60Resource.create%28%7B%22service.instance.id%22%3A%20...%7D%29%60%20runs%20%60OTELResourceDetector%60%20internally%2C%20so%20the%20second%20resource%20carries%20ALL%20env%20attributes%20at%20%22env%22%20priority.%20Because%20%60Resource.merge%60%20gives%20the%20%60other%60%20argument%20%28the%20pid%20resource%29%20precedence%2C%20any%20env%20value%20for%20%60service.name%60%2C%20%60service.version%60%2C%20or%20%60deployment.environment%60%20in%20%60OTEL_RESOURCE_ATTRIBUTES%60%20will%20silently%20override%20the%20%60service_name%60%2C%20%60service_version%60%2C%20and%20%60environment%60%20keyword%20arguments%20passed%20to%20%60init_otel_metrics%60.%20Today's%20call%20site%20passes%20no%20args%20so%20the%20values%20are%20identical%2C%20but%20a%20caller%20passing%20%60service_name%3D%22custom%22%60%20with%20%60OTEL_SERVICE_NAME%3Denv-name%60%20set%20would%20find%20%60custom%60%20discarded.%20Consider%20creating%20the%20pid%20resource%20with%20only%20the%20one%20explicit%20attribute%20and%20merging%20the%20detected%20env%20resource%20separately%20so%20precedence%20is%20unambiguous.%0A%0A&pr=305&platform=github"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInCursorDark.svg?v=3"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInCursor.svg?v=3"><img
alt="Fix All in Cursor"
src="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInCursor.svg?v=3"
height="20"></picture></a> <a
href="https://app.greptile.com/ide/claude-code?prompt=Fix%20the%20following%202%20code%20review%20issues.%20Work%20through%20them%20one%20at%20a%20time%2C%20proposing%20concise%20fixes.%0A%0A---%0A%0A%23%23%23%20Issue%201%20of%202%0Aagentex%2Fsrc%2Futils%2Fotel_metrics.py%3A110-114%0AThe%20comma-split%20on%20%60OTEL_RESOURCE_ATTRIBUTES%60%20doesn't%20account%20for%20quoted%20values%20%28e.g.%2C%20%60some.key%3D%22hello%2Cworld%22%60%29.%20The%20OTel%20spec%20allows%20commas%20inside%20quoted%20values%2C%20and%20the%20%60OTELResourceDetector%60%20handles%20them%20correctly.%20This%20bare%20%60split%28%22%2C%22%29%60%20would%20shatter%20a%20quoted%20entry%20into%20fragments%20and%20reassemble%20a%20corrupted%20string.%20If%20the%20operator%20ever%20injects%20a%20label%20whose%20value%20contains%20a%20comma%2C%20the%20attribute%20will%20be%20silently%20mangled%20in%20the%20environment%20for%20every%20subsequent%20call%20that%20reads%20it%20%28including%20%60initialize%28%29%60%20itself%29.%0A%0A%60%60%60suggestion%0A%20%20%20%20import%20re%0A%0A%20%20%20%20%23%20Split%20only%20on%20commas%20that%20are%20NOT%20inside%20double-quoted%20values.%0A%20%20%20%20raw%20%3D%20os.environ.get%28%22OTEL_RESOURCE_ATTRIBUTES%22%2C%20%22%22%29%0A%20%20%20%20parts%20%3D%20%5B%0A%20%20%20%20%20%20%20%20part.strip%28%29%0A%20%20%20%20%20%20%20%20for%20part%20in%20re.split%28r'%2C%28%3F%3D%28%3F%3A%5B%5E%22%5D*%22%5B%5E%22%5D*%22%29*%5B%5E%22%5D*%24%29'%2C%20raw%29%0A%20%20%20%20%20%20%20%20if%20part.strip%28%29%20and%20not%20part.strip%28%29.startswith%28f%22%7Bkey%7D%3D%22%29%0A%20%20%20%20%5D%0A%60%60%60%0A%0A%23%23%23%20Issue%202%20of%202%0Aagentex%2Fsrc%2Futils%2Fotel_metrics.py%3A273-277%0A**Resource%20merge%20silently%20drops%20explicit%20constructor%20args**%0A%0A%60Resource.create%28%7B%22service.instance.id%22%3A%20...%7D%29%60%20runs%20%60OTELResourceDetector%60%20internally%2C%20so%20the%20second%20resource%20carries%20ALL%20env%20attributes%20at%20%22env%22%20priority.%20Because%20%60Resource.merge%60%20gives%20the%20%60other%60%20argument%20%28the%20pid%20resource%29%20precedence%2C%20any%20env%20value%20for%20%60service.name%60%2C%20%60service.version%60%2C%20or%20%60deployment.environment%60%20in%20%60OTEL_RESOURCE_ATTRIBUTES%60%20will%20silently%20override%20the%20%60service_name%60%2C%20%60service_version%60%2C%20and%20%60environment%60%20keyword%20arguments%20passed%20to%20%60init_otel_metrics%60.%20Today's%20call%20site%20passes%20no%20args%20so%20the%20values%20are%20identical%2C%20but%20a%20caller%20passing%20%60service_name%3D%22custom%22%60%20with%20%60OTEL_SERVICE_NAME%3Denv-name%60%20set%20would%20find%20%60custom%60%20discarded.%20Consider%20creating%20the%20pid%20resource%20with%20only%20the%20one%20explicit%20attribute%20and%20merging%20the%20detected%20env%20resource%20separately%20so%20precedence%20is%20unambiguous.%0A%0A&repo=scaleapi%2Fscale-agentex&pr=305&platform=github"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInClaudeDark.svg?v=3"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInClaude.svg?v=3"><img
alt="Fix All in Claude Code"
src="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInClaude.svg?v=3"
height="20"></picture></a> <a
href="https://app.greptile.com/api/ide/codex?prompt=IMPORTANT%3A%20Work%20in%20the%20repository%20%22scaleapi%2Fscale-agentex%22%20on%20the%20existing%20branch%20%22jamesc-fix-auto-intrumentation%22.%20Checkout%20that%20branch%20%E2%80%94%20do%20NOT%20create%20a%20new%20branch%20or%20open%20a%20new%20PR.%20Push%20your%20changes%20to%20%22jamesc-fix-auto-intrumentation%22.%0A%0AFix%20the%20following%202%20code%20review%20issues.%20Work%20through%20them%20one%20at%20a%20time%2C%20proposing%20concise%20fixes.%0A%0A---%0A%0A%23%23%23%20Issue%201%20of%202%0Aagentex%2Fsrc%2Futils%2Fotel_metrics.py%3A110-114%0AThe%20comma-split%20on%20%60OTEL_RESOURCE_ATTRIBUTES%60%20doesn't%20account%20for%20quoted%20values%20%28e.g.%2C%20%60some.key%3D%22hello%2Cworld%22%60%29.%20The%20OTel%20spec%20allows%20commas%20inside%20quoted%20values%2C%20and%20the%20%60OTELResourceDetector%60%20handles%20them%20correctly.%20This%20bare%20%60split%28%22%2C%22%29%60%20would%20shatter%20a%20quoted%20entry%20into%20fragments%20and%20reassemble%20a%20corrupted%20string.%20If%20the%20operator%20ever%20injects%20a%20label%20whose%20value%20contains%20a%20comma%2C%20the%20attribute%20will%20be%20silently%20mangled%20in%20the%20environment%20for%20every%20subsequent%20call%20that%20reads%20it%20%28including%20%60initialize%28%29%60%20itself%29.%0A%0A%60%60%60suggestion%0A%20%20%20%20import%20re%0A%0A%20%20%20%20%23%20Split%20only%20on%20commas%20that%20are%20NOT%20inside%20double-quoted%20values.%0A%20%20%20%20raw%20%3D%20os.environ.get%28%22OTEL_RESOURCE_ATTRIBUTES%22%2C%20%22%22%29%0A%20%20%20%20parts%20%3D%20%5B%0A%20%20%20%20%20%20%20%20part.strip%28%29%0A%20%20%20%20%20%20%20%20for%20part%20in%20re.split%28r'%2C%28%3F%3D%28%3F%3A%5B%5E%22%5D*%22%5B%5E%22%5D*%22%29*%5B%5E%22%5D*%24%29'%2C%20raw%29%0A%20%20%20%20%20%20%20%20if%20part.strip%28%29%20and%20not%20part.strip%28%29.startswith%28f%22%7Bkey%7D%3D%22%29%0A%20%20%20%20%5D%0A%60%60%60%0A%0A%23%23%23%20Issue%202%20of%202%0Aagentex%2Fsrc%2Futils%2Fotel_metrics.py%3A273-277%0A**Resource%20merge%20silently%20drops%20explicit%20constructor%20args**%0A%0A%60Resource.create%28%7B%22service.instance.id%22%3A%20...%7D%29%60%20runs%20%60OTELResourceDetector%60%20internally%2C%20so%20the%20second%20resource%20carries%20ALL%20env%20attributes%20at%20%22env%22%20priority.%20Because%20%60Resource.merge%60%20gives%20the%20%60other%60%20argument%20%28the%20pid%20resource%29%20precedence%2C%20any%20env%20value%20for%20%60service.name%60%2C%20%60service.version%60%2C%20or%20%60deployment.environment%60%20in%20%60OTEL_RESOURCE_ATTRIBUTES%60%20will%20silently%20override%20the%20%60service_name%60%2C%20%60service_version%60%2C%20and%20%60environment%60%20keyword%20arguments%20passed%20to%20%60init_otel_metrics%60.%20Today's%20call%20site%20passes%20no%20args%20so%20the%20values%20are%20identical%2C%20but%20a%20caller%20passing%20%60service_name%3D%22custom%22%60%20with%20%60OTEL_SERVICE_NAME%3Denv-name%60%20set%20would%20find%20%60custom%60%20discarded.%20Consider%20creating%20the%20pid%20resource%20with%20only%20the%20one%20explicit%20attribute%20and%20merging%20the%20detected%20env%20resource%20separately%20so%20precedence%20is%20unambiguous.%0A%0A&repo=scaleapi%2Fscale-agentex&pr=305&platform=github"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInCodexDark.svg?v=3"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInCodex.svg?v=3"><img
alt="Fix All in Codex"
src="https://greptile-static-assets.s3.amazonaws.com/badges/FixAllInCodex.svg?v=3"
height="20"></picture></a>

<details><summary>Prompt To Fix All With AI</summary>

`````markdown
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
agentex/src/utils/otel_metrics.py:110-114
The comma-split on `OTEL_RESOURCE_ATTRIBUTES` doesn't account for quoted values (e.g., `some.key="hello,world"`). The OTel spec allows commas inside quoted values, and the `OTELResourceDetector` handles them correctly. This bare `split(",")` would shatter a quoted entry into fragments and reassemble a corrupted string. If the operator ever injects a label whose value contains a comma, the attribute will be silently mangled in the environment for every subsequent call that reads it (including `initialize()` itself).

```suggestion
    import re

    # Split only on commas that are NOT inside double-quoted values.
    raw = os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "")
    parts = [
        part.strip()
        for part in re.split(r',(?=(?:[^"]*"[^"]*")*[^"]*$)', raw)
        if part.strip() and not part.strip().startswith(f"{key}=")
    ]
```

### Issue 2 of 2
agentex/src/utils/otel_metrics.py:273-277
**Resource merge silently drops explicit constructor args**

`Resource.create({"service.instance.id": ...})` runs `OTELResourceDetector` internally, so the second resource carries ALL env attributes at "env" priority. Because `Resource.merge` gives the `other` argument (the pid resource) precedence, any env value for `service.name`, `service.version`, or `deployment.environment` in `OTEL_RESOURCE_ATTRIBUTES` will silently override the `service_name`, `service_version`, and `environment` keyword arguments passed to `init_otel_metrics`. Today's call site passes no args so the values are identical, but a caller passing `service_name="custom"` with `OTEL_SERVICE_NAME=env-name` set would find `custom` discarded. Consider creating the pid resource with only the one explicit attribute and merging the detected env resource separately so precedence is unambiguous.

`````

</details>

<sub>Reviews (5): Last reviewed commit: ["Merge branch &#39;main&#39;
into
jamesc-fix-auto..."](490a715)
| [Re-trigger
Greptile](https://app.greptile.com/api/retrigger?id=37157496)</sub>

<!-- /greptile_comment -->

Author: james-cardenas

agentex/src/api/app.py

@@ -1,3 +1,15 @@
+# ruff: noqa: E402
+# E402 suppressed: bootstrap_auto_instrumentation() must run before imports of
+# auto-instrumented libraries (FastAPI, httpx, SQLAlchemy, etc.).
+
+from src.utils.otel_metrics import (
+    bootstrap_auto_instrumentation,
+    init_otel_metrics,
+    shutdown_otel_metrics,
+)
+
+bootstrap_auto_instrumentation()
+
 import os
 from contextlib import asynccontextmanager
 from pathlib import Path
@@ -38,7 +50,6 @@
 from src.config.environment_variables import EnvVarKeys
 from src.domain.exceptions import GenericException
 from src.utils.logging import make_logger
-from src.utils.otel_metrics import init_otel_metrics, shutdown_otel_metrics
 
 logger = make_logger(__name__)
 

agentex/src/utils/otel_metrics.py

@@ -1,19 +1,45 @@
 """
-OpenTelemetry metrics configuration for Agentex.
-
-When auto-instrumentation (e.g. OTel Operator) has already installed a global
-MeterProvider, custom app metrics attach to it instead of replacing it.
-Otherwise this module creates its own provider with OTLP export when an endpoint
-is configured.
-
-Environment Variables:
-    OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: Metrics OTLP endpoint (falls back to
+OpenTelemetry bootstrap and custom metrics for Agentex.
+
+Two responsibilities:
+
+1. **Auto-instrumentation** — call ``bootstrap_auto_instrumentation()`` from
+   ``app.py`` before importing FastAPI or other auto-instrumented libraries so
+   ``initialize()`` runs in each uvicorn spawn worker when contrib packages
+   are installed.
+
+2. **Custom app metrics** — ``init_otel_metrics()`` registers Agentex instruments
+   (``auth_cache_*``, ``db_*``, etc.). Attaches to an existing global
+   ``MeterProvider`` from bootstrap/operator when present; otherwise creates a
+   standalone OTLP pipeline when an endpoint is configured.
+
+**Datadog ``ddtrace-run`` coexistence:** Neither OTel nor ddtrace detects the other's
+FastAPI patches. If both run in one process, ddtrace wraps the middleware stack
+first; OTel skips ``OpenTelemetryMiddleware`` with "unexpected middleware stack"
+and HTTP OTel metrics/traces are not emitted. Helm avoids this by using
+``ddtrace-run`` only when ``datadog.env`` is set (OTel-only otherwise). If both
+are required, set ``DD_TRACE_FASTAPI_ENABLED=false`` and
+``DD_TRACE_STARLETTE_ENABLED=false`` so OTel owns HTTP instrumentation.
+
+**Per-worker ``service.instance.id``:** Uvicorn spawn workers share pod-level
+``OTEL_RESOURCE_ATTRIBUTES``, so auto-instrumentation would otherwise emit all
+workers on the same metric timeseries (see `OTel #4390
+<https://github.com/open-telemetry/opentelemetry-python/issues/4390>`_).
+``bootstrap_auto_instrumentation()`` appends ``.<pid>`` to ``service.instance.id``
+in ``OTEL_RESOURCE_ATTRIBUTES`` before ``initialize()``; standalone
+``init_otel_metrics()`` applies the same via ``Resource.merge``. With
+``--workers 1``, operator ``sitecustomize`` may have already called
+``initialize()``; bootstrap calls it again (OTel providers and instrumentors
+are set-once; duplicate calls only produce log warnings).
+
+Environment variables (custom metrics / standalone mode):
+    OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: Metrics endpoint (falls back to
         OTEL_EXPORTER_OTLP_ENDPOINT)
-    OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: Metrics export protocol (falls back to
+    OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: Export protocol (falls back to
         OTEL_EXPORTER_OTLP_PROTOCOL; default: grpc)
     OTEL_EXPORTER_OTLP_ENDPOINT: General OTLP endpoint URL
-    OTEL_EXPORTER_OTLP_HEADERS: Optional headers for authentication
-    OTEL_SERVICE_NAME: Service name for metrics (default: agentex)
+    OTEL_EXPORTER_OTLP_HEADERS: Passed through by OTLP exporters when set
+    OTEL_SERVICE_NAME: Service name (default: agentex)
     OTEL_METRICS_EXPORT_INTERVAL_MS: Export interval in ms (default: 30000)
 """
 
@@ -23,7 +49,6 @@
 from typing import TYPE_CHECKING
 
 from opentelemetry import metrics
-from opentelemetry.metrics import NoOpMeterProvider
 from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
     OTLPMetricExporter as OTLPGrpcMetricExporter,
 )
@@ -32,7 +57,13 @@
 )
 from opentelemetry.sdk.metrics import MeterProvider
 from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
-from opentelemetry.sdk.resources import SERVICE_NAME, SERVICE_VERSION, Resource
+from opentelemetry.sdk.resources import (
+    SERVICE_NAME,
+    SERVICE_VERSION,
+    OTELResourceDetector,
+    Resource,
+    get_aggregated_resources,
+)
 
 from src.utils.logging import make_logger
 
@@ -42,15 +73,102 @@
 
 logger = make_logger(__name__)
 
-# Global state
-_meter_provider: MeterProvider | None = None # Set only when this module creates the provider
+# Module state
+_auto_instrumentation_bootstrapped = False
+_meter_provider: MeterProvider | None = None  # Set only when this module creates the provider
 _initialized: bool = False
 
-# Default configuration
 DEFAULT_SERVICE_NAME = "agentex"
 DEFAULT_EXPORT_INTERVAL_MS = 30000  # 30 seconds
 
 
+def _detected_resource() -> Resource:
+    """Resource attributes from OTEL_* env (operator-injected or local)."""
+    return get_aggregated_resources([OTELResourceDetector()])
+
+
+def _unique_instance_id(resource: Resource) -> str:
+    """Worker-unique service.instance.id (OTel #4390)."""
+    pid = os.getpid()
+    existing = resource.attributes.get("service.instance.id")
+    if existing:
+        existing = str(existing)
+        suffix = f".{pid}"
+        return existing if existing.endswith(suffix) else f"{existing}{suffix}"
+    service = (
+        resource.attributes.get("service.name")
+        or os.environ.get("OTEL_SERVICE_NAME")
+        or "unknown"
+    )
+    pod = resource.attributes.get("k8s.pod.name") or "unknown"
+    return f"{service}.{pod}.{pid}"
+
+
+def _sync_instance_id_to_env(instance_id: str) -> None:
+    """Write service.instance.id into OTEL_RESOURCE_ATTRIBUTES for auto-instrumentation."""
+    key = "service.instance.id"
+    parts = [
+        part.strip()
+        for part in os.environ.get("OTEL_RESOURCE_ATTRIBUTES", "").split(",")
+        if part.strip() and not part.strip().startswith(f"{key}=")
+    ]
+    parts.append(f"{key}={instance_id}")
+    os.environ["OTEL_RESOURCE_ATTRIBUTES"] = ",".join(parts)
+
+
+# --- Auto-instrumentation bootstrap ---
+
+
+def bootstrap_auto_instrumentation() -> bool:
+    """Call ``initialize()`` once per process when auto-instrumentation is available.
+
+    Call from ``app.py`` before any auto-instrumented library (FastAPI, httpx,
+    SQLAlchemy, etc.) — instrumentors patch at import time. Each uvicorn spawn
+    worker imports ``app.py`` fresh, so one call per worker is enough.
+
+    Runs when: contrib packages are installed (no ``ImportError``).
+    Skips when: bootstrap already succeeded in this process.
+    On ``ImportError`` or ``initialize()`` failure, returns False and leaves
+    the flag unset so a later call can retry.
+
+    Export config, ``OTEL_SDK_DISABLED``, and disabled instrumentations are
+    handled inside ``initialize()`` — not gated here. Custom app metrics use
+    ``init_otel_metrics()`` separately.
+
+    Returns:
+        True if ``initialize()`` completed; False if skipped or failed.
+    """
+    global _auto_instrumentation_bootstrapped
+
+    if _auto_instrumentation_bootstrapped:
+        return False
+
+    try:
+        from opentelemetry.instrumentation.auto_instrumentation import initialize
+    except ImportError:
+        return False
+
+    try:
+        _sync_instance_id_to_env(_unique_instance_id(_detected_resource()))
+        initialize()
+    except Exception:
+        logger.warning(
+            "OpenTelemetry auto-instrumentation bootstrap failed; continuing without it",
+            exc_info=True,
+        )
+        return False
+
+    _auto_instrumentation_bootstrapped = True
+    logger.debug(
+        "OpenTelemetry auto-instrumentation bootstrapped (pid=%s)",
+        os.getpid(),
+    )
+    return True
+
+
+# --- Custom application metrics ---
+
+
 def _global_meter_provider() -> MeterProvider | None:
     """Return the global MeterProvider if installed, else None (proxy is ignored)."""
     provider = metrics.get_meter_provider()
@@ -152,6 +270,10 @@ def init_otel_metrics(
             "deployment.environment": environment
             or os.environ.get("ENVIRONMENT", "development"),
         }
+    ).merge(
+        Resource.create(
+            {"service.instance.id": _unique_instance_id(_detected_resource())}
+        )
     )
     reader = PeriodicExportingMetricReader(
         exporter=_create_metric_exporter(endpoint, protocol),
@@ -167,9 +289,11 @@ def init_otel_metrics(
     _meter_provider = provider
     _initialized = True
     logger.info(
-        f"OpenTelemetry metrics initialized: endpoint={endpoint}, "
-        f"protocol={protocol}, service={resolved_service_name}, "
-        f"interval={resolved_export_interval_ms}ms"
+        "OpenTelemetry metrics initialized: endpoint=%s, protocol=%s, service=%s, interval=%sms",
+        endpoint,
+        protocol,
+        resolved_service_name,
+        resolved_export_interval_ms,
     )
     return _meter_provider
 
@@ -209,11 +333,6 @@ def shutdown_otel_metrics() -> None:
     except Exception:
         logger.exception("OpenTelemetry metrics shutdown failed")
     finally:
-        if _meter_provider is not None:
-            try:
-                metrics.set_meter_provider(NoOpMeterProvider())
-            except Exception:
-                logger.exception("Failed to reset global MeterProvider after shutdown")
         _meter_provider = None
         _initialized = False