Migrate agentserver tracing from azure-monitor-opentelemetry-exporter… (#46561)

* Migrate agentserver tracing from azure-monitor-opentelemetry-exporter to microsoft-opentelemetry

Replace manual Azure Monitor and OTLP exporter wiring (~100 lines) with a
single use_microsoft_opentelemetry() call via the new microsoft-opentelemetry
distro. The distro auto-detects exporters from environment variables.

Changes:
- core pyproject.toml: swap azure-monitor-opentelemetry-exporter + otlp-grpc
  for microsoft-opentelemetry>=0.5.0
- core _tracing.py: refactor _configure_tracing() to delegate to new
  _setup_distro_export() wrapper; remove 4 _setup_* functions and guard flags
- Update all test mocks in core and invocations to patch the single
  _setup_distro_export target instead of multiple _setup_* functions
- Remove azure-monitor-opentelemetry-exporter from dev_requirements.txt
  in both core and invocations packages

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add warning log when OTel resource creation fails

Log a warning in _configure_tracing() when _create_resource() returns None,
making it clear that tracing will not be configured.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments

- Fix docstring and inline comment to accurately describe logger suppression
  (only Azure Core HTTP logging policy, not generic OTel exporters)
- Make _ensure_trace_provider idempotent via _agentserver_processors_added
  guard to prevent duplicate processors on repeated calls
- Rename TestSetupAzureMonitor -> TestSetupDistroExport to match new impl

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: add create=True to _setup_distro_export patches for min-dep compat

The invocations min-dependency CI installs the published core package
from PyPI, which does not yet have _setup_distro_export. Adding
create=True allows the mock to succeed regardless of the installed
core version.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: use valid UUID in fake instrumentation keys for tests

The microsoft-opentelemetry distro validates that the instrumentation
key is a proper UUID. Replace 'InstrumentationKey=test' with
'InstrumentationKey=00000000-0000-0000-0000-000000000000' in all
test files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix: lower microsoft-opentelemetry version to >=0.1.0b1

The previous requirement >=0.5.0 doesn't exist on PyPI. The latest
available release is 0.1.0b1 (just published).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add INFO log on successful tracing configuration

The previous code logged INFO for each exporter configured (e.g.
'Application Insights trace exporter configured.'). This preserves
that pattern with a single INFO message when the distro setup succeeds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix: enable Azure Monitor export in distro call

The microsoft-opentelemetry distro has Azure Monitor disabled by
default. Pass enable_azure_monitor=True and the connection string
via azure_monitor_connection_string when a connection string is
provided.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add e2e span parenting test for App Insights

Adds test_span_parenting_in_appinsights which verifies that a child span
created inside request_span is correctly parented in App Insights:
- Captures child span ID locally during handler execution
- Queries dependencies table by child span ID
- Follows operation_ParentId to find the parent invoke_agent span in requests table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: defer set_tracer_provider to fixture to unblock e2e tests

Move module-level set_tracer_provider() calls in invocations unit tests
into autouse fixtures. When pytest collects tests it imports ALL modules,
even deselected ones. Module-level calls consumed the OTel Once guard
before the e2e test could configure the microsoft-opentelemetry distro,
causing spans to never reach App Insights.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: add microsoft-opentelemetry to shared_requirements.txt

The dependency analyzer requires all package dependencies to be listed
in shared_requirements.txt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: resolve pylint errors in _tracing.py

- Add :keyword: docstring entries for _setup_distro_export (C4758)
- Use dict literal instead of dict() call (R1735)
- Add :param: docstring entries for _ensure_trace_provider (C4739)
- Suppress protected-access warning on idempotency guard (W0212)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: add param types to _ensure_trace_provider docstring (C4740)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: singankit

sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/_tracing.py

@@ -13,9 +13,10 @@
 
   - Console ``StreamHandler`` on the **root** logger (so both SDK and
     user ``logging.info()`` calls are visible).
-  - Suppression of noisy Azure SDK / OTel exporter loggers.
-  - Azure Monitor trace + log export (when connection string available).
-  - OTLP trace + log export (when ``OTEL_EXPORTER_OTLP_ENDPOINT`` set).
+  - Suppression of noisy Azure Core HTTP logging policy output.
+  - Trace and log export via ``microsoft-opentelemetry`` distro (auto-detects
+    Azure Monitor from ``APPLICATIONINSIGHTS_CONNECTION_STRING`` and OTLP
+    from ``OTEL_EXPORTER_OTLP_ENDPOINT``).
 
   Users may pass a custom callable (or ``None``) via the
   ``configure_observability`` constructor parameter to override or
@@ -29,7 +30,7 @@
 - :func:`set_current_span` / :func:`detach_context` — explicit context management
 
 OpenTelemetry is a required dependency — these functions always create
-real spans.  Azure Monitor export is optional (lazy-imported).
+real spans.  Azure Monitor export is optional (auto-configured by the distro).
 """
 import logging
 import os
@@ -130,16 +131,15 @@ def configure_observability(
         setattr(_console, _CONSOLE_HANDLER_ATTR, True)
         root.addHandler(_console)
 
-    # Suppress noisy Azure SDK and OTel exporter logs
+    # Suppress the noisy Azure Core HTTP logging policy logger.
     logging.getLogger("azure.core.pipeline.policies.http_logging_policy").setLevel(logging.WARNING)
-    logging.getLogger("azure.monitor.opentelemetry.exporter").setLevel(logging.WARNING)
 
     # Tracing and OTel export
     _configure_tracing(connection_string=connection_string)
 
 
 def _configure_tracing(connection_string: Optional[str] = None) -> None:
-    """Configure OpenTelemetry exporters for Azure Monitor and OTLP.
+    """Configure OpenTelemetry exporters via the microsoft-opentelemetry distro.
 
     Internal helper called by :func:`configure_observability`.
 
@@ -148,20 +148,76 @@ def _configure_tracing(connection_string: Optional[str] = None) -> None:
     :type connection_string: str or None
     """
     resource = _create_resource()
-    provider = _ensure_trace_provider(resource)
+    if resource is None:
+        logger.warning("Failed to create OTel resource — tracing will not be configured.")
+        return
 
-    if provider is not None:
-        _register_enrichment_processor(provider)
+    # Build custom processors
+    agent_name = _config.resolve_agent_name() or None
+    agent_version = _config.resolve_agent_version() or None
+    project_id = _config.resolve_project_id() or None
 
+    if agent_name and agent_version:
+        agent_id = f"{agent_name}:{agent_version}"
+    elif agent_name:
+        agent_id = agent_name
+    else:
+        agent_id = None
+
+    span_processors = [
+        _FoundryEnrichmentSpanProcessor(
+            agent_name=agent_name, agent_version=agent_version,
+            agent_id=agent_id, project_id=project_id,
+        ),
+    ]
+    log_record_processors = [_BaggageLogRecordProcessor()]  # type: ignore[list-item]
+
+    try:
+        _setup_distro_export(
+            resource=resource,
+            span_processors=span_processors,
+            log_record_processors=log_record_processors,
+            connection_string=connection_string,
+        )
+        logger.info("Tracing configured successfully via microsoft-opentelemetry distro.")
+    except ImportError:
+        logger.warning("microsoft-opentelemetry is not installed — tracing export disabled.")
+        # Still set up TracerProvider with enrichment processor so spans are created
+        _ensure_trace_provider(resource, span_processors)
+
+
+def _setup_distro_export(
+    *,
+    resource: Any,
+    span_processors: list[Any],
+    log_record_processors: list[Any],
+    connection_string: Optional[str] = None,
+) -> None:
+    """Delegate to microsoft-opentelemetry distro for exporter configuration.
+
+    Separated into its own function so tests can easily mock it without
+    intercepting lazy imports.
+
+    :keyword resource: OTel resource describing this service.
+    :keyword span_processors: Span processors to register.
+    :keyword log_record_processors: Log record processors to register.
+    :keyword connection_string: Application Insights connection string.
+    """
+    from microsoft.opentelemetry import use_microsoft_opentelemetry
+
+    kwargs: dict[str, Any] = {
+        "resource": resource,
+        "span_processors": span_processors,
+        "log_record_processors": log_record_processors,
+    }
+
+    # Azure Monitor export is off by default in the distro — enable it
+    # when a connection string is available.
     if connection_string:
-        if resource is not None:
-            _setup_trace_export(provider, connection_string)
-            _setup_log_export(resource, connection_string)
+        kwargs["enable_azure_monitor"] = True
+        kwargs["azure_monitor_connection_string"] = connection_string
 
-    otlp_endpoint = _config.resolve_otlp_endpoint()
-    if otlp_endpoint and resource is not None:
-        _setup_otlp_trace_export(provider, otlp_endpoint)
-        _setup_otlp_log_export(resource, otlp_endpoint)
+    use_microsoft_opentelemetry(**kwargs)
 
 
 # ======================================================================
@@ -508,7 +564,16 @@ def _create_resource() -> Any:
     return Resource.create({_ATTR_SERVICE_NAME: service_name})
 
 
-def _ensure_trace_provider(resource: Any) -> Any:
+def _ensure_trace_provider(resource: Any, span_processors: Optional[list[Any]] = None) -> Any:
+    """Get or create a TracerProvider, optionally adding span processors.
+
+    Used as a fallback when the microsoft-opentelemetry distro is not installed.
+
+    :param resource: OTel resource describing this service.
+    :type resource: ~typing.Any
+    :param span_processors: Optional span processors to register.
+    :type span_processors: list[~typing.Any] or None
+    """
     if resource is None:
         return None
     try:
@@ -517,129 +582,15 @@ def _ensure_trace_provider(resource: Any) -> Any:
         return None
     current = trace.get_tracer_provider()
     if hasattr(current, "add_span_processor"):
-        return current
-    provider = SdkTracerProvider(resource=resource)
-    trace.set_tracer_provider(provider)
-    return provider
-
-
-_enrichment_configured = False
-_az_trace_configured = False
-_az_log_configured = False
-_otlp_trace_configured = False
-_otlp_log_configured = False
-
-
-def _register_enrichment_processor(provider: Any) -> None:
-    global _enrichment_configured  # pylint: disable=global-statement
-    if _enrichment_configured:
-        return
-    agent_name = _config.resolve_agent_name() or None
-    agent_version = _config.resolve_agent_version() or None
-    project_id = _config.resolve_project_id() or None
-
-    if agent_name and agent_version:
-        agent_id = f"{agent_name}:{agent_version}"
-    elif agent_name:
-        agent_id = agent_name
+        provider = current
     else:
-        agent_id = None
-
-    provider.add_span_processor(_FoundryEnrichmentSpanProcessor(
-        agent_name=agent_name, agent_version=agent_version,
-        agent_id=agent_id, project_id=project_id,
-    ))
-    _enrichment_configured = True
-
-
-def _setup_trace_export(provider: Any, connection_string: str) -> None:
-    global _az_trace_configured  # pylint: disable=global-statement
-    if _az_trace_configured or provider is None:
-        return
-    try:
-        from opentelemetry.sdk.trace.export import BatchSpanProcessor
-
-        from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter  # type: ignore[import-untyped]
-    except ImportError:
-        logger.warning("Trace export requires azure-monitor-opentelemetry-exporter.")
-        return
-    provider.add_span_processor(BatchSpanProcessor(
-        AzureMonitorTraceExporter(connection_string=connection_string)))
-    _az_trace_configured = True
-    logger.info("Application Insights trace exporter configured.")
-
-
-def _setup_log_export(resource: Any, connection_string: str) -> None:
-    global _az_log_configured  # pylint: disable=global-statement
-    if _az_log_configured:
-        return
-    try:
-        from opentelemetry._logs import set_logger_provider
-        from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
-        from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
-
-        from azure.monitor.opentelemetry.exporter import AzureMonitorLogExporter  # type: ignore[import-untyped]
-    except ImportError:
-        logger.warning("Log export requires azure-monitor-opentelemetry-exporter.")
-        return
-    log_provider = LoggerProvider(resource=resource)
-    set_logger_provider(log_provider)
-    log_provider.add_log_record_processor(BatchLogRecordProcessor(
-        AzureMonitorLogExporter(connection_string=connection_string)))
-    log_provider.add_log_record_processor(_BaggageLogRecordProcessor())  # type: ignore[arg-type]
-    logging.getLogger().addHandler(LoggingHandler(logger_provider=log_provider))
-    _az_log_configured = True
-    logger.info("Application Insights log exporter configured.")
-
-
-def _setup_otlp_trace_export(provider: Any, endpoint: str) -> None:
-    global _otlp_trace_configured  # pylint: disable=global-statement
-    if _otlp_trace_configured or provider is None:
-        return
-    try:
-        from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
-        from opentelemetry.sdk.trace.export import BatchSpanProcessor
-    except ImportError:
-        logger.warning("OTLP trace export requires opentelemetry-exporter-otlp-proto-grpc.")
-        return
-    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint)))
-    _otlp_trace_configured = True
-    logger.info("OTLP trace exporter configured (endpoint=%s).", endpoint)
-
-
-def _setup_otlp_log_export(resource: Any, endpoint: str) -> None:
-    global _otlp_log_configured  # pylint: disable=global-statement
-    if _otlp_log_configured:
-        return
-    try:
-        from opentelemetry._logs import get_logger_provider
-        from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
-        from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
-        from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
-    except ImportError:
-        logger.warning("OTLP log export requires opentelemetry-exporter-otlp-proto-grpc.")
-        return
-    current = get_logger_provider()
-    if hasattr(current, "add_log_record_processor"):
-        log_provider = current
-    else:
-        from opentelemetry._logs import set_logger_provider
-        log_provider = LoggerProvider(resource=resource)
-        set_logger_provider(log_provider)
-    log_provider.add_log_record_processor(  # type: ignore[union-attr]
-        BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint))
-    )
-    log_provider.add_log_record_processor(  # type: ignore[union-attr]
-        _BaggageLogRecordProcessor()  # type: ignore[arg-type]
-    )
-    # Note: LoggingHandler is NOT added here to avoid duplicating the
-    # handler already installed by _setup_log_export. The OTel LoggerProvider
-    # receives log records via the handler added there (or from direct OTel
-    # log API usage).  If OTLP is the only exporter, add a handler:
-    if not _az_log_configured:
-        logging.getLogger().addHandler(LoggingHandler(logger_provider=log_provider))
-    _otlp_log_configured = True
-    logger.info("OTLP log exporter configured (endpoint=%s).", endpoint)
+        provider = SdkTracerProvider(resource=resource)
+        trace.set_tracer_provider(provider)
+    if span_processors and not getattr(provider, "_agentserver_processors_added", False):
+        for proc in span_processors:
+            provider.add_span_processor(proc)
+        provider._agentserver_processors_added = True  # type: ignore[attr-defined]  # pylint: disable=protected-access
+    return provider
 
 
 def _extract_w3c_carrier(headers: Mapping[str, str]) -> dict[str, str]:

sdk/agentserver/azure-ai-agentserver-core/dev_requirements.txt

@@ -1,5 +1,4 @@
 -e ../../../eng/tools/azure-sdk-tools
--e ../../monitor/azure-monitor-opentelemetry-exporter
 -e ../../monitor/azure-monitor-query
 -e ../../identity/azure-identity
 pytest

sdk/agentserver/azure-ai-agentserver-core/pyproject.toml

@@ -25,8 +25,7 @@ dependencies = [
     "hypercorn>=0.17.0",
     "opentelemetry-api>=1.40.0",
     "opentelemetry-sdk>=1.40.0",
-    "opentelemetry-exporter-otlp-proto-grpc>=1.40.0",
-    "azure-monitor-opentelemetry-exporter>=1.0.0b49",
+    "microsoft-opentelemetry>=0.1.0b1",
 ]
 
 [build-system]
@@ -73,5 +72,4 @@ latestdependency = false
 pylint = true
 type_check_samples = false
 
-[tool.uv.sources]
-azure-monitor-opentelemetry-exporter = { path = "../../monitor/azure-monitor-opentelemetry-exporter" }
+[tool.uv.sources]

sdk/agentserver/azure-ai-agentserver-core/tests/test_tracing.py

@@ -53,11 +53,11 @@ def test_observability_always_called(self) -> None:
             mock_configure.assert_called_once()
 
     def test_observability_receives_appinsights_env_var(self) -> None:
-        with mock.patch.dict(os.environ, {"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=test"}):
+        with mock.patch.dict(os.environ, {"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=00000000-0000-0000-0000-000000000000"}):
             mock_configure = mock.MagicMock()
             AgentServerHost(configure_observability=mock_configure)
             mock_configure.assert_called_once()
-            assert mock_configure.call_args[1]["connection_string"] == "InstrumentationKey=test"
+            assert mock_configure.call_args[1]["connection_string"] == "InstrumentationKey=00000000-0000-0000-0000-000000000000"
 
     def test_observability_receives_otlp_env_var(self) -> None:
         with mock.patch.dict(os.environ, {"OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318"}):
@@ -78,7 +78,7 @@ def test_observability_receives_constructor_connection_string(self) -> None:
 
     def test_observability_disabled_when_none(self) -> None:
         """Passing configure_observability=None disables all SDK-managed observability."""
-        with mock.patch.dict(os.environ, {"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=test"}):
+        with mock.patch.dict(os.environ, {"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=00000000-0000-0000-0000-000000000000"}):
             # Should not raise even with App Insights configured
             AgentServerHost(configure_observability=None)
 
@@ -117,32 +117,30 @@ def test_explicit_overrides_env_var(self) -> None:
 
 
 # ------------------------------------------------------------------ #
-# _setup_azure_monitor (mocked)
+# _setup_distro_export (mocked)
 # ------------------------------------------------------------------ #
 
 
-class TestSetupAzureMonitor:
-    """Verify _configure_tracing calls the right exporter setup functions."""
-
-    def test_setup_azure_monitor_called_when_conn_str_provided(self) -> None:
-        with mock.patch("azure.ai.agentserver.core._tracing._setup_trace_export") as mock_trace:
-            with mock.patch("azure.ai.agentserver.core._tracing._setup_log_export"):
-                with mock.patch("azure.ai.agentserver.core._tracing._setup_otlp_trace_export"):
-                    with mock.patch("azure.ai.agentserver.core._tracing._setup_otlp_log_export"):
-                        from azure.ai.agentserver.core import _tracing
-                        _tracing._configure_tracing(connection_string="InstrumentationKey=test")
-                        mock_trace.assert_called_once()
-                        args = mock_trace.call_args[0]
-                        assert args[1] == "InstrumentationKey=test"
-
-    def test_setup_azure_monitor_not_called_when_no_conn_str(self) -> None:
-        with mock.patch("azure.ai.agentserver.core._tracing._setup_trace_export") as mock_trace:
-            with mock.patch("azure.ai.agentserver.core._tracing._setup_log_export"):
-                with mock.patch("azure.ai.agentserver.core._tracing._setup_otlp_trace_export"):
-                    with mock.patch("azure.ai.agentserver.core._tracing._setup_otlp_log_export"):
-                        from azure.ai.agentserver.core import _tracing
-                        _tracing._configure_tracing(connection_string=None)
-                        mock_trace.assert_not_called()
+class TestSetupDistroExport:
+    """Verify _configure_tracing calls the distro with the right args."""
+
+    def test_distro_called_when_conn_str_provided(self) -> None:
+        with mock.patch("azure.ai.agentserver.core._tracing._setup_distro_export") as mock_distro:
+            from azure.ai.agentserver.core import _tracing
+            _tracing._configure_tracing(connection_string="InstrumentationKey=00000000-0000-0000-0000-000000000000")
+            mock_distro.assert_called_once()
+            kwargs = mock_distro.call_args[1]
+            assert kwargs["connection_string"] == "InstrumentationKey=00000000-0000-0000-0000-000000000000"
+            assert len(kwargs["span_processors"]) >= 1
+            assert len(kwargs["log_record_processors"]) >= 1
+
+    def test_distro_called_without_conn_str(self) -> None:
+        with mock.patch("azure.ai.agentserver.core._tracing._setup_distro_export") as mock_distro:
+            from azure.ai.agentserver.core import _tracing
+            _tracing._configure_tracing(connection_string=None)
+            mock_distro.assert_called_once()
+            kwargs = mock_distro.call_args[1]
+            assert kwargs["connection_string"] is None
 
 
 # ------------------------------------------------------------------ #