GenAI Utils | Add agent metrics, and events by etserend · Pull Request #4329 · open-telemetry/opentelemetry-python-contrib

etserend · 2026-03-12T17:26:54Z

Description

Adds gen_ai.client.operation.duration and gen_ai.client.token.usage histogram metrics along with gen_ai.client.inference.operation.details log events for agent lifecycle operations (create_agent, invoke_agent).

Demo app PR: #4343
Sample metrics output:

Metrics — gen_ai.client.operation.duration (Histogram, unit: s)
├── Data Point 1 — create_agent (local, ~0ms)
│   ├── gen_ai.operation.name: create_agent
│   ├── gen_ai.request.model: gemini-2.5-flash
│   ├── gen_ai.provider.name: gcp_vertex_ai
│   └── server.address: us-central1-aiplatform.googleapis.com
│
├── Data Point 2 — create_agent (deploy, ~259.75s)
│   ├── gen_ai.operation.name: create_agent
│   ├── gen_ai.request.model: gemini-2.5-flash
│   ├── gen_ai.provider.name: gcp_vertex_ai
│   └── server.address: us-central1-aiplatform.googleapis.com
│
└── Data Point 3 — invoke_agent (~3.16s)
    ├── gen_ai.operation.name: invoke_agent
    ├── gen_ai.request.model: gemini-2.5-flash
    ├── gen_ai.provider.name: gcp_vertex_ai
    └── server.address: us-central1-aiplatform.googleapis.com

Metrics — gen_ai.client.token.usage (Histogram, unit: {token})
├── Data Point 1 — input tokens (invoke_agent)
│   ├── gen_ai.operation.name: invoke_agent
│   ├── gen_ai.request.model: gemini-2.5-flash
│   ├── gen_ai.provider.name: gcp_vertex_ai
│   ├── gen_ai.token.type: input
│   ├── server.address: us-central1-aiplatform.googleapis.com
│   └── value: 12
│
└── Data Point 2 — output tokens (invoke_agent)
    ├── gen_ai.operation.name: invoke_agent
    ├── gen_ai.request.model: gemini-2.5-flash
    ├── gen_ai.provider.name: gcp_vertex_ai
    ├── gen_ai.token.type: output
    ├── server.address: us-central1-aiplatform.googleapis.com
    └── value: 105

Sample events output:

Events — gen_ai.client.inference.operation.details (3 log records)
│
├── Event 1 — create_agent (trace: e7f8a7cd..., scope: opentelemetry.util.genai.handler v0.3b0.dev)
│   ├── gen_ai.operation.name: create_agent
│   ├── gen_ai.agent.name: Currency Exchange Agent
│   ├── gen_ai.agent.description: Deploying agent to Vertex AI Agent Engine
│   ├── gen_ai.provider.name: gcp_vertex_ai
│   ├── gen_ai.request.model: gemini-2.5-flash
│   └── server.address: us-central1-aiplatform.googleapis.com
│
├── Event 2 — invoke_agent (trace: 4dc8827b..., scope: opentelemetry.util.genai.handler v0.3b0.dev)
│   ├── gen_ai.operation.name: invoke_agent
│   ├── gen_ai.agent.name: Currency Exchange Agent
│   ├── gen_ai.agent.description: Currency exchange agent demo
│   ├── gen_ai.provider.name: gcp_vertex_ai
│   ├── gen_ai.request.model: gemini-2.5-flash
│   ├── gen_ai.response.finish_reasons:
│   ├── gen_ai.usage.input_tokens: 12
│   ├── gen_ai.usage.output_tokens: 105
│   └── server.address: us-central1-aiplatform.googleapis.com
│
└── Event 3 — chat (trace: 4dc8827b..., scope: opentelemetry.instrumentation.vertexai)
    ├── gen_ai.operation.name: chat
    ├── gen_ai.request.model: gemini-2.5-flash
    ├── gen_ai.response.model: gemini-2.5-flash
    ├── gen_ai.response.finish_reasons:
    ├── gen_ai.usage.input_tokens: 12
    ├── gen_ai.usage.output_tokens: 105
    ├── gen_ai.input.messages: [{"parts":[{"content":"What is the ex...
    ├── gen_ai.output.messages: [{"finish_reason":"stop","parts":[{"...
    ├── server.address: us-central1-aiplatform.googleapis.com
    └── server.port: 443

Fixes # (issue)

Type of change

New feature (non-breaking change which adds functionality)

Does This PR Require a Core Repo Change?

No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

adityamehra · 2026-03-18T19:22:46Z

+            attributes[error_attributes.ERROR_TYPE] = error_type
+
+        if duration_seconds is not None:
+            self._duration_histogram.record(


We might need to emit gen_ai.agent.duration metric to capture the agent duration and operation duration might not suffice. Is gen_ai.agent.duration part of the sem conv?

adityamehra · 2026-03-18T19:23:06Z

+    tool_definitions: list[dict[str, Any]] | None = None
+
+    # Span kind: CLIENT for remote agents, INTERNAL for in-process agents
+    is_remote: bool = True


Why is the default value True? Are we assuming the invoke agents will be a remote agent call by default and not an in-process agent?

Also, SpanContext already has a field to check for is remote - https://opentelemetry.io/docs/specs/otel/trace/api/#isremote. Maybe we can use that

github-actions · 2026-04-02T04:05:44Z

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

github-actions · 2026-04-17T04:16:52Z

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

… and events

Copilot

Pull request overview

This PR extends the opentelemetry-util-genai utility package to cover agent lifecycle telemetry by introducing first-class agent invocation types and wiring them into TelemetryHandler, including duration/token metrics and detailed inference-operation log events.

Changes:

Add AgentCreation and AgentInvocation invocation types with span attribute population, metrics recording, and event emission.
Add TelemetryHandler factory methods and context managers for create_agent and local/remote invoke_agent.
Add agent-focused unit tests and update the util changelog.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
util/opentelemetry-util-genai/tests/test_handler_agent.py	Adds unit tests for agent creation/invocation spans, metrics, and (intended) events.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/invocation.py	Re-exports `AgentCreation`/`AgentInvocation` as part of the public invocation surface.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py	Adds handler factory methods + context managers for agent operations.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/_content.py	Introduces shared content-serialization helper used by invocation types.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/_agent_invocation.py	Implements `invoke_agent` spans/metrics/events for local and remote agent calls.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/_agent_creation.py	Implements `create_agent` spans/metrics/events for agent creation.
util/opentelemetry-util-genai/CHANGELOG.md	Notes the new agent invocation types and handler methods.

Copilot · 2026-04-23T18:56:06Z

+from opentelemetry.util.genai._agent_creation import AgentCreation
+from opentelemetry.util.genai._agent_invocation import AgentInvocation


The handler imports AgentCreation/AgentInvocation from private modules (_agent_creation/_agent_invocation) even though they are now publicly re-exported from opentelemetry.util.genai.invocation (as done for EmbeddingInvocation/InferenceInvocation/etc.). Prefer importing them from the public invocation module to keep private modules internal and keep handler imports consistent.

Copilot · 2026-04-23T18:56:06Z

+from opentelemetry.util.genai._agent_creation import AgentCreation
+from opentelemetry.util.genai._agent_invocation import AgentInvocation


Since AgentCreation/AgentInvocation are now part of this module’s public re-export surface, the usage example in the module docstring (the import list) should be updated to include them; otherwise the docstring is misleading for users relying on this file for the canonical import path.

Copilot · 2026-04-23T18:56:06Z

+    def test_invoke_agent_emits_event(self, _mock_exp, _mock_emit):
+        inv = self.handler.start_invoke_local_agent(
+            "openai", request_model="gpt-4"
+        )
+        inv.agent_name = "Event Agent"
+        inv.stop()
+
+        # The logger.emit was called — verify via span attributes at minimum
+        spans = self.span_exporter.get_finished_spans()
+        assert len(spans) == 1
+        assert spans[0].attributes[GenAI.GEN_AI_AGENT_NAME] == "Event Agent"
+


These tests patch should_emit_event/is_experimental_mode but don’t actually assert that a log event was emitted (or validate its event_name/attributes). Consider wiring a LoggerProvider with an InMemoryLogExporter (similar to existing event tests) and asserting that exactly one log record was produced with event_name="gen_ai.client.inference.operation.details" and expected agent attributes.

etserend · 2026-04-28T14:57:52Z

Closing this PR — the gen_ai.client.operation.duration and gen_ai.client.token.usage histogram metrics for invoke_agent were already included in #4274 (_get_metric_attributes, _get_metric_token_counts, _metrics_recorder.record() in AgentInvocation._apply_finish, along with TestAgentInvocationMetrics tests). No additional changes needed.

Scope: opentelemetry.util.genai.handler
├── schema_url: https://opentelemetry.io/schemas/1.37.0
│
├── gen_ai.client.operation.duration (Histogram, unit: s)
│   ├── Data point [1/1]
│   │   ├── sum: 6.3
│   │   ├── min: 6.3
│   │   ├── max: 6.3
│   │   └── Attributes
│   │       ├── gen_ai.operation.name: invoke_agent
│   │       ├── gen_ai.provider.name: gcp_vertex_ai
│   │       ├── gen_ai.request.model: gemini-2.5-flash
│   │       ├── server.address: us-central1-aiplatform.googleapis.com
│   │       └── server.port: 443
│
└── gen_ai.client.token.usage (Histogram, unit: {token})
    ├── Data point [1/4] — input tokens (remote)
    │   ├── sum: 9.0
    │   └── Attributes
    │       ├── gen_ai.operation.name: invoke_agent
    │       ├── gen_ai.provider.name: gcp_vertex_ai
    │       ├── gen_ai.request.model: gemini-2.5-flash
    │       ├── server.address: us-central1-aiplatform.googleapis.com
    │       ├── server.port: 443
    │       └── gen_ai.token.type: input
    ├── Data point [2/4] — output tokens (remote)
    │   ├── sum: 9.0
    │   └── Attributes
    │       ├── ... (same as above)
    │       └── gen_ai.token.type: output
    ├── Data point [3/4] — input tokens (local)
    │   └── ... (without server.address / server.port)
    └── Data point [4/4] — output tokens (local)
        └── ... (without server.address / server.port)

github-actions Bot assigned DylanRussell and keith-decker Mar 12, 2026

github-actions Bot requested review from DylanRussell and keith-decker March 12, 2026 17:27

GenAI Utils | Agent metrics and events

8f2f3e1

etserend force-pushed the genai-utils/agent-metrics-events branch from d1959e6 to 8f2f3e1 Compare March 12, 2026 18:12

wrisa reviewed Mar 16, 2026

View reviewed changes

Comment thread util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py Outdated

Comment thread util/opentelemetry-util-genai/src/opentelemetry/util/genai/handler.py Outdated

adityamehra reviewed Mar 16, 2026

View reviewed changes

update context handler

6e23298

adityamehra reviewed Mar 18, 2026

View reviewed changes

github-actions Bot added the Stale label Apr 2, 2026

resolve comment related to is_remote attribute

9bd96d1

github-actions Bot assigned lmolkova Apr 2, 2026

github-actions Bot removed the Stale label Apr 3, 2026

github-actions Bot added the Stale label Apr 17, 2026

etserend mentioned this pull request Apr 17, 2026

Genai utils | Add AgentInvocation type #4274

Merged

10 tasks

feat(genai-utils): add AgentCreation and AgentInvocation with metrics…

13b73d9

… and events

github-actions Bot removed the Stale label Apr 18, 2026

lmolkova requested a review from Copilot April 23, 2026 18:50

Copilot started reviewing on behalf of lmolkova April 23, 2026 18:51 View session

Copilot AI reviewed Apr 23, 2026

View reviewed changes

etserend closed this Apr 28, 2026

		from opentelemetry.util.genai._agent_creation import AgentCreation
		from opentelemetry.util.genai._agent_invocation import AgentInvocation

Conversation

etserend commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Does This PR Require a Core Repo Change?

Checklist:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adityamehra Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

adityamehra Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 2, 2026

Uh oh!

github-actions Bot commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

etserend commented Apr 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

etserend commented Mar 12, 2026 •

edited

Loading