feat(telemetry) Instrument traces with more attributes and make them available to OTEL users (#20237)

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Jerop Kipruto <jerop@google.com>
Co-authored-by: MD. MOHIBUR RAHMAN <35300157+mrpmohiburrahman@users.noreply.github.com>
Co-authored-by: Jeffrey Ying <jeffrey.ying86@live.com>
Co-authored-by: Bryan Morgan <bryanmorgan@google.com>
Co-authored-by: joshualitt <joshualitt@google.com>
Co-authored-by: Dev Randalpura <devrandalpura@google.com>
Co-authored-by: Google Admin <github-admin@google.com>
Co-authored-by: Ben Knutson <benknutson@google.com>

Author: 10 people

docs/cli/telemetry.md

@@ -176,11 +176,12 @@ Sends telemetry directly to Google Cloud services. No collector needed.
    }
    ```
 2. Run Gemini CLI and send prompts.
-3. View logs and metrics:
+3. View logs, metrics, and traces:
    - Open the Google Cloud Console in your browser after sending prompts:
-     - Logs: https://console.cloud.google.com/logs/
-     - Metrics: https://console.cloud.google.com/monitoring/metrics-explorer
-     - Traces: https://console.cloud.google.com/traces/list
+     - Logs (Logs Explorer): https://console.cloud.google.com/logs/
+     - Metrics (Metrics Explorer):
+       https://console.cloud.google.com/monitoring/metrics-explorer
+     - Traces (Trace Explorer): https://console.cloud.google.com/traces/list
 
 ### Collector-based export (advanced)
 
@@ -208,11 +209,12 @@ forward data to Google Cloud.
    - Save collector logs to `~/.gemini/tmp/<projectHash>/otel/collector-gcp.log`
    - Stop collector on exit (e.g. `Ctrl+C`)
 3. Run Gemini CLI and send prompts.
-4. View logs and metrics:
+4. View logs, metrics, and traces:
    - Open the Google Cloud Console in your browser after sending prompts:
-     - Logs: https://console.cloud.google.com/logs/
-     - Metrics: https://console.cloud.google.com/monitoring/metrics-explorer
-     - Traces: https://console.cloud.google.com/traces/list
+     - Logs (Logs Explorer): https://console.cloud.google.com/logs/
+     - Metrics (Metrics Explorer):
+       https://console.cloud.google.com/monitoring/metrics-explorer
+     - Traces (Trace Explorer): https://console.cloud.google.com/traces/list
    - Open `~/.gemini/tmp/<projectHash>/otel/collector-gcp.log` to view local
      collector logs.
 
@@ -270,10 +272,10 @@ For local development and debugging, you can capture telemetry data locally:
 3. View traces at http://localhost:16686 and logs/metrics in the collector log
    file.
 
-## Logs and metrics
+## Logs, metrics, and traces
 
-The following section describes the structure of logs and metrics generated for
-Gemini CLI.
+The following section describes the structure of logs, metrics, and traces
+generated for Gemini CLI.
 
 The `session.id`, `installation.id`, `active_approval_mode`, and `user.email`
 (available only when authenticated with a Google account) are included as common
@@ -824,6 +826,32 @@ Optional performance monitoring for startup, CPU/memory, and phase timing.
     - `current_value` (number)
     - `baseline_value` (number)
 
+### Traces
+
+Traces offer a granular, "under-the-hood" view of every agent and backend
+operation. By providing a high-fidelity execution map, they enable precise
+debugging of complex tool interactions and deep performance optimization. Each
+trace captures rich, consistent metadata via custom span attributes:
+
+- `gen_ai.operation.name` (string): The high-level operation kind (e.g.
+  "tool_call", "llm_call").
+- `gen_ai.agent.name` (string): The service agent identifier ("gemini-cli").
+- `gen_ai.agent.description` (string): The service agent description.
+- `gen_ai.input.messages` (string): Input messages or metadata specific to the
+  operation.
+- `gen_ai.output.messages` (string): Output messages or metadata generated from
+  the operation.
+- `gen_ai.request.model` (string): The request model name.
+- `gen_ai.response.model` (string): The response model name.
+- `gen_ai.system_instructions` (json string): The system instructions.
+- `gen_ai.prompt.name` (string): The prompt name.
+- `gen_ai.tool.name` (string): The executed tool's name.
+- `gen_ai.tool.call_id` (string): The generated specific ID of the tool call.
+- `gen_ai.tool.description` (string): The executed tool's description.
+- `gen_ai.tool.definitions` (json string): The executed tool's description.
+- `gen_ai.conversation.id` (string): The current CLI session ID.
+- Additional user-defined Custom Attributes passed via the span's configuration.
+
 #### GenAI semantic convention
 
 The following metrics comply with [OpenTelemetry GenAI semantic conventions] for

docs/local-development.md

@@ -1,23 +1,21 @@
 # Local development guide
 
 This guide provides instructions for setting up and using local development
-features, such as development tracing.
+features, such as tracing.
 
-## Development tracing
+## Tracing
 
-Development traces (dev traces) are OpenTelemetry (OTel) traces that help you
-debug your code by instrumenting interesting events like model calls, tool
-scheduler, tool calls, etc.
+Traces are OpenTelemetry (OTel) records that help you debug your code by
+instrumenting key events like model calls, tool scheduler operations, and tool
+calls.
 
-Dev traces are verbose and are specifically meant for understanding agent
-behavior and debugging issues. They are disabled by default.
+Traces provide deep visibility into agent behavior and are invaluable for
+debugging complex issues. They are captured automatically when telemetry is
+enabled.
 
-To enable dev traces, set the `GEMINI_DEV_TRACING=true` environment variable
-when running Gemini CLI.
+### Viewing traces
 
-### Viewing dev traces
-
-You can view dev traces using either Jaeger or the Genkit Developer UI.
+You can view traces using either Jaeger or the Genkit Developer UI.
 
 #### Using Genkit
 
@@ -37,13 +35,12 @@ Genkit provides a web-based UI for viewing traces and other telemetry data.
     Genkit Developer UI: http://localhost:4000
     ```
 
-2.  **Run Gemini CLI with dev tracing:**
+2.  **Run Gemini CLI:**
 
-    In a separate terminal, run your Gemini CLI command with the
-    `GEMINI_DEV_TRACING` environment variable:
+    In a separate terminal, run your Gemini CLI command:
 
     ```bash
-    GEMINI_DEV_TRACING=true gemini
+    gemini
     ```
 
 3.  **View the traces:**
@@ -53,7 +50,7 @@ Genkit provides a web-based UI for viewing traces and other telemetry data.
 
 #### Using Jaeger
 
-You can view dev traces in the Jaeger UI. To get started, follow these steps:
+You can view traces in the Jaeger UI. To get started, follow these steps:
 
 1.  **Start the telemetry collector:**
 
@@ -67,13 +64,12 @@ You can view dev traces in the Jaeger UI. To get started, follow these steps:
     This command also configures your workspace for local telemetry and provides
     a link to the Jaeger UI (usually `http://localhost:16686`).
 
-2.  **Run Gemini CLI with dev tracing:**
+2.  **Run Gemini CLI:**
 
-    In a separate terminal, run your Gemini CLI command with the
-    `GEMINI_DEV_TRACING` environment variable:
+    In a separate terminal, run your Gemini CLI command:
 
     ```bash
-    GEMINI_DEV_TRACING=true gemini
+    gemini
     ```
 
 3.  **View the traces:**
@@ -84,10 +80,10 @@ You can view dev traces in the Jaeger UI. To get started, follow these steps:
 For more detailed information on telemetry, see the
 [telemetry documentation](./cli/telemetry.md).
 
-### Instrumenting code with dev traces
+### Instrumenting code with traces
 
-You can add dev traces to your own code for more detailed instrumentation. This
-is useful for debugging and understanding the flow of execution.
+You can add traces to your own code for more detailed instrumentation. This is
+useful for debugging and understanding the flow of execution.
 
 Use the `runInDevTraceSpan` function to wrap any section of code in a trace
 span.
@@ -96,29 +92,39 @@ Here is a basic example:
 
 ```typescript
 import { runInDevTraceSpan } from '@google/gemini-cli-core';
-
-await runInDevTraceSpan({ name: 'my-custom-span' }, async ({ metadata }) => {
-  // The `metadata` object allows you to record the input and output of the
-  // operation as well as other attributes.
-  metadata.input = { key: 'value' };
-  // Set custom attributes.
-  metadata.attributes['gen_ai.request.model'] = 'gemini-4.0-mega';
-
-  // Your code to be traced goes here
-  try {
-    const output = await somethingRisky();
-    metadata.output = output;
-    return output;
-  } catch (e) {
-    metadata.error = e;
-    throw e;
-  }
-});
+import { GeminiCliOperation } from '@google/gemini-cli-core/lib/telemetry/constants.js';
+
+await runInDevTraceSpan(
+  {
+    operation: GeminiCliOperation.ToolCall,
+    attributes: {
+      [GEN_AI_AGENT_NAME]: 'gemini-cli',
+    },
+  },
+  async ({ metadata }) => {
+    // The `metadata` object allows you to record the input and output of the
+    // operation as well as other attributes.
+    metadata.input = { key: 'value' };
+    // Set custom attributes.
+    metadata.attributes['custom.attribute'] = 'custom.value';
+
+    // Your code to be traced goes here
+    try {
+      const output = await somethingRisky();
+      metadata.output = output;
+      return output;
+    } catch (e) {
+      metadata.error = e;
+      throw e;
+    }
+  },
+);
 ```
 
 In this example:
 
-- `name`: The name of the span, which will be displayed in the trace.
+- `operation`: The operation type of the span, represented by the
+  `GeminiCliOperation` enum.
 - `metadata.input`: (Optional) An object containing the input data for the
   traced operation.
 - `metadata.output`: (Optional) An object containing the output data from the

integration-tests/acp-telemetry.test.ts

@@ -72,7 +72,6 @@ describe('ACP telemetry', () => {
           GEMINI_TELEMETRY_ENABLED: 'true',
           GEMINI_TELEMETRY_TARGET: 'local',
           GEMINI_TELEMETRY_OUTFILE: telemetryPath,
-          // GEMINI_DEV_TRACING not set: fake responses aren't instrumented for spans
         },
       },
     );

packages/cli/src/ui/hooks/useGeminiStream.test.tsx

@@ -25,6 +25,7 @@ import type {
   Config,
   EditorType,
   AnyToolInvocation,
+  SpanMetadata,
 } from '@google/gemini-cli-core';
 import {
   CoreToolCallStatus,
@@ -39,6 +40,7 @@ import {
   coreEvents,
   CoreEvent,
   MCPDiscoveryState,
+  GeminiCliOperation,
   getPlanModeExitMessage,
 } from '@google/gemini-cli-core';
 import type { Part, PartListUnion } from '@google/genai';
@@ -101,6 +103,19 @@ const MockValidationRequiredError = vi.hoisted(
     },
 );
 
+const mockRunInDevTraceSpan = vi.hoisted(() =>
+  vi.fn(async (opts, fn) => {
+    const metadata: SpanMetadata = {
+      name: opts.operation,
+      attributes: opts.attributes || {},
+    };
+    return await fn({
+      metadata,
+      endSpan: vi.fn(),
+    });
+  }),
+);
+
 vi.mock('@google/gemini-cli-core', async (importOriginal) => {
   const actualCoreModule = (await importOriginal()) as any;
   return {
@@ -113,6 +128,7 @@ vi.mock('@google/gemini-cli-core', async (importOriginal) => {
     tokenLimit: vi.fn().mockReturnValue(100), // Mock tokenLimit
     recordToolCallInteractions: vi.fn().mockResolvedValue(undefined),
     getCodeAssistServer: vi.fn().mockReturnValue(undefined),
+    runInDevTraceSpan: mockRunInDevTraceSpan,
   };
 });
 
@@ -794,6 +810,23 @@ describe('useGeminiStream', () => {
           item.text.includes('Got it. Focusing on tests only.'),
       ),
     ).toBe(true);
+
+    expect(mockRunInDevTraceSpan).toHaveBeenCalledWith(
+      expect.objectContaining({
+        operation: GeminiCliOperation.SystemPrompt,
+      }),
+      expect.any(Function),
+    );
+
+    const spanArgs = mockRunInDevTraceSpan.mock.calls[0];
+    const fn = spanArgs[1];
+    const metadata = { attributes: {} };
+    await act(async () => {
+      await fn({ metadata, endSpan: vi.fn() });
+    });
+    expect(metadata).toMatchObject({
+      input: sentParts,
+    });
   });
 
   it('should handle all tool calls being cancelled', async () => {
@@ -2452,6 +2485,11 @@ describe('useGeminiStream', () => {
     // This is the core fix validation: Rationale comes before tools are even scheduled (awaited)
     expect(rationaleIndex).toBeLessThan(scheduleIndex);
     expect(rationaleIndex).toBeLessThan(toolGroupIndex);
+
+    // Ensure all state updates from recursive submitQuery are settled
+    await waitFor(() => {
+      expect(result.current.streamingState).toBe(StreamingState.Idle);
+    });
   });
 
   it('should process @include commands, adding user turn after processing to prevent race conditions', async () => {
@@ -3554,4 +3592,31 @@ describe('useGeminiStream', () => {
       expect(result.current.pendingHistoryItems.length).toEqual(0);
     });
   });
+
+  it('should trace UserPrompt telemetry on submitQuery', async () => {
+    const { result } = renderTestHook();
+
+    mockSendMessageStream.mockReturnValue(
+      (async function* () {
+        yield { type: ServerGeminiEventType.Content, value: 'Response' };
+      })(),
+    );
+
+    await act(async () => {
+      await result.current.submitQuery('telemetry test query');
+    });
+
+    const userPromptCall = mockRunInDevTraceSpan.mock.calls.find(
+      (call) =>
+        call[0].operation === GeminiCliOperation.UserPrompt ||
+        call[0].operation === 'UserPrompt',
+    );
+    expect(userPromptCall).toBeDefined();
+
+    const spanMetadata = {} as SpanMetadata;
+    await act(async () => {
+      await userPromptCall![1]({ metadata: spanMetadata, endSpan: vi.fn() });
+    });
+    expect(spanMetadata.input).toBe('telemetry test query');
+  });
 });

packages/cli/src/ui/hooks/useGeminiStream.ts

@@ -36,6 +36,7 @@ import {
   CoreToolCallStatus,
   buildUserSteeringHintPrompt,
   generateSteeringAckMessage,
+  GeminiCliOperation,
   getPlanModeExitMessage,
 } from '@google/gemini-cli-core';
 import type {
@@ -1262,7 +1263,11 @@ export const useGeminiStream = (
       prompt_id?: string,
     ) =>
       runInDevTraceSpan(
-        { name: 'submitQuery' },
+        {
+          operation: options?.isContinuation
+            ? GeminiCliOperation.SystemPrompt
+            : GeminiCliOperation.UserPrompt,
+        },
         async ({ metadata: spanMetadata }) => {
           spanMetadata.input = query;