Metric: gen_ai.server.time_to_first_token

[spurs30]

What problem do you want to solve?

In Semantic Conventions 1.38.0, there is a standard definition for the TTFT (Time to First Transaction) metric in the metrics section. However, the current agent does not yet implement this metric. Is there any plan to implement this metric in the future? If not, what would be the approach to customizing the implementation of this metric?

Describe the solution you'd like

1. Follow-up plan for achieving the TTFT metric.
2. Developer's approach to customizing the TTFT metric.

Describe alternatives you've considered

No response

Additional Context

No response

Would you like to implement a fix?

None

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[xrmx]

First thing is that implemented stuff is per-instrumentation and not globally, second thing is that this is a server metric and AFAICS we only have client instrumentations.

[spurs30]

Yes, currently, the agent only has client-side metrics. How can we define this metric for a specific instrumentation? For example, vLLM, SGLang.

[Nik-Reddy]

Hi @spurs30, thanks for raising this. I've opened a PR that implements the gen_ai.server.time_to_first_token metric for the OpenAI v2 instrumentation: #4415.

To address @xrmx's point about this being a server metric: my implementation measures TTFT from the client's perspective , specifically, the elapsed time from request initiation to the first content chunk received during streaming. While this includes network latency and isn't a pure server-side measurement, it's the best approximation available from a client instrumentation and is the approach taken by other observability SDKs for this metric.

What's implemented:

• gen_ai.server.time_to_first_token histogram (seconds) with semconv v1.38.0 bucket boundaries
• Recorded only for successful streaming responses
• First-token detection in the stream wrapper, wired into both sync and async paths
• 8 test cases covering streaming, non-streaming exclusion, and tool-call streaming

For true server-side TTFT (e.g., vLLM, SGLang instrumentations), a separate server instrumentation would be needed, but this covers the common client-side use case.

Would appreciate a review when you get a chance. @xrmx, any feedback on the approach would be welcome as well.

[Nik-Reddy]

Following up on my earlier comment. I realized my client-side TTFT implementation in #4415 doesn't fully address what @spurs30 was asking about. The original question was specifically about server-side TTFT for inference engines like vLLM and SGLang, not client-observed TTFT.

To close that gap, I've now implemented a full vLLM server-side instrumentation: #4419.

This is the first server-side GenAI instrumentation in this repo. It instruments vllm.LLM.generate() and vllm.LLM.chat() and records true server-side metrics:

• gen_ai.server.time_to_first_token — actual server-side TTFT (no network latency)
• gen_ai.server.time_per_output_token — TPOT
• gen_ai.server.request.duration — total inference duration
• Spans use SpanKind.SERVER with full GenAI semconv attributes

So the two PRs together cover both sides:

PR	Scope	Measures
#4415	Client-side (OpenAI v2)	Client-observed TTFT (includes network)
#4419	Server-side (vLLM)	True server-side TTFT (model inference only)

@spurs30 the vLLM instrumentation should directly address your use case. A similar approach could be extended to SGLang as a follow-up. Would appreciate any feedback.