feat(chronicle): emit assistant.usage events for VS Code Chat sessions#317640
Closed
digitarald wants to merge 3 commits into
Closed
Conversation
Add assistant.usage event emission to the cloud session store so token consumption from the native VS Code Chat harness is visible for cost analysis. Changes: - translateSpan() now handles GenAiOperationName.CHAT spans, emitting assistant.usage events with model, inputTokens, outputTokens, and optional cacheReadTokens, timeToFirstTokenMs, duration fields - translateDebugLogEntry() handles llm_request entries for the reindex/backfill path - RemoteSessionExporter buffers pre-init CHAT spans (which complete before their parent invoke_agent) and replays them after session initialization to avoid losing first-turn usage data Fixes microsoft/vscode-internalbacklog#7766 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds assistant.usage event emission to Chronicle’s cloud session export pipeline so VS Code Chat sessions contribute token-usage data for cost/usage analysis (including support for both live OTel spans and debug-log reindexing).
Changes:
- Extend
translateSpan()to translateGenAiOperationName.CHATspans intoassistant.usageevents (tokens, model, optional TTFT/duration/cache). - Extend
translateDebugLogEntry()to emitassistant.usageforllm_requestentries during reindex/backfill. - Buffer pre-init CHAT spans in
RemoteSessionExporterand replay them after session initialization to avoid losing first-turn usage.
Show a summary per file
| File | Description |
|---|---|
| extensions/copilot/src/extension/chronicle/vscode-node/remoteSessionExporter.ts | Buffers CHAT spans that complete before invoke_agent initialization and replays them after init. |
| extensions/copilot/src/extension/chronicle/common/eventTranslator.ts | Emits assistant.usage events from CHAT spans and llm_request debug log entries. |
| extensions/copilot/src/extension/chronicle/common/test/eventTranslator.spec.ts | Adds tests for CHAT-span and llm_request → assistant.usage translation paths. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 3
- Preserve sub-agent attribution when replaying buffered CHAT spans by
storing { span, subagentId } in _pendingChatSpans instead of bare spans
- Accept both 'cachedTokens' (production debug logger format) and
'cacheReadTokens' field names in llm_request backfill path
- Map 'ttft' attribute to timeToFirstTokenMs in backfilled events
- Update test fixtures to use production attribute names (cachedTokens, ttft)
- Add test for cacheReadTokens fallback field name
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Usage events are transient metrics that should not be persisted as session content. Set ephemeral=true on both the span translation and debug log backfill paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
@copilot resolve the merge conflicts in this pull request |
vijayupadya
requested changes
May 21, 2026
| data.duration = span.endTime - span.startTime; | ||
| } | ||
|
|
||
| pushEvent(events, state, 'assistant.usage', data, /*ephemeral*/ false, subagentId); |
Contributor
There was a problem hiding this comment.
/ephemeral/ flag needs to be true else server could reject this payload.
| outputTokens, | ||
| }; | ||
| const cacheReadTokens = entry.attrs.cacheReadTokens; | ||
| if (typeof cacheReadTokens === 'number') { |
Contributor
There was a problem hiding this comment.
for reindex, we need to handle cachedTokens and ttf which is written by the chat debug logger.
| if (pendingChats) { | ||
| this._pendingChatSpans.delete(sessionId); | ||
| for (const chatSpan of pendingChats) { | ||
| const chatEvents = translateSpan(chatSpan, state, context); |
Contributor
There was a problem hiding this comment.
+1 on the CCR feedback regarding subAgent
Contributor
|
you can close this PR. these changes are merged from a different pr. |
digitarald
deleted the
agents/https-github-com-microsoft-vscode-internalbacklo-b40e0a7c
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
assistant.usageevent emission to the cloud session store so that token consumption from the native VS Code Chat harness (~85% of sessions) becomes visible for cost analysis.Problem
VS Code Chat sessions have turns data in the cloud session store but zero
assistant.usageevents, making token consumption invisible to thecost-optimizerskill and any usage analysis queries.Changes
eventTranslator.tstranslateSpan()now handlesGenAiOperationName.CHATspans, emittingassistant.usageevents with:model,inputTokens,outputTokenscacheReadTokens,timeToFirstTokenMs,durationagentIdtranslateDebugLogEntry()handlesllm_requestentries for the reindex/backfill pathcachedTokens(production debug logger format) andcacheReadTokensfield namesttftattribute totimeToFirstTokenMsremoteSessionExporter.tsinvoke_agentspan) and replays them after session initialization to avoid losing first-turn usage datasubagentId) for buffered CHAT spans during replayTests
eventTranslator.spec.tscovering: CHAT spans with/without usage tokens, model fallback, sub-agent attribution, debug log entry translation, and field name fallbackValidation
npx tsc --noEmit— zero errorsnpx vitest run src/extension/chronicle/— 223 tests pass (11 files)Fixes microsoft/vscode-internalbacklog#7766