feat: Token Usage Tracking implementation

[AjitPadhi-Microsoft]

Purpose

• This pull request adds comprehensive LLM token usage telemetry and monitoring to the codebase, enabling detailed tracking of token consumption at the agent, model, and user levels. It introduces robust extraction and aggregation of token usage from API responses and streaming updates, emits standardized telemetry events, and provides a suite of KQL queries for monitoring and cost analysis in Application Insights.

LLM Token Usage Telemetry and Monitoring

Token usage extraction and telemetry emission:

• Introduced helper functions (_extract_usage_from_dict, _extract_usage_from_update, _extract_usage_from_response, _track_token_usage) in chat.py to extract input, output, and total token counts from various response formats and emit three types of telemetry events: LLM_Token_Usage_Summary, LLM_Agent_Token_Usage, and LLM_Model_Token_Usage. These events include relevant metadata such as agent name, model deployment, user ID, and conversation ID.
• Added similar _track_title_token_usage helpers to both history.py and history_sql.py to emit token usage telemetry for title generation requests. [1] [2]

Integration into API flows:

• Updated stream_openai_text and stream_openai_text_workshop in chat.py to accumulate token usage across all response iterations/streaming updates and emit telemetry at the end of each request. [1] [2] [3] [4] [5] [6] [7]
• Integrated token usage telemetry emission into the title generation flow in both Cosmos and SQL history modules.

Monitoring and Analytics Support

Application Insights KQL queries:

• Added a new token-usage-queries.kql file with a comprehensive set of KQL queries for monitoring LLM token usage, top consumers, cost estimation, usage breakdowns by agent/model/user, and visualizations (area and pie charts). These queries support detailed analysis and reporting in Application Insights.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
chat.py	436	130	70%	91–92, 142, 199, 205–215, 225–229, 231–239, 241, 243–244, 246, 253, 260, 263–264, 282–286, 288–289, 291–299, 365, 390–391, 394, 398, 403, 430–432, 439, 452–454, 458–459, 462–465, 467, 471, 501–502, 504–509, 511–514, 520, 524–532, 541–542, 563, 576, 587–595, 597–598, 600–603, 687–688, 690, 694, 697, 703–708, 713–716
history.py	576	64	88%	92, 135, 211, 223, 260–261, 263, 270–272, 292–293, 332–333, 346, 427, 431, 441, 469, 503–504, 588, 600–601, 603, 624–625, 627, 630, 633–634, 636–639, 641–642, 644, 653, 757, 763, 874, 878, 935, 940, 977, 981, 986, 990, 1181, 1184, 1196–1198, 1202–1206, 1208–1211, 1213
history_sql.py	581	65	88%	45–49, 51–53, 55–56, 61, 63–74, 153–154, 219, 243–254, 256–262, 264–265, 269, 304–306, 419–421, 538–539, 579–580, 616, 623, 812–813, 1204, 1210, 1231
token_usage.py	126	16	87%	102, 123, 145–154, 165, 196, 248–249
TOTAL	1855	290	84%

Tests	Skipped	Failures	Errors	Time
520	0 💤	0 ❌	0 🔥	8.440s ⏱️

[github-actions]

Unit Test Results

520 tests   520 ✅  8s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 6a5bfce.

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

Adds LLM token usage telemetry for chat and title-generation flows, plus KQL snippets for monitoring usage and cost in Application Insights.

Changes:

• Added token usage extraction and telemetry emission helpers in chat/history modules.
• Integrated token accumulation into standard and workshop chat streams.
• Added Application Insights KQL queries for usage summaries, breakdowns, and cost estimates.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
src/api/python/chat.py	Adds chat token extraction, aggregation, and telemetry emission.
src/api/python/history.py	Adds Cosmos history title-generation token telemetry.
src/api/python/history_sql.py	Adds SQL history title-generation token telemetry.
infra/dashboards/token-usage-queries.kql	Adds monitoring and analytics queries for emitted token events.

Comments suppressed due to low confidence (1)

src/api/python/chat.py:177

• Telemetry extraction can raise before the chat response is streamed when usage is present but its token fields are missing or non-numeric (for example, the existing tests' mock Responses objects do not define usage fields). Because this helper is called directly from the request path, a telemetry-shape issue turns an otherwise successful chat response into a 500 instead of simply skipping token telemetry; validate/coerce defensively or catch extraction errors at the call sites.

    inp = getattr(usage_obj, "input_tokens", 0) or getattr(usage_obj, "prompt_tokens", 0) or 0
    out = getattr(usage_obj, "output_tokens", 0) or getattr(usage_obj, "completion_tokens", 0) or 0
    tot = getattr(usage_obj, "total_tokens", 0) or (inp + out)
    if tot > 0:
        return (int(inp), int(out), int(tot))

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.