Add usage.prompt_tokens_details.cached_tokens for prefix caching

[lvhan028]

Motivation

OpenAI reports prefix cache hits in usage.prompt_tokens_details.cached_tokens, which clients use for cost estimation and observability (cache_hit_rate = cached_tokens / prompt_tokens).

LMDeploy's supports prefix caching, but this hit count was not surfaced through the serving API. Without it, users cannot measure prefix cache effectiveness from chat completion responses or benchmark results.

Modification

PyTorch engine

• Record prefix cache hits in BlockTrie.match() as SchedulerSequence.cached_tokens.
• Propagate the value through RequestMetrics → GenOut / Response → OpenAI usage.
• Fix a scheduler bug in the migration path: block_trie.match(migration_waiting) → block_trie.match(seq).

Turbomind engine

TBD. After @lzhangzz refactor prefix caching

OpenAI API (/v1/chat/completions)

• Add PromptTokensDetails and UsageInfo.build() to always populate usage.prompt_tokens_details.cached_tokens (0 when prefix caching is disabled).
• Wire cached token counts into both streaming (stream_options.include_usage) and non-streaming responses.
• Not added to /v1/completions (deprecated by OpenAI).

Other endpoints(v1/messages, v1/response)

TBD

Metrics

• Add Prometheus metrics:
  • lmdeploy:request_cached_tokens
  • lmdeploy:request_cache_hit_ratio
  • lmdeploy:cached_tokens_total

Benchmark

• Parse cached_tokens from usage in benchmark/benchmark_chat_completion.py.
• Add per-request cached_tokens and aggregate total_cached_tokens / cache_hit_rate to summary output.
• Forward optional tools / tool_choice fields from JSONL inputs.

[Copilot]

Pull request overview

This PR adds end-to-end reporting for prefix-cache hit tokens (“cached tokens”) so that OpenAI-compatible responses can expose usage.prompt_tokens_details.cached_tokens, and internal metrics/benchmarks can track prefix caching effectiveness.

Changes:

• Add PromptTokensDetails(cached_tokens) to the OpenAI protocol and centralize usage construction via build_usage_info(...).
• Plumb cached_tokens from prefix-cache matching (BlockTrie.match) through PyTorch engine messaging/metrics to server responses (GenOut.cached_tokens).
• Add Prometheus metrics + benchmark support for cached token counts / cache hit ratio, and add unit tests for the new behavior.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
lmdeploy/serve/openai/protocol.py	Adds PromptTokensDetails and build_usage_info to include cached token details in UsageInfo.
lmdeploy/serve/openai/api_server.py	Switches usage construction to build_usage_info and forwards res.cached_tokens into OpenAI responses.
lmdeploy/serve/core/async_engine.py	Adds cached_tokens to GenOut and propagates it from engine outputs into responses.
lmdeploy/pytorch/paging/scheduler.py	Fixes migration scheduling to call block_trie.match(seq) correctly.
lmdeploy/pytorch/paging/block_trie.py	Records seq.prefix_cache_hit_tokens (and resets to 0 when prefix caching is disabled).
lmdeploy/pytorch/messages.py	Adds prefix_cache_hit_tokens to SchedulerSequence for prefix caching accounting.
lmdeploy/pytorch/engine/engine_loop.py	Emits cached_tokens in RequestMetrics based on msg.prefix_cache_hit_tokens.
lmdeploy/pytorch/engine/engine_instance.py	Extracts cached_tokens from req_metrics and forwards it via EngineOutput.
lmdeploy/messages.py	Adds cached_tokens fields to Response, RequestMetrics, and EngineOutput dataclasses.
lmdeploy/metrics/stats.py	Adds cached_tokens to per-request stats.
lmdeploy/metrics/metrics_processor.py	Copies outputs.cached_tokens into RequestStats.cached_tokens.
lmdeploy/metrics/loggers.py	Adds Prometheus histograms/counter for cached tokens and cache-hit ratio.
benchmark/benchmark_chat_completion.py	Forwards per-row tools/tool_choice and extracts cached_tokens from streamed usage payloads for reporting.
tests/test_lmdeploy/test_prefix_cache_hit_tokens.py	Adds a unit test ensuring disabled prefix caching sets hit tokens to 0.
tests/test_lmdeploy/serve/openai/test_usage_info.py	Adds unit tests verifying prompt_tokens_details.cached_tokens appears in built usage and dumps correctly.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.