Skip to content

[Bugfix][Helm]: update Grafana dashboards to current vLLM metric names#989

Open
bitnik wants to merge 1 commit into
vllm-project:mainfrom
bitnik:deprecated-metrics
Open

[Bugfix][Helm]: update Grafana dashboards to current vLLM metric names#989
bitnik wants to merge 1 commit into
vllm-project:mainfrom
bitnik:deprecated-metrics

Conversation

@bitnik

@bitnik bitnik commented Jul 1, 2026

Copy link
Copy Markdown
  • Make sure the code changes pass the pre-commit checks.
  • Sign-off your commit by using -s when doing git commit
  • Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

The bundled Grafana dashboards (helm/dashboards/vllm-dashboard.json and helm/dashboards/vllm-model-metrics-dashboard.json) referenced vLLM metrics that have since been renamed or removed. As a result, several panels showed No data on current vLLM versions.

This PR updates the dashboard queries to the current metric names and drops panels for metrics that no longer exist.

Renamed metrics:

Old metric New metric
vllm:time_per_output_token_seconds_{sum,count,bucket} vllm:request_time_per_output_token_seconds_{sum,count,bucket}
vllm:gpu_cache_usage_perc vllm:kv_cache_usage_perc

Removed panels / queries (metrics no longer emitted — vLLM V1 does not swap requests between GPU and CPU):

  • vllm-dashboard.json: removed the "Number of Swapped Requests" stat panel (vllm:num_requests_swapped).
  • vllm-model-metrics-dashboard.json: removed the vllm:num_requests_swapped query and the "CPU Cache Usage" query (vllm:cpu_cache_usage_perc) from the Cache Utilization panel.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the vLLM Grafana dashboards to use current metric names, replacing time_per_output_token with request_time_per_output_token and gpu_cache_usage_perc with kv_cache_usage_perc, while removing panels and queries related to swapped requests and CPU cache usage. The reviewer feedback highlights several key improvement opportunities: adding the percentunit formatting to panels displaying the new KV cache usage metric, updating deprecated prefix cache metrics, adjusting panel widths to fill empty layout spaces, fixing non-sequential query refIds, and removing outdated references to swapped requests in panel descriptions.

Comment thread helm/dashboards/vllm-dashboard.json
Comment thread helm/dashboards/vllm-dashboard.json
"title": "GPU KV Cache Hit Rate",
"type": "timeseries"
},
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There are two improvement opportunities here:

  1. Deprecated Metrics: The preceding panel "GPU KV Cache Hit Rate" (lines 800-836) uses the deprecated metrics vllm:gpu_prefix_cache_hits_total and vllm:gpu_prefix_cache_queries_total (at line 825). Since this PR updates the dashboard queries to current vLLM metric names, these should be updated to the current non-deprecated metric names: vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total respectively.
  2. Panel Layout: Since the "Number of Swapped Requests" panel (which was positioned at x: 12, y: 39 with width 12) has been removed, the "GPU KV Cache Hit Rate" panel (positioned at x: 0, y: 39 with width 12) now has an empty space of width 12 next to it. Consider increasing its width (w) to 24 (at line 802) to utilize the full width of the dashboard.

Comment thread helm/dashboards/vllm-model-metrics-dashboard.json
vLLM renamed several metrics and dropped the swap-related ones
(V1 no longer swaps between GPU and CPU),
so the shipped dashboards queried metrics that no longer exist and
rendered empty panels.
Update both dashboards accordingly:
- vllm:time_per_output_token_seconds_* -> vllm:request_time_per_output_token_seconds_*
- vllm:gpu_cache_usage_perc -> vllm:kv_cache_usage_perc
- remove the "Number of Swapped Requests" panel and
  the vllm:num_requests_swapped / vllm:cpu_cache_usage_perc queries

Signed-off-by: Kenan Erdogan <kenanerdogan@gmail.com>
@bitnik bitnik force-pushed the deprecated-metrics branch from ac12b30 to 9392413 Compare July 1, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant