Add per-model vLLM observability dashboard#948
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new Grafana dashboard for monitoring vLLM Inference Server metrics, such as request latency, token throughput, and cache utilization. The review feedback focuses on enhancing the dashboard's portability and accuracy by recommending the removal of hardcoded datasource UIDs and model names, setting the dashboard ID to null for better provisioning, and refining PromQL expressions to display averages per request. Additionally, improvements to the user interface were suggested, including better legend formatting and the addition of a default refresh interval.
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "expr": "rate(vllm:e2e_request_latency_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])\n/\nrate(vllm:e2e_request_latency_seconds_count{model_name=\"$model_name\"}[$__rate_interval])", |
There was a problem hiding this comment.
The average latency calculation should aggregate across all pods for the selected model to be consistent with the percentile calculations (P99, P95, etc.) in the same panel. Using sum ensures that the average is correctly calculated for the entire model deployment.
| "expr": "rate(vllm:e2e_request_latency_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])\n/\nrate(vllm:e2e_request_latency_seconds_count{model_name=\"$model_name\"}[$__rate_interval])", | |
| "expr": "sum(rate(vllm:e2e_request_latency_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])) / sum(rate(vllm:e2e_request_latency_seconds_count{model_name=\"$model_name\"}[$__rate_interval]))", |
There was a problem hiding this comment.
In high-performance GPU serving, the per-pod granularity is a feature, not a bug. If we aggregate everything into a single average, we lose the ability to see if one specific GPU is throttling or if a single pod is stuck in a 'zombie' state. I'm keeping the per-pod visualization to allow for outlier detection.
| }, | ||
| "disableTextWrap": false, | ||
| "editorMode": "code", | ||
| "expr": "rate(vllm:request_queue_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])", |
There was a problem hiding this comment.
The "Queue Time" panel currently shows the aggregate rate of time spent in the queue. To display the average queue time per request, the sum should be divided by the count. Additionally, aggregating across pods using sum is recommended for a per-model view.
| "expr": "rate(vllm:request_queue_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])", | |
| "expr": "sum(rate(vllm:request_queue_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])) / sum(rate(vllm:request_queue_time_seconds_count{model_name=\"$model_name\"}[$__rate_interval]))", |
There was a problem hiding this comment.
The current per-pod visualization is intentional for vLLM performance debugging. In production LLM serving, we need to identify 'unhealthy' or 'slow' pods (outliers). Aggregating into a single model average hides the variance that is critical for identifying GPU throttling or networking issues on specific Bare-Metal nodes.
| }, | ||
| "disableTextWrap": false, | ||
| "editorMode": "code", | ||
| "expr": "rate(vllm:request_prefill_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])", |
There was a problem hiding this comment.
The prefill time should be calculated as an average per request by dividing the sum rate by the count rate, aggregated across the model deployment.
| "expr": "rate(vllm:request_prefill_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])", | |
| "expr": "sum(rate(vllm:request_prefill_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])) / sum(rate(vllm:request_prefill_time_seconds_count{model_name=\"$model_name\"}[$__rate_interval]))", |
| "uid": "${DS_PROMETHEUS}" | ||
| }, | ||
| "editorMode": "code", | ||
| "expr": "rate(vllm:request_decode_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])", |
There was a problem hiding this comment.
The decode time should be calculated as an average per request by dividing the sum rate by the count rate, aggregated across the model deployment.
| "expr": "rate(vllm:request_decode_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])", | |
| "expr": "sum(rate(vllm:request_decode_time_seconds_sum{model_name=\"$model_name\"}[$__rate_interval])) / sum(rate(vllm:request_decode_time_seconds_count{model_name=\"$model_name\"}[$__rate_interval]))", |
| }, | ||
| "disableTextWrap": false, | ||
| "editorMode": "code", | ||
| "expr": "rate(vllm:request_max_num_generation_tokens_sum{model_name=\"$model_name\"}[$__rate_interval])", |
There was a problem hiding this comment.
The "Max Generation Token" panel should display the average number of tokens by dividing the sum rate by the count rate.
| "expr": "rate(vllm:request_max_num_generation_tokens_sum{model_name=\"$model_name\"}[$__rate_interval])", | |
| "expr": "sum(rate(vllm:request_max_num_generation_tokens_sum{model_name=\"$model_name\"}[$__rate_interval])) / sum(rate(vllm:request_max_num_generation_tokens_count{model_name=\"$model_name\"}[$__rate_interval]))", |
Adds granular per-model metrics dashboard for vLLM monitoring. Provides detailed observability at the model level. Tested on CoreWeave with production workloads. Addresses: vllm-project#828 Signed-off-by: Kosseila (CloudThrill) <klouddude@gmail.com>
2 participants
Adds granular per-model metrics dashboard for vLLM monitoring. Provides detailed observability at the model level. Tested on CoreWeave with production workloads.
Addresses this vllm dashboard feature request : #828
