Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 49 additions & 22 deletions docs/source/use_cases/autoscaling-keda.rst
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
Autoscaling with KEDA
=====================

This tutorial shows you how to automatically scale a vLLM deployment using `KEDA <https://keda.sh/>`_ and Prometheus-based metrics. With the vLLM Production Stack Helm chart (v0.1.9+), KEDA autoscaling is integrated directly into the chart, allowing you to enable it through simple ``values.yaml`` configuration.
This tutorial shows you how to automatically scale a vLLM deployment using `KEDA <https://keda.sh/>`_ and Prometheus-based metrics. With the vLLM Production Stack Helm chart (v0.1.11+), KEDA autoscaling and monitoring are integrated directly into the chart, allowing you to enable everything through simple ``values.yaml`` configuration.

Table of Contents
-----------------

- Prerequisites_
- Steps_

- `1. Deploy the Observability Stack`_
- `1. Enable the Integrated Monitoring Stack`_
- `2. Configure and Deploy vLLM`_
- `3. Install KEDA`_
- `4. Enable KEDA Autoscaling for vLLM`_
Expand All @@ -26,41 +26,50 @@ Prerequisites
- Access to a Kubernetes cluster with at least 2 GPUs
- ``kubectl`` and ``helm`` installed (v3.0+)
- Basic understanding of Kubernetes and Prometheus metrics
- vLLM Production Stack Helm chart v0.1.11+

Steps
-----

1. Deploy the Observability Stack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. Enable the Integrated Monitoring Stack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The observability stack (Prometheus, Grafana) is required for KEDA to query metrics.
Since v0.1.11, the vLLM Production Stack Helm chart includes ``kube-prometheus-stack`` and ``prometheus-adapter`` as optional sub-chart dependencies. There is no longer a need to manually install a separate observability stack — simply enable it in your ``values.yaml``:

.. code-block:: bash
.. code-block:: yaml

cd observability
bash install.sh
# Enable ServiceMonitors so Prometheus can scrape vLLM metrics
servingEngineSpec:
serviceMonitor:
enabled: true

Verify Prometheus is scraping the queue length metric ``vllm:num_requests_waiting``:
routerSpec:
serviceMonitor:
enabled: true

.. code-block:: bash
# Deploy the full kube-prometheus-stack (Prometheus + Grafana) as a sub-chart
kube-prometheus-stack:
enabled: true

kubectl port-forward svc/prometheus-operated -n monitoring 9090:9090
.. note::

In a separate terminal:
If you already have an existing Prometheus stack in your cluster (e.g., deployed via ``kube-prometheus-stack`` or VictoriaMetrics), you only need to enable the ``serviceMonitor`` resources. Set ``kube-prometheus-stack.enabled: false`` and your existing Prometheus Operator will automatically discover the ServiceMonitors.

.. code-block:: bash
.. note::

curl -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=vllm:num_requests_waiting'
``prometheus-adapter`` is **not** required for KEDA autoscaling. KEDA queries Prometheus directly via its built-in Prometheus scaler. You only need ``prometheus-adapter`` if you want to use the Kubernetes Custom Metrics API with a standard HPA.

2. Configure and Deploy vLLM
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Create a ``values.yaml`` file to deploy vLLM. Note that we'll enable KEDA autoscaling in a later step after KEDA is installed:
Create a ``values.yaml`` file to deploy vLLM with monitoring enabled. Note that we'll enable KEDA autoscaling in a later step after KEDA is installed:

.. code-block:: yaml

servingEngineSpec:
enableEngine: true
serviceMonitor:
enabled: true
modelSpec:
- name: "llama3"
repository: "lmcache/vllm-openai"
Expand All @@ -71,6 +80,13 @@ Create a ``values.yaml`` file to deploy vLLM. Note that we'll enable KEDA autosc
requestMemory: "64Gi"
requestGPU: 1

routerSpec:
serviceMonitor:
enabled: true

kube-prometheus-stack:
enabled: true

Deploy the chart:

.. code-block:: bash
Expand All @@ -87,7 +103,7 @@ Verify Prometheus is scraping the vLLM metrics:

.. code-block:: bash

kubectl port-forward svc/prometheus-operated -n monitoring 9090:9090
kubectl port-forward svc/vllm-kube-prometheus-stack-prometheus 9090:9090

In a separate terminal:

Expand Down Expand Up @@ -142,11 +158,22 @@ Update your ``values.yaml`` file to enable KEDA autoscaling:
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus-operated.monitoring.svc:9090
serverAddress: http://vllm-kube-prometheus-stack-prometheus:9090
metricName: vllm:num_requests_waiting
query: vllm:num_requests_waiting
threshold: '5'

routerSpec:
serviceMonitor:
enabled: true

kube-prometheus-stack:
enabled: true

.. note::

The ``serverAddress`` in the KEDA trigger must point to your Prometheus instance. When using the integrated sub-chart, the service name follows the pattern ``<release-name>-kube-prometheus-stack-prometheus`` (here: ``vllm-kube-prometheus-stack-prometheus``). If you use an external Prometheus, adjust the address accordingly.

Upgrade the chart to enable KEDA autoscaling:

.. code-block:: bash
Expand Down Expand Up @@ -231,14 +258,14 @@ Enable scale-to-zero by setting ``minReplicaCount: 0`` and adding a traffic-base
# Queue-based scaling
- type: prometheus
metadata:
serverAddress: http://prometheus-operated.monitoring.svc:9090
serverAddress: http://vllm-kube-prometheus-stack-prometheus:9090
metricName: vllm:num_requests_waiting
query: vllm:num_requests_waiting
threshold: '5'
# Traffic-based keepalive (prevents scale-to-zero when traffic exists)
- type: prometheus
metadata:
serverAddress: http://prometheus-operated.monitoring.svc:9090
serverAddress: http://vllm-kube-prometheus-stack-prometheus:9090
metricName: vllm:incoming_keepalive
query: sum(rate(vllm:num_incoming_requests_total[1m]) > bool 0)
threshold: "1"
Expand Down Expand Up @@ -295,16 +322,16 @@ To completely remove KEDA from the cluster:
helm uninstall keda -n keda
kubectl delete namespace keda

To remove the observability stack:
To disable the integrated monitoring, set ``kube-prometheus-stack.enabled: false`` in your ``values.yaml`` and upgrade:

.. code-block:: bash

cd observability
bash uninstall.sh
helm upgrade vllm vllm/vllm-stack -f values.yaml

Additional Resources
--------------------

- `KEDA Documentation <https://keda.sh/docs/>`_
- `KEDA ScaledObject Specification <https://keda.sh/docs/2.18/reference/scaledobject-spec/>`_
- `Helm Chart KEDA Configuration <https://github.com/vllm-project/production-stack/blob/main/helm/README.md#keda-autoscaling-configuration>`_
- `Monitoring Sub-Chart Integration (PR #860) <https://github.com/vllm-project/production-stack/pull/860>`_
Loading