You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+13-13Lines changed: 13 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,10 +4,10 @@
4
4
5
5
## Latest News
6
6
7
-
- 📄 Official documentation released for production-stack! [Link](https://docs.vllm.ai/projects/production-stack)
8
-
- ✨ Cloud Deployment Tutorials for Lambda Labs, AWS EKS, Google GCP are out! [Link](https://github.com/vllm-project/production-stack/blob/main/tutorials)
- 🔥 vLLM Production Stack is released! Checkout our [release blogs](https://blog.lmcache.ai/2025-01-21-stack-release)[01-22-2025]
7
+
- 📄 [Official documentation](https://docs.vllm.ai/projects/production-stack) released for production-stack!
8
+
- ✨ [Cloud Deployment Tutorials](https://github.com/vllm-project/production-stack/blob/main/tutorials) for Lambda Labs, AWS EKS, Google GCP are out!
9
+
- 🛤️ 2025 Q1 roadmap is released! [Join the discussion now](https://github.com/vllm-project/production-stack/issues/26)!
10
+
- 🔥 vLLM Production Stack is released! Check out our [release blogs](https://blog.lmcache.ai/2025-01-21-stack-release)posted on January 22, 2025.
11
11
12
12
## Community Events
13
13
@@ -21,15 +21,15 @@ Meetings alternate weekly between the two times. All are welcome to join!
21
21
22
22
**vLLM Production Stack** project provides a reference implementation on how to build an inference stack on top of vLLM, which allows you to:
23
23
24
-
- 🚀 Scale from single vLLM instance to distributed vLLM deployment without changing any application code
25
-
- 💻 Monitor the through a web dashboard
24
+
- 🚀 Scale from a single vLLM instance to a distributed vLLM deployment without changing any application code
25
+
- 💻 Monitor the metrics through a web dashboard
26
26
- 😄 Enjoy the performance benefits brought by request routing and KV cache offloading
27
27
28
28
## Step-By-Step Tutorials
29
29
30
30
0. How To [*Install Kubernetes (kubectl, helm, minikube, etc)*](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md)?
31
31
1. How to [*Deploy Production Stack on Major Cloud Platforms (AWS, GCP, Lambda Labs, Azure)*](https://github.com/vllm-project/production-stack/blob/main/tutorials/cloud_deployments)?
32
-
2. How To [*Setup a Minimal vLLM Production Stack*](https://github.com/vllm-project/production-stack/blob/main/tutorials/01-minimal-helm-installation.md)?
32
+
2. How To [*Set up a Minimal vLLM Production Stack*](https://github.com/vllm-project/production-stack/blob/main/tutorials/01-minimal-helm-installation.md)?
33
33
3. How To [*Customize vLLM Configs (optional)*](https://github.com/vllm-project/production-stack/blob/main/tutorials/02-basic-vllm-config.md)?
34
34
4. How to [*Load Your LLM Weights*](https://github.com/vllm-project/production-stack/blob/main/tutorials/03-load-model-from-pv.md)?
35
35
5. How to [*Launch Different LLMs in vLLM Production Stack*](https://github.com/vllm-project/production-stack/blob/main/tutorials/04-launch-multiple-model.md)?
@@ -39,7 +39,7 @@ Meetings alternate weekly between the two times. All are welcome to join!
39
39
40
40
The stack is set up using [Helm](https://helm.sh/docs/), and contains the following key parts:
41
41
42
-
-**Serving engine**: The vLLM engines that run different LLMs
42
+
-**Serving engine**: The vLLM engines that run different LLMs.
43
43
-**Request router**: Directs requests to appropriate backends based on routing keys or session IDs to maximize KV cache reuse.
44
44
-**Observability stack**: monitors the metrics of the backends through [Prometheus](https://github.com/prometheus/prometheus) + [Grafana](https://grafana.com/)
The deployed stack provides the same [**OpenAI API interface**](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html?ref=blog.mozilla.ai#openai-compatible-server) as vLLM, and can be accessed through kubernetes service.
78
78
79
-
To validate the installation and and send query to the stack, refer to [this tutorial](tutorials/01-minimal-helm-installation.md).
79
+
To validate the installation and send a query to the stack, refer to [this tutorial](tutorials/01-minimal-helm-installation.md).
80
80
81
81
For more information about customizing the helm chart, please refer to [values.yaml](https://github.com/vllm-project/production-stack/blob/main/helm/values.yaml) and our other [tutorials](https://github.com/vllm-project/production-stack/tree/main/tutorials).
82
82
@@ -114,18 +114,18 @@ The router ensures efficient request distribution among backends. It supports:
114
114
115
115
- Routing to endpoints that run different models
116
116
- Exporting observability metrics for each serving engine instance, including QPS, time-to-first-token (TTFT), number of pending/running/finished requests, and uptime
117
-
- Automatic service discovery and fault tolerance by Kubernetes API
117
+
- Automatic service discovery and fault tolerance via the Kubernetes API
118
118
- Model aliases
119
-
- Multiple different routing algorithms
119
+
- Multiple routing algorithms:
120
120
- Round-robin routing
121
121
- Session-ID based routing
122
-
-(WIP) prefix-aware routing
122
+
-Prefix-aware routing (WIP)
123
123
124
124
Please refer to the [router documentation](./src/vllm_router/README.md) for more details.
125
125
126
126
## Contributing
127
127
128
-
We welcome and value any contributions and collaborations. Please check out [CONTRIBUTING.md](CONTRIBUTING.md) for how to get involved.
128
+
We welcome and value any contributions and collaborations. Please check out [CONTRIBUTING.md](CONTRIBUTING.md) for how to get involved.
0 commit comments