Skip to content

Commit 18e34e2

Browse files
authored
doc update for kv aware routing (vllm-project#411)
Signed-off-by: Siddhant Ray <siddhant.r98@gmail.com>
1 parent 7aa131d commit 18e34e2

2 files changed

Lines changed: 12 additions & 2 deletions

File tree

docs/source/dev_guide/dev_api/router-logic.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,7 @@ Router Logic
1010
.. autoclass:: vllm_router.routers.routing_logic.RoundRobinRouter
1111
:members:
1212
:show-inheritance:
13+
14+
.. autoclass:: vllm_router.routers.routing_logic.KvawareRouter
15+
:members:
16+
:show-inheritance:

src/vllm_router/routers/routing_logic.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,11 @@ def route_request(
202202

203203

204204
class KvawareRouter(RoutingInterface):
205+
"""
206+
Route the request to the appropriate engine URL by where the KV cache
207+
of the longest prefix match is found.
208+
"""
209+
205210
def __init__(self, lmcache_controller_port: int):
206211
self.lmcache_controller_port = lmcache_controller_port
207212
self.kv_manager = controller_manager.LMCacheControllerManager(
@@ -239,12 +244,13 @@ async def route_request(
239244
of the longest prefix match is found.
240245
If there is no session id in the request header, it will pick a server
241246
with round robin.
247+
242248
Args:
243249
endpoints (List[EndpointInfo]): The list of engine URLs
244250
engine_stats (Dict[str, EngineStats]): The engine stats indicating
245-
the 'physical' load of each engine
251+
the 'physical' load of each engine
246252
request_stats (Dict[str, RequestStats]): The request stats
247-
indicating the request-level performance of each engine
253+
indicating the request-level performance of each engine
248254
request (Request): The incoming request
249255
request_json (Dict): The request body (needed for finding the
250256
longest prefix match)

0 commit comments

Comments
 (0)