Skip to content
This repository was archived by the owner on Oct 15, 2025. It is now read-only.

Commit dc1bfa4

Browse files
Gregory-PereiraGregory-Pereira
authored
using new NIXL only connector (#32)
* using new NIXL only connector Signed-off-by: greg pereira <grpereir@redhat.com> * runs but no cache hit Signed-off-by: greg pereira <grpereir@redhat.com> * no p/d services in prod example Signed-off-by: greg pereira <grpereir@redhat.com> * restore pd services deemed non-invasive Signed-off-by: greg pereira <grpereir@redhat.com> * keeping confimaps around but not using them in lmcache for dual connectors later Signed-off-by: greg pereira <grpereir@redhat.com> * downgrade to working image Signed-off-by: greg pereira <grpereir@redhat.com> * removing dead code placeholder sections Signed-off-by: greg pereira <grpereir@redhat.com> * linting Signed-off-by: greg pereira <grpereir@redhat.com> --------- Signed-off-by: greg pereira <grpereir@redhat.com>
1 parent 4ef6ab9 commit dc1bfa4

5 files changed

Lines changed: 245 additions & 99 deletions

File tree

‎charts/llm-d/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
apiVersion: v2
22
name: llm-d
33
type: application
4-
version: 0.7.0
4+
version: 0.7.1
55
appVersion: "0.0.1"
66
icon: data:null
77
description: A Helm chart for llm-d

‎charts/llm-d/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11

22
# llm-d Helm Chart for OpenShift
33

4-
![Version: 0.7.0](https://img.shields.io/badge/Version-0.7.0-informational?style=flat-square)
4+
![Version: 0.7.1](https://img.shields.io/badge/Version-0.7.1-informational?style=flat-square)
55
![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
66

77
A Helm chart for llm-d
@@ -174,7 +174,7 @@ Kubernetes: `>= 1.25.0-0`
174174
| modelservice.epp | Endpoint picker configuration | object | See below |
175175
| modelservice.epp.defaultEnvVars | Default environment variables for endpoint picker, use `extraEnvVars` to override default behavior by defining the same variable again. Ref: https://github.com/neuralmagic/gateway-api-inference-extension/tree/dev?tab=readme-ov-file#temporary-fork-configuration | list | `[{"name":"ENABLE_KVCACHE_AWARE_SCORER","value":"{{ .Values.redis.enabled }}"},{"name":"KVCACHE_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"KVCACHE_INDEXER_REDIS_ADDR","value":"{{ if .Values.redis.enabled }}{{ include \"redis.master.service.fullurl\" . }}{{ end }}"},{"name":"ENABLE_PREFIX_AWARE_SCORER","value":"true"},{"name":"PREFIX_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"ENABLE_LOAD_AWARE_SCORER","value":"true"},{"name":"LOAD_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"ENABLE_SESSION_AWARE_SCORER","value":"true"},{"name":"SESSION_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"PD_ENABLED","value":"true"},{"name":"PD_PROMPT_LEN_THRESHOLD","value":"10"},{"name":"PREFILL_ENABLE_KVCACHE_AWARE_SCORER","value":"true"},{"name":"PREFILL_KVCACHE_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"PREFILL_ENABLE_LOAD_AWARE_SCORER","value":"true"},{"name":"PREFILL_LOAD_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"PREFILL_ENABLE_PREFIX_AWARE_SCORER","value":"true"},{"name":"PREFILL_PREFIX_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"DECODE_ENABLE_KVCACHE_AWARE_SCORER","value":"true"},{"name":"DECODE_KVCACHE_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"DECODE_ENABLE_LOAD_AWARE_SCORER","value":"true"},{"name":"DECODE_LOAD_AWARE_SCORER_WEIGHT","value":"1.0"},{"name":"DECODE_ENABLE_PREFIX_AWARE_SCORER","value":"true"},{"name":"DECODE_PREFIX_AWARE_SCORER_WEIGHT","value":"1.0"}]` |
176176
| modelservice.epp.extraEnvVars | Additional environment variables for endpoint picker | list | `[]` |
177-
| modelservice.epp.image | Endpoint picker image used in ModelService CR presets | object | `{"imagePullPolicy":"Always","registry":"quay.io","repository":"llm-d/llm-d-inference-scheduler","tag":"0.0.1"}` |
177+
| modelservice.epp.image | Endpoint picker image used in ModelService CR presets | object | `{"imagePullPolicy":"Always","registry":"quay.io","repository":"llm-d/llm-d-gateway-api-inference-extension-dev","tag":"0.0.5"}` |
178178
| modelservice.epp.metrics.enabled | Enable metrics scraping from endpoint picker service, see `modelservice.serviceMonitor` for configuration | bool | `true` |
179179
| modelservice.fullnameOverride | String to fully override modelservice.fullname | string | `""` |
180180
| modelservice.image | Modelservice controller image, please change only if appropriate adjustments to the CRD are being made | object | `{"imagePullPolicy":"Always","registry":"quay.io","repository":"llm-d/llm-d-model-service","tag":"0.0.8"}` |
@@ -207,7 +207,7 @@ Kubernetes: `>= 1.25.0-0`
207207
| modelservice.serviceMonitor.port | ServiceMonitor endpoint port | string | `"vllm"` |
208208
| modelservice.serviceMonitor.selector | ServiceMonitor selector matchLabels </br> matchLabels must match labels on modelservice Services | object | `{"matchLabels":{}}` |
209209
| modelservice.vllm | vLLM container options | object | See below |
210-
| modelservice.vllm.image | vLLM image used in ModelService CR presets | object | `{"imagePullPolicy":"IfNotPresent","registry":"quay.io","repository":"llm-d/llm-d-dev","tag":"0.0.5"}` |
210+
| modelservice.vllm.image | vLLM image used in ModelService CR presets | object | `{"imagePullPolicy":"Always","registry":"quay.io","repository":"llm-d/llm-d-dev","tag":"vllm-nixl-0.0.6"}` |
211211
| modelservice.vllm.metrics.enabled | Enable metrics scraping from vllm service, see `modelservice.serviceMonitor` for configuration | bool | `true` |
212212
| modelservice.vllmSim | vLL sim container options | object | See below |
213213
| modelservice.vllmSim.image | vLLM sim image used in ModelService CR presets | object | `{"imagePullPolicy":"IfNotPresent","registry":"quay.io","repository":"llm-d/vllm-sim","tag":"0.0.4"}` |

‎charts/llm-d/templates/modelservice/presets/basic-gpu-with-nixl-preset.yaml‎

Lines changed: 42 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -16,42 +16,6 @@ metadata:
1616
{{- include "common.tplvalues.render" ( dict "value" .Values.modelservice.annotations "context" $) | nindent 4 }}
1717
{{- end }}
1818
data:
19-
configMaps: |
20-
- apiVersion: v1
21-
kind: ConfigMap
22-
metadata:
23-
name: {{ include "modelservice.fullname" . -}}-config-decoder
24-
data:
25-
lmcache-decoder-config.yaml: |
26-
local_cpu: False
27-
max_local_cpu_size: 0
28-
max_local_disk_size: 0
29-
remote_serde: NULL
30-
enable_nixl: True
31-
nixl_role: receiver
32-
nixl_peer_host: 0.0.0.0
33-
nixl_peer_port: 55555
34-
nixl_buffer_size: 524288
35-
nixl_buffer_device: "cuda"
36-
nixl_enable_gc: True
37-
- apiVersion: v1
38-
kind: ConfigMap
39-
metadata:
40-
name: {{ include "modelservice.fullname" . -}}-config-prefiller
41-
data:
42-
lmcache-prefiller-config.yaml: |
43-
local_cpu: False
44-
max_local_cpu_size: 0
45-
max_local_disk_size: 0
46-
remote_serde: NULL
47-
enable_nixl: True
48-
nixl_role: "sender"
49-
nixl_peer_host: {{`"{{ .DecodeServiceName }}"`}}
50-
nixl_peer_port: 55555
51-
nixl_buffer_size: 524288
52-
nixl_buffer_device: "cuda"
53-
nixl_enable_gc: True
54-
5519
decodeDeployment: |
5620
apiVersion: apps/v1
5721
kind: Deployment
@@ -62,54 +26,66 @@ data:
6226
tolerations:
6327
{{- toYaml .Values.modelservice.decode.tolerations | nindent 12 }}
6428
{{- end }}
65-
containers:
29+
initContainers:
6630
- name: routing-proxy
6731
image: {{ include "modelservice.routingProxyImage" . }}
6832
securityContext:
69-
allowPrivilegeEscalation: false
33+
capabilities:
34+
drop:
35+
- MKNOD
7036
runAsNonRoot: true
37+
allowPrivilegeEscalation: false
7138
args:
72-
- "--port=8001"
73-
- "--vllm-port=8000"
39+
- "--port=8000"
40+
- "--vllm-port=8001"
41+
- "--connector=nixl"
7442
ports:
7543
- containerPort: 8000
7644
protocol: TCP
45+
restartPolicy: Always
46+
imagePullPolicy: Always
47+
containers:
7748
- name: vllm
7849
image: {{ include "modelservice.vllmImage" . }}
7950
imagePullPolicy: {{ .Values.modelservice.vllm.image.imagePullPolicy }}
8051
securityContext:
52+
capabilities:
53+
drop:
54+
- MKNOD
55+
runAsNonRoot: true
8156
allowPrivilegeEscalation: false
8257
command:
8358
- vllm
8459
- serve
8560
- {{ `{{ default (print "/models/" .ModelPath) .HFModelName }}` }}
8661
args:
8762
- "--port"
88-
- "8000"
63+
- "8001"
64+
- "--enforce-eager"
8965
- "--kv-transfer-config"
90-
- '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_consumer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "consumer1"}}'
66+
- '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
9167
env:
9268
- name: HOME
9369
value: /home
70+
- name: VLLM_NIXL_SIDE_CHANNEL_PORT
71+
value: "5557"
72+
- name: VLLM_NIXL_SIDE_CHANNEL_HOST
73+
valueFrom:
74+
fieldRef:
75+
fieldPath: status.podIP
76+
- name: VLLM_LOGGING_LEVEL
77+
value: DEBUG
9478
- name: POD_IP
9579
valueFrom:
9680
fieldRef:
9781
apiVersion: v1
9882
fieldPath: status.podIP
9983
- name: LMCACHE_DISTRIBUTED_URL
100-
value: ${POD_IP}:80
84+
value: ${POD_IP}:8200
10185
- name: CUDA_VISIBLE_DEVICES
10286
value: "0"
10387
- name: UCX_TLS
10488
value: "cuda_ipc,cuda_copy,tcp"
105-
- name: LMCACHE_USE_EXPERIMENTAL
106-
value: "True"
107-
- name: VLLM_ENABLE_V1_MULTIPROCESSING
108-
value: "1"
109-
- name: VLLM_WORKER_MULTIPROC_METHOD
110-
value: spawn
111-
- name: LMCACHE_CONFIG_FILE
112-
value: /vllm-workspace/lmcache-decoder-config.yaml
11389
{{- if .Values.redis.enabled }}
11490
- name: LMCACHE_LOOKUP_URL
11591
value: {{ include "redis.master.service.fullurl" .}}
@@ -121,8 +97,6 @@ data:
12197
volumeMounts:
12298
- name: home
12399
mountPath: /home
124-
- name: config-decoder
125-
mountPath: /vllm-workspace
126100
{{ `{{- if .HFModelName }}` }}
127101
- name: model-cache
128102
mountPath: /models
@@ -132,16 +106,11 @@ data:
132106
readOnly: true
133107
{{ `{{- end }}` }}
134108
ports:
135-
- containerPort: 8001
136-
protocol: TCP
137-
- containerPort: 55555
109+
- containerPort: 5557
138110
protocol: TCP
139111
volumes:
140112
- name: home
141113
emptyDir: {}
142-
- name: config-decoder
143-
configMap:
144-
name: {{ include "modelservice.fullname" . -}}-config-decoder
145114
{{ `{{- if .HFModelName }}` }}
146115
- name: model-cache
147116
emptyDir: {}
@@ -158,17 +127,6 @@ data:
158127
{{- toYaml .Values.modelservice.prefill.tolerations | nindent 12 }}
159128
{{- end }}
160129
containers:
161-
- name: "routing-proxy"
162-
image: {{ include "modelservice.routingProxyImage" . }}
163-
securityContext:
164-
allowPrivilegeEscalation: false
165-
runAsNonRoot: true
166-
args:
167-
- "--port=8001"
168-
- "--vllm-port=8000"
169-
ports:
170-
- containerPort: 8000
171-
protocol: TCP
172130
- name: vllm
173131
image: {{ include "modelservice.vllmImage" . }}
174132
imagePullPolicy: {{ .Values.modelservice.vllm.image.imagePullPolicy }}
@@ -181,30 +139,31 @@ data:
181139
args:
182140
- "--port"
183141
- "8000"
142+
- "--enforce-eager"
184143
- "--kv-transfer-config"
185-
- '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_producer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "producer1"}}'
144+
- '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
186145
env:
187146
- name: HOME
188147
value: /home
148+
- name: VLLM_LOGGING_LEVEL
149+
value: DEBUG
150+
- name: VLLM_NIXL_SIDE_CHANNEL_PORT
151+
value: "5557"
152+
- name: VLLM_NIXL_SIDE_CHANNEL_HOST
153+
valueFrom:
154+
fieldRef:
155+
fieldPath: status.podIP
189156
- name: POD_IP
190157
valueFrom:
191158
fieldRef:
192159
apiVersion: v1
193160
fieldPath: status.podIP
194161
- name: LMCACHE_DISTRIBUTED_URL
195-
value: "${POD_IP}:80"
162+
value: ${POD_IP}:8200
196163
- name: CUDA_VISIBLE_DEVICES
197164
value: "0"
198165
- name: UCX_TLS
199166
value: "cuda_ipc,cuda_copy,tcp"
200-
- name: LMCACHE_USE_EXPERIMENTAL
201-
value: "True"
202-
- name: VLLM_ENABLE_V1_MULTIPROCESSING
203-
value: "1"
204-
- name: VLLM_WORKER_MULTIPROC_METHOD
205-
value: spawn
206-
- name: LMCACHE_CONFIG_FILE
207-
value: /vllm-workspace/lmcache-prefiller-config.yaml
208167
{{- if .Values.redis.enabled }}
209168
- name: LMCACHE_LOOKUP_URL
210169
value: {{ include "redis.master.service.fullurl" .}}
@@ -216,8 +175,6 @@ data:
216175
volumeMounts:
217176
- name: home
218177
mountPath: /home
219-
- name: config-prefiller
220-
mountPath: /vllm-workspace
221178
{{ `{{- if .HFModelName }}` }}
222179
- name: model-cache
223180
mountPath: /models
@@ -227,16 +184,11 @@ data:
227184
readOnly: true
228185
{{ `{{- end }}` }}
229186
ports:
230-
- containerPort: 8001
231-
protocol: TCP
232-
- containerPort: 55555
187+
- containerPort: 5557
233188
protocol: TCP
234189
volumes:
235190
- name: home
236191
emptyDir: {}
237-
- name: config-prefiller
238-
configMap:
239-
name: {{ include "modelservice.fullname" . -}}-config-prefiller
240192
{{ `{{ if .HFModelName }}` }}
241193
- name: model-cache
242194
emptyDir: {}
@@ -254,7 +206,7 @@ data:
254206
clusterIP: None
255207
ports:
256208
- name: nixl
257-
port: 55555
209+
port: 5557
258210
protocol: TCP
259211
- name: vllm
260212
port: 8000
@@ -272,7 +224,7 @@ data:
272224
clusterIP: None
273225
ports:
274226
- name: nixl
275-
port: 55555
227+
port: 5557
276228
protocol: TCP
277229
- name: vllm
278230
port: 8000

‎charts/llm-d/values.yaml‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -207,8 +207,10 @@ modelservice:
207207
# -- Endpoint picker image used in ModelService CR presets
208208
image:
209209
registry: quay.io
210-
repository: llm-d/llm-d-inference-scheduler
211-
tag: "0.0.1"
210+
repository: llm-d/llm-d-gateway-api-inference-extension-dev
211+
tag: 0.0.5
212+
# repository: llm-d/llm-d-inference-scheduler
213+
# tag: 0.0.1
212214
imagePullPolicy: "Always"
213215

214216
metrics:
@@ -300,9 +302,10 @@ modelservice:
300302
# -- vLLM image used in ModelService CR presets
301303
image:
302304
registry: quay.io
303-
repository: llm-d/llm-d-dev
304-
tag: "0.0.5"
305-
imagePullPolicy: "IfNotPresent"
305+
repository: "llm-d/llm-d-dev"
306+
tag: "vllm-nixl-0.0.6"
307+
imagePullPolicy: "Always"
308+
# imagePullPolicy: "IfNotPresent"
306309

307310
metrics:
308311

0 commit comments

Comments
 (0)