Skip to content

Commit baed0b5

Browse files
eero-tlianhao
eero-t
and
lianhao
authored
Add vLLM+HPA support to ChatQnA Helm chart (#610)
* Add monitoring support for the vLLM component Signed-off-by: Eero Tamminen <eero.t.tamminen@intel.com> * Initial vLLM support for ChatQnA For now vLLM replaces just TGI, but as it supports also embedding, also TEI-embed/-rerank may be replaceable later on. Signed-off-by: Eero Tamminen <eero.t.tamminen@intel.com> * Fix HPA comments in tgi/tei/tererank values files Signed-off-by: Eero Tamminen <eero.t.tamminen@intel.com> * Add HPA scaling support for ChatQnA / vLLM Signed-off-by: Eero Tamminen <eero.t.tamminen@intel.com> * Adapt to latest vllm changes - Remove --eager-enforce on hpu to improve performance - Refactor to the upstream docker entrypoint changes Fixes issue #631. Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> * Clean up ChatQnA vLLM Gaudi parameters Signed-off-by: Eero Tamminen <eero.t.tamminen@intel.com> --------- Signed-off-by: Eero Tamminen <eero.t.tamminen@intel.com> Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Co-authored-by: Lianhao Lu <lianhao.lu@intel.com>
1 parent a4a96ab commit baed0b5

25 files changed

Lines changed: 253 additions & 22 deletions

‎helm-charts/chatqna/Chart.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,11 @@ dependencies:
1818
- name: tgi
1919
version: 0-latest
2020
repository: "file://../common/tgi"
21+
condition: tgi.enabled
22+
- name: vllm
23+
version: 0-latest
24+
repository: "file://../common/vllm"
25+
condition: vllm.enabled
2126
- name: tei
2227
version: 0-latest
2328
repository: "file://../common/tei"

‎helm-charts/chatqna/README.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Helm chart for deploying ChatQnA service. ChatQnA depends on the following servi
1111
- [teirerank](../common/teirerank/README.md)
1212
- [llm-uservice](../common/llm-uservice/README.md)
1313
- [tgi](../common/tgi/README.md)
14+
- [vllm](../common/vllm/README.md)
1415

1516
## Installing the Chart
1617

@@ -26,13 +27,15 @@ export MODELNAME="Intel/neural-chat-7b-v3-3"
2627
# If you would like to use the traditional UI, please change the image as well as the containerport within the values
2728
# append these at the end of the command "--set chatqna-ui.image.repository=opea/chatqna-ui,chatqna-ui.image.tag=latest,chatqna-ui.containerPort=5173"
2829
helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME}
29-
# To use Gaudi device
30-
#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME} -f chatqna/gaudi-values.yaml
30+
# To use Gaudi device with TGI
31+
#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME} -f chatqna/gaudi-tgi-values.yaml
32+
# To use Gaudi device with vLLM
33+
#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME} -f chatqna/gaudi-vllm-values.yaml
3134
# To use Nvidia GPU
3235
#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME} -f chatqna/nv-values.yaml
33-
# To include guardrail component in chatqna on Xeon
36+
# To include guardrail component in chatqna on Xeon with TGI
3437
#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} -f chatqna/guardrails-values.yaml
35-
# To include guardrail component in chatqna on Gaudi
38+
# To include guardrail component in chatqna on Gaudi with TGI
3639
#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} -f chatqna/guardrails-gaudi-values.yaml
3740
```
3841

‎helm-charts/chatqna/ci-gaudi-tgi-values.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
gaudi-tgi-values.yaml

‎helm-charts/chatqna/ci-gaudi-values.yaml‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎helm-charts/chatqna/ci-gaudi-vllm-values.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
gaudi-vllm-values.yaml

helm-charts/chatqna/gaudi-values.yaml renamed to helm-charts/chatqna/gaudi-tgi-values.yaml

File renamed without changes.

‎helm-charts/chatqna/gaudi-vllm-values.yaml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Accelerate inferencing in heaviest components to improve performance
5+
# by overriding their subchart values
6+
7+
tgi:
8+
enabled: false
9+
10+
vllm:
11+
enabled: true
12+
accelDevice: "gaudi"
13+
image:
14+
repository: opea/vllm-gaudi
15+
tag: "latest"
16+
resources:
17+
limits:
18+
habana.ai/gaudi: 1
19+
startupProbe:
20+
initialDelaySeconds: 5
21+
periodSeconds: 5
22+
timeoutSeconds: 1
23+
failureThreshold: 120
24+
readinessProbe:
25+
initialDelaySeconds: 5
26+
periodSeconds: 5
27+
timeoutSeconds: 1
28+
livenessProbe:
29+
initialDelaySeconds: 5
30+
periodSeconds: 5
31+
timeoutSeconds: 1
32+
33+
PT_HPU_ENABLE_LAZY_COLLECTIVES: "true"
34+
OMPI_MCA_btl_vader_single_copy_mechanism: "none"
35+
36+
extraCmdArgs: [
37+
"--tensor-parallel-size", "1",
38+
"--block-size", "128",
39+
"--max-num-seqs", "256",
40+
"--max-seq_len-to-capture", "2048"
41+
]
42+
43+
44+
# Reranking: second largest bottleneck when reranking is in use
45+
# (i.e. query context docs have been uploaded with data-prep)
46+
#
47+
# TODO: could vLLM be used also for reranking / embedding?
48+
teirerank:
49+
accelDevice: "gaudi"
50+
OMPI_MCA_btl_vader_single_copy_mechanism: "none"
51+
MAX_WARMUP_SEQUENCE_LENGTH: "512"
52+
image:
53+
repository: ghcr.io/huggingface/tei-gaudi
54+
tag: 1.5.0
55+
resources:
56+
limits:
57+
habana.ai/gaudi: 1
58+
securityContext:
59+
readOnlyRootFilesystem: false
60+
livenessProbe:
61+
timeoutSeconds: 1
62+
readinessProbe:
63+
timeoutSeconds: 1

‎helm-charts/chatqna/hpa-values.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# Enable HorizontalPodAutoscaler (HPA)
55
#
66
# That will overwrite named PrometheusAdapter configMap with ChatQnA specific
7-
# custom metric queries for embedding, reranking, tgi services.
7+
# custom metric queries for embedding, reranking, and LLM services.
88
#
99
# Default upstream configMap is in:
1010
# - https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/deploy/manifests/config-map.yaml
@@ -15,6 +15,10 @@ autoscaling:
1515
# Override values in specific subcharts
1616

1717
# Enabling "autoscaling" for any of the subcharts requires enabling it also above!
18+
vllm:
19+
autoscaling:
20+
maxReplicas: 4
21+
enabled: true
1822
tgi:
1923
autoscaling:
2024
maxReplicas: 4

‎helm-charts/chatqna/templates/custom-metrics-configmap.yaml‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,27 @@ metadata:
1313
data:
1414
config.yaml: |
1515
rules:
16-
{{- if .Values.tgi.autoscaling.enabled }}
16+
{{- if and .Values.vllm.enabled .Values.vllm.autoscaling.enabled }}
1717
# check metric with:
1818
# kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/service/*/<metric> | jq
1919
#
20+
- seriesQuery: '{__name__="vllm:time_per_output_token_seconds_sum",service="{{ include "vllm.fullname" .Subcharts.vllm }}"}'
21+
# Average output token latency from vLLM histograms, over 1 min
22+
# (interval should be at least 4x serviceMonitor query interval,
23+
# 0.001 divider add is to make sure there's always a valid value)
24+
metricsQuery: 'rate(vllm:time_per_output_token_seconds_sum{service="{{ include "vllm.fullname" .Subcharts.vllm }}",<<.LabelMatchers>>}[1m]) / (0.001+rate(vllm:time_per_output_token_seconds_count{service="{{ include "vllm.fullname" .Subcharts.vllm }}",<<.LabelMatchers>>}[1m]))'
25+
name:
26+
matches: ^vllm:time_per_output_token_seconds_sum
27+
as: "{{ include "vllm.metricPrefix" .Subcharts.vllm }}_token_latency"
28+
resources:
29+
# HPA needs both namespace + suitable object resource for its query paths:
30+
# /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/service/*/<metric>
31+
# (pod is not suitable object type for matching as each instance has different name)
32+
overrides:
33+
namespace: {resource: "namespace"}
34+
service: {resource: "service"}
35+
{{- end }}
36+
{{- if and .Values.tgi.enabled .Values.tgi.autoscaling.enabled }}
2037
{{- if .Values.tgi.accelDevice }}
2138
- seriesQuery: '{__name__="tgi_queue_size",service="{{ include "tgi.fullname" .Subcharts.tgi }}"}'
2239
# TGI instances queue_size sum
@@ -27,16 +44,12 @@ data:
2744
{{- else }}
2845
- seriesQuery: '{__name__="tgi_request_inference_duration_sum",service="{{ include "tgi.fullname" .Subcharts.tgi }}"}'
2946
# Average request latency from TGI histograms, over 1 min
30-
# (0.001 divider add is to make sure there's always a valid value)
3147
metricsQuery: 'rate(tgi_request_inference_duration_sum{service="{{ include "tgi.fullname" .Subcharts.tgi }}",<<.LabelMatchers>>}[1m]) / (0.001+rate(tgi_request_inference_duration_count{service="{{ include "tgi.fullname" .Subcharts.tgi }}",<<.LabelMatchers>>}[1m]))'
3248
name:
3349
matches: ^tgi_request_inference_duration_sum
3450
as: "{{ include "tgi.metricPrefix" .Subcharts.tgi }}_request_latency"
3551
{{- end }}
3652
resources:
37-
# HPA needs both namespace + suitable object resource for its query paths:
38-
# /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/service/*/<metric>
39-
# (pod is not suitable object type for matching as each instance has different name)
4053
overrides:
4154
namespace: {resource: "namespace"}
4255
service: {resource: "service"}

‎helm-charts/chatqna/templates/deployment.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,19 @@ spec:
3535
- name: {{ .Release.Name }}
3636
env:
3737
- name: LLM_SERVER_HOST_IP
38+
{{- if .Values.vllm.enabled }}
39+
value: {{ .Release.Name }}-vllm
40+
{{- else }}
3841
value: {{ .Release.Name }}-tgi
42+
{{- end }}
3943
- name: LLM_SERVER_PORT
4044
value: "80"
4145
- name: LLM_MODEL
46+
{{- if .Values.vllm.enabled }}
47+
value: {{ .Values.vllm.LLM_MODEL_ID | quote }}
48+
{{- else }}
4249
value: {{ .Values.tgi.LLM_MODEL_ID | quote }}
50+
{{- end }}
4351
- name: RERANK_SERVER_HOST_IP
4452
value: {{ .Release.Name }}-teirerank
4553
- name: RERANK_SERVER_PORT

0 commit comments

Comments
 (0)