Skip to content

Commit 244f77a

Browse files
tintisimonetintisimone
committed
Add Intel Arc Battlemage (BMG) GPU XPU support
Add vLLM-based inference support for Intel Arc Battlemage (BMG) discrete GPUs to the single-node deployment: Helm values for the BMG vLLM serving container, XPU-aware model selection, Intel GPU device-plugin wiring, and setup documentation. Incorporates review feedback: run the vLLM container as non-root, use xpu*-style naming instead of bmg-specific names, replace hardcoded inventory/config values with placeholders, and de-duplicate the kubernetes SDK install tasks. The CodeTranslation changes were split into a separate PR (#117) to keep this one focused on XPU/BMG. Signed-off-by: tintisimone <simone.tinti@intel.com>
1 parent ba5141d commit 244f77a

37 files changed

Lines changed: 1701 additions & 274 deletions

‎README.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,22 +4,23 @@ Unleash the power of AI Inference on Intel Silicon
44

55
The Intel® AI for Enterprise Inference is aimed to streamline and enhance the deployment and management of AI inference services on Intel hardware. Utilizing the power of Kubernetes Orchestration, this solution automates the deployment of LLM models to run faster inference, provision compute resources, and configure the optimal settings to minimize the complexities and reduce manual efforts.
66

7-
It supports a broad range of Intel hardware platforms, including Intel® Xeon® Scalable processors and Intel® Gaudi® AI Accelerators, ensuring flexibility and scalability to meet diverse enterprise needs.
7+
It supports a broad range of Intel hardware platforms, including Intel® Xeon® Scalable processors, Intel® Gaudi® AI Accelerators, and **Intel® Arc™ Battlemage (BMG) GPUs**, ensuring flexibility and scalability to meet diverse enterprise needs.
88

9-
Intel® AI for Enterprise Inference, powered by OPEA, is compatible with OpenAI standard APIs, enabling seamless integration to enterprise applications both on-premises and in cloud-native environments. This compatibility allows businesses to leverage the full capabilities of Intel hardware while deploying AI models with ease. With this suite, enterprises can efficiently configure and evolve their AI infrastructure, adapting to new models and growing demands effortlessly.
9+
Intel® AI for Enterprise Inference, powered by OPEA, is compatible with OpenAI standard APIs, enabling seamless integration to enterprise applications both on-premises and in cloud-native environments. This compatibility allows businesses to leverage the full capabilities of Intel hardware while deploying AI models with ease. With this suite, enterprises can efficiently configure and evolve their AI infrastructure, adapting to new models and growing demands effortlessly.
1010

1111
![Intel AI for Enterprise Inference](docs/pictures/Enterprise-Inference-Architecture.png)
1212

1313
#### Key Components:
1414
- **Kubernetes**: A powerful container orchestration platform that automates the deployment, scaling, and management of containerized applications, ensuring high availability and efficient resource utilization.
1515
- **Intel Gaudi Base Operator**: A specialized operator that manages the lifecycle of Habana AI resources within the Kubernetes cluster, enabling efficient utilization of Intel® Gaudi® hardware for AI workloads. (Applicable only to Gaudi based deployments)
16+
- **Intel GPU Plugin**: A Kubernetes device plugin that manages Intel® Arc™ GPU resources within the cluster, enabling efficient utilization of Intel® Arc™ Battlemage (BMG) hardware for AI workloads. (Applicable only to BMG based deployments)
1617
- **Ingress NGINX Controller**: A high-performance reverse proxy and load balancer for traffic, responsible for routing incoming requests to the appropriate services within the Kubernetes cluster, ensuring seamless access to deployed AI models.
1718
- **Keycloak**: An open-source identity and access management solution that provides robust authentication and authorization capabilities, ensuring secure access to AI services and resources within the cluster.
1819
- **APISIX**: A cloud-native API gateway, handling API traffic and providing advanced features caching, and authentication, enabling efficient and secure access to deployed AI models.
1920
- **Observability**: An open-source monitoring solution designed to operate natively within Kubernetes clusters, providing comprehensive visibility into the performance, health, and resource utilization of deployed applications and cluster components through metrics, visualization, and alerting capabilities.
2021
- **Model Deployments**: Automated deployment and management of AI LLM models within the Kubernetes inference cluster, enabling scalable and reliable AI inference capabilities.
2122
- **GenAI Gateway**: An integrated gateway leveraging LiteLLM and Langfuse to provide flexible interfaces for routing and managing generative AI models. It enables user and key management, user token telemetry, and analytics for LLM inference workflows.
22-
23+
2324
## Table of Contents
2425
- [Usage](#usage)
2526
- [Support](#support)
@@ -32,6 +33,8 @@ Intel® AI for Enterprise Inference, powered by OPEA, is compatible with OpenAI
3233
The Usage instructions for the AI Inference as a Service Deployment Automation can be found in the [docs/README.md](docs/README.md) file.
3334
To setup, follow the step-by-step instructions provided in the `docs/README.md` file.
3435

36+
For Intel® Arc™ Battlemage (BMG) GPU setup, refer to [docs/intel-arc-bmg-setup.md](docs/intel-arc-bmg-setup.md).
37+
3538
## Support
3639
For feature requests, bugs or questions about the project, [open an issue](https://github.com/opea-project/Enterprise-Inference/issues) on the GitHub Issues page. Provide as much details as possible, including steps to reproduce the issue, expected behavior, and actual behavior.
3740

@@ -42,7 +45,7 @@ Intel® AI for Enterprise Inference is licensed under the [Apache License Versio
4245
The [Security Policy](SECURITY.md) outlines our guidelines and procedures for ensuring the highest level of security and trust for our users who consume Intel® AI for Enterprise Inference.
4346

4447
## Trademark Information
45-
Intel, the Intel logo, Xeon, and Gaudi are trademarks of Intel Corporation or its subsidiaries.
48+
Intel, the Intel logo, Xeon, Gaudi, and Arc are trademarks of Intel Corporation or its subsidiaries.
4649

4750
* Other names and brands may be claimed as the property of others.
4851
&copy; Intel Corporation

‎core/helm-charts/vllm/bmg-values.yaml‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# Copyright (C) 2025-2026 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Intel® Arc™ Battlemage (BMG) GPU optimized override values for vLLM deployments.
5+
# This file contains BMG-specific overrides for Intel Arc B-series GPU (e.g., B580, B770).
6+
# Requires the Intel GPU Plugin (intel-device-plugins-gpu) to be installed on the cluster.
7+
8+
# Intel XPU accelerator device (Arc GPU)
9+
accelDevice: "xpu"
10+
# Kubernetes resource name exposed by the Intel GPU device plugin.
11+
# Use "gpu.intel.com/xe" with the newer Xe kernel driver (Ubuntu 25.10 / Arc B-series),
12+
# or "gpu.intel.com/i915" on hosts still using the i915 driver.
13+
xpuDeviceResource: "gpu.intel.com/xe"
14+
15+
block_size: 64 # XPU-optimised KV cache block size (must be >= 64 for 0.14.1-xpu IPEX chunked prefill)
16+
max_num_seqs: 128 # Max concurrent sequences (tuned for Arc B-series VRAM)
17+
max_seq_len_to_capture: 2048
18+
d_type: "float16"
19+
max_model_len: 8192
20+
tensor_parallel_size: "1" # Arc Pro B50 has 1 GPU; adjust for multi-GPU setups
21+
22+
image:
23+
repository: intel/vllm
24+
tag: "0.17.0-xpu"
25+
pullPolicy: IfNotPresent
26+
command: ["vllm", "serve"]
27+
28+
# --- Security context (non-root, scanner-compliant) ---
29+
# The intel/vllm:0.17.0-xpu image defaults to root, but the workload runs fine as
30+
# a non-root user (UID 1001) provided it can reach the Arc render node
31+
# (/dev/dri/renderD*). That device is owned by root:render on the host, so the pod
32+
# must join the host's "render" (and typically "video") group via supplementalGroups.
33+
# HF_HOME is set to /data (the model-volume mount); fsGroup: 1001 makes that volume
34+
# and the /tmp + /dev/shm emptyDirs group-writable, so caches work without root.
35+
#
36+
# IMPORTANT — validate the host GIDs on the BMG node before deploying (they vary by
37+
# distro/kernel). On the target node run:
38+
# getent group render video # e.g. render:x:993 video:x:44
39+
# stat -c '%g %G' /dev/dri/renderD128
40+
# then set supplementalGroups below to the matching numeric GIDs. A GPU permission
41+
# error at startup almost always means the render GID here does not match the host.
42+
podSecurityContext:
43+
runAsNonRoot: true
44+
runAsUser: 1001
45+
runAsGroup: 1001
46+
fsGroup: 1001
47+
# [<render GID>, <video GID>] — verify per host with `getent group render video`.
48+
# Validated on the Arc Pro B50 test host (Ubuntu 25.10): render=992, video=44.
49+
# These GIDs are NOT guaranteed across distros/kernels — re-check on each node.
50+
supplementalGroups: [992, 44]
51+
seccompProfile:
52+
type: RuntimeDefault
53+
54+
securityContext:
55+
allowPrivilegeEscalation: false
56+
capabilities:
57+
drop:
58+
- ALL
59+
add:
60+
- SYS_NICE
61+
# vLLM/IPEX writes SYCL and compile caches at runtime, so the root filesystem
62+
# cannot be fully read-only; model/HF caches live on the mounted /data volume.
63+
readOnlyRootFilesystem: false
64+
runAsNonRoot: true
65+
runAsUser: 1001
66+
runAsGroup: 1001
67+
68+
# Node affinity for BMG inference nodes
69+
affinity:
70+
nodeAffinity:
71+
requiredDuringSchedulingIgnoredDuringExecution:
72+
nodeSelectorTerms:
73+
- matchExpressions:
74+
- key: ei-inference-eligible
75+
operator: In
76+
values: ["true"]
77+
78+
# Intel XPU runtime settings
79+
VLLM_NO_USAGE_STATS: 1
80+
DO_NOT_TRACK: 1
81+
82+
# vLLM device backend - set via env var in 0.14.1-xpu (VLLM_TARGET_DEVICE=xpu is already baked in)
83+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
84+
85+
LLM_MODEL_ID: "Qwen/Qwen2.5-Coder-3B-Instruct"
86+
87+
modelConfigs:
88+
89+
"meta-llama/Llama-3.1-8B-Instruct":
90+
configMapValues:
91+
VLLM_NO_USAGE_STATS: "1"
92+
DO_NOT_TRACK: "1"
93+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
94+
HF_HUB_DISABLE_XET: "1"
95+
extraCmdArgs:
96+
[
97+
"--dtype", "float16",
98+
"--block-size", "64",
99+
"--max-model-len", "8192",
100+
"--gpu-memory-utilization", "0.90",
101+
"--max-num-seqs", "128",
102+
"--enforce-eager",
103+
"--enable-auto-tool-choice",
104+
"--tool-call-parser", "llama3_json",
105+
]
106+
tensor_parallel_size: "1"
107+
pipeline_parallel_size: "1"
108+
109+
"mistralai/Mistral-7B-Instruct-v0.3":
110+
configMapValues:
111+
VLLM_NO_USAGE_STATS: "1"
112+
DO_NOT_TRACK: "1"
113+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
114+
HF_HUB_DISABLE_XET: "1"
115+
extraCmdArgs:
116+
[
117+
"--dtype", "float16",
118+
"--block-size", "64",
119+
"--max-model-len", "8192",
120+
"--gpu-memory-utilization", "0.90",
121+
"--max-num-seqs", "128",
122+
"--enforce-eager",
123+
"--enable-auto-tool-choice",
124+
"--tool-call-parser", "mistral",
125+
]
126+
tensor_parallel_size: "1"
127+
pipeline_parallel_size: "1"
128+
129+
"deepseek-ai/DeepSeek-R1-Distill-Llama-8B":
130+
configMapValues:
131+
VLLM_NO_USAGE_STATS: "1"
132+
DO_NOT_TRACK: "1"
133+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
134+
HF_HUB_DISABLE_XET: "1"
135+
extraCmdArgs:
136+
[
137+
"--dtype", "float16",
138+
"--block-size", "64",
139+
"--max-model-len", "8192",
140+
"--gpu-memory-utilization", "0.90",
141+
"--max-num-seqs", "128",
142+
"--enforce-eager",
143+
]
144+
tensor_parallel_size: "1"
145+
pipeline_parallel_size: "1"
146+
147+
"Qwen/Qwen2.5-7B-Instruct":
148+
configMapValues:
149+
VLLM_NO_USAGE_STATS: "1"
150+
DO_NOT_TRACK: "1"
151+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
152+
HF_HUB_DISABLE_XET: "1"
153+
extraCmdArgs:
154+
[
155+
"--dtype", "float16",
156+
"--block-size", "64",
157+
"--max-model-len", "8192",
158+
"--gpu-memory-utilization", "0.90",
159+
"--max-num-seqs", "128",
160+
"--enforce-eager",
161+
"--enable-auto-tool-choice",
162+
"--tool-call-parser", "hermes",
163+
]
164+
tensor_parallel_size: "1"
165+
pipeline_parallel_size: "1"
166+
167+
"Qwen/Qwen2.5-Coder-3B-Instruct":
168+
configMapValues:
169+
VLLM_NO_USAGE_STATS: "1"
170+
DO_NOT_TRACK: "1"
171+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
172+
HF_HUB_DISABLE_XET: "1"
173+
extraCmdArgs:
174+
[
175+
"--dtype", "float16",
176+
"--block-size", "64",
177+
"--max-model-len", "8192",
178+
"--gpu-memory-utilization", "0.90",
179+
"--max-num-seqs", "128",
180+
"--enforce-eager",
181+
"--enable-auto-tool-choice",
182+
"--tool-call-parser", "hermes",
183+
]
184+
tensor_parallel_size: "1"
185+
pipeline_parallel_size: "1"
186+
187+
"tiiuae/Falcon3-7B-Instruct":
188+
configMapValues:
189+
VLLM_NO_USAGE_STATS: "1"
190+
DO_NOT_TRACK: "1"
191+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
192+
HF_HUB_DISABLE_XET: "1"
193+
extraCmdArgs:
194+
[
195+
"--dtype", "float16",
196+
"--block-size", "64",
197+
"--max-model-len", "8192",
198+
"--gpu-memory-utilization", "0.90",
199+
"--max-num-seqs", "128",
200+
"--enforce-eager",
201+
]
202+
tensor_parallel_size: "1"
203+
pipeline_parallel_size: "1"
204+
205+
defaultModelConfigs:
206+
configMapValues:
207+
VLLM_NO_USAGE_STATS: "1"
208+
DO_NOT_TRACK: "1"
209+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
210+
HF_HUB_DISABLE_XET: "1"
211+
HOME: "/tmp" # Non-root UID 1001 not in /etc/passwd; HOME avoids getpass.getuser() in PyTorch cache
212+
PYTORCH_DISABLE_TORCH_INDUCTOR: "1" # Workaround: inductor cache requires getpass.getuser(), which fails for non-root UID 1001
213+
extraCmdArgs:
214+
[
215+
"--dtype", "float16",
216+
"--block-size", "16",
217+
"--max-model-len", "8192",
218+
"--gpu-memory-utilization", "0.90",
219+
"--max-num-seqs", "128",
220+
"--enforce-eager",
221+
]
222+
tensor_parallel_size: "{{ .Values.tensor_parallel_size }}"
223+
pipeline_parallel_size: "{{ .Values.pipeline_parallel_size }}"

‎core/helm-charts/vllm/templates/deployment.yaml‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -126,11 +126,12 @@ spec:
126126
{{- if .Values.image.pullPolicy }}
127127
imagePullPolicy: {{ .Values.image.pullPolicy }}
128128
{{- end }}
129-
# command:
130-
# - /bin/bash
131-
# - -c
132-
# - |
133-
# python3 -m vllm.entrypoints.openai.api_server --dtype {{ .Values.d_type }} --model {{ .Values.LLM_MODEL_ID }} --port {{ .Values.port }} --tensor-parallel-size {{ .Values.tensor_parallel_size }} --block-size {{ .Values.block_size }} --max-model-len {{ .Values.max_model_len }} --disable-log-requests
129+
{{- if .Values.image.command }}
130+
command:
131+
{{- range .Values.image.command }}
132+
- {{ . | quote }}
133+
{{- end }}
134+
{{- end }}
134135
args:
135136
{{- $modelConfig := (index .Values.modelConfigs $modelName | default dict) }}
136137
{{- $modelArgs := $modelConfig.extraCmdArgs | default .Values.defaultModelConfigs.extraCmdArgs }}
@@ -195,7 +196,7 @@ spec:
195196
{{- end }}
196197
{{- else }}
197198
limits:
198-
habana.ai/gaudi: {{ .Values.tensor_parallel_size | default (index .Values.modelConfigs .Values.LLM_MODEL_ID | default dict).tensor_parallel_size | default .Values.defaultModelConfigs.tensor_parallel_size | quote}}
199+
{{ .Values.xpuDeviceResource | default "habana.ai/gaudi" }}: {{ .Values.tensor_parallel_size | default (index .Values.modelConfigs .Values.LLM_MODEL_ID | default dict).tensor_parallel_size | default .Values.defaultModelConfigs.tensor_parallel_size | quote}}
199200
{{- end }}
200201
{{- end }}
201202

‎core/helm-charts/vllm/values.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ autoscaling:
1717

1818
# empty for CPU (longer latencies are tolerated before HPA scaling unaccelerated service)
1919
accelDevice: ""
20+
# Kubernetes resource name for the accelerator (e.g. habana.ai/gaudi, gpu.intel.com/xe or gpu.intel.com/i915)
21+
xpuDeviceResource: ""
2022

2123
port: 2080
2224
shmSize: 1Gi

‎core/inference-stack-deploy.sh‎

100644100755
Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ NC=$(tput sgr0)
5050
# --keycloak-admin-password <password>: The Keycloak admin password.
5151
# --hugging-face-token <token>: The token for Huggingface.
5252
# --models <models>: The models to deploy (comma-separated list of model numbers or names).
53-
# --cpu-or-gpu <c/g>: Specify whether to run on CPU or GPU.
53+
# --device <cpu/hpu/xpu>: Specify the target device. 'cpu' for Xeon, 'hpu' for Gaudi GPU, 'xpu' for Intel Arc Battlemage GPU.
5454

5555
# Main Menu
5656

@@ -86,7 +86,7 @@ NC=$(tput sgr0)
8686

8787
# Example
8888
# To perform a fresh installation with specific parameters, you can run:
89-
# ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/to/cert.pem" --key-file "/path/to/key.pem" --keycloak-client-id "my-client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --cpu-or-gpu "g"
89+
# ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/to/cert.pem" --key-file "/path/to/key.pem" --keycloak-client-id "my-client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --device "hpu"
9090

9191
##############################################################################
9292

@@ -118,6 +118,7 @@ source "$SCRIPT_DIR/lib/cluster/drv-fw-update.sh"
118118
# Components deployment
119119
source "$SCRIPT_DIR/lib/components/kubernetes-setup.sh"
120120
source "$SCRIPT_DIR/lib/components/intel-base-operator.sh"
121+
source "$SCRIPT_DIR/lib/components/intel-gpu-plugin.sh"
121122
source "$SCRIPT_DIR/lib/components/ingress-controller.sh"
122123
source "$SCRIPT_DIR/lib/components/keycloak-controller.sh"
123124
source "$SCRIPT_DIR/lib/components/genai-gateway-controller.sh"
@@ -166,10 +167,11 @@ Options:
166167
--keycloak-admin-password <pw> Keycloak admin password.
167168
--hugging-face-token <token> Huggingface token.
168169
--models <models> Models to deploy (comma-separated).
169-
--cpu-or-gpu <c/g> Run on CPU (c) or GPU (g).
170+
--device <cpu/hpu/xpu> Target device: cpu (Xeon), hpu (Gaudi GPU), xpu (Intel Arc Battlemage GPU).
170171
171172
Examples:
172-
Setup cluster: ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/cert.pem" --key-file "/path/key.pem" --keycloak-client-id "client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --cpu-or-gpu "g"
173+
Setup cluster (Gaudi GPU): ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/cert.pem" --key-file "/path/key.pem" --keycloak-client-id "client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --device "hpu"
174+
Setup cluster (BMG GPU): ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/cert.pem" --key-file "/path/key.pem" --keycloak-client-id "client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "36" --device "xpu"
173175
174176
###############################################################################
175177
EOF

‎core/inventory/hosts.yaml‎

Lines changed: 7 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,19 @@
11
all:
22
hosts:
3-
master:
4-
ansible_host: "{{ private_ip_control_plane_node }}"
5-
ansible_user: "username_of_user_running_automation"
6-
ansible_ssh_private_key_file: "/home/ubuntu/.ssh/id_rsa"
7-
worker1:
8-
ansible_host: "{{ private_ip_workload_node_1 }}"
9-
ansible_user: "username_of_user_running_automation"
10-
ansible_ssh_private_key_file: "/home/ubuntu/.ssh/id_rsa"
11-
worker2:
12-
ansible_host: "{{ private_ip_workload_node_2 }}"
13-
ansible_user: "username_of_user_running_automation"
14-
ansible_ssh_private_key_file: "/home/ubuntu/.ssh/id_rsa"
3+
master1:
4+
ansible_connection: local
5+
# ansible_user: your-username
6+
ansible_become: true
157
children:
168
kube_control_plane:
179
hosts:
18-
master:
10+
master1:
1911
kube_node:
2012
hosts:
21-
worker1:
22-
worker2:
13+
master1:
2314
etcd:
2415
hosts:
25-
master:
16+
master1:
2617
k8s_cluster:
2718
children:
2819
kube_control_plane:

0 commit comments

Comments
 (0)