Skip to content

Commit 19ea8fa

Browse files
tintisimoneclaude
tintisimone
and
claude
committed
Add Intel Arc Battlemage (BMG) GPU XPU support and Code Translation enhancements
Add Intel Arc Battlemage (BMG) GPU support with XPU backend: - Add Intel GPU Plugin for Kubernetes deployment - Support XPU device type alongside CPU and HPU (Gaudi) - Add vLLM XPU backend configuration with intel/vllm:0.14.0-xpu image - Add BMG-specific Helm values and deployment templates - Add 6 BMG-optimized models (Llama, Mistral, DeepSeek, Qwen, Falcon3) - Default to Qwen/Qwen2.5-Coder-3B-Instruct for testing - Add gpu.intel.com/xe resource management for Ubuntu 25.10 - Add comprehensive BMG deployment documentation - Add single-node XPU configuration examples Code Translation improvements: - Update API client for XPU deployment compatibility - Update docker-compose configuration Refactoring: - Replace cpu_or_gpu parameter with device (cpu/hpu/xpu) - Update all deployment scripts and configuration files - Add XPU-specific environment variables for vLLM - Improve model selection logic for multi-device support Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent ba5141d commit 19ea8fa

38 files changed

Lines changed: 1695 additions & 305 deletions

‎README.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,27 @@
1+
<<<<<<< HEAD
12
# Intel® AI for Enterprise Inference
23

34
Unleash the power of AI Inference on Intel Silicon
45

56
The Intel® AI for Enterprise Inference is aimed to streamline and enhance the deployment and management of AI inference services on Intel hardware. Utilizing the power of Kubernetes Orchestration, this solution automates the deployment of LLM models to run faster inference, provision compute resources, and configure the optimal settings to minimize the complexities and reduce manual efforts.
67

7-
It supports a broad range of Intel hardware platforms, including Intel® Xeon® Scalable processors and Intel® Gaudi® AI Accelerators, ensuring flexibility and scalability to meet diverse enterprise needs.
8+
It supports a broad range of Intel hardware platforms, including Intel® Xeon® Scalable processors, Intel® Gaudi® AI Accelerators, and **Intel® Arc™ Battlemage (BMG) GPUs**, ensuring flexibility and scalability to meet diverse enterprise needs.
89

9-
Intel® AI for Enterprise Inference, powered by OPEA, is compatible with OpenAI standard APIs, enabling seamless integration to enterprise applications both on-premises and in cloud-native environments. This compatibility allows businesses to leverage the full capabilities of Intel hardware while deploying AI models with ease. With this suite, enterprises can efficiently configure and evolve their AI infrastructure, adapting to new models and growing demands effortlessly.
10+
Intel® AI for Enterprise Inference, powered by OPEA, is compatible with OpenAI standard APIs, enabling seamless integration to enterprise applications both on-premises and in cloud-native environments. This compatibility allows businesses to leverage the full capabilities of Intel hardware while deploying AI models with ease. With this suite, enterprises can efficiently configure and evolve their AI infrastructure, adapting to new models and growing demands effortlessly.
1011

1112
![Intel AI for Enterprise Inference](docs/pictures/Enterprise-Inference-Architecture.png)
1213

1314
#### Key Components:
1415
- **Kubernetes**: A powerful container orchestration platform that automates the deployment, scaling, and management of containerized applications, ensuring high availability and efficient resource utilization.
1516
- **Intel Gaudi Base Operator**: A specialized operator that manages the lifecycle of Habana AI resources within the Kubernetes cluster, enabling efficient utilization of Intel® Gaudi® hardware for AI workloads. (Applicable only to Gaudi based deployments)
17+
- **Intel GPU Plugin**: A Kubernetes device plugin that manages Intel® Arc™ GPU resources within the cluster, enabling efficient utilization of Intel® Arc™ Battlemage (BMG) hardware for AI workloads. (Applicable only to BMG based deployments)
1618
- **Ingress NGINX Controller**: A high-performance reverse proxy and load balancer for traffic, responsible for routing incoming requests to the appropriate services within the Kubernetes cluster, ensuring seamless access to deployed AI models.
1719
- **Keycloak**: An open-source identity and access management solution that provides robust authentication and authorization capabilities, ensuring secure access to AI services and resources within the cluster.
1820
- **APISIX**: A cloud-native API gateway, handling API traffic and providing advanced features caching, and authentication, enabling efficient and secure access to deployed AI models.
1921
- **Observability**: An open-source monitoring solution designed to operate natively within Kubernetes clusters, providing comprehensive visibility into the performance, health, and resource utilization of deployed applications and cluster components through metrics, visualization, and alerting capabilities.
2022
- **Model Deployments**: Automated deployment and management of AI LLM models within the Kubernetes inference cluster, enabling scalable and reliable AI inference capabilities.
2123
- **GenAI Gateway**: An integrated gateway leveraging LiteLLM and Langfuse to provide flexible interfaces for routing and managing generative AI models. It enables user and key management, user token telemetry, and analytics for LLM inference workflows.
22-
24+
2325
## Table of Contents
2426
- [Usage](#usage)
2527
- [Support](#support)
@@ -32,6 +34,8 @@ Intel® AI for Enterprise Inference, powered by OPEA, is compatible with OpenAI
3234
The Usage instructions for the AI Inference as a Service Deployment Automation can be found in the [docs/README.md](docs/README.md) file.
3335
To setup, follow the step-by-step instructions provided in the `docs/README.md` file.
3436

37+
For Intel® Arc™ Battlemage (BMG) GPU setup, refer to [docs/intel-arc-bmg-setup.md](docs/intel-arc-bmg-setup.md).
38+
3539
## Support
3640
For feature requests, bugs or questions about the project, [open an issue](https://github.com/opea-project/Enterprise-Inference/issues) on the GitHub Issues page. Provide as much details as possible, including steps to reproduce the issue, expected behavior, and actual behavior.
3741

@@ -42,7 +46,7 @@ Intel® AI for Enterprise Inference is licensed under the [Apache License Versio
4246
The [Security Policy](SECURITY.md) outlines our guidelines and procedures for ensuring the highest level of security and trust for our users who consume Intel® AI for Enterprise Inference.
4347

4448
## Trademark Information
45-
Intel, the Intel logo, Xeon, and Gaudi are trademarks of Intel Corporation or its subsidiaries.
49+
Intel, the Intel logo, Xeon, Gaudi, and Arc are trademarks of Intel Corporation or its subsidiaries.
4650

4751
* Other names and brands may be claimed as the property of others.
4852
&copy; Intel Corporation

‎core/helm-charts/vllm/bmg-values.yaml‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
# Copyright (C) 2025-2026 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Intel® Arc™ Battlemage (BMG) GPU optimized override values for vLLM deployments.
5+
# This file contains BMG-specific overrides for Intel Arc B-series GPU (e.g., B580, B770).
6+
# Requires the Intel GPU Plugin (intel-device-plugins-gpu) to be installed on the cluster.
7+
8+
# Intel XPU accelerator device (Arc GPU)
9+
accelDevice: "xpu"
10+
# Kubernetes resource name exposed by the Intel GPU device plugin
11+
accelDeviceResource: "gpu.intel.com/xe"
12+
13+
block_size: 64 # XPU-optimised KV cache block size (must be >= 64 for 0.14.1-xpu IPEX chunked prefill)
14+
max_num_seqs: 128 # Max concurrent sequences (tuned for Arc B-series VRAM)
15+
max_seq_len_to_capture: 2048
16+
d_type: "float16"
17+
max_model_len: 8192
18+
19+
image:
20+
repository: intel/vllm
21+
tag: "0.14.1-xpu"
22+
pullPolicy: IfNotPresent
23+
command: ["vllm", "serve"]
24+
25+
# intel/vllm:0.14.1-xpu runs as root (no user defined in image)
26+
podSecurityContext:
27+
fsGroup: 0
28+
runAsUser: 0
29+
30+
securityContext:
31+
allowPrivilegeEscalation: false
32+
capabilities:
33+
drop:
34+
- ALL
35+
add:
36+
- SYS_NICE
37+
readOnlyRootFilesystem: false
38+
runAsNonRoot: false
39+
runAsUser: 0
40+
41+
# Node affinity for BMG inference nodes
42+
affinity:
43+
nodeAffinity:
44+
requiredDuringSchedulingIgnoredDuringExecution:
45+
nodeSelectorTerms:
46+
- matchExpressions:
47+
- key: ei-inference-eligible
48+
operator: In
49+
values: ["true"]
50+
51+
# Intel XPU runtime settings
52+
VLLM_NO_USAGE_STATS: 1
53+
DO_NOT_TRACK: 1
54+
55+
# vLLM device backend - set via env var in 0.14.1-xpu (VLLM_TARGET_DEVICE=xpu is already baked in)
56+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
57+
58+
LLM_MODEL_ID: "Qwen/Qwen2.5-Coder-3B-Instruct"
59+
60+
modelConfigs:
61+
62+
"meta-llama/Llama-3.1-8B-Instruct":
63+
configMapValues:
64+
VLLM_NO_USAGE_STATS: "1"
65+
DO_NOT_TRACK: "1"
66+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
67+
HF_HUB_DISABLE_XET: "1"
68+
extraCmdArgs:
69+
[
70+
"--dtype", "float16",
71+
"--block-size", "64",
72+
"--max-model-len", "8192",
73+
"--gpu-memory-utilization", "0.90",
74+
"--max-num-seqs", "128",
75+
"--enforce-eager",
76+
"--enable-auto-tool-choice",
77+
"--tool-call-parser", "llama3_json",
78+
]
79+
tensor_parallel_size: "1"
80+
pipeline_parallel_size: "1"
81+
82+
"mistralai/Mistral-7B-Instruct-v0.3":
83+
configMapValues:
84+
VLLM_NO_USAGE_STATS: "1"
85+
DO_NOT_TRACK: "1"
86+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
87+
HF_HUB_DISABLE_XET: "1"
88+
extraCmdArgs:
89+
[
90+
"--dtype", "float16",
91+
"--block-size", "64",
92+
"--max-model-len", "8192",
93+
"--gpu-memory-utilization", "0.90",
94+
"--max-num-seqs", "128",
95+
"--enforce-eager",
96+
"--enable-auto-tool-choice",
97+
"--tool-call-parser", "mistral",
98+
]
99+
tensor_parallel_size: "1"
100+
pipeline_parallel_size: "1"
101+
102+
"deepseek-ai/DeepSeek-R1-Distill-Llama-8B":
103+
configMapValues:
104+
VLLM_NO_USAGE_STATS: "1"
105+
DO_NOT_TRACK: "1"
106+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
107+
HF_HUB_DISABLE_XET: "1"
108+
extraCmdArgs:
109+
[
110+
"--dtype", "float16",
111+
"--block-size", "64",
112+
"--max-model-len", "8192",
113+
"--gpu-memory-utilization", "0.90",
114+
"--max-num-seqs", "128",
115+
"--enforce-eager",
116+
]
117+
tensor_parallel_size: "1"
118+
pipeline_parallel_size: "1"
119+
120+
"Qwen/Qwen2.5-7B-Instruct":
121+
configMapValues:
122+
VLLM_NO_USAGE_STATS: "1"
123+
DO_NOT_TRACK: "1"
124+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
125+
HF_HUB_DISABLE_XET: "1"
126+
extraCmdArgs:
127+
[
128+
"--dtype", "float16",
129+
"--block-size", "64",
130+
"--max-model-len", "8192",
131+
"--gpu-memory-utilization", "0.90",
132+
"--max-num-seqs", "128",
133+
"--enforce-eager",
134+
"--enable-auto-tool-choice",
135+
"--tool-call-parser", "hermes",
136+
]
137+
tensor_parallel_size: "1"
138+
pipeline_parallel_size: "1"
139+
140+
"Qwen/Qwen2.5-Coder-3B-Instruct":
141+
configMapValues:
142+
VLLM_NO_USAGE_STATS: "1"
143+
DO_NOT_TRACK: "1"
144+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
145+
HF_HUB_DISABLE_XET: "1"
146+
extraCmdArgs:
147+
[
148+
"--dtype", "float16",
149+
"--block-size", "64",
150+
"--max-model-len", "8192",
151+
"--gpu-memory-utilization", "0.90",
152+
"--max-num-seqs", "128",
153+
"--enforce-eager",
154+
"--enable-auto-tool-choice",
155+
"--tool-call-parser", "hermes",
156+
]
157+
tensor_parallel_size: "1"
158+
pipeline_parallel_size: "1"
159+
160+
"tiiuae/Falcon3-7B-Instruct":
161+
configMapValues:
162+
VLLM_NO_USAGE_STATS: "1"
163+
DO_NOT_TRACK: "1"
164+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
165+
HF_HUB_DISABLE_XET: "1"
166+
extraCmdArgs:
167+
[
168+
"--dtype", "float16",
169+
"--block-size", "64",
170+
"--max-model-len", "8192",
171+
"--gpu-memory-utilization", "0.90",
172+
"--max-num-seqs", "128",
173+
"--enforce-eager",
174+
]
175+
tensor_parallel_size: "1"
176+
pipeline_parallel_size: "1"
177+
178+
defaultModelConfigs:
179+
configMapValues:
180+
VLLM_NO_USAGE_STATS: "1"
181+
DO_NOT_TRACK: "1"
182+
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
183+
HF_HUB_DISABLE_XET: "1"
184+
extraCmdArgs:
185+
[
186+
"--dtype", "float16",
187+
"--block-size", "16",
188+
"--max-model-len", "8192",
189+
"--gpu-memory-utilization", "0.90",
190+
"--max-num-seqs", "128",
191+
"--enforce-eager",
192+
]
193+
tensor_parallel_size: "{{ .Values.tensor_parallel_size }}"
194+
pipeline_parallel_size: "{{ .Values.pipeline_parallel_size }}"

‎core/helm-charts/vllm/templates/deployment.yaml‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -126,11 +126,12 @@ spec:
126126
{{- if .Values.image.pullPolicy }}
127127
imagePullPolicy: {{ .Values.image.pullPolicy }}
128128
{{- end }}
129-
# command:
130-
# - /bin/bash
131-
# - -c
132-
# - |
133-
# python3 -m vllm.entrypoints.openai.api_server --dtype {{ .Values.d_type }} --model {{ .Values.LLM_MODEL_ID }} --port {{ .Values.port }} --tensor-parallel-size {{ .Values.tensor_parallel_size }} --block-size {{ .Values.block_size }} --max-model-len {{ .Values.max_model_len }} --disable-log-requests
129+
{{- if .Values.image.command }}
130+
command:
131+
{{- range .Values.image.command }}
132+
- {{ . | quote }}
133+
{{- end }}
134+
{{- end }}
134135
args:
135136
{{- $modelConfig := (index .Values.modelConfigs $modelName | default dict) }}
136137
{{- $modelArgs := $modelConfig.extraCmdArgs | default .Values.defaultModelConfigs.extraCmdArgs }}
@@ -195,7 +196,7 @@ spec:
195196
{{- end }}
196197
{{- else }}
197198
limits:
198-
habana.ai/gaudi: {{ .Values.tensor_parallel_size | default (index .Values.modelConfigs .Values.LLM_MODEL_ID | default dict).tensor_parallel_size | default .Values.defaultModelConfigs.tensor_parallel_size | quote}}
199+
{{ .Values.accelDeviceResource | default "habana.ai/gaudi" }}: {{ .Values.tensor_parallel_size | default (index .Values.modelConfigs .Values.LLM_MODEL_ID | default dict).tensor_parallel_size | default .Values.defaultModelConfigs.tensor_parallel_size | quote}}
199200
{{- end }}
200201
{{- end }}
201202

‎core/helm-charts/vllm/values.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ autoscaling:
1717

1818
# empty for CPU (longer latencies are tolerated before HPA scaling unaccelerated service)
1919
accelDevice: ""
20+
# Kubernetes resource name for the accelerator (e.g. habana.ai/gaudi, gpu.intel.com/xe)
21+
accelDeviceResource: ""
2022

2123
port: 2080
2224
shmSize: 1Gi

‎core/inference-stack-deploy.sh‎

100644100755
Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ NC=$(tput sgr0)
5050
# --keycloak-admin-password <password>: The Keycloak admin password.
5151
# --hugging-face-token <token>: The token for Huggingface.
5252
# --models <models>: The models to deploy (comma-separated list of model numbers or names).
53-
# --cpu-or-gpu <c/g>: Specify whether to run on CPU or GPU.
53+
# --device <cpu/hpu/xpu>: Specify the target device. 'cpu' for Xeon, 'hpu' for Gaudi GPU, 'xpu' for Intel Arc Battlemage GPU.
5454

5555
# Main Menu
5656

@@ -86,7 +86,7 @@ NC=$(tput sgr0)
8686

8787
# Example
8888
# To perform a fresh installation with specific parameters, you can run:
89-
# ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/to/cert.pem" --key-file "/path/to/key.pem" --keycloak-client-id "my-client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --cpu-or-gpu "g"
89+
# ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/to/cert.pem" --key-file "/path/to/key.pem" --keycloak-client-id "my-client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --device "hpu"
9090

9191
##############################################################################
9292

@@ -118,6 +118,7 @@ source "$SCRIPT_DIR/lib/cluster/drv-fw-update.sh"
118118
# Components deployment
119119
source "$SCRIPT_DIR/lib/components/kubernetes-setup.sh"
120120
source "$SCRIPT_DIR/lib/components/intel-base-operator.sh"
121+
source "$SCRIPT_DIR/lib/components/intel-gpu-plugin.sh"
121122
source "$SCRIPT_DIR/lib/components/ingress-controller.sh"
122123
source "$SCRIPT_DIR/lib/components/keycloak-controller.sh"
123124
source "$SCRIPT_DIR/lib/components/genai-gateway-controller.sh"
@@ -166,10 +167,11 @@ Options:
166167
--keycloak-admin-password <pw> Keycloak admin password.
167168
--hugging-face-token <token> Huggingface token.
168169
--models <models> Models to deploy (comma-separated).
169-
--cpu-or-gpu <c/g> Run on CPU (c) or GPU (g).
170+
--device <cpu/hpu/xpu> Target device: cpu (Xeon), hpu (Gaudi GPU), xpu (Intel Arc Battlemage GPU).
170171
171172
Examples:
172-
Setup cluster: ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/cert.pem" --key-file "/path/key.pem" --keycloak-client-id "client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --cpu-or-gpu "g"
173+
Setup cluster (Gaudi GPU): ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/cert.pem" --key-file "/path/key.pem" --keycloak-client-id "client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "1,3,5" --device "hpu"
174+
Setup cluster (BMG GPU): ./inference-stack-deploy.sh --cluster-url "https://example.com" --cert-file "/path/cert.pem" --key-file "/path/key.pem" --keycloak-client-id "client-id" --keycloak-admin-user "user" --keycloak-admin-password "password" --hugging-face-token "token" --models "36" --device "xpu"
173175
174176
###############################################################################
175177
EOF

‎core/inventory/hosts.yaml‎

Lines changed: 7 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,19 @@
11
all:
22
hosts:
3-
master:
4-
ansible_host: "{{ private_ip_control_plane_node }}"
5-
ansible_user: "username_of_user_running_automation"
6-
ansible_ssh_private_key_file: "/home/ubuntu/.ssh/id_rsa"
7-
worker1:
8-
ansible_host: "{{ private_ip_workload_node_1 }}"
9-
ansible_user: "username_of_user_running_automation"
10-
ansible_ssh_private_key_file: "/home/ubuntu/.ssh/id_rsa"
11-
worker2:
12-
ansible_host: "{{ private_ip_workload_node_2 }}"
13-
ansible_user: "username_of_user_running_automation"
14-
ansible_ssh_private_key_file: "/home/ubuntu/.ssh/id_rsa"
3+
master1:
4+
ansible_connection: local
5+
ansible_user: gta
6+
ansible_become: true
157
children:
168
kube_control_plane:
179
hosts:
18-
master:
10+
master1:
1911
kube_node:
2012
hosts:
21-
worker1:
22-
worker2:
13+
master1:
2314
etcd:
2415
hosts:
25-
master:
16+
master1:
2617
k8s_cluster:
2718
children:
2819
kube_control_plane:

‎core/inventory/inference-config.cfg‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,14 +2,15 @@ cluster_url=api.example.com
22
cert_file=~/certs/cert.pem
33
key_file=~/certs/key.pem
44
keycloak_client_id=my-client-id
5-
keycloak_admin_user=your-keycloak-admin-user
5+
keycloak_admin_user=admin
66
keycloak_admin_password=changeme
7-
hugging_face_token=your_hugging_face_token
8-
hugging_face_token_falcon3=your_hugging_face_token
9-
models=11
10-
cpu_or_gpu=cpu
7+
hugging_face_token=
8+
hugging_face_token_falcon3=
9+
models=36
10+
# Hardware selection: 'cpu' for Xeon CPU, 'hpu'/'gpu'/'gaudi2'/'gaudi3' for Gaudi GPU, 'xpu'/'bmg' for Intel Arc Battlemage GPU
11+
device=xpu
1112
vault_pass_code=place-holder-123
12-
deploy_kubernetes_fresh=on
13+
deploy_kubernetes_fresh=yes
1314
deploy_ingress_controller=on
1415
deploy_keycloak_apisix=on
1516
deploy_genai_gateway=off
@@ -20,4 +21,7 @@ deploy_istio=off
2021
uninstall_ceph=off
2122

2223
# Agentic AI Plugin
23-
deploy_agenticai_plugin=off
24+
deploy_agenticai_plugin=off
25+
http_proxy=""
26+
https_proxy=""
27+
no_proxy="localhost,127.0.0.1,10.96.0.0/12,10.244.0.0/16,192.168.0.0/16,.svc,.cluster.local,10.233.0.1,10.233.0.0/18,10.0.0.0/8,api.example.com"

0 commit comments

Comments
 (0)