Skip to content

Commit 4764291

Browse files
arpannookala-12alexsin368HarikaHarikaDev296
authored
cld2labs/llama-3.1-8b-instruct (#98)
* add model-deployment folder Signed-off-by: alexsin368 <alex.sin@intel.com> Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * feat: add llama-3.1-8b-instruct model card and deployment guide for Dell EI Signed-off-by: arpannookala-12 <ganesh.arpan.nookala@cloud2labs.com> Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * updated llama 3.1 8b instruct deployment.md Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * updating llama 3.1 8b instruct deployment.md Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * update llama 3.1 8b instruct deployment.md Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * Add model deployment troubleshooting guide for 504 gateway timeout Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * Remove em dashes from troubleshooting guide Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * update troubleshooting.md Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * Remove README.md from model-deployment folder Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * update troubleshooting.md Signed-off-by: Harika <harika.devulapally@cloud2labs.com> * Update third_party/Dell/model-deployment/troubleshooting.md Co-authored-by: alexsin368 <109180236+alexsin368@users.noreply.github.com> Signed-off-by: Harika <harika.devulapally@cloud2labs.com> --------- Signed-off-by: alexsin368 <alex.sin@intel.com> Signed-off-by: Harika <harika.devulapally@cloud2labs.com> Signed-off-by: arpannookala-12 <ganesh.arpan.nookala@cloud2labs.com> Co-authored-by: alexsin368 <alex.sin@intel.com> Co-authored-by: Harika <codewith3@gmail.com> Co-authored-by: Harika <harika.devulapally@cloud2labs.com> Co-authored-by: alexsin368 <109180236+alexsin368@users.noreply.github.com>
1 parent 7feabe0 commit 4764291

3 files changed

Lines changed: 244 additions & 0 deletions

File tree

‎third_party/Dell/model-deployment/llama-3.1-8b-instruct/deployment.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
## Step 1: Prerequisites to Deploy Llama-3.1-8B-Instruct Model on Xeon with Keycloak
2+
3+
Ensure the Enterprise Inference stack with Keycloak is already deployed before proceeding.
4+
5+
Edit `core/scripts/generate-token.sh` and set your values before sourcing it:
6+
7+
| Variable | Description |
8+
| ------------------------- | ------------------------------------------------------------------------ |
9+
| `BASE_URL` | Hostname of your cluster (e.g. `api.example.com`), without `https://` |
10+
| `KEYCLOAK_ADMIN_USERNAME` | Keycloak admin username |
11+
| `KEYCLOAK_PASSWORD` | Keycloak admin password |
12+
| `KEYCLOAK_CLIENT_ID` | Keycloak client ID configured during EI deployment |
13+
14+
Then run:
15+
16+
```bash
17+
export HUGGING_FACE_HUB_TOKEN="your_token_here"
18+
19+
cd ~/Enterprise-Inference
20+
source core/scripts/generate-token.sh
21+
```
22+
23+
This exports: `BASE_URL`, `KEYCLOAK_CLIENT_ID`, `KEYCLOAK_CLIENT_SECRET`, and `TOKEN`.
24+
25+
## Step 2: Deploy Llama-3.1-8B-Instruct Model
26+
27+
```bash
28+
helm install vllm-llama-8b ./core/helm-charts/vllm \
29+
--values ./core/helm-charts/vllm/xeon-values.yaml \
30+
--set LLM_MODEL_ID="meta-llama/Llama-3.1-8B-Instruct" \
31+
--set global.HUGGINGFACEHUB_API_TOKEN="$HUGGING_FACE_HUB_TOKEN" \
32+
--set ingress.enabled=true \
33+
--set ingress.secretname="${BASE_URL}" \
34+
--set ingress.host="${BASE_URL}" \
35+
--set oidc.client_id="$KEYCLOAK_CLIENT_ID" \
36+
--set oidc.client_secret="$KEYCLOAK_CLIENT_SECRET" \
37+
--set apisix.enabled=true \
38+
--set tensor_parallel_size="1" \
39+
--set pipeline_parallel_size="1"
40+
```
41+
42+
## Step 3: Verify the Deployment
43+
44+
```bash
45+
kubectl get pods
46+
kubectl get apisixroutes
47+
```
48+
49+
Expected Output:
50+
51+
```
52+
NAME READY STATUS RESTARTS
53+
keycloak-0 1/1 Running 0
54+
keycloak-postgresql-0 1/1 Running 0
55+
vllm-llama-8b-<hash>-<hash> 1/1 Running 0
56+
```
57+
58+
> Note: The pod name suffix `<hash>-<hash>` is auto-generated by Kubernetes and will differ on each deployment. Ensure all pods show `1/1 Running`.
59+
60+
```
61+
NAME HOSTS
62+
vllm-llama-8b-apisixroute api.example.com
63+
```
64+
65+
## Step 4: Test the Deployed Model
66+
67+
```bash
68+
curl -k https://${BASE_URL}/Llama-3.1-8B-Instruct-vllmcpu/v1/completions \
69+
-X POST \
70+
-H "Content-Type: application/json" \
71+
-H "Authorization: Bearer $TOKEN" \
72+
-d '{
73+
"model": "meta-llama/Llama-3.1-8B-Instruct",
74+
"prompt": "What is Deep Learning?",
75+
"max_tokens": 25,
76+
"temperature": 0
77+
}'
78+
```
79+
80+
If successful, the model will return a completion response.
81+
82+
## To undeploy the model
83+
84+
```bash
85+
helm uninstall vllm-llama-8b
86+
```
87+
88+
## Parameters
89+
90+
| Parameter | Description |
91+
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
92+
| `--set LLM_MODEL_ID="meta-llama/Llama-3.1-8B-Instruct"` | Defines the target model from **Hugging Face** to deploy. |
93+
| `--set global.HUGGINGFACEHUB_API_TOKEN="..."` | Authenticates access to gated or private Hugging Face models. Replace with your own secure token. |
94+
| `--set ingress.enabled=true` | Enables Kubernetes **Ingress** to expose the model service externally. |
95+
| `--set ingress.host="${BASE_URL}"` | Public hostname or FQDN for the inference endpoint (maps to your Ingress controller IP). |
96+
| `--set ingress.secretname="${BASE_URL}"` | Kubernetes **TLS Secret** used for HTTPS termination at the ingress layer. |
97+
| `--set oidc.client_id="..."` | Keycloak OIDC client ID used for token-based authentication. |
98+
| `--set oidc.client_secret="..."` | Keycloak OIDC client secret corresponding to the client ID. |
99+
| `--set apisix.enabled=true` | Enables **APISIX** as the API gateway for routing and authentication. |
100+
| `--set tensor_parallel_size="1"` | Number of tensor parallel workers. Set to the number of available Gaudi cards per node. |
101+
| `--set pipeline_parallel_size="1"` | Number of pipeline parallel stages. Typically `1` for single-node deployments. |

‎third_party/Dell/model-deployment/llama-3.1-8b-instruct/model-card.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Llama-3.1-8B-Instruct
2+
3+
This model uses Llama-3.1-8B-Instruct, a 8 billion-parameter instruction-tuned model from Meta Platforms, Inc. (Meta AI). It belongs to the Llama 3.1 model family and is optimized for multilingual dialogue, code tasks, and general instruction-following across a large context window.
4+
5+
For full details including model specifications, licensing, intended use, safety guidance, and example prompts, please visit the official Hugging Face page: **Official Hugging Face Page**
6+
7+
https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
8+
9+
This model provides inference services only; weights are hosted by Hugging Face under Meta’s license.
10+
11+
Ensure compliance with the Llama 2 Community License Agreement before using this model.
12+
13+
### Model Attribution
14+
15+
**Developer:** Meta Platforms, Inc. (Meta AI)
16+
17+
**purpose:** Instruction-following model for dialogue, code generation/completion, multilingual tasks
18+
19+
**Sizes/Variants:** 8 B parameters (instruction tuned); the Llama 3.1 family also includes 70 B and 405 B parameter variants
20+
21+
**Modalities:** Text input → Text (including code) output
22+
23+
**Parameter Size:** ~8 billion
24+
25+
**Max Context:** Up to ~128 k tokens (for the 3.1 family)
26+
27+
**License:** Llama 3.1 Community License (custom commercial license)
28+
29+
**Minimum required CPU Cores:** 157
30+
31+
**Minimum required PCIe Cards:** 1
32+
33+
### Usage Notice
34+
35+
**By using this model, you agree that:**
36+
37+
- Inputs and outputs are processed through Llama-3.1-8B-Instruct under Meta’s Community License.
38+
- You will comply with Meta’s licensing terms, including restrictions on redistribution, commercial scale-use thresholds, attribution (“Built with Llama”), and acceptable use policy.
39+
- All generated content (text or code) must be reviewed for accuracy, compliance, and safety before deployment.
40+
- The model should not be used for generating malicious content, disallowed content, or automating decisions in high-risk or regulated systems without appropriate safeguards.
41+
42+
### Intended Applications
43+
44+
- Instruction-following chatbots and assistants (multilingual)
45+
- Code generation, completion, refactoring tasks (Python, Java, JavaScript, etc.)
46+
- Multilingual support (English, German, French, Italian, Portuguese, Hindi, Spanish, Thai) and potentially others with fine-tuning.
47+
- Large-context tasks: summarization of long documents, dialog over long history, RAG (retrieve-and-generate) over extended context.
48+
- Research, prototyping, and commercial workflows (subject to license terms).
49+
50+
### Limitations
51+
52+
- Although capable, the 8 B size still has trade-offs: accuracy and depth of reasoning may lag behind much larger models.
53+
- As with all large language models, risk of hallucinations (incorrect statements), biases, or unsafe outputs remains.
54+
- The custom license restricts certain uses (e.g., if your product has > 700 million monthly active users you may require a special license) as described in Meta’s license terms.
55+
- The model does not guarantee tool-use, vision/multimodal input (unless you fine-tune or wrap appropriately) – it is primarily text → text.
56+
- Running it efficiently still requires significant hardware/resources for full context and best performance
57+
58+
### References
59+
60+
“Introducing Llama 3.1: Our most capable models to date”. https://ai.meta.com/blog/meta-llama-3-1
61+
62+
Hugging Face Model Card: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
63+
64+
Meta Llama GitHub Repository & License Details. https://github.com/meta-llama/llama3

‎third_party/Dell/model-deployment/troubleshooting.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Troubleshooting Guide
2+
3+
This section provides common issues observed when running inference against models deployed via Helm commands on Intel® AI for Enterprise Inference, along with step-by-step resolutions.
4+
5+
**Issues:**
6+
1. [Gateway Timeout (504) on Inference Requests](#1-gateway-timeout-504-on-inference-requests)
7+
8+
---
9+
10+
### 1. Gateway Timeout (504) on Inference Requests
11+
12+
**Context:** Model deployed via Helm commands. Inference request sent through the ingress stack (ingress-nginx -> APISIX -> vLLM service).
13+
14+
**Error:** Inference requests return `504 Gateway Timeout` after 60 seconds:
15+
16+
```
17+
"POST /<model-name>/v1/completions HTTP/2.0" 504
18+
upstream timed out (110: Operation timed out) ... 60.001
19+
```
20+
21+
**Cause:**
22+
23+
Model inference using CPU-based configurations may encounter performance variances where processing times exceed the default 60-second upstream timeout enforced by ingress-nginx and APISIX, resulting in a timeout error.
24+
25+
**Fix:**
26+
27+
**Step 1 - Increase the nginx ingress timeout**
28+
29+
Apply to both the `default` and `auth-apisix` namespaces. To find ingress names:
30+
31+
```bash
32+
kubectl get ingress -A | grep <model-name>
33+
```
34+
35+
Then annotate **EACH** ingress:
36+
37+
```bash
38+
kubectl annotate ingress <ingress-name> -n <namespace> \
39+
nginx.ingress.kubernetes.io/proxy-read-timeout="300" \
40+
nginx.ingress.kubernetes.io/proxy-send-timeout="300" \
41+
nginx.ingress.kubernetes.io/proxy-connect-timeout="60" \
42+
--overwrite
43+
```
44+
45+
**Step 2 - Increase the APISIX route timeout**
46+
47+
To find the route name:
48+
49+
```bash
50+
kubectl get apisixroute -n auth-apisix | grep <model-name>
51+
```
52+
53+
Edit the route:
54+
55+
```bash
56+
kubectl edit apisixroute <route-name> -n auth-apisix
57+
```
58+
59+
Update the timeout section under the route:
60+
61+
```yaml
62+
spec:
63+
http:
64+
- name: <route-name>
65+
timeout:
66+
connect: 60s
67+
send: 300s
68+
read: 300s
69+
```
70+
71+
**Verification:**
72+
73+
Re-run the inference request and confirm a `200 OK` response is returned within the new timeout window.
74+
75+
**Notes:**
76+
77+
- The nginx ingress annotation takes effect immediately; no pod restart required.
78+
- GPU-based deployments typically achieve significantly higher throughput, making this timeout adjustment rarely necessary compared to CPU configurations.
79+
- If requests still time out after increasing both timeouts, reduce `max_tokens` in the request payload to limit response length.

0 commit comments

Comments
 (0)