Skip to content

Commit 9e3c97d

Browse files
author
Harika
committed
updated llama 3.1 8b instruct deployment.md
1 parent d68922a commit 9e3c97d

1 file changed

Lines changed: 101 additions & 71 deletions

File tree

  • third_party/Dell/model-deployment/llama-3.1-8b-instruct

‎third_party/Dell/model-deployment/llama-3.1-8b-instruct/deployment.md‎

Lines changed: 101 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -1,71 +1,101 @@
1-
# Deployed with EI Version-1.2
2-
3-
## Step 1: Set Environment Variables
4-
5-
```bash
6-
# Export Hugging Face token
7-
export HUGGING_FACE_HUB_TOKEN="your_token_here"
8-
9-
# Set your base URL and API token
10-
export BASE_HOST="your-cluster-url"
11-
12-
#generate keyclock token
13-
export BASE_URL="https://your-cluster-url"
14-
export KEYCLOAK_CLIENT_ID=api
15-
export KEYCLOAK_CLIENT_SECRET="your keyclock client secret"
16-
export TOKEN=$(curl -k -X POST $BASE_URL/token -H 'Content-Type: application/x-www-form-urlencoded' -d "grant_type=client_credentials&client_id=${KEYCLOAK_CLIENT_ID}&client_secret=${KEYCLOAK_CLIENT_SECRET}" | jq -r .access_token)
17-
```
18-
19-
## Step 2: Deploy Llama-3.1-8B-Instruct Model
20-
21-
```bash
22-
helm install vllm-llama-8b ./core/helm-charts/vllm \
23-
--values ./core/helm-charts/vllm/gaudi3-values.yaml \
24-
--set LLM_MODEL_ID="meta-llama/Llama-3.1-8B-Instruct" \
25-
--set global.HUGGINGFACEHUB_API_TOKEN="$HUGGING_FACE_HUB_TOKEN" \
26-
--set ingress.enabled=true \
27-
--set ingress.host="$BASE_HOST" \
28-
--set ingress.secretname="$BASE_HOST" \
29-
--force
30-
```
31-
32-
## Step 3: Test the Deployed Model
33-
34-
```bash
35-
curl -k ${BASE_URL}/Llama-3.1-8B-Instruct/v1/completions \
36-
-X POST \
37-
-H "Content-Type: application/json" \
38-
-H "Authorization: Bearer $TOKEN" \
39-
-d '{
40-
"model": "meta-llama/Llama-3.1-8B-Instruct",
41-
"prompt": "What is Deep Learning?",
42-
"max_tokens": 25,
43-
"temperature": 0
44-
}'
45-
```
46-
## To undeploy the model
47-
48-
```bash
49-
helm uninstall vllm-llama-8b
50-
```
51-
## Parameters
52-
53-
| Parameter | Description |
54-
| --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
55-
| `--set LLM_MODEL_ID="meta-llama/Llama-3.1-8B-Instruct"` | Defines the target model from **Hugging Face** to deploy. |
56-
| `--set global.HUGGINGFACEHUB_API_TOKEN="..."` | Authenticates access to gated or private Hugging Face models. Replace with your own secure token. |
57-
| `--set ingress.enabled=true` | Enables Kubernetes **Ingress** to expose the model service externally. |
58-
| `--set ingress.host="replace-ingress"` | Public hostname or FQDN for the inference endpoint (maps to your Ingress controller IP). |
59-
| `--set ingress.secretname="replace-secret"` | Kubernetes **TLS Secret** used for HTTPS termination at the ingress layer. |
60-
61-
62-
63-
64-
65-
66-
67-
68-
69-
70-
71-
1+
## Step 1: Prerequisites to Deploy Llama-3.1-8B-Instruct Model on Gaudi with Keycloak
2+
3+
Ensure the Enterprise Inference stack with Keycloak is already deployed before proceeding.
4+
5+
Edit `core/scripts/generate-token.sh` and set your values before sourcing it:
6+
7+
| Variable | Description |
8+
| ------------------------- | ------------------------------------------------------------------------ |
9+
| `BASE_URL` | Hostname of your cluster (e.g. `api.example.com`), without `https://` |
10+
| `KEYCLOAK_ADMIN_USERNAME` | Keycloak admin username |
11+
| `KEYCLOAK_PASSWORD` | Keycloak admin password |
12+
| `KEYCLOAK_CLIENT_ID` | Keycloak client ID configured during EI deployment |
13+
14+
Then run:
15+
16+
```bash
17+
export HUGGING_FACE_HUB_TOKEN="your_token_here"
18+
19+
cd ~/Enterprise-Inference
20+
source core/scripts/generate-token.sh
21+
```
22+
23+
This exports: `BASE_URL`, `KEYCLOAK_CLIENT_ID`, `KEYCLOAK_CLIENT_SECRET`, and `TOKEN`.
24+
25+
## Step 2: Deploy Llama-3.1-8B-Instruct Model
26+
27+
```bash
28+
helm install vllm-llama-8b ./core/helm-charts/vllm \
29+
--values ./core/helm-charts/vllm/gaudi3-values.yaml \
30+
--set LLM_MODEL_ID="meta-llama/Llama-3.1-8B-Instruct" \
31+
--set global.HUGGINGFACEHUB_API_TOKEN="$HUGGING_FACE_HUB_TOKEN" \
32+
--set ingress.enabled=true \
33+
--set ingress.secretname="${BASE_URL}" \
34+
--set ingress.host="${BASE_URL}" \
35+
--set oidc.client_id="$KEYCLOAK_CLIENT_ID" \
36+
--set oidc.client_secret="$KEYCLOAK_CLIENT_SECRET" \
37+
--set apisix.enabled=true \
38+
--set tensor_parallel_size="1" \
39+
--set pipeline_parallel_size="1"
40+
```
41+
42+
## Step 3: Verify the Deployment
43+
44+
```bash
45+
kubectl get pods
46+
kubectl get apisixroutes
47+
```
48+
49+
Expected Output:
50+
51+
```
52+
NAME READY STATUS RESTARTS
53+
keycloak-0 1/1 Running 0
54+
keycloak-postgresql-0 1/1 Running 0
55+
vllm-llama-8b-<hash>-<hash> 1/1 Running 0
56+
```
57+
58+
> Note: The pod name suffix `<hash>-<hash>` is auto-generated by Kubernetes and will differ on each deployment. Ensure all pods show `1/1 Running`.
59+
60+
```
61+
NAME HOSTS
62+
vllm-llama-8b-apisixroute api.example.com
63+
```
64+
65+
## Step 4: Test the Deployed Model
66+
67+
```bash
68+
curl -k https://${BASE_URL}/Llama-3.1-8B-Instruct/v1/completions \
69+
-X POST \
70+
-H "Content-Type: application/json" \
71+
-H "Authorization: Bearer $TOKEN" \
72+
-d '{
73+
"model": "meta-llama/Llama-3.1-8B-Instruct",
74+
"prompt": "What is Deep Learning?",
75+
"max_tokens": 25,
76+
"temperature": 0
77+
}'
78+
```
79+
80+
If successful, the model will return a completion response.
81+
82+
## To undeploy the model
83+
84+
```bash
85+
helm uninstall vllm-llama-8b
86+
```
87+
88+
## Parameters
89+
90+
| Parameter | Description |
91+
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
92+
| `--set LLM_MODEL_ID="meta-llama/Llama-3.1-8B-Instruct"` | Defines the target model from **Hugging Face** to deploy. |
93+
| `--set global.HUGGINGFACEHUB_API_TOKEN="..."` | Authenticates access to gated or private Hugging Face models. Replace with your own secure token. |
94+
| `--set ingress.enabled=true` | Enables Kubernetes **Ingress** to expose the model service externally. |
95+
| `--set ingress.host="${BASE_URL}"` | Public hostname or FQDN for the inference endpoint (maps to your Ingress controller IP). |
96+
| `--set ingress.secretname="${BASE_URL}"` | Kubernetes **TLS Secret** used for HTTPS termination at the ingress layer. |
97+
| `--set oidc.client_id="..."` | Keycloak OIDC client ID used for token-based authentication. |
98+
| `--set oidc.client_secret="..."` | Keycloak OIDC client secret corresponding to the client ID. |
99+
| `--set apisix.enabled=true` | Enables **APISIX** as the API gateway for routing and authentication. |
100+
| `--set tensor_parallel_size="1"` | Number of tensor parallel workers. Set to the number of available Gaudi cards per node. |
101+
| `--set pipeline_parallel_size="1"` | Number of pipeline parallel stages. Typically `1` for single-node deployments. |

0 commit comments

Comments
 (0)