Skip to content

Commit be1cb51

Browse files
arpannookala-12Harika
arpannookala-12
and
Harika
authored
cld2labs/Mistral-7B-v0.3 (#93)
* feat: add Mistral-7B-Instruct-v0.3 model card and deployment guide for Dell EI Signed-off-by: arpannookala-12 <ganesh.arpan.nookala@cloud2labs.com> * feat: add Mistral-7B-v0.3 model card and deployment guide for Dell EI Signed-off-by: arpannookala-12 <ganesh.arpan.nookala@cloud2labs.com> * update mistral 7b 0.3 deployment.md * Remove README.md from model-deployment folder --------- Signed-off-by: arpannookala-12 <ganesh.arpan.nookala@cloud2labs.com> Co-authored-by: Harika <codewith3@gmail.com>
1 parent 89adb2b commit be1cb51

2 files changed

Lines changed: 177 additions & 0 deletions

File tree

‎third_party/Dell/model-deployment/Mistral-7B-v0.3/deployment.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
## Step 1: Prerequisites to Deploy Mistral-7B-v0.3 Model on Xeon with Keycloak
2+
3+
Ensure the Enterprise Inference stack with Keycloak is already deployed before proceeding.
4+
5+
Edit `core/scripts/generate-token.sh` and set your values before sourcing it:
6+
7+
| Variable | Description |
8+
| ------------------------- | ------------------------------------------------------------------------ |
9+
| `BASE_URL` | Hostname of your cluster (e.g. `api.example.com`), without `https://` |
10+
| `KEYCLOAK_ADMIN_USERNAME` | Keycloak admin username |
11+
| `KEYCLOAK_PASSWORD` | Keycloak admin password |
12+
| `KEYCLOAK_CLIENT_ID` | Keycloak client ID configured during EI deployment |
13+
14+
Then run:
15+
16+
```bash
17+
export HUGGING_FACE_HUB_TOKEN="your_token_here"
18+
19+
cd ~/Enterprise-Inference
20+
source core/scripts/generate-token.sh
21+
```
22+
23+
This exports: `BASE_URL`, `KEYCLOAK_CLIENT_ID`, `KEYCLOAK_CLIENT_SECRET`, and `TOKEN`.
24+
25+
## Step 2: Deploy Mistral-7B-v0.3 Model
26+
27+
```bash
28+
helm install mistral-7b-v3 ./core/helm-charts/vllm \
29+
--values ./core/helm-charts/vllm/xeon-values.yaml \
30+
--set LLM_MODEL_ID="mistralai/Mistral-7B-v0.3" \
31+
--set global.HUGGINGFACEHUB_API_TOKEN="$HUGGING_FACE_HUB_TOKEN" \
32+
--set ingress.enabled=true \
33+
--set ingress.secretname="${BASE_URL}" \
34+
--set ingress.host="${BASE_URL}" \
35+
--set oidc.client_id="$KEYCLOAK_CLIENT_ID" \
36+
--set oidc.client_secret="$KEYCLOAK_CLIENT_SECRET" \
37+
--set apisix.enabled=true \
38+
--set tensor_parallel_size="1" \
39+
--set pipeline_parallel_size="1"
40+
```
41+
42+
## Step 3: Verify the Deployment
43+
44+
```bash
45+
kubectl get pods
46+
kubectl get apisixroutes
47+
```
48+
49+
Expected Output:
50+
51+
```
52+
NAME READY STATUS RESTARTS
53+
keycloak-0 1/1 Running 0
54+
keycloak-postgresql-0 1/1 Running 0
55+
mistral-7b-v3-<hash>-<hash> 1/1 Running 0
56+
```
57+
58+
> Note: The pod name suffix `<hash>-<hash>` is auto-generated by Kubernetes and will differ on each deployment. Ensure all pods show `1/1 Running`.
59+
60+
```
61+
NAME HOSTS
62+
mistral-7b-v3-apisixroute api.example.com
63+
```
64+
65+
## Step 4: Test the Deployed Model
66+
67+
```bash
68+
curl -k https://${BASE_URL}/Mistral-7B-v0.3-vllmcpu/v1/completions \
69+
-X POST \
70+
-H "Content-Type: application/json" \
71+
-H "Authorization: Bearer $TOKEN" \
72+
-d '{
73+
"model": "mistralai/Mistral-7B-v0.3",
74+
"prompt": "What is Deep Learning?",
75+
"max_tokens": 25,
76+
"temperature": 0
77+
}'
78+
```
79+
80+
If successful, the model will return a completion response.
81+
82+
## To undeploy the model
83+
84+
```bash
85+
helm uninstall mistral-7b-v3
86+
```
87+
88+
## Parameters
89+
90+
| Parameter | Description |
91+
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------- |
92+
| `--set LLM_MODEL_ID="mistralai/Mistral-7B-v0.3"` | Defines the target model from **Hugging Face** to deploy. |
93+
| `--set global.HUGGINGFACEHUB_API_TOKEN="..."` | Authenticates access to gated or private Hugging Face models. Replace with your own secure token. |
94+
| `--set ingress.enabled=true` | Enables Kubernetes **Ingress** to expose the model service externally. |
95+
| `--set ingress.host="${BASE_URL}"` | Public hostname or FQDN for the inference endpoint (maps to your Ingress controller IP). |
96+
| `--set ingress.secretname="${BASE_URL}"` | Kubernetes **TLS Secret** used for HTTPS termination at the ingress layer. |
97+
| `--set oidc.client_id="..."` | Keycloak OIDC client ID used for token-based authentication. |
98+
| `--set oidc.client_secret="..."` | Keycloak OIDC client secret corresponding to the client ID. |
99+
| `--set apisix.enabled=true` | Enables **APISIX** as the API gateway for routing and authentication. |
100+
| `--set tensor_parallel_size="1"` | Number of tensor parallel workers. Set to the number of available CPUs/GPUs per node. |
101+
| `--set pipeline_parallel_size="1"` | Number of pipeline parallel stages. Typically `1` for single-node deployments. |

‎third_party/Dell/model-deployment/Mistral-7B-v0.3/model-card.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
## Mistral 7B v0.3
2+
3+
This model uses **Mistral-7B v0.3**, a next-generation 7-billion-parameter transformer language model developed by **Mistral AI**. It represents a compact, efficient, and high-performance LLM architecture optimized for general-purpose text generation, research, and downstream fine-tuning. Compared to earlier releases, the v0.3 iteration integrates tokenizer improvements, extended context length support, and architectural refinements for stronger performance and interoperability in modern LLM ecosystems.
4+
5+
For full details including model specifications, licensing, intended use, and technical documentation, please visit the official Hugging Face page: **Official Hugging Face Page**
6+
7+
https://huggingface.co/mistralai/Mistral-7B-v0.3
8+
9+
---
10+
11+
### Model Attribution
12+
13+
**Developer:** Mistral AI
14+
15+
**Purpose:** Foundation model for general NLP tasks, downstream fine-tuning, and integration into custom pipelines
16+
17+
**Sizes / Variants:**
18+
7B (≈ 7 billion parameters)
19+
20+
**Modalities:**
21+
Text → Text (autoregressive language modeling)
22+
23+
**Parameter Size:**
24+
~7 billion
25+
26+
**Max Context:**
27+
Extended context window supported (exact length may depend on inference backend and configuration)
28+
29+
**License:**
30+
Apache 2.0 (open-weight release)
31+
32+
**Minimum Required PCIe Cards:**
33+
1–2 (varies by precision, quantization, and inference framework)
34+
35+
---
36+
37+
### Usage Notice
38+
39+
By using this model, you agree that:
40+
41+
- Inputs and outputs are processed via the Mistral-7B v0.3 model and you accept its licensing terms under Apache 2.0.
42+
- Model outputs must be reviewed for accuracy, suitability, and safety before use in commercial or production contexts.
43+
- This base model does not include alignment or instruction-fine-tuning, and therefore may produce literal, unfiltered, or undesired content without safety conditioning.
44+
- You remain responsible for monitoring, filtering, and enforcing compliance, especially in sensitive, regulated, or user-facing deployments.
45+
46+
---
47+
48+
### Intended Applications
49+
50+
- Research in transformer and LLM architectures
51+
- Pre-training or continued training for domain-specific LLMs
52+
- Fine-tuning for instruction following, chat roles, code, or domain tasks
53+
- General-purpose text generation and language modeling
54+
- Embedding into autonomous or semi-autonomous agents with external alignment layers
55+
- Experimental or academic benchmarking on open-weight LLMs
56+
57+
---
58+
59+
### Limitations
60+
61+
- As a base model, it lacks instruction tuning and safety alignment, making outputs potentially unstructured or unsafe without further processing.
62+
- May generate hallucinated, biased, or factually incorrect content; human validation is recommended.
63+
- Safety-critical and regulated use cases require external safeguards, filtering, or moderation systems.
64+
- Operational performance varies with context length, quantization, and hardware backend; optimization may be required for real-time workloads.
65+
66+
---
67+
68+
### References
69+
70+
- Official Model Card on Hugging Face: https://huggingface.co/mistralai/Mistral-7B-v0.3
71+
72+
- Open model documentation by Mistral AI.
73+
https://docs.mistral.ai/getting-started/models
74+
75+
- “Mistral 7B” announcement blog post.
76+
https://mistral.ai/news/announcing-mistral-7b

0 commit comments

Comments
 (0)