Skip to content

Commit e53bf11

Browse files
authored
update readme for llamacpp (#99)
update readme for llamacpp Signed-off-by: AhmedSeemalK <ahmed.seemal@intel.com>
1 parent 24e12af commit e53bf11

1 file changed

Lines changed: 37 additions & 3 deletions

File tree

blueprints/llamacpp/README.md

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,42 @@
1-
# llama.cpp — Intel oneAPI with VMWare Foundation
1+
# llama.cpp — Intel oneAPI with VMware Foundation
22

3-
Runs [llama.cpp](https://github.com/ggml-org/llama.cpp) `llama-server` built with Intel oneAPI compilers and oneMKL BLAS for optimized CPU inference.
3+
## Overview
44

5-
> **Note:** You can use any GGUF model from Hugging Face. The examples below use specific models, but you can replace them with any GGUF model by setting the appropriate environment variables.
5+
This guide covers deploying [llama.cpp](https://github.com/ggml-org/llama.cpp) `llama-server` as a Docker container built with Intel oneAPI compilers and oneMKL BLAS for optimized CPU inference on Intel Xeon systems
6+
7+
llama.cpp exposes an OpenAI-compatible REST API, making it a drop-in inference backend for applications that consume `/v1/completions` and `/v1/chat/completions` endpoints.
8+
9+
> **Note:** You can use any GGUF model from Hugging Face. The examples below use specific models, but you can replace them with GGUF model by setting the appropriate environment variables.
10+
11+
---
12+
13+
## Prerequisites
14+
15+
### Hardware Requirements
16+
17+
For best performance, Intel® Xeon® 6 Processors are recommended.
18+
19+
| Component | Specification |
20+
|--------------|-----------------------------------------------------------------------------------------------|
21+
| CPU Cores | Minimum 8 cores recommended; 16+ cores for optimal inference throughput |
22+
| RAM Memory | Minimum 32 GB; 64 GB or more recommended for larger models |
23+
| Disk Space | Minimum 100 GB free (SSD recommended); model files range from ~1 GB to 20+ GB per GGUF file |
24+
25+
### Software Requirements
26+
27+
| Tool | Version | Notes |
28+
|-------------------|------------------|--------------------------------------------------------------|
29+
| Docker Engine | v20.10 or later | [Install Docker](https://docs.docker.com/engine/install/) |
30+
| HuggingFace Token || Required to download gated or private models (optional for public GGUF repos) |
31+
| Internet Access || Needed on first run to pull the GGUF model from Hugging Face |
32+
33+
Verify Docker is running before proceeding:
34+
35+
```bash
36+
docker info
37+
```
38+
39+
---
640

741
## Build & Run
842

0 commit comments

Comments
 (0)