|
1 | | -# llama.cpp — Intel oneAPI with VMWare Foundation |
| 1 | +# llama.cpp — Intel oneAPI with VMware Foundation |
2 | 2 |
|
3 | | -Runs [llama.cpp](https://github.com/ggml-org/llama.cpp) `llama-server` built with Intel oneAPI compilers and oneMKL BLAS for optimized CPU inference. |
| 3 | +## Overview |
4 | 4 |
|
5 | | -> **Note:** You can use any GGUF model from Hugging Face. The examples below use specific models, but you can replace them with any GGUF model by setting the appropriate environment variables. |
| 5 | +This guide covers deploying [llama.cpp](https://github.com/ggml-org/llama.cpp) `llama-server` as a Docker container built with Intel oneAPI compilers and oneMKL BLAS for optimized CPU inference on Intel Xeon systems |
| 6 | + |
| 7 | +llama.cpp exposes an OpenAI-compatible REST API, making it a drop-in inference backend for applications that consume `/v1/completions` and `/v1/chat/completions` endpoints. |
| 8 | + |
| 9 | +> **Note:** You can use any GGUF model from Hugging Face. The examples below use specific models, but you can replace them with GGUF model by setting the appropriate environment variables. |
| 10 | +
|
| 11 | +--- |
| 12 | + |
| 13 | +## Prerequisites |
| 14 | + |
| 15 | +### Hardware Requirements |
| 16 | + |
| 17 | +For best performance, Intel® Xeon® 6 Processors are recommended. |
| 18 | + |
| 19 | +| Component | Specification | |
| 20 | +|--------------|-----------------------------------------------------------------------------------------------| |
| 21 | +| CPU Cores | Minimum 8 cores recommended; 16+ cores for optimal inference throughput | |
| 22 | +| RAM Memory | Minimum 32 GB; 64 GB or more recommended for larger models | |
| 23 | +| Disk Space | Minimum 100 GB free (SSD recommended); model files range from ~1 GB to 20+ GB per GGUF file | |
| 24 | + |
| 25 | +### Software Requirements |
| 26 | + |
| 27 | +| Tool | Version | Notes | |
| 28 | +|-------------------|------------------|--------------------------------------------------------------| |
| 29 | +| Docker Engine | v20.10 or later | [Install Docker](https://docs.docker.com/engine/install/) | |
| 30 | +| HuggingFace Token | — | Required to download gated or private models (optional for public GGUF repos) | |
| 31 | +| Internet Access | — | Needed on first run to pull the GGUF model from Hugging Face | |
| 32 | + |
| 33 | +Verify Docker is running before proceeding: |
| 34 | + |
| 35 | +```bash |
| 36 | +docker info |
| 37 | +``` |
| 38 | + |
| 39 | +--- |
6 | 40 |
|
7 | 41 | ## Build & Run |
8 | 42 |
|
|
0 commit comments