Skip to content

README.md

Latest commit

 

History

History
216 lines (161 loc) · 10 KB

README.md

File metadata and controls

216 lines (161 loc) · 10 KB

LLM Text Generation Microservice

This microservice, designed for Language Model Inference (LLM), processes input consisting of a query string and associated reranked documents. It constructs a prompt based on the query and documents, which is then used to perform inference with a large language model. The service delivers the inference results as output.

A prerequisite for using this microservice is that users must have a LLM text generation service (etc., TGI, vLLM) already running. Users need to set the LLM service's endpoint into an environment variable. The microservice utilizes this endpoint to create an LLM object, enabling it to communicate with the LLM service for executing language model operations.

Overall, this microservice offers a streamlined way to integrate large language model inference into applications, requiring minimal setup from the user beyond initiating a TGI/vLLM service and configuring the necessary environment variables. This allows for the seamless processing of queries and documents to generate intelligent, context-aware responses.

Table of Contents

  1. Validated LLM Models
  2. Start Microservice
  3. Consume Microservice

Validated LLM Models

Model TGI-Gaudi vLLM-CPU vLLM-Gaudi OVMS Optimum-Habana SGLANG-CPU
Intel/neural-chat-7b-v3-3 ✓ ✓ ✓ ✓ ✓ -
meta-llama/Llama-2-7b-chat-hf ✓ ✓ ✓ ✓ ✓ ✓
meta-llama/Llama-2-70b-chat-hf ✓ - ✓ - ✓ ✓
meta-llama/Meta-Llama-3-8B-Instruct ✓ ✓ ✓ ✓ ✓ ✓
meta-llama/Meta-Llama-3-70B-Instruct ✓ - ✓ - ✓ ✓
Phi-3 ✗ Limit 4K Limit 4K Limit 4K ✓ -
Phi-4 ✗ ✗ ✗ ✗ ✓ -
deepseek-ai/DeepSeek-R1-Distill-Llama-8B ✓ - ✓ - ✓ -
deepseek-ai/DeepSeek-R1-Distill-Llama-70B ✓ - ✓ - ✓ -
deepseek-ai/DeepSeek-R1-Distill-Qwen-14B ✓ - ✓ - ✓ -
deepseek-ai/DeepSeek-R1-Distill-Qwen-32B ✓ - ✓ - ✓ -
mistralai/Mistral-Small-24B-Instruct-2501 ✓ - ✓ - ✓ -
mistralai/Mistral-Large-Instruct-2411 ✗ - ✓ - ✓ -
[meta-llama/Llama-4-Scout-17B-16E-Instruct] - - - - - ✓
[meta-llama/Llama-4-Maverick-17B-128E-Instruct] - - - - - ✓

System Requirements for LLM Models

Model Minimum Number of Gaudi Cards
Intel/neural-chat-7b-v3-3 1
meta-llama/Llama-2-7b-chat-hf 1
meta-llama/Llama-2-70b-chat-hf 2
meta-llama/Meta-Llama-3-8B-Instruct 1
meta-llama/Meta-Llama-3-70B-Instruct 2
Phi-3 -
Phi-4 -
deepseek-ai/DeepSeek-R1-Distill-Llama-8B 1
deepseek-ai/DeepSeek-R1-Distill-Llama-70B 8
deepseek-ai/DeepSeek-R1-Distill-Qwen-14B 2
deepseek-ai/DeepSeek-R1-Distill-Qwen-32B 4
mistralai/Mistral-Small-24B-Instruct-2501 1
mistralai/Mistral-Large-Instruct-2411 4

Note: Detailed hardware requirements will be provided soon.

Start Microservice

Support integrations

In this microservices, we have supported following backend LLM service as integrations, we will include TGI/vLLM/Ollama in this readme, for others, please refer to corresponding readmes.

Clone OPEA GenAIComps

Clone Repository

git clone https://github.com/opea-project/GenAIComps.git

export OPEA_GENAICOMPS_ROOT=$(pwd)/GenAIComps

Prerequisites

Build Docker Image

Backend LLM Image

  • For vLLM, refer to the vLLM Build Guide to build the Docker images first.

  • TGI and Ollama are not needed.

Build TextGen Microservice Image

cd ${OPEA_GENAICOMPS_ROOT}

docker build \
  --build-arg https_proxy=$https_proxy \
  --build-arg http_proxy=$http_proxy \
  -t opea/llm-textgen:latest \
  -f comps/llms/src/text-generation/Dockerfile .

Run Docker Service

You can start the service using either the CLI or Docker Compose. The compose_text-generation.yaml file will automatically start both endpoint and the microservice docker.

Setup Environment Variables

In order to start services, you need to setup the following environment variables first.

export LLM_ENDPOINT_PORT=8008
export TEXTGEN_PORT=9000
export host_ip=${host_ip}
export HF_TOKEN=${HF_TOKEN}
export LLM_ENDPOINT="http://${host_ip}:${LLM_ENDPOINT_PORT}"
export LLM_MODEL_ID="Intel/neural-chat-7b-v3-3"

Option A: Run Docker with CLI

1: Start backend LLM service (TGI, vLLM, Ollama).

2: Start TextGen Microservice:

export LLM_COMPONENT_NAME="OpeaTextGenService"
docker run \
  --name="llm-textgen-server" \
  -p $TEXTGEN_PORT:9000 \
  --ipc=host \
  -e http_proxy=$http_proxy \
  -e https_proxy=$https_proxy \
  -e no_proxy=${no_proxy} \
  -e LLM_ENDPOINT=$LLM_ENDPOINT \
  -e HF_TOKEN=$HF_TOKEN \
  -e LLM_MODEL_ID=$LLM_MODEL_ID \
  -e LLM_COMPONENT_NAME=$LLM_COMPONENT_NAME \
  opea/llm-textgen:latest

Option B: Run with Docker Compose

Set service_name to match backend service.

export service_name="textgen-service-tgi"
# export service_name="textgen-service-tgi-gaudi"
# export service_name="textgen-service-vllm"
# export service_name="textgen-service-vllm-gaudi"
# export service_name="textgen-service-ollama"

cd ../../deployment/docker_compose/
docker compose -f compose_text-generation.yaml up ${service_name} -d

Consume Microservice

Check Service Status

curl http://${host_ip}:${TEXTGEN_PORT}/v1/health_check\
  -X GET \
  -H 'Content-Type: application/json'

Verify microservice

You can set the following model parameters according to your actual needs, such as max_tokens, stream.

The stream parameter determines the format of the data returned by the API. It will return text string with stream=false, return text stream flow with stream=true.

# stream mode
curl http://${host_ip}:${TEXTGEN_PORT}/v1/chat/completions \
    -X POST \
    -d '{"model": "${LLM_MODEL_ID}", "messages": "What is Deep Learning?", "max_tokens":17}' \
    -H 'Content-Type: application/json'

curl http://${host_ip}:${TEXTGEN_PORT}/v1/chat/completions \
    -X POST \
    -d '{"model": "${LLM_MODEL_ID}", "messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens":17}' \
    -H 'Content-Type: application/json'

#Non-stream mode
curl http://${host_ip}:${TEXTGEN_PORT}/v1/chat/completions \
    -X POST \
    -d '{"model": "${LLM_MODEL_ID}", "messages": "What is Deep Learning?", "max_tokens":17, "stream":false}' \
    -H 'Content-Type: application/json'