Skip to content

Commit a150312

Browse files
RazvanLiviuVarzaruRazvanLiviuVarzaru
committed
Add ChatQnA docker-compose example on AIPC using MariaDB Vector
Example on how to deploy the ChatBot on AIPC by using MariaDB Server as a vectorstore. - use MariaDB Server as the backend database. Minimum required version is 11.7 - use the OPEA_DATAPREP_MARIADBVECTOR component for dataprep microservice - use the OPEA_RETRIEVER_MARIADBVECTOR component for retriever microservice How to test Set the HF API token environment variable and: ``` cd ChatQnA/tests bash _test_compose_mariadb_on_aipc.sh ``` Signed-off-by: Razvan-Liviu Varzaru <razvan@mariadb.org>
1 parent b467a13 commit a150312

4 files changed

Lines changed: 622 additions & 0 deletions

File tree

‎ChatQnA/docker_compose/intel/cpu/aipc/README_mariadb.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Build Mega Service of ChatQnA on AIPC
2+
3+
This document outlines the deployment process for a ChatQnA application utilizing the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline on AIPC. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as `embedding`, `retriever`, `rerank`, and `llm`.
4+
5+
## Quick Start:
6+
7+
1. Set up the environment variables.
8+
2. Run Docker Compose.
9+
3. Consume the ChatQnA Service.
10+
11+
### Quick Start: 1. Set up environment variables
12+
13+
1. Set the default environment:
14+
15+
```bash
16+
source ./set_env_mariadb.sh
17+
```
18+
19+
2. You need to set a Hugging Face access token for your account:
20+
21+
```bash
22+
export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
23+
```
24+
25+
3. _Set up the model (optional)_
26+
27+
_By default, **llama3.2** is used for LLM serving, the default model can be changed to other LLM models. Please pick a [validated llm models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#validated-llm-models) from the table._
28+
\*To change the default model defined in `set_env_mariadb.sh`, overwrite it by exporting `OLLAMA_MODEL` to the new model or by modifying `set_env_mariadb.sh`.
29+
30+
_Example, in order to use [DeepSeek-R1-Distill-Llama-8B model](https://ollama.com/library/deepseek-r1:8b), set:_
31+
32+
```bash
33+
export OLLAMA_MODEL="deepseek-r1:8b"
34+
```
35+
36+
### Quick Start: 2. Run Docker Compose
37+
38+
```bash
39+
docker compose -f compose_mariadb.yaml up -d
40+
```
41+
42+
---
43+
44+
_You should build docker image from source by yourself if_:
45+
46+
- _You are developing off the git main branch (as the container's ports in the repo may be different from the published docker image)._
47+
- _You can't download the docker image._
48+
- _You want to use a specific docker image version_
49+
50+
Please refer to ['Build Docker Images'](#🚀-build-docker-images) in below.
51+
52+
---
53+
54+
### Quick Start:3. Consume the ChatQnA Service
55+
56+
Once the services are up, open the following URL from your browser: `http://localhost` and fiddle around with prompts.
57+
58+
## 🚀 Build Docker Images
59+
60+
```bash
61+
# Clone the AI components repository
62+
mkdir ~/OPEA -p
63+
cd ~/OPEA
64+
git clone https://github.com/opea-project/GenAIComps.git
65+
cd GenAIComps
66+
```
67+
68+
If you are in a proxy environment, set the proxy-related environment variables:
69+
70+
```bash
71+
export http_proxy="Your_HTTP_Proxy"
72+
export https_proxy="Your_HTTPs_Proxy"
73+
```
74+
75+
### 1. Build Retriever Image
76+
77+
```bash
78+
docker build --no-cache -t opea/retriever:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/src/Dockerfile .
79+
```
80+
81+
### 2. Build Dataprep Image
82+
83+
```bash
84+
docker build --no-cache -t opea/dataprep:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/src/Dockerfile .
85+
cd ..
86+
```
87+
88+
### 3. Build MegaService Docker Image
89+
90+
To construct the Mega Service, we utilize the [GenAIComps](https://github.com/opea-project/GenAIComps.git) microservice pipeline within the `chatqna.py` Python script. Build MegaService Docker image via below command:
91+
92+
```bash
93+
cd ~/OPEA
94+
git clone https://github.com/opea-project/GenAIExamples.git
95+
cd GenAIExamples/ChatQnA
96+
docker build --no-cache -t opea/chatqna:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
97+
```
98+
99+
### 4. Build UI Docker Image
100+
101+
Build frontend Docker image via below command:
102+
103+
```bash
104+
cd ~/OPEA/GenAIExamples/ChatQnA/ui
105+
docker build --no-cache -t opea/chatqna-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
106+
```
107+
108+
### 5. Build Nginx Docker Image
109+
110+
```bash
111+
cd GenAIComps
112+
docker build -t opea/nginx:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/third_parties/nginx/src/Dockerfile .
113+
```
114+
115+
Finally, `docker image ls` would return:
116+
117+
```
118+
opea/dataprep:latest
119+
opea/retriever:latest
120+
opea/chatqna:latest
121+
opea/chatqna-ui:latest
122+
opea/nginx:latest
123+
```
124+
125+
After starting the services, if you want to check each service individually, please refer to the section below.
126+
127+
### Validate Microservices
128+
129+
You can validate each microservice by making individual requests.
130+
For details on how to verify the correctness of the response, refer to [how-to-validate_service](../../hpu/gaudi/how_to_validate_service.md).
131+
132+
1. TEI Embedding Service
133+
134+
```bash
135+
curl ${host_ip}:6006/embed \
136+
-X POST \
137+
-d '{"inputs":"What is Deep Learning?"}' \
138+
-H 'Content-Type: application/json'
139+
```
140+
141+
2. Retriever Microservice
142+
143+
```bash
144+
# Mock the embedding vector
145+
export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
146+
```
147+
148+
```bash
149+
# Perform a similarity search
150+
curl http://${host_ip}:7000/v1/retrieval \
151+
-X POST \
152+
-d '{"text":"What is the revenue of Nike in 2023?","embedding":"'"${your_embedding}"'"}' \
153+
-H 'Content-Type: application/json'
154+
```
155+
156+
3. TEI Reranking Service
157+
158+
```bash
159+
curl http://${host_ip}:8808/rerank \
160+
-X POST \
161+
-d '{"query":"What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."]}' \
162+
-H 'Content-Type: application/json'
163+
```
164+
165+
4. Ollama Service
166+
167+
```bash
168+
curl http://${host_ip}:11434/api/generate -d '{"model": "llama3.2", "prompt":"What is Deep Learning?"}'
169+
```
170+
171+
5. MegaService
172+
173+
```bash
174+
curl http://${host_ip}:8888/v1/chatqna -H "Content-Type: application/json" -d '{
175+
"messages": "What is the revenue of Nike in 2023?"
176+
}'
177+
```
178+
179+
6. DataPrep Service
180+
181+
Try the `ingest - get - delete` endpoints:
182+
183+
```bash
184+
# Get a sample file to ingest
185+
wget https://raw.githubusercontent.com/opea-project/GenAIComps/v1.1/comps/retrievers/redis/data/nke-10k-2023.pdf
186+
```
187+
188+
```bash
189+
# Ingest
190+
curl -X POST "http://${host_ip}:6007/v1/dataprep/ingest" \
191+
-H "Content-Type: multipart/form-data" \
192+
-F "files=@./nke-10k-2023.pdf"
193+
```
194+
195+
```bash
196+
# Get
197+
curl -X POST "http://${host_ip}:6007/v1/dataprep/get" \
198+
-H "Content-Type: application/json"
199+
```
200+
201+
```bash
202+
# Delete all
203+
curl -X POST "http://${host_ip}:6007/v1/dataprep/delete" \
204+
-H "Content-Type: application/json" \
205+
-d '{"file_path": "all"}'
206+
```

‎ChatQnA/docker_compose/intel/cpu/aipc/compose_mariadb.yaml‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# Copyright (C) 2025 MariaDB Foundation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
mariadb-server:
6+
image: mariadb:latest
7+
container_name: mariadb-server
8+
ports:
9+
- "3306:3306"
10+
environment:
11+
- MARIADB_DATABASE=${MARIADB_DATABASE}
12+
- MARIADB_USER=${MARIADB_USER}
13+
- MARIADB_PASSWORD=${MARIADB_PASSWORD}
14+
- MARIADB_RANDOM_ROOT_PASSWORD=1
15+
healthcheck:
16+
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
17+
start_period: 10s
18+
interval: 10s
19+
timeout: 5s
20+
retries: 3
21+
22+
dataprep-mariadb-vector:
23+
image: ${REGISTRY:-opea}/dataprep:${TAG:-latest}
24+
container_name: dataprep-mariadb-vector
25+
depends_on:
26+
mariadb-server:
27+
condition: service_healthy
28+
tei-embedding-service:
29+
condition: service_started
30+
ports:
31+
- "6007:5000"
32+
environment:
33+
no_proxy: ${no_proxy}
34+
http_proxy: ${http_proxy}
35+
https_proxy: ${https_proxy}
36+
DATAPREP_COMPONENT_NAME: "OPEA_DATAPREP_MARIADBVECTOR"
37+
MARIADB_CONNECTION_URL: mariadb+mariadbconnector://${MARIADB_USER}:${MARIADB_PASSWORD}@mariadb-server:3306/${MARIADB_DATABASE}
38+
TEI_ENDPOINT: http://tei-embedding-service:80
39+
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
40+
healthcheck:
41+
test: ["CMD-SHELL", "curl -f http://localhost:5000/v1/health_check || exit 1"]
42+
interval: 10s
43+
timeout: 5s
44+
retries: 50
45+
restart: unless-stopped
46+
tei-embedding-service:
47+
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
48+
container_name: tei-embedding-server
49+
ports:
50+
- "6006:80"
51+
volumes:
52+
- "./data:/data"
53+
shm_size: 1g
54+
environment:
55+
no_proxy: ${no_proxy}
56+
http_proxy: ${http_proxy}
57+
https_proxy: ${https_proxy}
58+
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate
59+
retriever:
60+
image: ${REGISTRY:-opea}/retriever:${TAG:-latest}
61+
container_name: retriever-mariadb-vector
62+
depends_on:
63+
mariadb-server:
64+
condition: service_healthy
65+
ports:
66+
- "7000:7000"
67+
ipc: host
68+
environment:
69+
no_proxy: ${no_proxy}
70+
http_proxy: ${http_proxy}
71+
https_proxy: ${https_proxy}
72+
MARIADB_CONNECTION_URL: mariadb+mariadbconnector://${MARIADB_USER}:${MARIADB_PASSWORD}@mariadb-server:3306/${MARIADB_DATABASE}
73+
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
74+
LOGFLAG: ${LOGFLAG}
75+
RETRIEVER_COMPONENT_NAME: "OPEA_RETRIEVER_MARIADBVECTOR"
76+
restart: unless-stopped
77+
tei-reranking-service:
78+
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
79+
container_name: tei-reranking-server
80+
ports:
81+
- "8808:80"
82+
volumes:
83+
- "./data:/data"
84+
shm_size: 1g
85+
environment:
86+
no_proxy: ${no_proxy}
87+
http_proxy: ${http_proxy}
88+
https_proxy: ${https_proxy}
89+
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
90+
HF_HUB_DISABLE_PROGRESS_BARS: 1
91+
HF_HUB_ENABLE_HF_TRANSFER: 0
92+
command: --model-id ${RERANK_MODEL_ID} --auto-truncate
93+
ollama-service:
94+
image: ollama/ollama
95+
container_name: ollama
96+
ports:
97+
- "11434:11434"
98+
volumes:
99+
- ollama:/root/.ollama
100+
entrypoint: ["bash", "-c"]
101+
command: ["ollama serve & sleep 10 && ollama run ${OLLAMA_MODEL} & wait"]
102+
environment:
103+
no_proxy: ${no_proxy}
104+
https_proxy: ${https_proxy}
105+
OLLAMA_MODEL: ${OLLAMA_MODEL}
106+
107+
chatqna-aipc-backend-server:
108+
image: ${REGISTRY:-opea}/chatqna:${TAG:-latest}
109+
container_name: chatqna-aipc-backend-server
110+
depends_on:
111+
mariadb-server:
112+
condition: service_healthy
113+
dataprep-mariadb-vector:
114+
condition: service_healthy
115+
tei-embedding-service:
116+
condition: service_started
117+
retriever:
118+
condition: service_started
119+
tei-reranking-service:
120+
condition: service_started
121+
ollama-service:
122+
condition: service_started
123+
ports:
124+
- "8888:8888"
125+
environment:
126+
- no_proxy=${no_proxy}
127+
- https_proxy=${https_proxy}
128+
- http_proxy=${http_proxy}
129+
- MEGA_SERVICE_HOST_IP=chatqna-aipc-backend-server
130+
- EMBEDDING_SERVER_HOST_IP=tei-embedding-service
131+
- EMBEDDING_SERVER_PORT=80
132+
- RETRIEVER_SERVICE_HOST_IP=retriever
133+
- RERANK_SERVER_HOST_IP=tei-reranking-service
134+
- RERANK_SERVER_PORT=80
135+
- LLM_SERVER_HOST_IP=ollama-service
136+
- LLM_SERVER_PORT=11434
137+
- LLM_MODEL=${OLLAMA_MODEL}
138+
- LOGFLAG=${LOGFLAG}
139+
ipc: host
140+
restart: always
141+
chatqna-aipc-ui-server:
142+
image: ${REGISTRY:-opea}/chatqna-ui:${TAG:-latest}
143+
container_name: chatqna-aipc-ui-server
144+
depends_on:
145+
- chatqna-aipc-backend-server
146+
ports:
147+
- "5173:5173"
148+
environment:
149+
- no_proxy=${no_proxy}
150+
- https_proxy=${https_proxy}
151+
- http_proxy=${http_proxy}
152+
ipc: host
153+
restart: always
154+
chatqna-aipc-nginx-server:
155+
image: ${REGISTRY:-opea}/nginx:${TAG:-latest}
156+
container_name: chatqna-aipc-nginx-server
157+
depends_on:
158+
- chatqna-aipc-backend-server
159+
- chatqna-aipc-ui-server
160+
ports:
161+
- "${NGINX_PORT:-80}:80"
162+
environment:
163+
- no_proxy=${no_proxy}
164+
- https_proxy=${https_proxy}
165+
- http_proxy=${http_proxy}
166+
- FRONTEND_SERVICE_IP=chatqna-aipc-ui-server
167+
- FRONTEND_SERVICE_PORT=5173
168+
- BACKEND_SERVICE_NAME=chatqna
169+
- BACKEND_SERVICE_IP=chatqna-aipc-backend-server
170+
- BACKEND_SERVICE_PORT=8888
171+
- DATAPREP_SERVICE_IP=dataprep-mariadb-vector
172+
- DATAPREP_SERVICE_PORT=5000
173+
ipc: host
174+
restart: always
175+
176+
volumes:
177+
ollama:
178+
179+
networks:
180+
default:
181+
driver: bridge

0 commit comments

Comments
 (0)