Skip to content
This repository was archived by the owner on Jun 22, 2025. It is now read-only.

Commit e8aa78b

Browse files
kkrishTareachaadipre-commit-ci[bot]lvliang-intel
authored
Feature/elasticsearch vector store integration - Infosys (opea-project#972)
* Feature/elastic Elasticsearch vectorstore, dataprep and retriever --------- Co-authored-by: Adarsh <reachaadi@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Liang Lv <liang1.lv@intel.com>
1 parent ff8b858 commit e8aa78b

21 files changed

Lines changed: 1229 additions & 0 deletions

‎.github/workflows/docker/compose/dataprep-compose.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,3 +63,7 @@ services:
6363
build:
6464
dockerfile: comps/dataprep/multimedia2text/audio2text/Dockerfile
6565
image: ${REGISTRY:-opea}/dataprep-audio2text:${TAG:-latest}
66+
dataprep-elasticsearch:
67+
build:
68+
dockerfile: comps/dataprep/elasticsearch/langchain/Dockerfile
69+
image: ${REGISTRY:-opea}/dataprep-elasticsearch:${TAG:-latest}

‎.github/workflows/docker/compose/retrievers-compose.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,7 @@ services:
4747
build:
4848
dockerfile: comps/retrievers/neo4j/llama_index/Dockerfile
4949
image: ${REGISTRY:-opea}/retriever-neo4j-llamaindex:${TAG:-latest}
50+
retriever-elasticsearch:
51+
build:
52+
dockerfile: comps/retrievers/elasticsearch/langchain/Dockerfile
53+
image: ${REGISTRY:-opea}/retriever-elasticsearch:${TAG:-latest}

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
11
__pycache__
22
*.egg-info/
33
.DS_Store
4+
.idea/
5+
.venv/
6+
build/

‎comps/dataprep/elasticsearch/langchain/Dockerfile‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
FROM python:3.11-slim
5+
6+
ENV LANG=C.UTF-8
7+
8+
ARG ARCH="cpu"
9+
10+
RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
11+
build-essential \
12+
default-jre \
13+
libgl1-mesa-glx \
14+
libjemalloc-dev
15+
16+
RUN useradd -m -s /bin/bash user && \
17+
mkdir -p /home/user && \
18+
chown -R user /home/user/
19+
20+
USER user
21+
22+
COPY comps /home/user/comps
23+
24+
RUN pip install --no-cache-dir --upgrade pip setuptools && \
25+
if [ ${ARCH} = "cpu" ]; then pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cpu; fi && \
26+
pip install --no-cache-dir -r /home/user/comps/dataprep/elasticsearch/langchain/requirements.txt
27+
28+
ENV PYTHONPATH=$PYTHONPATH:/home/user
29+
30+
USER root
31+
32+
RUN mkdir -p /home/user/comps/dataprep/elasticsearch/langchain/uploaded_files && chown -R user /home/user/comps/dataprep/elasticsearch/langchain/uploaded_files
33+
34+
USER user
35+
36+
WORKDIR /home/user/comps/dataprep/elasticsearch/langchain
37+
38+
ENTRYPOINT ["python", "prepare_doc_elasticsearch.py"]

‎comps/dataprep/elasticsearch/langchain/README.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Dataprep Microservice with Elasticsearch
2+
3+
## 🚀1. Start Microservice with Python（Option 1）
4+
5+
### 1.1 Install Requirements
6+
7+
```bash
8+
pip install -r requirements.txt
9+
```
10+
11+
### 1.2 Setup Environment Variables
12+
13+
```bash
14+
export ES_CONNECTION_STRING=http://localhost:9200
15+
export INDEX_NAME=${your_index_name}
16+
```
17+
18+
### 1.3 Start Elasticsearch
19+
20+
Please refer to this [readme](../../../vectorstores/elasticsearch/README.md).
21+
22+
### 1.4 Start Document Preparation Microservice for Elasticsearch with Python Script
23+
24+
Start document preparation microservice for Elasticsearch with below command.
25+
26+
```bash
27+
python prepare_doc_elastic.py
28+
```
29+
30+
## 🚀2. Start Microservice with Docker (Option 2)
31+
32+
### 2.1 Start Elasticsearch
33+
34+
Please refer to this [readme](../../../vectorstores/elasticsearch/README.md).
35+
36+
### 2.2 Setup Environment Variables
37+
38+
```bash
39+
export ES_CONNECTION_STRING=http://localhost:9200
40+
export INDEX_NAME=${your_index_name}
41+
```
42+
43+
### 2.3 Build Docker Image
44+
45+
```bash
46+
cd GenAIComps
47+
docker build -t opea/dataprep-elasticsearch:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/elasticsearch/langchain/Dockerfile .
48+
```
49+
50+
### 2.4 Run Docker with CLI (Option A)
51+
52+
```bash
53+
docker run --name="dataprep-elasticsearch" -p 6011:6011 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e ES_CONNECTION_STRING=$ES_CONNECTION_STRING -e INDEX_NAME=$INDEX_NAME -e TEI_ENDPOINT=$TEI_ENDPOINT opea/dataprep-elastic:latest
54+
```
55+
56+
### 2.5 Run with Docker Compose (Option B)
57+
58+
```bash
59+
cd comps/dataprep/elasticsearch/langchain
60+
docker compose -f docker-compose-dataprep-elastic.yaml up -d
61+
```
62+
63+
## 🚀3. Consume Microservice
64+
65+
### 3.1 Consume Upload API
66+
67+
Once document preparation microservice for Elasticsearch is started, user can use below command to invoke the
68+
microservice to convert the document to embedding and save to the database.
69+
70+
```bash
71+
curl -X POST \
72+
-H "Content-Type: application/json" \
73+
-d '{"path":"/path/to/document"}' \
74+
http://localhost:6011/v1/dataprep
75+
```
76+
77+
### 3.2 Consume get_file API
78+
79+
To get uploaded file structures, use the following command:
80+
81+
```bash
82+
curl -X POST \
83+
-H "Content-Type: application/json" \
84+
http://localhost:6011/v1/dataprep/get_file
85+
```
86+
87+
Then you will get the response JSON like this:
88+
89+
```json
90+
[
91+
{
92+
"name": "uploaded_file_1.txt",
93+
"id": "uploaded_file_1.txt",
94+
"type": "File",
95+
"parent": ""
96+
},
97+
{
98+
"name": "uploaded_file_2.txt",
99+
"id": "uploaded_file_2.txt",
100+
"type": "File",
101+
"parent": ""
102+
}
103+
]
104+
```
105+
106+
### 4.3 Consume delete_file API
107+
108+
To delete uploaded file/link, use the following command.
109+
110+
The `file_path` here should be the `id` get from `/v1/dataprep/get_file` API.
111+
112+
```bash
113+
# delete link
114+
curl -X POST \
115+
-H "Content-Type: application/json" \
116+
-d '{"file_path": "https://www.ces.tech/.txt"}' \
117+
http://localhost:6011/v1/dataprep/delete_file
118+
119+
# delete file
120+
curl -X POST \
121+
-H "Content-Type: application/json" \
122+
-d '{"file_path": "uploaded_file_1.txt"}' \
123+
http://localhost:6011/v1/dataprep/delete_file
124+
125+
# delete all files and links
126+
curl -X POST \
127+
-H "Content-Type: application/json" \
128+
-d '{"file_path": "all"}' \
129+
http://localhost:6011/v1/dataprep/delete_file
130+
```

‎comps/dataprep/elasticsearch/langchain/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0

‎comps/dataprep/elasticsearch/langchain/config.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
import os
5+
6+
ES_CONNECTION_STRING = os.getenv("ES_CONNECTION_STRING", "http://localhost:9200")
7+
UPLOADED_FILES_PATH = os.getenv("UPLOADED_FILES_PATH", "./uploaded_files/")
8+
9+
# Embedding model
10+
EMBED_MODEL = os.getenv("EMBED_MODEL", "BAAI/bge-base-en-v1.5")
11+
12+
# TEI Embedding endpoints
13+
TEI_ENDPOINT = os.getenv("TEI_ENDPOINT", "")
14+
15+
# Vector Index Configuration
16+
INDEX_NAME = os.getenv("INDEX_NAME", "rag-elastic")
17+
18+
# chunk parameters
19+
CHUNK_SIZE = os.getenv("CHUNK_SIZE", 1500)
20+
CHUNK_OVERLAP = os.getenv("CHUNK_OVERLAP", 100)
21+
22+
# Logging enabled
23+
LOG_FLAG = os.getenv("LOGFLAG", False)

‎comps/dataprep/elasticsearch/langchain/docker-compose.yaml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
version: "3"
5+
services:
6+
elasticsearch-vector-db:
7+
hostname: db
8+
container_name: elasticsearch-vector-db
9+
image: docker.elastic.co/elasticsearch/elasticsearch:8.16.0
10+
ports:
11+
- "9200:9200"
12+
- "9300:9300"
13+
restart: always
14+
ipc: host
15+
environment:
16+
- ES_JAVA_OPTS=-Xms1g -Xmx1g
17+
- discovery.type=single-node
18+
- xpack.security.enabled=false
19+
- bootstrap.memory_lock=false
20+
- no_proxy= ${no_proxy}
21+
- http_proxy= ${http_proxy}
22+
- https_proxy= ${https_proxy}
23+
24+
dataprep-elasticsearch:
25+
image: opea/dataprep-elasticsearch:latest
26+
container_name: dataprep-elasticsearch
27+
ports:
28+
- "6011:6011"
29+
ipc: host
30+
environment:
31+
http_proxy: ${http_proxy}
32+
https_proxy: ${https_proxy}
33+
ES_CONNECTION_STRING: ${ES_CONNECTION_STRING}
34+
INDEX_NAME: ${INDEX_NAME}
35+
TEI_ENDPOINT: ${TEI_ENDPOINT}
36+
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
37+
restart: unless-stopped
38+
39+
networks:
40+
default:
41+
driver: bridge

0 commit comments

Comments
 (0)