|
| 1 | +# Retrieval Augmented Generation with OpenVINO Model Server demo |
| 2 | + |
| 3 | +This demo shows how to deploy in Kubernetes or OpenShift a service based on OpenVINO Model Server with a generative use cases. OpenVINO Model Server has a feature of serving MediaPipe graphs. Here we are going to use a graph with a Python calculator analyzing documents based on RAG algorithm. It indexes the documents provided to the service and expose KServe gRPC endpoint which can be used to query about content. The response is generated by the server using LLM model and Hugging Face library powered by OpenVINO Runtime. |
| 4 | + |
| 5 | +Here are the steps we are going to follow: |
| 6 | +- building the container image of the model server together with required python dependencies (public image has minimal server of python components) |
| 7 | +- adding the RAG servable configuration to a ConfigMap (it includes the model server configuraiton, MediaPipe graph, python code and also a list of documents to be analysed) |
| 8 | +- optionally adding Hugging Face token to a secret, it will be attached to the deployed service to pull the models in case they requires authentication |
| 9 | +- deploying the service via the operator and ModelServer custom resource |
| 10 | +- running the client to query the document |
| 11 | +- online updates of the documents and rerunning the client query |
| 12 | + |
| 13 | + |
| 14 | +## Building the container image of the model server |
| 15 | + |
| 16 | +The public image of Model Server is enabled for python execution but it has a minimal list of python dependencies. Depending on the use cases, a custom dependencies might need to be included in the image. Follow the instructions from https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/rag_chatbot |
| 17 | + |
| 18 | +```bash |
| 19 | +git clone https://github.com/openvinotoolkit/model_server.git |
| 20 | +cd model_server |
| 21 | +# for ubi base image |
| 22 | +make python_image BASE_OS=redhat OVMS_CPP_DOCKER_IMAGE=registry.connect.redhat.com/intel/openvino-model-server OVMS_CPP_IMAGE_TAG=2024.1-gpu |
| 23 | +# or for ubuntu22 base image |
| 24 | +make python_image BASE_OS=ubuntu OVMS_CPP_DOCKER_IMAGE=openvino/model_server OVMS_CPP_IMAGE_TAG=2024.1-gpu |
| 25 | +``` |
| 26 | +Push the build image to your image registry. |
| 27 | + |
| 28 | +In case of OpenShift, this build operation can be automated using a BuildConfig resource. After creating an ImageStream called `ovms`, create build config with the below spec: |
| 29 | + |
| 30 | +``` |
| 31 | +spec: |
| 32 | + nodeSelector: null |
| 33 | + output: |
| 34 | + to: |
| 35 | + kind: ImageStreamTag |
| 36 | + name: 'ovms:py' |
| 37 | + resources: {} |
| 38 | + strategy: |
| 39 | + type: Docker |
| 40 | + dockerStrategy: |
| 41 | + dockerfilePath: Dockerfile.redhat |
| 42 | + buildArgs: |
| 43 | + - name: IMAGE_NAME |
| 44 | + value: 'registry.connect.redhat.com/intel/openvino-model-server@sha256:b8721a65da98bd9b354680ecb2dca8a4d60a1097babe9822ab264f8e951190f7' |
| 45 | + postCommit: {} |
| 46 | + source: |
| 47 | + type: Git |
| 48 | + git: |
| 49 | + uri: 'https://github.com/openvinotoolkit/model_server' |
| 50 | + ref: main |
| 51 | + contextDir: demos/python_demos |
| 52 | +``` |
| 53 | +After triggering the build for that configuration, an image will be automatically pushed to the local cluster container registry in the OpenShift. |
| 54 | + |
| 55 | +## Adding the RAG servable configuration to a ConfigMap |
| 56 | + |
| 57 | +The Model Server configuration including servable configuration needs to be attached to deployed pods. In this demo they will be passed via a ConfigMap which can store arbitrary files. The operator automatically mounts the content of a ConfigMap to the serving pod. |
| 58 | + |
| 59 | +```bash |
| 60 | +pushd $(pwd) |
| 61 | +cd model_server/demos/python_demos/rag_chatbot/servable_stream |
| 62 | +echo https://gist.githubusercontent.com/ryanloney/42b8ebe29f95ebd4382ee0b2bb50bea2/raw/cfbb679fefb6babec675c7806254a5fff29a5e6b/aipc.txt > docs.txt |
| 63 | +oc create configmap rag-demo --from-file=config.json=config.json --from-file=graph.pbtxt=graph.pbtxt --from-file=model.py=model.py \ |
| 64 | +--from-file=config.py=config.py --from-file=ov_embedding_model.py=ov_embedding_model.py \ |
| 65 | +--from-file=ov_llm_model.py=ov_llm_model.py --from-file=docs.txt=docs.txt |
| 66 | +popd |
| 67 | +``` |
| 68 | + |
| 69 | +## Adding Hugging Face token to a Secret and the model name to the ConfigMap |
| 70 | +The RAG demo can work with several models and it can be defined as an environment variable passed to the pod. Some models might require a token to authorize the model pull from Hugging Face hub. Extra environment variables can be added to the deployment via a ConfigMap or a secret. |
| 71 | + |
| 72 | +```bash |
| 73 | +oc create secret generic rag-env --from-literal=HF_TOKEN=hf_GFb... |
| 74 | +oc create configmap rag-env --from-literal=SELECTED_MODEL=tiny-llama-2-chat-7b |
| 75 | +``` |
| 76 | + |
| 77 | +## Deploying the service |
| 78 | + |
| 79 | +Now we can deploy the Model Server by applying the custom resource: |
| 80 | +```bash |
| 81 | +oc apply -f - <<EOF |
| 82 | +apiVersion: intel.com/v1alpha1 |
| 83 | +kind: ModelServer |
| 84 | +metadata: |
| 85 | + name: ovms-rag |
| 86 | +spec: |
| 87 | + image_name: image-registry.openshift-image-registry.svc:5000/<your project name>/ovms:py |
| 88 | + deployment_parameters: |
| 89 | + replicas: 1 |
| 90 | + extra_envs_secret: "rag-env" |
| 91 | + extra_envs_configmap: "rag-env" |
| 92 | + models_settings: |
| 93 | + single_model_mode: false |
| 94 | + config_configmap_name: "rag-demo" |
| 95 | + server_settings: |
| 96 | + log_level: "INFO" |
| 97 | + service_parameters: |
| 98 | + grpc_port: 8080 |
| 99 | + rest_port: 8081 |
| 100 | +EOF |
| 101 | +``` |
| 102 | +It will deploy new pod and service `ovms-rag`. In the pod initialization, it will pull and load the LLM and embedding models from Hugging Face. Then the documents from provided docs.txt file will be downloaded and analyzed. The server will be updating the analysis when the docs.txt file is modified. |
| 103 | + |
| 104 | +## Running the client |
| 105 | + |
| 106 | +Confirm if the RAG servable is ready for processing. Below is a command to be executed from a client Pod in the same namespace: |
| 107 | +```bash |
| 108 | +curl http://ovms-rag:8081/v2/models/python_model/ready |
| 109 | +``` |
| 110 | + |
| 111 | +The query can be also executed via a gRPC client with steaming capabilities. That way you can read the text as it gets generated. |
| 112 | + |
| 113 | +```bash |
| 114 | +pushd $(pwd) |
| 115 | +cd model_server/demos/python_demos/llm_text_generation |
| 116 | +pip install -r client_requirements.txt |
| 117 | +python3 client_stream.py --url ovms-rag:8080 --prompt "Summarize what is AIPC." |
| 118 | +popd |
| 119 | +``` |
| 120 | + |
| 121 | +Alternatively, deploy a service based on `gradio` component, which provide easy to use GUI interface. |
| 122 | +In Openshift you can build the image with gradio client using the following BuildConfig spec. It assumes the image stream name `rag-gradio`: |
| 123 | +``` |
| 124 | +spec: |
| 125 | + nodeSelector: null |
| 126 | + output: |
| 127 | + to: |
| 128 | + kind: ImageStreamTag |
| 129 | + name: 'rag-gradio:latest' |
| 130 | + resources: {} |
| 131 | + successfulBuildsHistoryLimit: 5 |
| 132 | + failedBuildsHistoryLimit: 5 |
| 133 | + strategy: |
| 134 | + type: Docker |
| 135 | + dockerStrategy: |
| 136 | + dockerfilePath: Dockerfile.gradio |
| 137 | + postCommit: {} |
| 138 | + source: |
| 139 | + type: Git |
| 140 | + git: |
| 141 | + uri: 'https://github.com/openvinotoolkit/model_server' |
| 142 | + ref: main |
| 143 | + contextDir: demos/python_demos/rag_chatbot |
| 144 | +``` |
| 145 | + |
| 146 | +Launch the pod: |
| 147 | +```bash |
| 148 | +oc run gradio --image image-registry.openshift-image-registry.svc:5000/<your project name>/rag-gradio --port 9000 --command -- python app.py --ovms_url ovms-rag:8080 --web_url 0.0.0.0:7860 |
| 149 | + |
| 150 | +oc apply -f - <<EOF |
| 151 | +kind: Service |
| 152 | +apiVersion: v1 |
| 153 | +metadata: |
| 154 | + name: rag-gradio |
| 155 | +spec: |
| 156 | + ports: |
| 157 | + - protocol: TCP |
| 158 | + port: 80 |
| 159 | + targetPort: 7860 |
| 160 | + type: ClusterIP |
| 161 | + selector: |
| 162 | + run: gradio |
| 163 | +EOF |
| 164 | +``` |
| 165 | +The created service can be not exposed via Routes resources to connect to gradio interface from the browser: |
| 166 | + |
| 167 | + |
| 168 | + |
| 169 | +## Updating the documents |
| 170 | +Sometimes you might want to change the scope of the RAG analysis and change the documents. It can be done without reloading of the model and restarting the service. |
| 171 | +Just the `docs.txt` needs to be updated. The RAG servable has an extra thread checking regularly if the file is modified. In such case documentes are downloaded and indexed again. |
| 172 | + |
| 173 | +```bash |
| 174 | +pushd $(pwd) |
| 175 | +cd model_server/demos/python_demos/rag_chatbot/servable_stream |
| 176 | +echo https://gist.githubusercontent.com/dtrawins/2956a7a77aa6732b52b8ae6eab0be205/raw/e05f2ab8fea9c8631ac5f20b8dd640074ae429c7/genai.txt > docs.txt |
| 177 | +oc delete configmap rag-demo && \ |
| 178 | +oc create configmap rag-demo --from-file=config.json=config.json --from-file=graph.pbtxt=graph.pbtxt --from-file=model.py=model.py \ |
| 179 | +--from-file=config.py=config.py --from-file=ov_embedding_model.py=ov_embedding_model.py \ |
| 180 | +--from-file=ov_llm_model.py=ov_llm_model.py --from-file=docs.txt=docs.txt |
| 181 | +popd |
| 182 | +``` |
| 183 | + |
| 184 | +Wait a few moments and rerun the query: |
| 185 | +```bash |
| 186 | +pushd $(pwd) |
| 187 | +cd model_server/demos/python_demos/llm_text_generation |
| 188 | +python3 client_stream.py --url ovms-rag:8080 --prompt "What are the features of Gaudi3?" |
| 189 | +popd |
| 190 | +``` |
| 191 | + |
| 192 | +## Deploying on GPU |
| 193 | + |
| 194 | +The same demo can be adjusted to run the inference on GPU cards. It would require the following changes: |
| 195 | +- installing [Intel Device Plugin for Kubernetes](https://github.com/intel/intel-device-plugins-for-kubernetes) |
| 196 | +- adding to the ModelServer resource requirements |
| 197 | +- adding extra environment variable DEVICE=gpu |
| 198 | + |
| 199 | +``` |
| 200 | +deployment_parameters: |
| 201 | + resources: |
| 202 | + limits: |
| 203 | + xpu_device: gpu.intel.com/i915 |
| 204 | + xpu_device_quantity: "1" |
| 205 | +``` |
| 206 | +- added extra environment variable `DEVICE=gpu` to the configmap `rag-env`. |
| 207 | + |
| 208 | +## Deploying on Persistent Volume Claim |
| 209 | + |
| 210 | +The demo can be used also with the cluster storage and PVC in the cluster. In such scenario the model can be stored in the same location with the servable configuration files. That way model downloading Hugging Face hub is needed. Locally stored model can be in compressed IR format. It speeds up the model loading time. |
| 211 | + |
| 212 | +In such scenario, copy the content of `servable_stream` from the [RAG demo](https://github.com/openvinotoolkit/model_server/tree/main/demos/python_demos/rag_chatbot) to the volume claim togather with the downloaded and compressed models in IR format. |
| 213 | +It can be then mounted to the model server containers using the parameter with dropped `config_configmap_name`: |
| 214 | + |
| 215 | +``` |
| 216 | +models_repository: |
| 217 | + models_volume_claim: <pvc name> |
| 218 | +models_settings: |
| 219 | + config_path: /models/<path in pvc>/config.json |
| 220 | +``` |
| 221 | + |
| 222 | + |
| 223 | + |
0 commit comments