Add monitoring for AudioQnA deployed by Docker compose

Signed-off-by: Yi Yao <yi.a.yao@intel.com>

Author: joshuayao

AudioQnA/docker_compose/intel/cpu/xeon/README.md

@@ -15,12 +15,19 @@ Note: The default LLM is `meta-llama/Meta-Llama-3-8B-Instruct`. Before deploying
 
 This section describes how to quickly deploy and test the AudioQnA service manually on an Intel® Xeon® processor. The basic steps are:
 
-1. [Access the Code](#access-the-code)
-2. [Configure the Deployment Environment](#configure-the-deployment-environment)
-3. [Deploy the Services Using Docker Compose](#deploy-the-services-using-docker-compose)
-4. [Check the Deployment Status](#check-the-deployment-status)
-5. [Validate the Pipeline](#validate-the-pipeline)
-6. [Cleanup the Deployment](#cleanup-the-deployment)
+- [Deploying AudioQnA on Intel® Xeon® Processors](#deploying-audioqna-on-intel-xeon-processors)
+  - [Table of Contents](#table-of-contents)
+  - [AudioQnA Quick Start Deployment](#audioqna-quick-start-deployment)
+    - [Access the Code](#access-the-code)
+    - [Configure the Deployment Environment](#configure-the-deployment-environment)
+    - [Deploy the Services Using Docker Compose](#deploy-the-services-using-docker-compose)
+    - [Check the Deployment Status](#check-the-deployment-status)
+    - [Validate the Pipeline](#validate-the-pipeline)
+    - [Cleanup the Deployment](#cleanup-the-deployment)
+  - [AudioQnA Docker Compose Files](#audioqna-docker-compose-files)
+    - [Running LLM models with remote endpoints](#running-llm-models-with-remote-endpoints)
+  - [Validate MicroServices](#validate-microservices)
+  - [Conclusion](#conclusion)
 
 ### Access the Code
 
@@ -59,7 +66,7 @@ To deploy the AudioQnA services, execute the `docker compose up` command with th
 
 ```bash
 cd docker_compose/intel/cpu/xeon
-docker compose -f compose.yaml up -d
+docker compose -f compose_tgi.yaml up -d
 ```
 
 > **Note**: developers should build docker image from source when:
@@ -80,6 +87,13 @@ Please refer to the table below to build different microservices from source:
 | MegaService  | [MegaService build guide](../../../../README_miscellaneous.md#build-megaservice-docker-image)                                     |
 | UI           | [Basic UI build guide](../../../../README_miscellaneous.md#build-ui-docker-image)                                                 |
 
+(Optional) Enabling monitoring using the command:
+
+```bash
+cd docker_compose/intel/cpu/xeon
+docker compose -f compose_tgi.yaml -f compose.monitoring.yaml up -d
+```
+
 ### Check the Deployment Status
 
 After running docker compose, check if all the containers launched via docker compose have started:
@@ -127,7 +141,13 @@ curl http://${host_ip}:3008/v1/audioqna \
 To stop the containers associated with the deployment, execute the following command:
 
 ```bash
-docker compose -f compose.yaml down
+docker compose -f compose_tgi.yaml down
+```
+
+If monitoring is enabled, stop the containers using the following command:
+
+```bash
+docker compose -f compose_tgi.yaml -f compose.monitoring.yaml down
 ```
 
 ## AudioQnA Docker Compose Files
@@ -140,6 +160,7 @@ In the context of deploying an AudioQnA pipeline on an Intel® Xeon® platform,
 | [compose_tgi.yaml](./compose_tgi.yaml)             | The LLM serving framework is TGI. All other configurations remain the same as the default                                                                                                                                    |
 | [compose_multilang.yaml](./compose_multilang.yaml) | The TTS component is GPT-SoVITS. All other configurations remain the same as the default                                                                                                                                     |
 | [compose_remote.yaml](./compose_remote.yaml)       | The LLM used is hosted on a remote server and an endpoint is used to access this model. Additional environment variables need to be set before running. See [instructions](#running-llm-models-with-remote-endpoints) below. |
+| [compose.monitoring.yaml](./compose.monitoring.yaml) | Helper file for monitoring features. Can be used along with any compose files          |
 
 ### Running LLM models with remote endpoints
 

AudioQnA/docker_compose/intel/cpu/xeon/README_vllm.md

@@ -74,7 +74,7 @@ export HF_TOKEN='your_huggingfacehub_token'
 ### Setting variables in the file set_env_vllm.sh
 
 ```bash
-cd cd cd ~/searchqna-test/GenAIExamples/SearchQnA/docker_compose/amd/gpu/rocm
+cd ~/searchqna-test/GenAIExamples/SearchQnA/docker_compose/amd/gpu/rocm
 ### The example uses the Nano text editor. You can use any convenient text editor
 nano set_env_vllm.sh
 ```
@@ -107,7 +107,7 @@ export https_proxy="Your_HTTPs_Proxy"
 
 ```bash
 cd cd ~/audioqna-test/GenAIExamples/AudioQnA/docker_compose/amd/gpu/rocm/
-docker compose -f compose_vllm up -d
+docker compose up -d
 ```
 
 After starting the containers, you need to view their status with the command:
@@ -126,6 +126,12 @@ The following containers should be running:
 
 Containers should not restart.
 
+(Optional) Enabling monitoring using the commmand:
+```bash
+docker compose -f compose.yaml -f compose.monitoring.yaml up -d
+```
+
+
 #### 3.1.1. Configuring GPU forwarding
 
 By default, in the Docker Compose file, compose_vllm.yaml is configured to forward all GPUs to the audioqna-vllm-service container.

AudioQnA/docker_compose/intel/cpu/xeon/compose.monitoring.yaml

@@ -0,0 +1,59 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+services:
+  prometheus:
+    image: prom/prometheus:v2.52.0
+    container_name: opea_prometheus
+    user: root
+    volumes:
+      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml
+      - ./prometheus_data:/prometheus
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yaml'
+    ports:
+      - '9090:9090'
+    ipc: host
+    restart: unless-stopped
+
+  grafana:
+    image: grafana/grafana:11.0.0
+    container_name: grafana
+    volumes:
+      - ./grafana_data:/var/lib/grafana
+      - ./grafana/dashboards:/var/lib/grafana/dashboards
+      - ./grafana/provisioning:/etc/grafana/provisioning
+    user: root
+    environment:
+      GF_SECURITY_ADMIN_PASSWORD: admin
+      GF_RENDERING_CALLBACK_URL: http://grafana:3000/
+      GF_LOG_FILTERS: rendering:debug
+      no_proxy: ${no_proxy}
+      host_ip: ${host_ip}
+    depends_on:
+      - prometheus
+    ports:
+      - '3000:3000'
+    ipc: host
+    restart: unless-stopped
+
+  node-exporter:
+    image: prom/node-exporter
+    container_name: node-exporter
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+    command:
+      - '--path.procfs=/host/proc'
+      - '--path.sysfs=/host/sys'
+      - --collector.filesystem.ignored-mount-points
+      - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)"
+    environment:
+      no_proxy: ${no_proxy}
+    ports:
+      - 9100:9100
+    ipc: host
+    restart: always
+    deploy:
+      mode: global

AudioQnA/docker_compose/intel/cpu/xeon/grafana/dashboards/download_opea_dashboard.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+if ls *.json 1> /dev/null 2>&1; then
+    rm *.json
+fi
+
+wget https://raw.githubusercontent.com/opea-project/GenAIEval/refs/heads/main/evals/benchmark/grafana/vllm_grafana.json
+wget https://raw.githubusercontent.com/opea-project/GenAIEval/refs/heads/main/evals/benchmark/grafana/tgi_grafana.json
+wget https://raw.githubusercontent.com/opea-project/GenAIEval/refs/heads/main/evals/benchmark/grafana/audioqna_megaservice_grafana.json
+wget https://raw.githubusercontent.com/opea-project/GenAIEval/refs/heads/main/evals/benchmark/grafana/node_grafana.json

AudioQnA/docker_compose/intel/cpu/xeon/grafana/provisioning/dashboards/local.yaml

@@ -0,0 +1,14 @@
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: 1
+
+providers:
+- name: 'default'
+  orgId: 1
+  folder: ''
+  type: file
+  disableDeletion: false
+  updateIntervalSeconds: 10 #how often Grafana will scan for changed dashboards
+  options:
+    path: /var/lib/grafana/dashboards

AudioQnA/docker_compose/intel/cpu/xeon/grafana/provisioning/datasources/datasource.yaml

@@ -0,0 +1,54 @@
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+# config file version
+apiVersion: 1
+
+# list of datasources that should be deleted from the database
+deleteDatasources:
+  - name: Prometheus
+    orgId: 1
+
+# list of datasources to insert/update depending
+# what's available in the database
+datasources:
+  # <string, required> name of the datasource. Required
+- name: Prometheus
+  # <string, required> datasource type. Required
+  type: prometheus
+  # <string, required> access mode. direct or proxy. Required
+  access: proxy
+  # <int> org id. will default to orgId 1 if not specified
+  orgId: 1
+  # <string> url
+  url: http://$host_ip:9090
+  # <string> database password, if used
+  password:
+  # <string> database user, if used
+  user:
+  # <string> database name, if used
+  database:
+  # <bool> enable/disable basic auth
+  basicAuth: false
+  # <string> basic auth username, if used
+  basicAuthUser:
+  # <string> basic auth password, if used
+  basicAuthPassword:
+  # <bool> enable/disable with credentials headers
+  withCredentials:
+  # <bool> mark as default datasource. Max one per org
+  isDefault: true
+  # <map> fields that will be converted to json and stored in json_data
+  jsonData:
+     httpMethod: GET
+     graphiteVersion: "1.1"
+     tlsAuth: false
+     tlsAuthWithCACert: false
+  # <string> json object of data that will be encrypted.
+  secureJsonData:
+    tlsCACert: "..."
+    tlsClientCert: "..."
+    tlsClientKey: "..."
+  version: 1
+  # <bool> allow users to edit datasources from the UI.
+  editable: true

AudioQnA/docker_compose/intel/cpu/xeon/prometheus.yaml

@@ -0,0 +1,27 @@
+# Copyright (C) 2025 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+# [IP_ADDR]:{PORT_OUTSIDE_CONTAINER} -> {PORT_INSIDE_CONTAINER} / {PROTOCOL}
+global:
+  scrape_interval: 5s
+  external_labels:
+    monitor: "my-monitor"
+scrape_configs:
+  - job_name: "prometheus"
+    static_configs:
+      - targets: ["opea_prometheus:9090"]
+  - job_name: "vllm"
+    metrics_path: /metrics
+    static_configs:
+      - targets: ["vllm-service:80"]
+  - job_name: "tgi"
+    metrics_path: /metrics
+    static_configs:
+      - targets: ["tgi-service:80"]
+  - job_name: "docsum-backend-server"
+    metrics_path: /metrics
+    static_configs:
+      - targets: ["audioqna-xeon-backend-server:8888"]
+  - job_name: "prometheus-node-exporter"
+    metrics_path: /metrics
+    static_configs:
+      - targets: ["node-exporter:9100"]

AudioQnA/docker_compose/intel/cpu/xeon/set_env.sh

@@ -3,6 +3,8 @@
 # Copyright (C) 2024 Intel Corporation
 # SPDX-License-Identifier: Apache-2.0
 
+SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)
+
 # export host_ip=<your External Public IP>
 export host_ip=$(hostname -I | awk '{print $1}')
 export HF_TOKEN=${HF_TOKEN}
@@ -21,3 +23,7 @@ export SPEECHT5_SERVER_PORT=7055
 export LLM_SERVER_PORT=3006
 
 export BACKEND_SERVICE_ENDPOINT=http://${host_ip}:3008/v1/audioqna
+
+pushd "${SCRIPT_DIR}/grafana/dashboards" > /dev/null
+source download_opea_dashboard.sh
+popd > /dev/null

AudioQnA/docker_compose/intel/hpu/gaudi/README.md

@@ -15,12 +15,18 @@ Note: The default LLM is `meta-llama/Meta-Llama-3-8B-Instruct`. Before deploying
 
 This section describes how to quickly deploy and test the AudioQnA service manually on an Intel® Gaudi® processor. The basic steps are:
 
-1. [Access the Code](#access-the-code)
-2. [Configure the Deployment Environment](#configure-the-deployment-environment)
-3. [Deploy the Services Using Docker Compose](#deploy-the-services-using-docker-compose)
-4. [Check the Deployment Status](#check-the-deployment-status)
-5. [Validate the Pipeline](#validate-the-pipeline)
-6. [Cleanup the Deployment](#cleanup-the-deployment)
+- [Deploying AudioQnA on Intel® Gaudi® Processors](#deploying-audioqna-on-intel-gaudi-processors)
+  - [Table of Contents](#table-of-contents)
+  - [AudioQnA Quick Start Deployment](#audioqna-quick-start-deployment)
+    - [Access the Code](#access-the-code)
+    - [Configure the Deployment Environment](#configure-the-deployment-environment)
+    - [Deploy the Services Using Docker Compose](#deploy-the-services-using-docker-compose)
+    - [Check the Deployment Status](#check-the-deployment-status)
+    - [Validate the Pipeline](#validate-the-pipeline)
+    - [Cleanup the Deployment](#cleanup-the-deployment)
+  - [AudioQnA Docker Compose Files](#audioqna-docker-compose-files)
+  - [Validate MicroServices](#validate-microservices)
+  - [Conclusion](#conclusion)
 
 ### Access the Code
 
@@ -79,6 +85,13 @@ Please refer to the table below to build different microservices from source:
 | MegaService  | [MegaService build guide](../../../../README_miscellaneous.md#build-megaservice-docker-image)                        |
 | UI           | [Basic UI build guide](../../../../README_miscellaneous.md#build-ui-docker-image)                                    |
 
+(Optional) Enabling monitoring using the command:
+
+```bash
+cd docker_compose/intel/cpu/xeon
+docker compose -f compose.yaml -f compose.monitoring.yaml up -d
+```
+
 ### Check the Deployment Status
 
 After running docker compose, check if all the containers launched via docker compose have started:
@@ -128,6 +141,12 @@ To stop the containers associated with the deployment, execute the following com
 docker compose -f compose.yaml down
 ```
 
+If monitoring is enabled, stop the containers using the following command:
+
+```bash
+docker compose -f compose.yaml -f compose.monitoring.yaml down
+```
+
 ## AudioQnA Docker Compose Files
 
 In the context of deploying an AudioQnA pipeline on an Intel® Gaudi® platform, we can pick and choose different large language model serving frameworks. The table below outlines the various configurations that are available as part of the application. These configurations can be used as templates and can be extended to different components available in [GenAIComps](https://github.com/opea-project/GenAIComps.git).