Use GPRC/ACAP Runtime to capture video frames

Angelo DelliSanti · Angelo DelliSanti · commit c3ff5233484a · 2024-04-17T17:36:24.000+02:00
Change-Id: I9d69abe9b5ff9c9edb03e77cbf96362f3153556a
diff --git a/hello-world-python/Dockerfile b/hello-world-python/Dockerfile
@@ -2,7 +2,7 @@
 
 ARG ARCH=armv7hf
 ARG REPO=axisecp
-ARG SDK_VERSION=1.12
+ARG SDK_VERSION=1.13
 ARG UBUNTU_VERSION=22.04
 
 FROM arm32v7/ubuntu:${UBUNTU_VERSION} as runtime-image-armv7hf
diff --git a/hello-world-python/README.md b/hello-world-python/README.md
@@ -29,8 +29,8 @@ Meet the following requirements to ensure compatibility with the example:
 
 * Axis device
   * Chip: ARTPEC-{7-8} DLPU devices (e.g., Q1615 MkIII)
-  * Firmware: 10.9 or higher
-  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) installed and started, using TLS and SD card as storage
+  * Firmware: 11.10 or higher
+  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) version 3.0 installed and started, using TLS with TCP and IPC socket and SD card as storage
 * Computer
   * Either [Docker Desktop](https://docs.docker.com/desktop/) version 4.11.1 or higher,
   * or [Docker Engine](https://docs.docker.com/engine/) version 20.10.17 or higher with BuildKit enabled using Docker Compose version 1.29.2 or higher
diff --git a/minimal-ml-inference/Dockerfile b/minimal-ml-inference/Dockerfile
@@ -2,7 +2,7 @@
 
 ARG ARCH=armv7hf
 ARG REPO=axisecp
-ARG SDK_VERSION=1.12
+ARG SDK_VERSION=1.13
 ARG UBUNTU_VERSION=22.04
 
 FROM arm32v7/ubuntu:${UBUNTU_VERSION} as runtime-image-armv7hf
@@ -17,11 +17,12 @@ FROM runtime-image-${ARCH}
 # Get packages from the CV SDK
 COPY --from=cv-sdk /axis/python /
 COPY --from=cv-sdk /axis/python-tfserving /
+COPY --from=cv-sdk /axis/python-vdoproto /
 COPY --from=cv-sdk /axis/python-numpy /
 COPY --from=cv-sdk /axis/opencv /
 COPY --from=cv-sdk /axis/openblas /
 
 # Copy the application script to the container
 WORKDIR /app
 COPY simple_inference.py .
-ENTRYPOINT ["python3", "simple_inference.py"]
+CMD ["python3", "simple_inference.py"]
diff --git a/minimal-ml-inference/README.md b/minimal-ml-inference/README.md
@@ -8,7 +8,7 @@ This example demonstrates how to create a simple Python application that perform
 
 This example composes three different container images into an application that performs an inference using a deep learning model.
 
-The first container contains the actual program built in this example. It uses [OpenCV](https://opencv.org/) to capture pictures from the camera and modifies them to fit the input required by the model. It then uses [gRPC](https://grpc.io/)/[protobuf](https://developers.google.com/protocol-buffers) to call the second container, the *inference-server*, that performs the actual inference by implementing the [TensorFlow Serving API](https://github.com/tensorflow/serving). You can find more documentation on the [Machine Learning API documentation page](https://axiscommunications.github.io/acap-documentation/docs/api/computer-vision-sdk-apis.html#machine-learning-api). This example uses a containerized version of the [ACAP Runtime](https://github.com/AxisCommunications/acap-runtime#containerized-version) as the *inference-server*.
+The first container contains the actual program built in this example. It then uses [gRPC](https://grpc.io/)/[protobuf](https://developers.google.com/protocol-buffers) to call the second container, the *inference-server*, that is used to capture images from the camera and perform the actual inference by implementing the [TensorFlow Serving API](https://github.com/tensorflow/serving). You can find more documentation on the [Machine Learning API documentation page](https://axiscommunications.github.io/acap-documentation/docs/api/computer-vision-sdk-apis.html#machine-learning-api). This example uses a containerized version of the [ACAP Runtime](https://github.com/AxisCommunications/acap-runtime#containerized-version) as the *inference-server*.
 
 Lastly, there is a third container that holds the deep learning model, which is put into a volume that is accessible by the other two images. The layout of the Docker image containing the model is shown below. The *MODEL_PATH* variable in the configuration file you're using specifies what model to use. By default, the armv7hf configuration file uses the edgetpu model, while the aarch64 configuration file uses the vanilla model.
 
@@ -47,8 +47,8 @@ Meet the following requirements to ensure compatibility with the example:
 
 * Axis device
   * Chip: ARTPEC-{7-8} DLPU devices (e.g., Q1615 MkIII)
-  * Firmware: 10.9 or higher
-  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) installed and started, using TLS and SD card as storage
+  * Firmware: 11.10 or higher
+  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) version 3.0 installed and started, using TLS with TCP and IPC socket and SD card as storage
 * Computer
   * Either [Docker Desktop](https://docs.docker.com/desktop/) version 4.11.1 or higher,
   * or [Docker Engine](https://docs.docker.com/engine/) version 20.10.17 or higher with BuildKit enabled using Docker Compose version 1.29.2 or higher
diff --git a/minimal-ml-inference/config/env.aarch64.artpec8 b/minimal-ml-inference/config/env.aarch64.artpec8
@@ -1,3 +1,3 @@
 MODEL_PATH=/models/ssd_mobilenet_v2_coco_quant_postprocess.tflite
 INFERENCE_SERVER_IMAGE=axisecp/acap-runtime:1.3.1-aarch64-containerized
-INFERENCE_CHIP=12
+INFERENCE_CHIP=12
diff --git a/minimal-ml-inference/config/env.aarch64.cpu b/minimal-ml-inference/config/env.aarch64.cpu
@@ -1,3 +1,3 @@
 MODEL_PATH=/models/ssd_mobilenet_v2_coco_quant_postprocess.tflite
 INFERENCE_SERVER_IMAGE=axisecp/acap-runtime:1.3.1-aarch64-containerized
-INFERENCE_CHIP=2
+INFERENCE_CHIP=2
diff --git a/minimal-ml-inference/config/env.armv7hf.cpu b/minimal-ml-inference/config/env.armv7hf.cpu
@@ -1,3 +1,3 @@
 MODEL_PATH=/models/ssd_mobilenet_v2_coco_quant_postprocess.tflite
 INFERENCE_SERVER_IMAGE=axisecp/acap-runtime:1.3.1-armv7hf-containerized
-INFERENCE_CHIP=2
+INFERENCE_CHIP=2
diff --git a/minimal-ml-inference/config/env.armv7hf.tpu b/minimal-ml-inference/config/env.armv7hf.tpu
@@ -1,3 +1,3 @@
 MODEL_PATH=/models/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite
 INFERENCE_SERVER_IMAGE=axisecp/acap-runtime:1.3.1-armv7hf-containerized
-INFERENCE_CHIP=4
+INFERENCE_CHIP=4
diff --git a/minimal-ml-inference/docker-compose.yml b/minimal-ml-inference/docker-compose.yml
@@ -8,10 +8,8 @@ services:
       - INFERENCE_PORT=0
       - MODEL_PATH=${MODEL_PATH}
     volumes:
-      - /usr/lib/libvdostream.so.1:/usr/lib/libvdostream.so.1
       - acap_dl-models:/models:ro
       - /tmp:/output
-      - /var/run/dbus:/var/run/dbus:rw
       - inference-server:/tmp
 
   inference-server:
@@ -37,4 +35,4 @@ services:
 
 volumes:
   acap_dl-models: {}
-  inference-server: {}
+  inference-server: {}
diff --git a/minimal-ml-inference/simple_inference.py b/minimal-ml-inference/simple_inference.py
@@ -1,24 +1,22 @@
 import os
-import cv2
 import numpy as np
-
 from tf_proto_utils import InferenceClient
+from vdo_proto_utils import VideoCaptureClient
 
-# Create the inference client with model and socket from environment variables
-inf_client = InferenceClient(os.environ['INFERENCE_HOST'])
 
-# Create a video stream that fits the model inputs
-cap = cv2.VideoCapture(0)
-cap.set(cv2.CAP_PROP_FRAME_WIDTH, 300)
-cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 300)
-cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"RGB3"))
+grpc_socket = os.environ['INFERENCE_HOST']
+stream_width, stream_height, stream_framerate = (224, 224, 10)
+capture_client = VideoCaptureClient(socket=grpc_socket,
+                                  stream_width=stream_width,
+                                  stream_height=stream_height,
+                                  stream_framerate=stream_framerate)
+inf_client = InferenceClient(grpc_socket)
 
 # The capture -> inference loop
 while True:
     # Read an image from the video stream
-    _, image = cap.read()
+    image = capture_client.get_frame()
 
-    # Give the image a batch dimension such that it has the shape (1, H, W, C)
     image = np.expand_dims(image, axis=0)
 
     # Perform inference with a model specified as an environment variable
diff --git a/object-detector-python/Dockerfile b/object-detector-python/Dockerfile
@@ -2,7 +2,7 @@
 
 ARG ARCH=armv7hf
 ARG REPO=axisecp
-ARG SDK_VERSION=1.12
+ARG SDK_VERSION=1.13
 ARG UBUNTU_VERSION=22.04
 
 FROM arm32v7/ubuntu:${UBUNTU_VERSION} as runtime-image-armv7hf
@@ -14,6 +14,7 @@ FROM runtime-image-${ARCH}
 COPY --from=cv-sdk /axis/python /
 COPY --from=cv-sdk /axis/python-numpy /
 COPY --from=cv-sdk /axis/python-tfserving /
+COPY --from=cv-sdk /axis/python-vdoproto /
 COPY --from=cv-sdk /axis/opencv /
 COPY --from=cv-sdk /axis/openblas /
 
diff --git a/object-detector-python/README.md b/object-detector-python/README.md
@@ -51,8 +51,8 @@ Meet the following requirements to ensure compatibility with the example:
 
 * Axis device
   * Chip: ARTPEC-{7-8} DLPU devices (e.g., Q1615 MkIII)
-  * Firmware: 10.9 or higher
-  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) installed and started, using TLS and SD card as storage
+  * Firmware: 11.10 or higher
+  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) version 3.0 installed and started, using TLS with TCP and IPC socket and SD card as storage
 * Computer
   * Either [Docker Desktop](https://docs.docker.com/desktop/) version 4.11.1 or higher,
   * or [Docker Engine](https://docs.docker.com/engine/) version 20.10.17 or higher with BuildKit enabled using Docker Compose version 1.29.2 or higher
diff --git a/object-detector-python/app/detector.py b/object-detector-python/app/detector.py
@@ -20,7 +20,7 @@
 import numpy as np
 import cv2
 from tf_proto_utils import InferenceClient
-
+from vdo_proto_utils import VideoCaptureClient
 
 # Detector object
 # Send images for object detection
@@ -83,8 +83,9 @@ def draw_bounding_boxes(self, image, bounding_boxes, obj_classes, color=(1, 190,
 
     # Read environment variables
     def read_enviroment(self):
+        self.grpc_socket = os.environ['INFERENCE_HOST']
         self.threshold = float(os.environ.get('DETECTION_THRESHOLD', 0.5))
-        self.inference_client =  InferenceClient(os.environ['INFERENCE_HOST'])
+        self.inference_client =  InferenceClient(self.grpc_socket)
 
         self.model_path = os.environ['MODEL_PATH']
         image_path = os.environ.get('IMAGE_PATH')
@@ -93,8 +94,8 @@ def read_enviroment(self):
 
     # Run object detection
     def run(self):
-        print(f"object-detector-python connect to: {os.environ['INFERENCE_HOST']}")
         image_path, object_list_path = self.read_enviroment()
+        print(f"object-detector-python connect to: {self.grpc_socket}")
         self.read_object_list(object_list_path)
         if image_path is not None:
             while True:
@@ -104,13 +105,14 @@ def run(self):
 
     # Run object detection on video stream
     def run_camera_source(self):
-        cap = cv2.VideoCapture(1)
-        cap.set(cv2.CAP_PROP_FPS, 5)
-        cap.set(cv2.CAP_PROP_FRAME_WIDTH, 480)
-        cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 320)
-        cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"RGB3"))
+        stream_width, stream_height, stream_framerate = (480, 320, 10)
+        capture_client = VideoCaptureClient(socket=self.grpc_socket,
+                                            stream_width=stream_width,
+                                            stream_height=stream_height,
+                                            stream_framerate=stream_framerate)
+
         while True:
-            _, frame = cap.read()
+            frame = capture_client.get_frame()
             succeed, bounding_boxes, obj_classes, _ = self.detect(frame)
             if not succeed:
                 time.sleep(1)
diff --git a/object-detector-python/docker-compose.yml b/object-detector-python/docker-compose.yml
@@ -14,10 +14,8 @@ services:
       - MODEL_PATH=${MODEL_PATH}
       - OBJECT_LIST_PATH=/models/coco_labels.txt
     volumes:
-      - /usr/lib/libvdostream.so.1:/usr/lib/libvdostream.so.1
       - acap_dl-models:/models:ro
       - /tmp:/output
-      - /var/run/dbus:/var/run/dbus:rw
       - inference-server:/tmp
 
   inference-server:
diff --git a/opencv-qr-decoder-python/Dockerfile b/opencv-qr-decoder-python/Dockerfile
@@ -2,7 +2,7 @@
 
 ARG ARCH=armv7hf
 ARG REPO=axisecp
-ARG SDK_VERSION=1.12
+ARG SDK_VERSION=1.13
 ARG UBUNTU_VERSION=22.04
 
 FROM arm32v7/ubuntu:${UBUNTU_VERSION} as runtime-image-armv7hf
@@ -13,6 +13,8 @@ FROM runtime-image-${ARCH}
 
 COPY --from=cv-sdk /axis/opencv /
 COPY --from=cv-sdk /axis/openblas /
+COPY --from=cv-sdk /axis/python-tfserving /
+COPY --from=cv-sdk /axis/python-vdoproto /
 COPY --from=cv-sdk /axis/python /
 COPY --from=cv-sdk /axis/python-numpy /
 
diff --git a/opencv-qr-decoder-python/README.md b/opencv-qr-decoder-python/README.md
@@ -3,12 +3,12 @@
 ## OpenCV QR Decoder
 
 This example shows a minimal QR code detector and decoder application written in Python.
-It involves using OpenCV to create a video stream, NumPy to preprocess
-the images and OpenCV to detect and decode any QR codes within the image. The example uses the
-[ACAP Computer Vision SDK](https://github.com/AxisCommunications/acap-computer-vision-sdk) to get
-OpenCV with VDO integration, allowing capture of a video stream through OpenCV's standard
-VideoCapture class. Additionally, Python, NumPy and OpenBLAS are also copied
-from the Computer Vision SDK.
+It composes two different container images into an application that performs an inference using a deep learning model.
+
+The first container contains the actual program built in this example. It then uses [gRPC](https://grpc.io/)/[protobuf](https://developers.google.com/protocol-buffers) to call the second container, that is used to capture images from the camera.
+
+NumPy is used to preprocess the images and OpenCV to detect and decode any QR codes within the image.
+Additionally, Python, NumPy and OpenBLAS are also copied from the Computer Vision SDK.
 
 ## Application layout
 
@@ -31,8 +31,8 @@ Meet the following requirements to ensure compatibility with the example:
 
 * Axis device
   * Chip: ARTPEC-{7-8} DLPU devices (e.g., Q1615 MkIII)
-  * Firmware: 10.9 or higher
-  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) installed and started, using TLS and SD card as storage
+  * Firmware: 11.10 or higher
+  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) version 3.0 installed and started, using TLS with TCP and IPC socket and SD card as storage
 * Computer
   * Either [Docker Desktop](https://docs.docker.com/desktop/) version 4.11.1 or higher,
   * or [Docker Engine](https://docs.docker.com/engine/) version 20.10.17 or higher with BuildKit enabled using Docker Compose version 1.29.2 or higher
diff --git a/opencv-qr-decoder-python/app/qr.py b/opencv-qr-decoder-python/app/qr.py
@@ -15,28 +15,23 @@
 """
 
 import cv2
-import numpy as np
+import os
+from vdo_proto_utils import VideoCaptureClient
 
 # Setup VDO stream
-cap = cv2.VideoCapture(0)
-
-# Set resolution
-cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
-cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 360)
-
-# Set image format
-cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc('R', 'G', 'B', '3'))
-
-# Set a small buffer size
-cap.set(cv2.CAP_PROP_UNIMATRIX_MAX_BUFFERS, 1)
+grpc_socket = os.environ['INFERENCE_HOST']
+stream_width, stream_height, stream_framerate = (640, 360, 10)
+capture_client = VideoCaptureClient(socket=grpc_socket,
+                                  stream_width=stream_width,
+                                  stream_height=stream_height,
+                                  stream_framerate=stream_framerate)
 
 # Instatiate our detector
 qr_detector = cv2.QRCodeDetector()
 
-got_frame, _ = cap.read()
-while got_frame:
+while True:
   # Read new frame
-  got_frame, frame = cap.read()
+  frame = capture_client.get_frame()
 
   # Preprocessing
   # RGB to BGR to accommodate the QR detectors input signature
diff --git a/opencv-qr-decoder-python/docker-compose.yml b/opencv-qr-decoder-python/docker-compose.yml
@@ -1,15 +1,30 @@
 version: '3.3'
-
 services:
   qr-decoder-app:
     image: ${APP_NAME}
-    privileged: true
+    logging:
+      driver: "json-file"
+      options:
+        max-file: "5"
+        max-size: "100k"
     environment:
-      - DBUS_SYSTEM_BUS_ADDRESS=unix:path=/var/run/dbus/system_bus_socket
       - PYTHONUNBUFFERED=1
+      - INFERENCE_HOST=unix:///tmp/acap-runtime.sock
     volumes:
-      - /usr/lib/libvdostream.so.1:/usr/lib/libvdostream.so.1
-      - /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:rw
-      - /var/run/statuscache:/var/run/statuscache:rw
-    devices:
-      - /dev/datacache0:/dev/datacache0:rw
+      - inference-server:/tmp
+  inference-server:
+    image: axisecp/acap-runtime:1.3.1-${ARCH}-containerized
+    logging:
+      driver: "json-file"
+      options:
+        max-file: "5"
+        max-size: "100k"
+    entrypoint: ["/opt/app/acap_runtime/acapruntime"]
+    volumes:
+      - /run/dbus/system_bus_socket:/run/dbus/system_bus_socket
+      - /run/parhand/parhandsocket:/run/parhand/parhandsocket
+      - /usr/lib:/usr/lib
+      - /usr/bin:/usr/bin
+      - inference-server:/tmp
+volumes:
+  inference-server: {}
diff --git a/parameter-api-python/Dockerfile b/parameter-api-python/Dockerfile
@@ -2,7 +2,7 @@
 
 ARG ARCH=armv7hf
 ARG REPO=axisecp
-ARG SDK_VERSION=1.12
+ARG SDK_VERSION=1.13
 ARG UBUNTU_VERSION=22.04
 
 FROM arm32v7/ubuntu:${UBUNTU_VERSION} as runtime-image-armv7hf
diff --git a/parameter-api-python/README.md b/parameter-api-python/README.md
@@ -38,8 +38,8 @@ Meet the following requirements to ensure compatibility with the example:
 
 * Axis device
   * Chip: ARTPEC-{7-8} DLPU devices (e.g., Q1615 MkIII)
-  * Firmware: 10.9 or higher
-  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) installed and started, using TLS and SD card as storage
+  * Firmware: 11.10 or higher
+  * [Docker ACAP](https://github.com/AxisCommunications/docker-acap#installing) version 3.0 installed and started, using TLS with TCP and IPC socket and SD card as storage
   * [ACAP Runtime](https://github.com/AxisCommunications/acap-runtime#native-acap-application) installed and started
 * Computer
   * Either [Docker Desktop](https://docs.docker.com/desktop/) version 4.11.1 or higher,
diff --git a/pose-estimator-with-flask/Dockerfile b/pose-estimator-with-flask/Dockerfile
@@ -2,7 +2,7 @@
 
 ARG ARCH=armv7hf
 ARG REPO=axisecp
-ARG SDK_VERSION=1.12
+ARG SDK_VERSION=1.13
 ARG UBUNTU_VERSION=22.04
 
 FROM arm32v7/ubuntu:${UBUNTU_VERSION} as runtime-image-armv7hf
@@ -14,6 +14,7 @@ FROM runtime-image-${ARCH}
 COPY --from=cv-sdk /axis/python /
 COPY --from=cv-sdk /axis/python-numpy /
 COPY --from=cv-sdk /axis/python-tfserving /
+COPY --from=cv-sdk /axis/python-vdoproto /
 COPY --from=cv-sdk /axis/opencv /
 COPY --from=cv-sdk /axis/openblas /
 
diff --git a/pose-estimator-with-flask/README.md b/pose-estimator-with-flask/README.md
diff --git a/pose-estimator-with-flask/app/detector_with_flask.py b/pose-estimator-with-flask/app/detector_with_flask.py
diff --git a/pose-estimator-with-flask/docker-compose.yml b/pose-estimator-with-flask/docker-compose.yml
diff --git a/web-server/README.md b/web-server/README.md
