DMD Depth Material

Depth information was also captured in the DMD recordings. However, this data is different from the RGB and IR available material. In this document, it will explain how this data is handled and prepared, and how you can access it, modify and use it.

Depth data extraction

The recorded information is inside a rosbag. To extract Depth data (z16 format), the msg is converted to an OpenCV compatible format with 16UC1 encoding. This extracted data is an array or an image of 720x1280, where each pixel represents a distance in mm. For example, depth_img[200,300] = 560 means that there was a distance of 560mm from the camera to the object at the 200th-column and 300th-row pixel.

import cv2
bridge = CvBridge()
depth_img = bridge.imgmsg_to_cv2(msg, desired_encoding="16UC1")

Then, it is saved as a .tif image with OpenCV.

cv2.imwrite("depth_%06i.tif", depth_img)

Depth images to video

Because the DMD is a video-format dataset, we wanted to provide depth data as a video as well. The 16UC1 images are then encoded in a .avi video with FFMPEG, using the FFV1 encoder, which is lossless and compatible with gray16le pixel format.

ffmpeg -i depth_%06d.tif -c:v ffv1 -crf 0 output.avi

Using depth data with DExTool

DEx is our dataset explorer tool, this allows accessing annotations and videos from the DMD and exporting material for training. It has been prepared to handle depth videos so you can export images and sub-videoclips from this material. To export depth with DEx you should follow these steps:

1. Add "depth" to the list of @channels. You can find the @channels variable at the init() function of file accessDMDAnn.py under “-- CONTROL VARIABLES --“ comment. You can specify whether you want to get images or videos with the @material variable.
2. Run DExTool.py and select the "export material for training:[0]" option.
3. Enter the destination path.
4. Choose the option of how you want to export material: from an entire group, just from one session, or material of one specific VCD.
5. Paste the corresponding path of the material you chose above (group's material path, session path or VCD path).
6. Go to your destination path and the data should be inside "dmd-depth" folder following the same basic DMD structure.

What DEx is doing

All happens in accessDMDAnn.py. To access depth video data, we have included the ffmpeg-python library.

Images

For images, an array containing all of the frames from the depth video is obtained, this is done in getDepthVideoArray(). Then, the desired frame is taken from this array and saved as a .tif image with OpenCV, as done in depthFrameIntervalToImages(). Must be TIFF format since it supports gray16le pixel format.

Videos

Videos are trimmed from a start time to an end time, this is how FFMPEG works. Since we only have frame intervals as numbers, this number is divided by the frame rate to get time. This is done in depthFrameIntervalToVideo().

How to properly read depth images

As mentioned before, these depth images are in 16UC1 format. To load an image with OpenCV, you must set the read function to load it as they are, with the -1 flag.

depth_data = cv2.imread("depth_%06i.tif", -1)

This should return an image of 720x1280 where each pixel represents a distance in mm.

Visualize depth

Since a depth image is actually distance data, it is not something to be visualized. However, by applying a colour map it can be better interpreted by the human eye. After getting the depth array from the previous step, do the following:

cv_depth_img = cv2.applyColorMap(cv2.convertScaleAbs(depth_data, alpha=0.25), cv2.COLORMAP_JET)

For depth images, most examples show an alpha of 0.03, but 0.25 seems to work better for DMD :) This is an example: