RGB node

Function

The Rgb node is responsible for locating candidate holes in a RGB image.

Edge detection

Here, a standard OpenCV edge detector cannot be directly applied to the image: the input RGB image must first be processed because of the heterogeneity of colours, even inside defined objects, like wooden walls. In order to successfully locate edges of objects in the RGB image, two techniques have been employed: segmentation and watersheding via backprojection.

Segmentation

The RGB image is segmented by the standard OpenCV method pyrMeanShiftFiltering(). As stated in the method's documentation:

The function implements the filtering stage of meanshift segmentation, that is, 
the output of the function is the filtered “posterized” image with color gradients 
and fine-grain texture flattened. [...]

This means that colours closely related to one another are "grouped", which is something that contributes massively to the homogeneity of each object seen. After performing segmentation, there is a choice of further colour-flattening, much like a k-means algorithms would do, called posterization. In essence it fills all regions where colours are limited in a band with one colour. Experimentally, its usage is discouraged due to the large variance in colour of the segmented image in the borders of objects. There, the posterization is unable to find a way to consider which pixels reside where, resulting in a very poor and low approximation of the objects' edges, which is the next step taken. Posterization or not, the image is then transformed to grayscale and its edges are detected by a standard OpenCV edge detector.

Watersheding via backprojection

At first, the backprojection of the input RGB image with respect to the pre-calculated histogram of images of walls is obtained. The backprojection image is a grayscale image where each pixel is assigned a value which corresponds to the degree of relativity of the colour of a particular pixel to the histogram of images of walls. The backprojection image, even if dilated or/and multiply thresholded, cannot employ homogenous regions because of imperfections either in the input image, the images used to produce the histogram of walls or the backprojection mechanism itself. What is mostly observed is individual non-zero pixels scattered across a black image.

So, the backprojection itself is not the means to achieve a result upon which edge detection will be applied: it has to be used as a means by another algorithm. The backprojection of the input RGB image is used as a means to obtain the markers needed by the cv::watershed() method, which can automatically detect the edges of objects in the RGB image, with a sufficient backprojection image. This link will provide more information to the mechanisms of the process than this wiki can. Applying watersheding on the input RGB image, based on a backprojection image will ultimately provide the edges of objects of interest, walls in our case, without having to apply a standard OpenCV edge detector.

However the image of edges is obtained, it is then thresholded in order to suppress any noisy edges.

Now, the outline of holes (if any) can be seen rather clearly. Without loss of generality, because the outline of holes in space is not connected to any other detected edge, all pixels that are not iteratively neighboring the borders of the image can be eliminated as being the outline of holes. What we end up with is standalone closed-ended and open-ended shapes.

Because the shape of holes' edges seen from an angle can end up being open-ended (if, for example, the robot is travelling in parallel to a wall with a hole, the far end of the hole has a mild depth gradient, hence no edges can be detected), the ends of all open-ended shapes must be closed in a non-linear way, so as to approximate the hole's actual outline.

After the connection of the ends of open-ended shapes, all shapes are pruned of small non-substantial cluttering pixels that are connected to the actual outline of holes, so as to obtain clear shapes.

Blob detection

The blob detection is performed on the denoised images of edges, via the standard OpenCV blob detector. The output of the blob detector is a series of keypoints of blobs found. What is needed next, is to locate the outline of each blob. This can be performed by either starting a brush fire from each keypoint, that expands up to the blob's borders - it's outline - or by means of raycasting. The raycast algorithm approximates the blob's actual outline by casting rays from the blob's keypoint around 360 degrees in 360 / n intervals. Lastly, having found the keypoint and outline of a blob, its least area rotated rectangle is found.

All necessary pieces of information about holes found have been obtained up to this point. To sum up, a hole is identified by its:

• Keypoint
• Outline points
• The vertices of its bounding box

These pieces of information, along with the denoised depth image are then published to the Hole Fusion node for further, higher level, processing.

Interfaces

Input

Under

rgb_node_topics.yaml:subscribed_topics

the rgb_image_topic is the name of the topic where the Rgb node is subscribed to, where the input RGB image is published by the Synchronizer node. Its value must comply with synchronizer_node_topics.yaml/published_topics/rgb_image_topic.

Output

Under

rgb_node_topics.yaml:published_topics

the candidate_holes_topic is the name of the topic where the RGB node publishes information about holes found, along with the input RGB image. Its type is vision_communications::CandidateHolesVectorMsg.msg.

How to construct the crucial histogram of images of walls.

Use the image_saver_by_utility package, under pandora_vision.

Launch the openni or openni2 driver:

roslaunch openni[2]_launch openni[2].launch

Its usage is:

pandora_vision_support image_saver_by_topic $image_topic $folder_for_imgs $prefix $start_number $suffix [$width] [$height]

with the below parameters:

$image_topic : /camera/rgb/image
$folder_for_imgs : $pandora_vision_hole_detector/walls
$prefix : ""
$start_number : 0
$suffix : png
$width : 500
$height : 500

The topic's name may vary depending on the driver used. The starting number of the images should be exactly zero and the images' type should be ".png". The width and height of the images should be the same; the number 500 is suggested but not compulsory.

Create bounding boxes only around walls, so as to obtain their texture. Take care to include different light conditions.