Depth node
The Depth node is responsible for locating candidate holes in a Depth image. The conceptual steps made in order to transform the input Depth image to candidate holes is described in the following block diagram.
The input depth image published to the Depth node by the Synchronizer node is cluttered with black pixels, due to the depth sensor's principle of locating the depth distance of an object. In order for any processing of the depth image to be reliable, the noise must somehow vanish and be replaced with actual meaningful values of depth.
The noise elimination techniques used are three and differ by the amount of noise present in the depth image. If the percentage of noise exceeds the 70% mark, all black pixels are transformed to white, given an invalid but meaningful value of 4.0 meters. This value has a meaning in that for the presence of noise to be that high, the depth sensor is closer to an object that obstructs its field of view than its minimum operation length, which is 0.5 meters. If the percentage of noise in the depth image does not exceed the 70% of total pixels, but the mean depth observed in the depth image is lower than 0.7 meters, all regions of noise are identified and replaced by the minimum observed depth value that encloses each one of them. Lastly, if none of the above are true, all black pixels are first identified and then, progressively, assigned the mean value of their non-noisy neighbors, until every pixel has a non-zero (zero means noise) value.
After the elimination of noise, a selected standard OpenCV edge detector is applied on the denoised depth image, and the image of edges is then thresholded in order to suppress any noisy edges.
Now, the outline of holes (if any) can be seen rather clearly. Without loss of generality, because the outline of holes in space is not connected to any other detected edge, all pixels that are not iteratively neighboring the borders of the image can be eliminated as being the outline of holes. What we end up with is standalone closed-ended and open-ended shapes.
Because the shape of holes' edges seen from an angle can end up being open-ended (if, for example, the robot is travelling in parallel to a wall with a hole, the far end of the hole has a mild depth gradient, hence no edges can be detected), the ends of all open-ended shapes must be closed in a non-linear way, so as to approximate the hole's actual outline.
After the connection of the ends of open-ended shapes, all shapes are pruned of small non-substantial cluttering pixels that are connected to the actual outline of holes, so as to obtain clear shapes.
The blob detection is performed on the denoised images of edges, via the standard OpenCV blob detector. The output of the blob detector is a series of keypoints of blobs found. What is needed next, is to locate the outline of each blob. This can be performed by either starting a brush fire from each keypoint, that expands up to the blob's borders - it's outline - or by means of raycasting. The raycast algorithm approximates the blob's actual outline by casting rays from the blob's keypoint around 360 degrees in 360 / n intervals. Lastly, having found the keypoint and outline of a blob, its least area rotated rectangle is found.
All necessary pieces of information about holes found have been obtained up to this point. To sum up, a hole is identified by its:
- Keypoint
- Outline points
- The vertices of its bounding box
These pieces of information, along with the denoised depth image are then published to the Hole Fusion node for further, higher level, processing.
Under
depth_node_topics.yaml:subscribed_topics
the depth_image_topic is the name of the topic where the Depth node is subscribed to, where the input depth image is published by the Synchronizer node. Its value must comply with synchronizer_node_topics.yaml/published_topics/depth_image_topic.
Under
depth_node_topics.yaml:published_topics
the candidate_holes_topic is the name of the topic where the Depth node publishes information about holes found, along with the denoised input depth image. Its type is vision_communications::CandidateHolesVectorMsg.msg.