[TASK] Detecting an ArUco marker obscured by a grid

[sophia-mina]

Description of task

During the last competition, one of the ArUco markers on the structure was obscured by a grid, causing the detector to fail. This is because successful scanning requires that the detector can first identify the marker’s encoded pattern.

The current filtering approach can remove the grid and reconstruct the obscured parts if the image is cropped to contain only the marker and the grid.

original image:

reconstructed image:

However, when the image includes the background, the filter can’t isolate the marker correctly. The next step is to automatically crop or segment the image to focus on the marker before reconstruction.

original image:

reconstructed image:

Current filter: removeGrid.py

Suggested Workflow

• Crop the image to only contain the marker and not the background.
• Translate the current Python code to C++
• Make the filter robust
• Integrate in vortex-image-filtering

Contacts

@sophia-mina

Code Quality

• Every function in header files are documented (inputs/returns/exceptions)
• The project has automated tests that cover MOST of the functions and branches in functions (pytest/gtest)
• The code is documented on the wiki (provide link)

[sophia-mina]

The current code (with threshold_green, inpaint, k-means and binary threshold) works alright for images from the simulator, but fails with "real" images since the whole image contains a yellow hue.

What I've tested since:

Using three thresholding values (RGB). The best manual tuning result was:
threshold_red = 0.435
threshold_green = 0.35
threshold_blue = 0.2

Also, manually finding the RGB values of the darkest area which should be black but isn't and subtracting them from the image. Then using threshold_green. The best result was:

Tried changing the contrast of the image both using multiplication (point process) and converting to YCrCb colorspace and stretching the Y channel. This made it very hard to find suitable threshold-values.
multiplication:

contrast stretching:

Radon transforming the image, then suppressing some of the vertical and horizontal lines to filter out the grid. This worked to some degree, but not well enough to make a working mask of the grid. The mask was made by subtracting the difference between the original image and the reconstructed image from radon transform. This didn't work since the filter also filtered out part of the marker.

[kluge7]

@sophia-mina you can make a branch on this repository: https://github.com/vortexntnu/vortex-image-filtering/tree/main and add a function for your filter (can start with what worked in the simulator) here https://github.com/vortexntnu/vortex-image-filtering/blob/main/image-filtering/src/image_processing.cpp. Then we can test it while driving around in the simulator

[kluge7]

We will attempt to define a box on the image received, and only inside this box we will run the filter. Then the job will be for the pilot to align the aruco marker with the grid.

• Add functionality for the filter to only run inside part of the image (a box area in the middle of the image)
• Add functionality to specify the size of the box
• Add functionality to rotate the box

Ideally the pilot should be able to change the dimensions of the box through the GUI/Foxglove.