Clarification on Cone Label Structure and Introduction of New Label Type

[Lazystinkdog]

Context:

Hi there! I'm from the OTH Regensburg Formula Student Team and we are looking to contribute and utilize the dataset from your repository. Currently, I am in the process of exporting our data into the required format. However, I have encountered some confusion regarding the cone labels.

Questions:

1. Cone Bounding Box Values:
   The instructions mention using the mmdetection3d label standard, which is primarily designed for bounding boxes. Upon inspecting the generate_scene.py file, I noticed the following hardcoded values:

# Save to file
with open(f"{folder_path}/labels/{filenum:07d}.txt", "w") as f:
    for x, y, c in zip(x_values, y_values, colors):
        if c == YELLOR_C:
            cone_type = 'Cone_Yellow'
        elif c == BLUE_C:
            cone_type = 'Cone_Blue'
        elif c == BIG_C:
            cone_type = 'Cone_Big'
        elif c == ORANGE_C:
            cone_type = 'Cone_Orange'
        else:
            continue
        f.write(f"{x} {y} 0.0 0.23 0.23 0.33 0.0 {cone_type}\n")

Should everyone use these fixed bounding box values (0.0 0.23 0.23 0.33 0.0) for all cones, or should we create individual bounding boxes based on the point cloud points associated with each cone?

2. Unlabeled Point Clouds:
   Are unlabeled point clouds required or optional? Currently, we have labels for every point cloud.

3. Introducing a New Label Type:
   In our current data, we do not have left/right labels for our cone positions, as this is handled independently by our pathplanning module. To accommodate this, would it be possible to introduce a new label type such as Cone_Unsorted or something similar? This would help us integrate our data without the need for left/right classification.

Best regards,
Tim

[bertaveira]

Hi! Glad to see some interest. I am happy to help and answer any questions and I am open to changes, pull requests and contribuitions.

About your specific questions here is my opinion:

1. Cone Bounding Box Values

I am open to suggestions there. So far I don't really use the size for anything. As a FS team the cone size is known so I was even experimenting with removing size from the detection networks. One can simply estimate the position altought the size might help the network learn. I left all of them to 23cm size since that way I can automate things opposed to having to manually label things.

I am all for having adjusted sizes if you want that. But it makes the process much harder and in reality i think the size should reflect the object size not the cloud of points size. More importantly is if you aggregate pointclouds you would have a fuller cloud so the bounding box size has to reflect the object size not just the volume occupied by the points of an object in a single pointcloud.

2. Unlabeled Point Clouds

I would say optional. But I don't think one should use consecutive pointclouds for training unless you are training something like a recurrent neural network or a transformer based with multiple scans. It just polutes the dataset with too similar scenarios.

Therefore I would still put every 10th or 20th as the main ones and then add the others as "unlabeled clouds". I think the naming is not the best. Perhaps we should call them unused pointclouds and let them have anotations or not (the teams can choose). That way they are not part of the main dataset but they are there if anyone wants to use them.

What do you think of this? If you have them labeled then great and I think it can be a great contribuition to the dataset but to keep it diverse only every 10th is part of the core dataset and the others are also labeled but part of the "unused clouds" (we can rename from unlabeled).

3. Introducing a New Label Type

Since we want to grow this for now I am fine with that. But I think we have to add to the data.json in the repo a boolean in a scene saying if it has a color label or not so people can know. If you are willing to add that and make a pull request here to add that boolean and make the current code add that boolean as well then I would agree with it.

If you use the generate_scene.py script you have a way to set colors to cones anyway. The recordings I used do not have cone color. That was set in generate_scene.py but if you choose to use nother method to annotate the pointclouds I am fine with not havign color. I would be itnerested to know how you generated the data if yuo would be willing to share with everyone so everyone can benefit from it

[Lazystinkdog]

First of all, thanks for your quick and detailed response. I really appreciate the explanations and insights :)

1. Cone Bounding Box Values

Your explanation about the bounding box values makes sense now. I was just very confused when I first encountered that code snippet. Keeping the size consistent for automation purposes sounds like a reasonable approach.

2. Unlabeled Point Clouds

That's fair. "Intermediate Pointclouds" could also be a fitting name. I hadn't considered the similarity issue between point clouds and was initially thinking that more data would always be better, but your explanation makes a lot of sense.

3. Cone Labels

Adding a boolean to the data.json file is a good idea. I’m happy to create a pull request for this, though it might take me a couple of weeks to get around to it. Regarding labeling in general, I haven’t used the generate_scene.py script due to some export issues with our custom ROS2 messages. Additionally, our tracks often feature a mix of green, blue, and yellow cones, which complicates the labeling process.

I’m also uncertain about using the SLAM map as ground truth. Initially, I thought it was a viable approach, but I’m concerned that our EKF SLAM might not be accurate enough. Moreover, SLAM cones don’t necessarily have to align with LiDAR cones at every timestamp. I’ve been considering projecting cone positions from future point clouds into the current point cloud as a potential alternative. When I have time I will try to implement both approaches and check if they work and with what accuracy.

As I’m considering writing my bachelor’s thesis on evaluating different approaches to 3D point cloud object detection. I would be interested to know if you have any experience using this dataset for such work and what your experiences were. Specifically, is the dataset sufficiently large for this kind of research?

Best regards,
Tim