evaluate

Deep Object Pose Estimation (DOPE) - Evaluation

IMPORTANT NOTE

These utilities currently require NVISII for visualization.

Simple Performance Evaluation

This directory contains code to measure the performance of your trained DOPE model. Below is an example of running the basic evaluation script:

python evaluate.py --data_prediction ../inference/output --data ../sample_data 

Arguments

--data:

Path to ground-truth data for the predictions that you wish to evaluate.

--data_prediction:

Path to predictions that were generated from running inference on your trained model. To support the evaluation of multiple sets of weights at once, this path can point to a folder containing the outputs of multiple inference results.

--models:

Path to 3D model files. These models are loaded before running evaluation and are rendered to compute the 3D error between the predicted results and ground truth. Point this argument at the root of the folder containing all of your different model files. Below is a sample folder structure:

/PATH_TO_MODELS_FOLDER
├── 002_master_chef_can
│   ├── 002_master_chef_can.xml
│   ├── points.xyz
│   ├── textured.mtl
│   ├── textured.obj
│   ├── textured_simple.obj
│   ├── textured_simple.obj.mtl
│   └── texture_map.png
└── 035_power_drill
    ├── 035_power_drill.xml
    ├── points.xyz
    ├── textured.mtl
    ├── textured.obj
    ├── textured_simple.obj
    ├── textured_simple.obj.mtl
    └── texture_map.png

If you trained DOPE on a new object and want to evaluate its performance, make sure to include the 3D model files in a folder that matches "class_name" in the ground truth .json file.

Multiple models can be loaded at once as the script will recursively search for any 3D models in the folder specified in --models.

--adds:

The average distance computed using the closest point distance between the predicted pose and the ground truth pose. This takes a while to compute. If you are only looking for a fast approximation, use --cuboid.

--cuboid:

Computes average distance using the 8 cuboid points of the 3D models. It is much faster than --adds but is only an approximation for the metric. It should be used for testing purposes.

More Complex ADD Metrics and Figure Generation

Requirements

Run the download content file: ./download_content.sh, this downloads a simple scene with annotation rendered by NViSII and with DOPE predictions.

How to run

If you downloaded the previous content you can execute the following:

python add_compute.py

which should generate the following results:

mean 0.0208515107260977 std 0.016006083915162977 ratio 17/22
auc at  0.02 : 0.5
auc at  0.04 : 0.6818181818181818
auc at  0.06 : 0.7272727272727273
auc 0.6115249999999999

This means the area under the curve, auc from 0 cm to 10 cm is 0.61. This script also produces graphs such as:

These are the metrics we reported in the original DOPE paper. I will refer to the paper for explaining the graph.

Assumptions

We make a few assumptions in this script.

1. We assume the folders structures are the same and there are only scenes in the folder. See data/ folder example from downloading the content.
2. We assume the notation folder is in the OpenGL format and that it is using the nvisii outputs from the data generation pipeline. If you use a different file format please update the script or your data.
3. We assume the inferences are from DOPE inference, e.g., the poses are in the OpenGL format. These conventions are easy to change, e.g., look for the line visii_gu.get_transform().rotate_around in add_compute.py to change the pose convention.

If the script takes to long to run, please run with --cuboid, instead of using the 3d models vertices to compare the metric, it uses the 3d cuboid of the 3d model to compute the metric.

2D image-based metric

If you do not have a 3d model of your model and you would prefer to just measure the quality of your detections with a simple euclidean distance for the predicted keypoints. You can use python kpd_compute.py, this is very similar to add_compute.py and it behaves very similarly. The metric used here is the euclidean (L2) distance from predicted keypoint and the ground truth keypoint. Then we propose to use a threshold plot to evaluate the data, similar to the ADD metric.

Rendering 3d predictions using NViSII

We added a script for you to add render of the 3d model to your predictions. It uses a version of NViSII that is not released yet. Please manually install this wheel.

# for scenes with DOPE inference
python render_json.py --path_json data/table_dope_results/scene1/00300.json --scale 0.01 --opencv --contour --gray
# for scenes generated by nvisii 
python render_json.py --path_json data/table_ground_truth/scene1/00100.json --scale 0.01 --contour --gray

--gray render the 3d model as a gray image and --contour adds the 3d model contour in green.

Rendering BOP format on images

Using the same argument, you can use this script on the BOP annotation with 3d models. The script simply rebuilds the data structure that is needed to load the scene.

python render_json.py --path_json /PATH/TO/hope_bop/hope_val/val/000001/scene_gt.json --bop --objs_folder /PATH/TO/hope_bop/hope_models/models/ --gray --contour --bop_scene 0

Only --bop is needed to be passed to load a bop scene. You can pass which scene you want to load with --bop_scene. The rest is the same behavior. This was only tested on the HOPE data.

Assumptions

We assume that you have the intrinsics stored in the camera data. If you do not have them, the script uses 512 x 512 with a fov of 0.78. If the camera data is complete, like with NViSII data, it will use the camera intrinsics.