docs(env): clean dataprocess env with only dataset api.

* remove evalai & open3d etc.
* fix #9
* visualization add vis raw point cloud.

Author: Kin-Zhang

dataprocess/README.md

@@ -12,47 +12,47 @@ We've updated the process dataset for:
 - [x] Waymo: check [here](#waymo-dataset). The process script was involved from [SeFlow](https://github.com/KTH-RPL/SeFlow).
 - [ ] nuScenes: done coding, public after review. Will be involved later by another paper.
 
-If you want to use all datasets above, there is a specific process environment in [envprocess.yml](../envprocess.yml) to install all the necessary packages. As Waymo package have different configuration and conflict with the main environment. Setup through the following command:
+If you want to use all datasets above, there is a specific process environment in [envprocess.yaml](../envprocess.yaml) to install all the necessary packages. As Waymo package have different configuration and conflict with the main environment. Setup through the following command:
 
 ```bash
-conda env create -f envprocess.yml
+conda env create -f envprocess.yaml
 conda activate dataprocess
+# NOTE we need **manually reinstall numpy** (higher than 1.22)
+# * since waymo package force numpy==1.21.5, BUT!
+# * hdbscan w. numpy<1.22.0 will raise error: 'numpy.float64' object cannot be interpreted as an integer
+# * av2 need numpy >=1.22.0, waymo with numpy==1.22.0 will be fine on code running.
+pip install numpy==1.22
 ```
 
 ## Download
 
 ### Argoverse 2.0
 
-Install their download tool:
-```bash
-mamba install s5cmd -c conda-forge
-```
-
-Download the dataset:
+Install their download tool `s5cmd`, already in our envprocess.yaml. Then download the dataset:
 ```bash
 # train is really big (750): totally 966 GB
-s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/sensor/train/*" sensor/train
+s5cmd --numworkers 12 --no-sign-request cp "s3://argoverse/datasets/av2/sensor/train/*" av2/sensor/train 
 
 # val (150) and test (150): totally 168GB + 168GB
-s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/sensor/val/*" sensor/val
-s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/sensor/test/*" sensor/test
+s5cmd --numworkers 12 --no-sign-request cp "s3://argoverse/datasets/av2/sensor/val/*" av2/sensor/val
+s5cmd --numworkers 12 --no-sign-request cp "s3://argoverse/datasets/av2/sensor/test/*" av2/sensor/test
 
 # for local and online eval mask from official repo
 s5cmd --no-sign-request cp "s3://argoverse/tasks/3d_scene_flow/zips/*" .
 ```
 
 Then to quickly pre-process the data, we can [read these commands](#process) on how to generate the pre-processed data for training and evaluation. This will take around 0.5-2 hour for the whole dataset (train & val) based on how powerful your CPU is.
 
-More [self-supervised data in AV2 LiDAR only](https://www.argoverse.org/av2.html#lidar-link), note: It **does not** include **imagery or 3D annotations**. The dataset is designed to support research into self-supervised learning in the lidar domain, as well as point cloud forecasting.
+Optional: More [self-supervised data in AV2 LiDAR only](https://www.argoverse.org/av2.html#lidar-link), note: It **does not** include **imagery or 3D annotations**. The dataset is designed to support research into self-supervised learning in the lidar domain, as well as point cloud forecasting. 
 ```bash
 # train is really big (16000): totally 4 TB
-s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/lidar/train/*" lidar/train
+s5cmd --numworkers 12 --no-sign-request cp "s3://argoverse/datasets/av2/lidar/train/*" av2/lidar/train
 
 # val (2000): totally 0.5 TB
-s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/lidar/val/*" lidar/val
+s5cmd --numworkers 12 --no-sign-request cp "s3://argoverse/datasets/av2/lidar/val/*" av2/lidar/val
 
 # test (2000): totally 0.5 TB
-s5cmd --no-sign-request cp "s3://argoverse/datasets/av2/lidar/test/*" lidar/test
+s5cmd --numworkers 12 --no-sign-request cp "s3://argoverse/datasets/av2/lidar/test/*" av2/lidar/test
 ``` 
 
 #### Dataset frames

dataprocess/extract_waymo.py

@@ -346,7 +346,6 @@ def create_group_data(group, pc, pose, gm = None, flow_0to1=None, flow_valid=Non
     first_frame = dataset_pb2.Frame.FromString(bytearray(all_data[0]))
     scene_id = first_frame.context.name
     total_lens = len(all_data)
-    # for data_idx in tqdm(range(1, total_lens), ncols=100):
     for data_idx in range(1, total_lens):
         if data_idx >= total_lens - 2:
             # 0: no correct flow label, end(total_lens - 1) - 1: no correct pose flow
@@ -384,7 +383,6 @@ def process_logs(data_dir: Path, map_dir: Path, output_dir: Path, nproc: int):
          data_dir: Argoverse 2.0 directory
          output_dir: Output directory.
     """
-    
     if not (data_dir).exists():
         print(f'{data_dir} not found')
         return
@@ -408,7 +406,7 @@ def process_logs(data_dir: Path, map_dir: Path, output_dir: Path, nproc: int):
 def main(
     flow_data_dir: str = "/home/kin/data/waymo/flowlabel",
     mode: str = "test",
-    map_dir: str = "/home/kin/data/waymo/flowlabel/maps",
+    map_dir: str = "/home/kin/data/waymo/flowlabel/map",
     output_dir: str ="/home/kin/data/waymo/flowlabel/preprocess",
     nproc: int = (multiprocessing.cpu_count() - 1),
     create_index_only: bool = False,

envprocess.yaml

@@ -6,25 +6,24 @@ dependencies:
   - python=3.8
   - pytorch::pytorch=2.0.0
   - pytorch::torchvision
+  - mkl==2024.0.0
   - numba
-  - numpy==1.22
+  - numpy
   - pandas
   - pip
   - scipy
   - tqdm
-  - scikit-learn
   - fire
+  - hdbscan
+  - s5cmd
   - pip:
     - nuscenes-devkit
     - av2==0.2.1
     - waymo-open-dataset-tf-2.11.0==1.5.0
-    - open3d==0.18.0
+    - dufomap==1.0.0
     - linefit
     - dztimer
-    - dufomap==1.0.0
-    - evalai
 
 # Reason about the version fixed:
 # numpy==1.22: package conflicts, need numpy higher or same 1.22
-# open3d==0.18.0: because 0.17.0 have bug on set the view json file
-# dufomap==1.0.0: in case later updating may not compatible with the code.
+# mkl==2024.0.0: https://github.com/pytorch/pytorch/issues/123097

tools/visualization.py

@@ -62,7 +62,7 @@ def check_flow(
 def vis(
     data_dir: str ="/home/kin/data/av2/preprocess/sensor/mini",
     res_name: str = "flow", # "flow", "flow_est"
-    start_id: int = -1,
+    start_id: int = 0,
     point_size: float = 2.0,
 ):
     dataset = HDF5Data(data_dir, vis_name=res_name, flow_view=True)
@@ -88,7 +88,10 @@ def vis(
         pose_flow = pc0[:, :3] @ ego_pose[:3, :3].T + ego_pose[:3, 3] - pc0[:, :3]
 
         pcd = o3d.geometry.PointCloud()
-        if res_name in ['dufo_label', 'label']:
+        if res_name == 'raw': # no result, only show **raw point cloud**
+            pcd.points = o3d.utility.Vector3dVector(pc0[:, :3])
+            pcd.paint_uniform_color([1.0, 1.0, 1.0])
+        elif res_name in ['dufo_label', 'label']:
             labels = data[res_name]
             pcd_i = o3d.geometry.PointCloud()
             for label_i in np.unique(labels):