Merge DeltaFlow into codebase (#21)

* cuda(histlib): cuda library from icp-flow project.

* docs: update README with icp-flow in the official implementation.
* conf(optimization-based): update all config files.
* todo: update model file, double check with yancong and Qingwen confirm that: icp-flow results can be reproduced and tested.

* docs: fix small typo, and starting updating model file.

* feat(icp): core icp files, tested successfully.

* feat(deflowpp): update deflowpp model.
* hotfix(ssf): bug fixes in seflow+ssf as ssf concat two pc and forget to - index.

* feat(autolabel): update more autolabel through himo paper.
* add data with lidar_id and lidar_dt
* add flow_instances for afterward easy deltaflow update.

* feat(seflowpp): the full seflowpp process and train scripts.
* tested with demo good, need double check the results on av2 and update the training time & weight link also.

* fix(trainer): add seflowpp into trainer for folder name.

* data(zod): update zod extraction scripts.
* it could be a good reference for users to extract other data into h5 files etc.

* !feat(lr): update optimizer to new structure.
- update from DeltaFlow.
- align some format with copilot comments.
* update other submodule repo to align the lr changed.

* dcos(slurm): update slurm script, for reader to easy check how is the trainining setup.

* fix(env): fix some potiential env issue later
* add some notes

* fix(av2): instance label typo.

* fix(process): update key name in new version for seflow-variant process.

* hotfix(eval): updating num_frames into eval.

* hotfix(eval/test): for history frames, we update keys' name and it need rm ground mask also.

* small fix on ssl_label to None if under supervise training.

* feat(aug): merge data aug strategy from DeltaFlow project.

* docs(README): update readme.

align with paper.

* docs(README): update arxiv link

* feat(deltaflow): update deltaflow model file.
* checked with trained weight.

* conf: update deltaflow conf files
update README to show the progress.

* hotfix(eval): not assert but return if no gt class pt etc.
to avoid killed during eval.

* loss(deltaflow): add deltaflow loss.

* docs(README): update readme.

* revert to OpenSceneFlow readme for the codebase.

* docs: update README.
* update av2_mode to data_mode.
* revert back for zod process file
* runner metric is good in last version as it's range_bucket have different meaning.

* docs: update for diff opt methods.

* hotfix(dataset): fix eval_mask in dataset.
* we only evaluated points on non-ground point.
* no need print ssf_metrics since our train range is out of evaluation.

* docs(bib): add deltaflow bib back.

* docs(readme): update new bib file.

* feat(log): save the evaluation log and add inlineTee for output file.
* for easy sharing score afterward etc.
* update docs about upcoming works also.

* style: clean up the developed code previously.
* add readh5 and create_eval_pkl

* docs(bib): update bib.

* docs(data): update README demo data link.

Author: Kin-Zhang

README.md

@@ -11,10 +11,15 @@
 OpenSceneFlow is a codebase for point cloud scene flow estimation. 
 It is also an official implementation of the following papers (sorted by the time of publication):
 
+<!-- - **TeFlow: An Efficient Multi-frame Scene Flow Estimation Method**   
+*Qingwen Zhang, Chenhan Jiang, Xiaomeng Zhu, Yunqi Miao, Yushan Zhang, Olov Andersson, Patric Jensfelt*  
+Under Review   
+[ Strategy ] [ Self-Supervised ] - [ [OpenReview](https://openreview.net/forum?id=h70FLgnIAw) ] [ [Project](https://github.com/Kin-Zhang/TeFlow) ]&rarr; [here](#teflow) -->
+
 - **DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method**   
 *Qingwen Zhang, Xiaomeng Zhu, Yushan Zhang, Yixi Cai, Olov Andersson, Patric Jensfelt*  
 Conference on Neural Information Processing Systems (**NeurIPS**) 2025 - Spotlight   
-[ Backbone ] [ Supervised ] - [ [arXiv](https://arxiv.org/abs/2508.17054) ] [ [Project](https://github.com/Kin-Zhang/DeltaFlow) ]
+[ Backbone ] [ Supervised ] - [ [arXiv](https://arxiv.org/abs/2508.17054) ] [ [Project](https://github.com/Kin-Zhang/DeltaFlow) ]&rarr; [here](#deltaflow)
 
 - **HiMo: High-Speed Objects Motion Compensation in Point Clouds** (SeFlow++)   
 *Qingwen Zhang, Ajinkya Khoche, Yi Yang, Li Ling, Sina Sharif Mansouri, Olov Andersson, Patric Jensfelt*  
@@ -103,11 +108,11 @@ If you prefer to build the Docker image by yourself, Check [build-docker-image](
 
 ## 1. Data Preparation
 
-Refer to [dataprocess/README.md](dataprocess/README.md) for dataset download instructions. Currently, we support **Argoverse 2**, **Waymo**, **nuScenes**, [**MAN-TruckScene**](https://github.com/TUMFTM/truckscenes-devkit), [**ZOD**](https://github.com/zenseact/zod) and **custom datasets** (more datasets will be added in the future). 
+Refer to [dataprocess/README.md](dataprocess/README.md) for dataset download instructions. Currently, we support [**Argoverse 2**](https://www.argoverse.org/av2.html), [**Waymo**](https://waymo.com/open/), [**nuScenes**](https://www.nuscenes.org/), [**MAN-TruckScene**](https://github.com/TUMFTM/truckscenes-devkit), [**ZOD**](https://github.com/zenseact/zod) and **custom datasets** (more datasets will be added in the future). 
 
 After downloading, convert the raw data to `.h5` format for easy training, evaluation, and visualization. Follow the steps in [dataprocess/README.md#process](dataprocess/README.md#process). 
 
-For a quick start, use our **mini processed dataset**, which includes one scene in `train` and `val`. It is pre-converted to `.h5` format with label data ([HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/blob/main/demo_data.zip)/[Zenodo](https://zenodo.org/records/13744999/files/demo_data.zip)).
+For a quick start, use our **mini processed dataset**, which includes one scene in `train` and `val`. It is pre-converted to `.h5` format with label data ([HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/demo-data-v2.zip)).
 
 
 ```bash
@@ -125,14 +130,26 @@ Some tips before running the code:
 * If you want to use [wandb](wandb.ai), replace all `entity="kth-rpl",` to your own entity otherwise tensorboard will be used locally.
 * Set correct data path by passing the config, e.g. `train_data=/home/kin/data/av2/h5py/demo/train val_data=/home/kin/data/av2/h5py/demo/val`.
 
-And free yourself from trainning, you can download the pretrained weight from [HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow) and we provided the detail `wget` command in each model section. For optimization-based method, it's train-free so you can directly run with [3. Evaluation](#3-evaluation) (check more in the evaluation section).
+And free yourself from trainning, you can download the pretrained weight from [**HuggingFace - OpenSceneFlow**](https://huggingface.co/kin-zhang/OpenSceneFlow) and we provided the detail `wget` command in each model section. For optimization-based method, it's train-free so you can directly run with [3. Evaluation](#3-evaluation) (check more in the evaluation section).
 
 ```bash
 conda activate opensf
 ```
 
 ### Supervised Training
 
+#### DeltaFlow
+
+Train DeltaFlow with the leaderboard submit config. [Runtime: Around 18 hours in 10x RTX 3080 GPUs.]
+
+```bash
+# total bz then it's 10x2 under above training setup.
+python train.py model=deltaFlow optimizer.lr=2e-3 epochs=20 batch_size=2 num_frames=5 loss_fn=deflowLoss train_aug=True "voxel_size=[0.15, 0.15, 0.15]" "point_cloud_range=[-38.4, -38.4, -3.2, 38.4, 38.4, 3.2]" +optimizer.scheduler.name=WarmupCosLR +optimizer.scheduler.max_lr=2e-3 +optimizer.scheduler.total_steps=20000
+
+# Pretrained weight can be downloaded through (av2), check all other datasets in the same folder.
+wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/deltaflow/deltaflow-av2.ckpt
+```
+
 #### Flow4D
 
 Train Flow4D with the leaderboard submit config. [Runtime: Around 18 hours in 4x RTX 3090 GPUs.]
@@ -181,7 +198,7 @@ wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/deflow_best.ckp
 ### Feed-Forward Self-Supervised Model Training
 
 Train Feed-forward SSL methods (e.g. SeFlow/SeFlow++/VoteFlow etc), we needed to:
-1) process auto-label process.
+1) process auto-label process for training. Check [dataprocess/README.md#self-supervised-process](dataprocess/README.md#self-supervised-process) for more details. We provide these inside the demo dataset already.
 2) specify the loss function, we set the config here for our best model in the leaderboard.
 
 #### SeFlow
@@ -240,7 +257,8 @@ python save.py model=fastnsf
 
 ## 3. Evaluation
 
-You can view Wandb dashboard for the training and evaluation results or upload result to online leaderboard.
+You can view Wandb dashboard for the training and evaluation results or upload result to online leaderboard. 
+<!-- Three-way EPE and Dynamic Bucket-normalized are evaluated within a 70x70m range (followed Argoverse 2 online leaderboard). No ground points are considered in the evaluation. -->
 
 Since in training, we save all hyper-parameters and model checkpoints, the only thing you need to do is to specify the checkpoint path. Remember to set the data path correctly also.
 
@@ -249,7 +267,7 @@ Since in training, we save all hyper-parameters and model checkpoints, the only
 python eval.py checkpoint=/home/kin/seflow_best.ckpt data_mode=val
 
 # (optimization-based): it might need take really long time, maybe tmux for run it.
-python eval.py model=nsfp
+python eval.py model=nsfp +master_port=12344 # change diff port if you want to have multiple runners.
 
 # it will output the av2_submit.zip or av2_submit_v2.zip for you to submit to leaderboard
 python eval.py checkpoint=/home/kin/seflow_best.ckpt data_mode=test leaderboard_version=1
@@ -326,6 +344,7 @@ https://github.com/user-attachments/assets/07e8d430-a867-42b7-900a-11755949de21
 ## Cite Us
 
 [*OpenSceneFlow*](https://github.com/KTH-RPL/OpenSceneFlow) is originally designed by [Qingwen Zhang](https://kin-zhang.github.io/) from DeFlow and SeFlow. 
+It is actively maintained and developed by the community (ref. below works).
 If you find it useful, please cite our works:
 
 ```bibtex
@@ -347,16 +366,26 @@ If you find it useful, please cite our works:
   doi={10.1109/ICRA57147.2024.10610278}
 }
 @article{zhang2025himo,
-    title={HiMo: High-Speed Objects Motion Compensation in Point Clouds},
-    author={Zhang, Qingwen and Khoche, Ajinkya and Yang, Yi and Ling, Li and Sina, Sharif Mansouri and Andersson, Olov and Jensfelt, Patric},
-    year={2025},
-    journal={arXiv preprint arXiv:2503.00803},
+  title={{HiMo}: High-Speed Objects Motion Compensation in Point Cloud},
+  author={Zhang, Qingwen and Khoche, Ajinkya and Yang, Yi and Ling, Li and Mansouri, Sina Sharif and Andersson, Olov and Jensfelt, Patric},
+  journal={IEEE Transactions on Robotics}, 
+  year={2025},
+  volume={41},
+  pages={5896-5911},
+  doi={10.1109/TRO.2025.3619042}
+}
+@inproceedings{zhang2025deltaflow,
+  title={{DeltaFlow}: An Efficient Multi-frame Scene Flow Estimation Method},
+  author={Zhang, Qingwen and Zhu, Xiaomeng and Zhang, Yushan and Cai, Yixi and Andersson, Olov and Jensfelt, Patric},
+  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
+  year={2025},
+  url={https://openreview.net/forum?id=T9qNDtvAJX}
 }
-@article{zhang2025deltaflow,
-    title={{DeltaFlow}: An Efficient Multi-frame Scene Flow Estimation Method},
-    author={Zhang, Qingwen and Zhu, Xiaomeng and Zhang, Yushan and Cai, Yixi and Andersson, Olov and Jensfelt, Patric},
-    year={2025},
-    journal={arXiv preprint arXiv:2508.17054},
+@misc{zhang2025teflow,
+  title={{TeFlow}: Enabling Multi-frame Supervision for Feed-forward Scene Flow Estimation},
+  author={Zhang, Qingwen and Jiang, Chenhan and Zhu, Xiaomeng and Miao, Yunqi and Zhang, Yushan and Andersson, Olov and Jensfelt, Patric},
+  year={2025},
+  url={https://openreview.net/forum?id=h70FLgnIAw}
 }
 ```
 
@@ -373,13 +402,14 @@ And our excellent collaborators works contributed to this codebase also:
   pages={3462-3469},
   doi={10.1109/LRA.2025.3542327}
 }
-@article{khoche2025ssf,
-  title={SSF: Sparse Long-Range Scene Flow for Autonomous Driving},
+@inproceedings{khoche2025ssf,
+  title={{SSF}: Sparse Long-Range Scene Flow for Autonomous Driving},
   author={Khoche, Ajinkya and Zhang, Qingwen and Sanchez, Laura Pereira and Asefaw, Aron and Mansouri, Sina Sharif and Jensfelt, Patric},
-  journal={arXiv preprint arXiv:2501.17821},
-  year={2025}
+  booktitle={2025 IEEE International Conference on Robotics and Automation (ICRA)}, 
+  year={2025},
+  pages={6394-6400},
+  doi={10.1109/ICRA55743.2025.11128770}
 }
-
 @inproceedings{lin2025voteflow,
   title={VoteFlow: Enforcing Local Rigidity in Self-Supervised Scene Flow},
   author={Lin, Yancong and Wang, Shiming and Nan, Liangliang and Kooij, Julian and Caesar, Holger},

assets/slurm/dufolabel_sbatch.py

@@ -18,7 +18,7 @@ cd /proj/berzelius-2023-154/users/x_qinzh/OpenSceneFlow
 
 
 # data directory containing the extracted h5py files
-DATA_DIR="/proj/berzelius-2023-364/data/truckscenes/h5py/val"
+DATA_DIR="/proj/berzelius-2023-364/data/av2/h5py/sensor/train"
 
 TOTAL_SCENES=$(ls ${DATA_DIR}/*.h5 | wc -l)
 # Process every n-th frame into DUFOMap, no need to change at least for now.

assets/slurm/ssl-process.sh

@@ -9,6 +9,7 @@ wandb_mode: disabled # [offline, disabled, online]
 wandb_project_name: seflow
 
 train_data: /home/kin/data/av2/h5py/demo/train
+train_aug: False # involved by deltaflow
 val_data: /home/kin/data/av2/h5py/demo/val
 
 output: ${model.name}-${slurm_id}
@@ -28,8 +29,8 @@ gradient_clip_val: 5.0
 # optimizer ==> Adam
 optimizer:
   name: Adam # [Adam, AdamW]
-  lr: 1e-4
-loss_fn: seflowLoss # choices: [ff3dLoss, zeroflowLoss, deflowLoss, seflowLoss]
+  lr: 2e-4
+loss_fn: deflowLoss # choices: [ff3dLoss, zeroflowLoss, deflowLoss, seflowLoss]
 # add_seloss: {chamfer_dis: 1.0, static_flow_loss: 1.0, dynamic_chamfer_dis: 1.0, cluster_based_pc0pc1: 1.0}
 # ssl_label:
 

conf/config.yaml

@@ -1,6 +1,6 @@
 
 dataset_path: /home/kin/data/av2/h5py/sensor
-checkpoint: /home/kin/model_zoo/deflow.ckpt
+checkpoint: /home/kin/data/model_zoo/deltaflow_public/deltaflow-av2.ckpt
 data_mode: val # [val, test]
 save_res: False # [True, False]
 
@@ -15,7 +15,7 @@ output: ${model.name}-${slurm_id}
 gpus: 1
 seed: 42069
 eval_only: True
-wandb_mode: offline # [offline, disabled, online]
+wandb_mode: disabled # [offline, disabled, online]
 defaults:
   - hydra: default
   - model: deflow

conf/eval.yaml

@@ -0,0 +1,15 @@
+name: deltaflow
+
+target:
+  _target_: src.models.DeltaFlow
+  voxel_size: ${voxel_size}
+  point_cloud_range: ${point_cloud_range}
+  num_frames: ${num_frames}
+  planes: [16, 32, 64, 128, 256, 256, 128, 64, 32, 16] # 1st is #input channel, last is #output channel
+  # num_layer: [1, 1, 1, 1, 1, 1, 1, 1, 1] # the smallest model
+  num_layer: [2, 2, 2, 2, 2, 2, 2, 2, 2] # MinkUnet 18
+  # num_layer: [2, 3, 4, 6, 2, 2, 2, 2, 2] # MinkUnet 34
+  decay_factor: 0.4
+  decoder_option: default # choices: [default, deflow]
+
+val_monitor: val/Dynamic/Mean

conf/model/deltaflow.yaml

@@ -247,3 +247,6 @@ Process train data for self-supervised learning. Only training data needs this s
 ```bash
 python process.py --data_dir /home/kin/data/av2/h5py/sensor/train --scene_range 0,701
 ```
+
+As some users must have multi-nodes for running, here I provide an example SLURM script to run the data process in parallel. 
+Check [assets/slurm/ssl-process.sh](../assets/slurm/ssl-process.sh) for more details.

dataprocess/README.md

@@ -13,12 +13,13 @@
 import torch
 from torch.utils.data import DataLoader
 import lightning.pytorch as pl
-from lightning.pytorch.loggers import WandbLogger
+from lightning.pytorch.loggers import TensorBoardLogger, WandbLogger
 from omegaconf import DictConfig
 import hydra, wandb, os, sys
 from hydra.core.hydra_config import HydraConfig
 from src.dataset import HDF5Dataset
 from src.trainer import ModelWrapper
+from src.utils import InlineTee
 
 def precheck_cfg_valid(cfg):
     if os.path.exists(cfg.dataset_path + f"/{cfg.data_mode}") is False:
@@ -36,8 +37,8 @@ def main(cfg):
 
     if 'iter_only' in cfg.model and cfg.model.iter_only:
         from src.runner import launch_runner
-        print(f"---LOG[eval]: Run optmization-based method: {cfg.model.name}")
-        launch_runner(cfg, cfg.data_mode)
+        launch_runner(cfg, cfg.data_mode, output_dir)
+        print(f"---LOG[eval]: Finished optimization-based evaluation. Logging saved to {output_dir}/output.log")
         return
 
     if not os.path.exists(cfg.checkpoint):
@@ -47,27 +48,39 @@ def main(cfg):
     torch_load_ckpt = torch.load(cfg.checkpoint)
     checkpoint_params = DictConfig(torch_load_ckpt["hyper_parameters"])
     cfg.output = checkpoint_params.cfg.output + f"-e{torch_load_ckpt['epoch']}-{cfg.data_mode}-v{cfg.leaderboard_version}"
+    # replace output_dir ${old_output_dir} with ${output_dir}
+    output_dir = output_dir.replace(HydraConfig.get().runtime.output_dir.split('/')[-2], checkpoint_params.cfg.output.split('/')[-1])
     cfg.model.update(checkpoint_params.cfg.model)
     cfg.num_frames = cfg.model.target.get('num_frames', checkpoint_params.cfg.get('num_frames', cfg.get('num_frames', 2)))
 
     mymodel = ModelWrapper.load_from_checkpoint(cfg.checkpoint, cfg=cfg, eval=True)
-    print(f"\n---LOG[eval]: Loaded model from {cfg.checkpoint}. The backbone network is {checkpoint_params.cfg.model.name}.\n")
+    os.makedirs(output_dir, exist_ok=True)
+    sys.stdout = InlineTee(f"{output_dir}/output.log")
+    print(f"---LOG[eval]: Loaded model from {cfg.checkpoint}. The backbone network is {checkpoint_params.cfg.model.name}.")
+    print(f"---LOG[eval]: Evaluation data: {cfg.dataset_path}/{cfg.data_mode} set.\n")
 
-    wandb_logger = WandbLogger(save_dir=output_dir,
-                               entity="kth-rpl",
-                               project=f"deflow-eval", 
-                               name=f"{cfg.output}",
-                               offline=(cfg.wandb_mode == "offline"))
+    if cfg.wandb_mode != "disabled":
+        logger = WandbLogger(save_dir=output_dir,
+                            entity="kth-rpl",
+                            project=f"opensf-eval", 
+                            name=f"{cfg.output}",
+                            offline=(cfg.wandb_mode == "offline"))
+        logger.watch(mymodel, log_graph=False)
+    else:
+        # check local tensorboard logging: tensorboard --logdir logs/jobs/{log folder}
+        logger = TensorBoardLogger(save_dir=output_dir, name="logs")
 
-    trainer = pl.Trainer(logger=wandb_logger, devices=1)
+    trainer = pl.Trainer(logger=logger, devices=1)
     # NOTE(Qingwen): search & check: def eval_only_step_(self, batch, res_dict)
     trainer.validate(model = mymodel, \
                      dataloaders = DataLoader( \
                                             HDF5Dataset(cfg.dataset_path + f"/{cfg.data_mode}", \
                                                         n_frames=cfg.num_frames, \
                                                         eval=True, leaderboard_version=cfg.leaderboard_version), \
                                             batch_size=1, shuffle=False))
-    wandb.finish()
+    if cfg.wandb_mode != "disabled":
+        wandb.finish()
+    print(f"---LOG[eval]: Finished feed-forward evaluation. Logging saved to {output_dir}/output.log")
 
 if __name__ == "__main__":
     main()

eval.py

@@ -186,7 +186,6 @@ def main(
     if not os.path.exists(gm_config_path) and run_gm:
         raise FileNotFoundError(f"Ground segmentation config file not found: {gm_config_path}. Please check folder")
 
-    
     data_path = Path(data_dir)
     dataset = HDF5Data(data_path) # single frame reading.
     all_scene_ids = list(dataset.scene_id_bounds.keys())