docs: update data link, and data-demo img.

* link author-response file also.
* test(backup): upload test script (mainly for gt zip file for online evaluaiton.
终于快整理完了！差visualization part

Author: Kin-Zhang

README.md

@@ -5,16 +5,16 @@ HiMo: High-Speed Objects Motion Compensation in Point Clouds
 [![page](https://img.shields.io/badge/Project-Page-green)](https://kin-zhang.github.io/HiMo)
 [![video](https://img.shields.io/badge/video-YouTube-FF0000?logo=youtube&logoColor=white)](https://youtu.be/rofaKfezIx0?si=59mMPLYUMgvrkRGj)
 
-Note: I knew sometime we might want to see codes asap, so I upload all my experiment codes without cleaning up (some lib might missing etc). 
+Note 2025-08-28: I knew sometime we might want to see codes asap, so I upload all my experiment codes without cleaning up (some lib might missing etc). 
 I will try my best to cleanup TBD list here:
 
-Update 2025-12-30 I'm back and updating the script now.... Hope I can finish all it before 2026-01-13.
+Update 2025-12-30: I'm back and updating the script now.... Hope I can finish all it before 2026-01-13.
 
 - [x] Update the repo README.
 - [x] Update OpenSceneFlow repo for dataprocess and SeFlow++.
 - [x] Test successfully evaluation codes on Scania and Argoverse2.
 - [ ] Test successfully visualization codes.
-- [ ] Upload Scania validation set (w/o gt).
+- [x] Upload Scania validation set (w/o gt).
 - [x] Setup leaderboard for users get their Scania val score.
 - [x] Downstream task two repos README update.
 - [x] Public the [author-response file](https://github.com/KTH-RPL/HiMo/discussions/1) for readers to check some discussion and future directions etc.
@@ -68,7 +68,14 @@ For further method, you can refer this script for the same format saving.
 
 ### Scania
 
-You need upload your result files to the public leaderboard page, we present the best model we have in the paper:
+First download the Scania validation set from [huggingface](https://huggingface.co/datasets/KTH/HiMo) (~ 2GB) with 10 scenes.
+```bash
+# setup hf cli if you don't have it
+# curl -LsSf https://hf.co/cli/install.sh | bash
+hf download KTH/HiMo --repo-type dataset
+```
+
+Get the result files with HiMo by following the best model we have in the paper, and save the .zip files for afterward online evaluation:
 ```bash
 cd OpenSceneFlow
 # (feed-forward): load ckpt
@@ -139,9 +146,7 @@ For Video animation example, we use [manim](https://www.manim.community/). I may
 }
 ```
 
-💞 Thanks to Bogdan Timus and Magnus Granström from Scania and Ci Li from KTH RPL, who helped with this work. 
-We also thank Yixi Cai, Yuxuan Liu, Peizheng Li and Shenghai Yuan for helpful discussions during revision.
-We also thank the anonymous reviewers for their useful comments.
+💞 We sincerely thank Bogdan Timus and Magnus Granström (Scania) and Ci Li (KTH RPL) for their contributions to this work. We also appreciate Yixi Cai, Yuxuan Liu, Peizheng Li, and Shenghai Yuan for their insightful discussions, as well as the anonymous reviewers for their valuable feedback.
 
 This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation and Prosense (2020-02963) funded by Vinnova. 
 The computations were enabled by the supercomputing resource Berzelius provided by National Supercomputer Centre at Linköping University and the Knut and Alice Wallenberg Foundation, Sweden.

assets/docs/scania-val.png

@@ -0,0 +1,145 @@
+"""
+Repack Scania h5 files:
+1. Rename 'SensorsCenter' to 'lidar_center'
+2. Change lidar_center to (N, 4, 4) with identity rotation
+3. Delete 'himu_seflowpp' key
+4. Fix data types for PyTorch compatibility (uint32 -> int64, etc.)
+5. Repack the h5 file to reclaim space
+
+Usage:
+    python tools/repack_h5_scania.py --data_dir /home/kin/data/scania/val_v1repacked
+"""
+
+import os
+import h5py
+import numpy as np
+from pathlib import Path
+from tqdm import tqdm
+import argparse
+import shutil
+
+# Define expected dtypes for each key (PyTorch compatible)
+# Supported types: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, bool
+DTYPE_MAP = {
+    # 'timestamp': np.int64,
+    # 'flow_instance_id': np.int32,  # uint32 -> int32
+    'lidar_id': np.uint8,
+    # 'flow_category_indices': np.uint8,
+    # 'flow_is_valid': np.bool_,
+    'ground_mask': np.bool_,
+    'lidar': np.float32,
+    'lidar_dt': np.float32,
+    # 'flow': np.float32,
+    'pose': np.float64,
+    'ego_motion': np.float32,
+    'lidar_center': np.float32,
+}
+
+
+def process_single_h5(h5_path: Path, output_path: Path):
+    """Process a single h5 file and save to output path."""
+    
+    with h5py.File(h5_path, 'r') as f_in:
+        with h5py.File(output_path, 'w') as f_out:
+            for timestamp_key in f_in.keys():
+                group_in = f_in[timestamp_key]
+                group_out = f_out.create_group(timestamp_key)
+                
+                for sub_key in group_in.keys():
+                    # Skip himu_seflowpp
+                    if sub_key in ['seflowpp_best']:
+                        continue
+                    
+                    # Rename SensorsCenter to lidar_center and convert to (N, 4, 4)
+                    # Also handle existing lidar_center that needs to be converted
+                    if sub_key == 'SensorsCenter' or sub_key == 'lidar_center':
+                        data = group_in[sub_key][:]
+                        # Convert to (N, 4, 4) with identity rotation and translation from original data
+                        if len(data.shape) == 3 and data.shape[1] == 4 and data.shape[2] == 4:
+                            # Already (N, 4, 4), keep as is
+                            new_data = data.astype(np.float32)
+                        elif len(data.shape) == 2 and data.shape[1] == 3:
+                            # (N, 3) -> (N, 4, 4) with identity rotation
+                            N = data.shape[0]
+                            new_data = np.zeros((N, 4, 4), dtype=np.float32)
+                            new_data[:, :3, :3] = np.eye(3)  # identity rotation
+                            new_data[:, :3, 3] = data  # translation
+                            new_data[:, 3, 3] = 1.0
+                        else:
+                            print(f"Warning: Unexpected shape {data.shape} for {sub_key} in {h5_path}/{timestamp_key}")
+                            new_data = data.astype(np.float32)
+                        
+                        group_out.create_dataset('lidar_center', data=new_data)
+                    else:
+                        if sub_key not in DTYPE_MAP:
+                            print(f"Warning: {sub_key} not in DTYPE_MAP, skip this key.")
+                            continue
+                        # Copy other datasets with proper dtype conversion
+                        # Handle scalar datasets (e.g., timestamp)
+                        if group_in[sub_key].shape == ():
+                            data = group_in[sub_key][()]
+                        else:
+                            data = group_in[sub_key][:]
+                        
+                        # Convert dtype if needed for PyTorch compatibility
+                        if sub_key in DTYPE_MAP:
+                            data = np.array(data, dtype=DTYPE_MAP[sub_key])
+                        elif data.dtype == np.uint32:
+                            # uint32 not supported by PyTorch, convert to int64
+                            data = data.astype(np.int64)
+                        elif data.dtype == np.uint64:
+                            # uint64 not supported by PyTorch, convert to int64
+                            data = data.astype(np.int64)
+                        
+                        group_out.create_dataset(sub_key, data=data)
+
+
+def main(data_dir: str, in_place: bool = True):
+    data_path = Path(data_dir)
+    
+    if not data_path.exists():
+        print(f"Error: {data_path} does not exist")
+        return
+    
+    h5_files = sorted(list(data_path.glob("*.h5")))
+    print(f"Found {len(h5_files)} h5 files in {data_path}")
+    
+    if len(h5_files) == 0:
+        print("No h5 files found!")
+        return
+    
+    # Create temp directory for repacked files
+    temp_dir = data_path / "_temp_repack"
+    temp_dir.mkdir(exist_ok=True)
+    
+    for h5_file in tqdm(h5_files, desc="Processing h5 files"):
+        temp_output = temp_dir / h5_file.name
+        
+        try:
+            process_single_h5(h5_file, temp_output)
+            
+            if in_place:
+                # Replace original file with repacked one
+                shutil.move(str(temp_output), str(h5_file))
+        except Exception as e:
+            print(f"Error processing {h5_file}: {e}")
+            if temp_output.exists():
+                temp_output.unlink()
+            continue
+    
+    # Clean up temp directory
+    if temp_dir.exists():
+        shutil.rmtree(temp_dir)
+    
+    print("Done!")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Repack Scania h5 files")
+    parser.add_argument("--data_dir", type=str, required=True, 
+                        help="Path to directory containing h5 files")
+    parser.add_argument("--no_in_place", action="store_true",
+                        help="If set, don't replace original files (for testing)")
+    
+    args = parser.parse_args()
+    main(args.data_dir, in_place=not args.no_in_place)

tools/test/repack_h5_scania.py

@@ -0,0 +1,185 @@
+"""
+# Created: 2024-04-15 17:32
+# Copyright (C) 2024-now, RPL, KTH Royal Institute of Technology
+# Author: Qingwen Zhang  (https://kin-zhang.github.io/)
+
+# Description: 
+# Save the ground truth compensation distance into feather files inside a zip file for 
+# afterward easy evaluation on the public benchmark.
+
+
+PUT THIS FILE INTO HIMO FOLDER AND RUN.
+"""
+
+import fire, time, json
+from tqdm import tqdm
+import numpy as np
+import numpy.typing as npt
+NDArrayFloat = npt.NDArray[np.float64]
+
+from typing import Tuple
+from pathlib import Path
+import pandas as pd
+from zipfile import ZipFile
+from io import BytesIO
+import time, os, sys
+
+BASE_DIR = os.path.abspath(os.path.join( os.path.dirname( __file__ ), 'OpenSceneFlow' ))
+sys.path.append(BASE_DIR)
+from src.dataset import HDF5Dataset
+from src.utils.av2_eval import CLOSE_DISTANCE_THRESHOLD
+from utils import check_valid, ego_pts_mask, flow2compDis, refine_pts
+
+
+def read_output_zip(
+    zip_path: str,
+    sweep_uuid: Tuple[str, int],
+) -> np.ndarray:
+    """Read compensation distance predictions from a zip file.
+
+    Args:
+        zip_path: Path to the zip file containing predictions.
+        sweep_uuid: Identifier of the sweep being predicted (log_id, timestamp_ns).
+
+    Returns:
+        compensation_dis: (N, 3) compensation distance predictions.
+    """
+    with ZipFile(zip_path, 'r') as myzip:
+        feather_path = f"{sweep_uuid[0]}/{sweep_uuid[1]}.feather"
+        with myzip.open(feather_path) as f:
+            df = pd.read_feather(BytesIO(f.read()))
+    
+    compensation_dis = np.stack([
+        df['comp_dis_x_m'].values.astype(np.float32),
+        df['comp_dis_y_m'].values.astype(np.float32),
+        df['comp_dis_z_m'].values.astype(np.float32),
+    ], axis=1)
+    
+    # If category/instance columns are present, they will be available in the DataFrame
+    eval_mask = df['eval_mask'].values.astype(bool)
+    flow_category = df['flow_category_indices'].values.astype(np.uint8) if 'flow_category_indices' in df.columns else None
+    flow_instance = df['flow_instance_id'].values.astype(np.uint32) if 'flow_instance_id' in df.columns else None
+    return compensation_dis, eval_mask, flow_category, flow_instance
+    
+def write_output_file(
+    compensation_dis: NDArrayFloat,
+    sweep_uuid: Tuple[str, int],
+    output_dir: Path,
+    eval_mask: NDArrayFloat,
+    flow_category_indices: np.ndarray = None,
+    flow_instance_id: np.ndarray = None,
+    gt_flow_norm: np.ndarray = None,
+    pc0: np.ndarray = None,
+) -> None:
+    """Write an output predictions file in the correct format for submission.
+
+    Args:
+        compensation_dis: (N,3) compensation_dis predictions.
+        sweep_uuid: Identifier of the sweep being predicted (log_id, timestamp_ns).
+        output_dir: Top level directory containing all predictions.
+        pc0: (N,3) original point cloud coordinates (needed for Chamfer calculation).
+    """
+    output_log_dir = output_dir / sweep_uuid[0]
+    output_log_dir.mkdir(exist_ok=True, parents=True)
+    compensation_dis_x_m = compensation_dis[:, 0].astype(np.float32)
+    compensation_dis_y_m = compensation_dis[:, 1].astype(np.float32)
+    compensation_dis_z_m = compensation_dis[:, 2].astype(np.float32)
+
+    # Build the output DataFrame; include category/instance if provided
+    data_dict = {
+        "comp_dis_x_m": compensation_dis_x_m,
+        "comp_dis_y_m": compensation_dis_y_m,
+        "comp_dis_z_m": compensation_dis_z_m,
+        "eval_mask": eval_mask.astype(np.uint8),
+    }
+    if flow_category_indices is not None:
+        data_dict['flow_category_indices'] = flow_category_indices.astype(np.uint8)
+    if flow_instance_id is not None:
+        data_dict['flow_instance_id'] = flow_instance_id.astype(np.uint32)
+    if gt_flow_norm is not None:
+        data_dict['gt_flow_norm'] = gt_flow_norm.astype(np.float32)
+    # Save pc0 for Chamfer calculation (GT only)
+    if pc0 is not None:
+        data_dict['pc0_x'] = pc0[:, 0].astype(np.float32)
+        data_dict['pc0_y'] = pc0[:, 1].astype(np.float32)
+        data_dict['pc0_z'] = pc0[:, 2].astype(np.float32)
+
+    output = pd.DataFrame(data_dict)
+    output.to_feather(output_log_dir / f"{sweep_uuid[1]}.feather")
+
+
+def zip_res(res_folder, output_file="submit.zip"):
+    all_scenes = [f for f in os.listdir(res_folder) if os.path.isdir(os.path.join(res_folder, f))]
+    with ZipFile(output_file, "w") as myzip:
+        for scene in all_scenes:
+            scene_folder = os.path.join(res_folder, scene)
+            # only directory
+            all_logs = [f for f in os.listdir(scene_folder) if os.path.isfile(os.path.join(scene_folder, f)) and f.endswith('.feather')]
+            for log in all_logs:
+                file_path = os.path.join(scene, log)
+                myzip.write(os.path.join(res_folder, file_path), arcname=file_path)
+    # remove the folder after zipping
+    for scene in all_scenes:
+        scene_folder = os.path.join(res_folder, scene)
+        os.system(f"rm -rf {scene_folder}")
+    print(f"Zipped results to {res_folder} into {output_file}. Submit your result by uploading this zip file.")
+    # print()
+    return output_file
+
+def main(
+    # data_dir: str ="/home/kin/data/Scania/preprocess/val",
+    data_dir: str ="/home/kin/data/av2/h5py/sensor/himo/demo",
+    output_dir: str = "/home/kin/data/av2/h5py/sensor/himo/results",
+    res_name: str = "flow"
+):
+    data_dir = Path(data_dir)
+    output_dir = Path(output_dir)
+    output_dir.mkdir(exist_ok=True, parents=True)
+    data_name, _ = check_valid(str(data_dir), res_name, None)
+
+    dataset = HDF5Dataset(data_dir, vis_name=res_name, eval=True)
+    for data_id in tqdm(range(0, len(dataset)), ncols=120, desc=f"Extracting {res_name} from {data_dir}"):
+        data = dataset[data_id]
+        pc0, pose0, pose1 = data['pc0'], data['pose0'], data['pose1']
+        ego_pose = np.linalg.inv(pose1) @ pose0
+        pose_flow = pc0[:, :3] @ ego_pose[:3, :3].T + ego_pose[:3, 3] - pc0[:, :3]
+        pc_dis = np.linalg.norm(pc0[:, :2], axis=1)
+        dis_mask = pc_dis <= CLOSE_DISTANCE_THRESHOLD
+        notgm_mask = ~data['gm0']
+
+        # scania
+        if data_name == "scania":
+            mask_eval = dis_mask & data['flow_is_valid'] & notgm_mask & ego_pts_mask(pc0)
+        else:
+            mask_eval = dis_mask & notgm_mask & ego_pts_mask(pc0, min_bound=[-1.5, -1.5, -2.0], max_bound=[1.5, 1.5, 2.0])
+        
+        try:
+            est_flow = np.zeros_like(pose_flow) if res_name == "raw" else (data[res_name] - pose_flow)
+        except:
+            print(f"Warning: {data['scene_id']} {data['timestamp']} has no result for {res_name}, set zero flow.")
+            est_flow = np.zeros_like(pose_flow)
+
+        # NOTE: we compensated to the latest observation. dts: (N,1) Nanosecond offsets _from_ the start of the sweep.
+        # NOTE: we compensated to the latest observation. dts: (N,1) Nanosecond offsets _from_ the start of the sweep.
+        dt0 = max(data['lidar_dt']) - data['lidar_dt']
+        
+        # GT flow and GT compensation distance
+        gt_flow = data['flow'] - pose_flow  # GT flow in ego-motion compensated frame
+        gt_comp_dis = flow2compDis(gt_flow, dt0, sensor_dt=0.1)  # GT compensation distance
+        gt_flow_norm = np.linalg.norm(gt_flow, axis=1).astype(np.float32)
+
+        # Attach category/instance arrays so the saved feather contains labels for GT
+        flow_cat = data['flow_category_indices'] if 'flow_category_indices' in data else None
+        flow_inst = data['flow_instance_id'] if 'flow_instance_id' in data else None
+        
+        # Save GT compensation distance (for GT zip file)
+        write_output_file(gt_comp_dis, (data['scene_id'], str(data['timestamp'])), output_dir, mask_eval, 
+                          flow_category_indices=flow_cat, flow_instance_id=flow_inst, 
+                          gt_flow_norm=gt_flow_norm, pc0=pc0[:, :3])
+
+    zip_res(output_dir, output_file=f"{output_dir}/{res_name}-submit.zip")
+
+if __name__ == '__main__':
+    start_time = time.time()
+    fire.Fire(main)
+    print(f"Time used: {time.time() - start_time:.2f} s")