Lidar Waveforms are Worth 40×128×33 Words

This repository provides the official code for "Lidar Waveforms are Worth 40×128×33 Words" (ICCV 2025).

Included Components:

• Neural DSP: The transformer architecture for lidar waveform processing.
• Synthetic CARLA Dataset: Simulation-based training data. Note: The real lidar waveform data contains proprietary sensor information that the sensor provider is not allowing for release as of now.

Install

Create the conda env:

conda create -n neuraldsp python=3.9
conda activate neuraldsp 

Install the required packages:

pip install fvcore iopath
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py39_cu117_pyt1131/download.html
pip install -e .

Data

Prepare the Dataset

• Download the synthetic CARLA dataset here.
• Create a data folder in the root directory of this package.
• Organize the downloaded data according to the filetree below.

📁 data

└── carla/
    ├── centroid_tof/
    │   ├── Town01_Opt-scene1-frame0.npz
    │   └── ...
    ├── gt_tof/
    │   ├── Town01_Opt-scene1-frame0.npz
    │   └── ...
    ├── sensor_specs/
    │   ├── matched_filter_values.npy
    │   └── view_direction_carla_60deg.npy
    ├── splits/
    │   ├── test.txt
    │   ├── train.txt
    │   └── val.txt
    └── waveform_8bit/
        ├── Town01_Opt-scene1-frame0.npy
        └── ...

Explore the Data

You can explore the waveform data and the ground truth in the interactive viewer:

python scripts/viewer_data.py

Run a Training

Launch a training on a single GPU with:

python scripts/train.py --cfg_name neuraldsp.yaml --extra_tag initial_run

Eval Results

After training, evaluate model performance on the synthetic test set:

python scripts/eval_training.py \
    --cfg_path <path_to_log_dir>/neuraldsp.yaml \
    --ckpt_path <path_to_log_dir>/<ckpt_name>.pth

The logs are saved by default in the logs directory in the root of the package.

Note The quantitative results differ slightly from those reported in the paper. The simulated data used proprietary sensor parameters that are removed for this release. We obtain a Chamfer Distance of 0.298m and a Recall 91.95%.

Results can also be visualized in the interactive viewer:

python scripts/viewer_results.py \
    --cfg_name <path_to_log_dir>/neuraldsp.yaml  \
    --ckpt_path <path_to_log_dir>/<ckpt_name>.pth 

Citation(s)

If you find this work useful, please consider citing our paper.

@inproceedings{scheuble2025neuralDSP,
title={Lidar Waveforms are Worth 40x128x33 Words},
author={Dominik Scheuble and Hanno Holzhueter and Steven Peters and Mario Bijelic and Felix Heide},
journal={2025 IEEE/CVF International Conference on Computer Vision (ICCV},
year={2025}
}