Preprocessing of the MVSEC dataset skips all data

[vbrebion]

Hi!

Thank you very much for your work and for taking the time to release the associated code!

I am currently trying to preprocess the MVSEC dataset using your MVSEC_to_frames_clear.py script.
However, all data seems to be skipped, and no output file is produced.

To give a bit more details, the code keeps outputting messages similar to the following one:

- Skipping ts: 10 ||| 1506117902840734 ||| num_patches tensor([[0]]) ||| skipped 9

Looking into the code, it seems that num_patches is always 0, which itself is due to iter_events_repr.sum() also always being 0.
Looking further into the events_to_frames.process_event_stream function, the issue seems to come from the computation of diff (line 61): maxTime is defined as 256000, while time_steps are ROS timestamps which have a value of around 1e+15. Therefore, diff will always be around -1e+15, making the values in frames being zeroed out (lines 63-64).

Have I made a mistake somewhere, or is there an issue with the code?

Best regards,
Vincent

[anusha-devulapally]

Hi @vbrebion, I am facing the same issue. Were you able to fix this? If so, it would be really helpful if you could share the working file.

Thanks,
Anusha

[vbrebion]

Hi @anusha-devulapally

Sadly, I wasn't able to fix this issue, and I don't really have time at the moment to look into it any further.
I am tagging @AlbertoSabater in the hope that he may see this exchange and have an answer.

I guess there is an issue with how the timestamps are processed (maybe an incorrect conversion between the ROS and python formats?) as noted in my first message, but I am unsure how to solve it properly.

Best,
Vincent

[AlbertoSabater]

It is a common behavior to skip/merge time steps that do not contain enough event information. It is not if all the time steps are skipped. If the latter happens, try commenting lines 127 and 137 and un-comment lines 128 and 138 of the events_to_frames.py script.
maxTime is applied to time differences, so it is agnostic of the absolute time magnitude. It is used to normalize the time dimension to this value.

Unfortunately, I am no longer working at the lab where I did the research so I do not have the time and resources for downloading the dataset and debugging.

[Harin99]

I suffer this same issue.

In events_to_frames.py code (61 line~), maxTime is 256000, time_steps is rosbag timestamp (e.g., 1.5061179e+15). So the value of "diff" variable can be negative. So the frames values become all zero.

So I thought that the timestamp conversion was not done well. Therefore, before starting line 61, I normalized time_steps and frames to the range of maxTime as shown below, and data is created only the first time and then skipped again.. I am asking because I am confused about whether this method is the correct method. Maybe it's not.

How can i normalize well those variables (frames , time_steps)

# Make each window in the range (0, maxTime)
    ## Normalization 
    min_ts = time_steps.min()
    max_ts = time_steps.max()
    time_steps = (time_steps - min_ts) * maxTime / (max_ts - min_ts)
    
    min_ts = frames.min()
    max_ts = frames.max()
    frames = (frames - min_ts) * maxTime / (max_ts - min_ts)
    ## Normalization
    
    diff = maxTime - time_steps
    diff = diff[:,None,None,None,None].astype('float32')
    frames = frames + diff           # Make newer events to have higher value than the older ones
    frames[frames < 0] = 0
    frames = (frames / maxTime).astype('float32')             # Make newer events to have a value close to 1 and older ones a value close to 0
    if return_mem: return (frames, min_max_values), (pos_fifo, neg_fifo)
    else: 
        if return_tw_ends: return frames, min_max_values, tw_ends
        else: return frames, min_max_values

Thank you so much :)

[leejielong]

It seems like the issue is with the large rosbag timestamps that are in the 1e+15 range. This leads to numerical overflows as the timestamps are processed. Currently, the cleanest method for me seems to be to offset all timestamps (events,depths,images) by the timestamp of the first event in the sequence. I do it by drawing the first event packet from the event generator at the beginning of each sequence. Then as we read the events, depths and image timestamps from the respective generators, we subtract the initial timestamp (init_ts) from all the timestamp readings. This seems to work for me.

Another small bug fix:

1. line 125: should be max()-maxTime so that we actually filter off events outside the 256ms window.
   buffer_events = buffer_events[buffer_events[:,2] > (buffer_events[:,2].max()-maxTime)]

Cheers!

[SharineLee]

It seems like the issue is with the large rosbag timestamps that are in the 1e+15 range. This leads to numerical overflows as the timestamps are processed. Currently, the cleanest method for me seems to be to offset all timestamps (events,depths,images) by the timestamp of the first event in the sequence. I do it by drawing the first event packet from the event generator at the beginning of each sequence. Then as we read the events, depths and image timestamps from the respective generators, we subtract the initial timestamp (init_ts) from all the timestamp readings. This seems to work for me.似乎问题出在 1e+15 范围内的大型 rosbag 时间戳上。这导致在处理时间戳时出现数值溢出。目前对我来说，最干净的方法似乎是偏移所有时间戳（事件、深度、图像）到序列中第一个事件的时刻。我通过在序列开始时从事件生成器中绘制第一个事件数据包来实现。然后，当我们从各自的生成器中读取事件、深度和图像时间戳时，我们从所有时间戳读取中减去初始时间戳（init_ts）。这似乎对我有效。

Another small bug fix:另一个小错误修复：

1. line 125: should be max()-maxTime so that we actually filter off events outside the 256ms window.行 125：应该是 max()-maxTime，以便我们实际上过滤掉 256ms 窗口外的事件。
   buffer_events = buffer_events[buffer_events[:,2] > (buffer_events[:,2].max()-maxTime)]

Cheers!  干杯！

Did you solve the above problem? Can you share the code for this part? thanks.