Skip to content

Overwriting small amount of data during multi-day campaign #27

Description

@gseropian

During multi-day campaign, a small amount (up to one hour) of the data is overwritten and lost during the conversion to MSEED during the hour at which data was downloaded (or battery changed), because of the '--force-overwrite' option of mseedrename.

It's probably easier to explain with an example: say you start recording on March 1st at 8am and come back to download the data every following day at 10:15 am. By the time you collect the data on the 3rd day, the file structure in the datacube will look like this:

+--20260301
|   |
|   +-- 03010800.C3X
|   +-- 03020000.C3X
|
+--20260302
|   |
|   +-- 03021015.C3X
|   +-- 03030000.C3X

If on the 2nd day, you run the cube_convert.py script with 20260301 as input directory, you will create hourly MSEED files up to March 2nd 10:00 - 11:00 (with name looking like *.061.10). This last MSEED file will only contain 15 min of data (10:00 - 10:15).

Now on the third day, when you run cube_convert.py again, with 20260302 as input directory, the *.061.10 MSEED file will be overwritten and now contain data from 10:15 - 11:00, i.e. 15 minutes of data have been lost! (or at least not included)

I hope it makes sense?

Depending at what part of the hour the data downloading/battery unplug happens the amount of lost data can vary between 0-60 minutes.

I wrote a fix for this for my current purpose, where the '--force-overwrite' option is removed and instead the code checks for pre-existing MSEED files and merges them. Another fix for this (without touching the code) is to move the last datacube file from each folder into the next one (in my example, move 03020000.C3X into 20260302/).

This is a fairly minor issue overall, not sure how relevant it is for most people, happy to share my code if of interest!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions