Describe the bug:
While reading a long pd0 file from a SentinelV I noticed that under certain circumstances read_buffer() enters into an infinite loop whereby the pointer in the data stream cycles back over the same number of ensembles.
To Reproduce:
The actual file is rather large. At any rate, I have a fix destailed at the bottom that has the added bonus of increasing the processing rate by about 60% (likely due to the precaching of the whole ensemble to execute che checksum.
Expected behavior:
A clear and concise description of what you expected to happen.
Screenshots:
This is time vs. ensemble number if the file is read correctly
Note how time increases monotonically with ensemble number.
This is time vs. ensemble number when the file is not read correctly. Note how after about 2000 ensembles, time stops increasing.
Zooming in, it shows that the time is stuck in a loop.
Desktop (please complete the following information):
- OS: Linux, UBUNTU 24.10
- MHKiT Version: 0.8.2
Additional context:
The solution I found is to add the following after line 454 in rea_buffer()
make sure that the checksum for this ensemble is correct
noBytesInEnsemble=fd.read_i16(1)
# go back to start of ensemble
fd.seek(-4,1)
# pack the entire ensemble into a bytearray
bytesInEnsemble = bytearray(fd.read_ui8(noBytesInEnsemble))
# get checksum (2 bytes unsigned integer)
checksum = fd.read_ui16(1)
# calculate checksum and check
# if the checksum is wrong, back up 100 bytes and search for the next
# ensemble
if (sum(bytesInEnsemble) & 0xFFFF ) != checksum:
logging.warning("Ensemble starting at startpos {} has a checksum error".format(startpos))
logging.warning("checksum calculated = %s, actual checksum = %s\n" % ((sum(bytesInEnsemble) & 0xFFFF), checksum))
fd.seek(-100, 1)
self.read_buffer()
else:
# go back to start of ensemble
fd.seek(-noBytesInEnsemble, 1)
and the following
before the return statement at the end of the function.
Describe the bug:
While reading a long pd0 file from a SentinelV I noticed that under certain circumstances read_buffer() enters into an infinite loop whereby the pointer in the data stream cycles back over the same number of ensembles.
To Reproduce:
The actual file is rather large. At any rate, I have a fix destailed at the bottom that has the added bonus of increasing the processing rate by about 60% (likely due to the precaching of the whole ensemble to execute che checksum.
Expected behavior:
A clear and concise description of what you expected to happen.
Screenshots:
This is time vs. ensemble number if the file is read correctly
Note how time increases monotonically with ensemble number.
This is time vs. ensemble number when the file is not read correctly. Note how after about 2000 ensembles, time stops increasing.
Zooming in, it shows that the time is stuck in a loop.
Desktop (please complete the following information):
Additional context:
The solution I found is to add the following after line 454 in rea_buffer()
make sure that the checksum for this ensemble is correct
and the following
before the return statement at the end of the function.