Skip to content

Chunk_buffer handling on timeouts #249

Description

@dwest77a

Currently there's no handling for where a timeout occurs within aiohttp/fsspec, so the response object is interpreted as a bytes-like object on decompression. Ideally this can be caught further up the chain of events.

Traceback (most recent call last):
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/file_access.py", line 87, in <module>
    raise exc
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/file_access.py", line 84, in <module>
    s = future.result()
  File "/opt/homebrew/Cellar/python@3.13/3.13.9_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.9_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/homebrew/Cellar/python@3.13/3.13.9_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/file_access.py", line 25, in _iterate_range
    return ds[i*multiplier:(i+1)*multiplier]  # np.min(ds[0:i])
           ~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/high_level.py", line 445, in __getitem__
    data = self.id.get_data(args, self.fillvalue)
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 425, in get_data
    return self._get_selection_via_chunks(args)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 932, in _get_selection_via_chunks
    self._select_chunks(indexer, out, dtype)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 131, in _select_chunks
    self._read_bulk_fsspec(fh, chunks, out, dtype)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 218, in _read_bulk_fsspec
    out[out_sel] = self._decode_chunk(
                   ~~~~~~~~~~~~~~~~~~^
        chunk_buffer, storeinfo.filter_mask, dtype
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )[chunk_sel]
    ^
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/h5d.py", line 104, in _decode_chunk
    chunk_buffer = BTreeV1RawDataChunks._filter_chunk(
        chunk_buffer,
    ...<2 lines>...
        self.dtype.itemsize,
    )
  File "/Users/daniel.westwood/Documents/CEDA_Documents/project_tracking_details/testing/.venv/lib/python3.13/site-packages/pyfive/btree.py", line 282, in _filter_chunk
    chunk_buffer = zlib.decompress(chunk_buffer)
TypeError: a bytes-like object is required, not 'ConnectionTimeoutError'

Wherever the chunk_buffer object originates, a check to determine if the object is bytes-like would be good. There can then be a correct timeout error raised, or alternatively a second request to retry?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions