[Feature Request] - Handle multiple parquet hived partition as an input dataset

[lbesnard]

Summary

Handle multiple parquet hived partition as an input dataset

---

Problem / Motivation

AMSA vessel data from @khannakarishma is converted to yearly Parquet files.

The CO library can't handle this at the moment

Proposed Solution

• Modify the parquet handler to handle it

Further info

• @khannakarishma to provide test Parquet dataset, link to an S3 location is enough

[thommodin]

Somewhere in here related:

aodn_cloud_optimised/aodn_cloud_optimised/lib/GenericParquetHandler.py

Line 263 in 7205022

def preprocess_data_parquet(

[lbesnard]

#257

[thommodin]

To discuss in pre-planning relevancy for future

[thommodin]

Put back into the backlog to decide if it is still required in the future.