feat: rolling writer in s3 destination

[mihir-datazip]

Description

This change is regarding https://datazip.atlassian.net/browse/OL-2905

The parquet destination previously buffered an entire partition in the writer's memory and uploaded one file per partition at the end of the sync. Memory grew linearly with partition size and produced a single oversized object. This branch streams each partition to a local file and rolls it into multiple size-bounded files, uploading each as it's sealed.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

• Unit tests to verify rolling, without rolling and partitioned rolling
• Integration tests to verify rolling on a limited size(1MB)
• For benchmarking multiple file uploads, I added a temporary time.Sleep() of 10sec in each file upload and compared master and the new code.

Following are the benchmark results (with mocked upload latency)

Old Writer

New Writer

Screenshots or Recordings

Old writer(without rolling) -> 2GB single parquet

New Writer(with 512 MB rolling) -> 4 files with ~512MB size per file

Documentation

• Documentation Link: https://github.com/datazip-inc/olake-docs/
• N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

datazip-inc/olake-docs#450