Feature Request
During an AWS S3 to S3 transfer using the Connector, the current transfer behaviour is for the S3 DataSource to get the transfer payload from the source bucket into memory and push the payload into the destination bucket. This creates a bottleneck in the dataplane as the payload can consume all the host memory resources.
As a feature request, i'd like to explore the idea of levering the AWS S3 API to trigger a direct transfer from one bucket to another, without any payload even reaching the dataplane. In this case, the connector would act as a transfer coordinator only.
This only applies to AWS S3 to S3 transfers.
Which Areas Would Be Affected?
S3 DataSource and Sink.
S3 Provisioners would have to be changed aswell to correctly setup the environment for the transfer (permission management).
Why Is the Feature Desired?
Benefits in performance.
Solution Proposal
N/A
Feature Request
During an AWS S3 to S3 transfer using the Connector, the current transfer behaviour is for the S3 DataSource to get the transfer payload from the source bucket into memory and push the payload into the destination bucket. This creates a bottleneck in the dataplane as the payload can consume all the host memory resources.
As a feature request, i'd like to explore the idea of levering the AWS S3 API to trigger a direct transfer from one bucket to another, without any payload even reaching the dataplane. In this case, the connector would act as a transfer coordinator only.
This only applies to AWS S3 to S3 transfers.
Which Areas Would Be Affected?
S3 DataSource and Sink.
S3 Provisioners would have to be changed aswell to correctly setup the environment for the transfer (permission management).
Why Is the Feature Desired?
Benefits in performance.
Solution Proposal
N/A