Add CrtS3TransferManager implementation of upload to support correct progress updates#6341
Conversation
|
Would it be possible to add an integration test that verifies the progress listener actually reports incremental progress (e.g., 0% → X% → X+Y% → X+Y+Z% → 100%) rather than jumping directly from 0% to 100%? |
|
Yeah - that's a good idea but might be slightly tricky to get sure it's not flaky - let me see what I can do. Edit: added a test that is similar to existing tests and should be reliable. |
|
|
This pull request has been closed and the conversation has been locked. Comments on closed PRs are hard for our team to see. If you need more assistance, please open a new issue that references this one. |



Add CrtS3TransferManager implementation of upload to support correct progress updates.
Motivation and Context
see: #3670
When a non-file source (eg
ByteBuffer/bytes[]) is used with theS3TransferManager.upload, the progress reported is inaccurate. This is because the TransferProgressUpdater.wrapRequestBody implementation reports progress when the bytes are read from the body rather than after they are sent on the wire.Modifications
This PR adds a CRT specific implementation of the
uploadmethod that allows us to use the CRT progress listenter - this correctly reports progress on uploads.Note - this does not address the bug in the Java S3TransferManager - doing so would require significant changes and rework of the ProgressUpdater and requires some mechanism to determine when bytes are actually sent - this likely would require additional event hooks be added to the HTTP client interface.
Testing
New and existing unit and integration tests.
Screenshots (if appropriate)
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License