Skip to content

feat: S3 transfer manager improvements#3247

Merged
stobrien89 merged 24 commits into
aws:masterfrom
yenfryherrerafeliz:s3_transfer_manager_improvements
Mar 30, 2026
Merged

feat: S3 transfer manager improvements#3247
stobrien89 merged 24 commits into
aws:masterfrom
yenfryherrerafeliz:s3_transfer_manager_improvements

Conversation

@yenfryherrerafeliz

Copy link
Copy Markdown
Contributor

Description of changes:

  • Includes resumable capabilities to resume downloads and uploads.
  • Concurrent downloads to enhance download speed.
  • Directory Transfer Improvements.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@yenfryherrerafeliz yenfryherrerafeliz changed the title S3 transfer manager improvements feat: S3 transfer manager improvements Feb 24, 2026
Comment thread src/S3/S3Transfer/AbstractMultipartDownloader.php Outdated
Comment thread src/S3/S3Transfer/Models/DownloadDirectoryResult.php Outdated
Comment thread src/S3/S3Transfer/AbstractMultipartDownloader.php
Comment thread src/S3/S3Transfer/AbstractMultipartDownloader.php
Comment thread src/S3/CalculatesChecksumTrait.php Outdated
Comment thread src/S3/S3Transfer/Progress/TransferProgressSnapshot.php Outdated
Comment thread src/S3/S3Transfer/Models/ResumableDownload.php
Comment thread src/S3/S3Transfer/Progress/DirectoryTransferProgressAggregator.php
Comment thread src/S3/S3Transfer/Progress/DirectoryProgressTracker.php
Comment thread src/S3/S3Transfer/Progress/DirectoryTransferProgressSnapshot.php
- Address some formatting issues.
- Fixed arguments passed to ResumableDownload
- Add test coverage for:
  - DirectoryProgressTracker
  - DirectoryTransferProgressAggregator
  - DirectoryTransferProgressSnapshot
  - DirectoryDownloader
  - DirectoryUploader
@yenfryherrerafeliz

Copy link
Copy Markdown
Contributor Author
vendor/bin/behat --format=progress --suite=smoke --tags='~@noassumerole'
...................................................................... 70
...................................................................... 140
...................................................................... 210
............

111 scenarios (111 passed)
222 steps (222 passed)
0m35.66s (96.23Mb)
vendor/bin/behat --format=progress --tags=integ
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
...................................................................... 350
...................................................................... 420
......................................................

136 scenarios (136 passed)
474 steps (474 passed)
27m28.00s (720.09Mb)

@stobrien89 stobrien89 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few more questions/comments

*
* @return bool
*/
private function writePartToDestinationHandle(array $response): bool

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: do part numbers start at 1 or 0? If zero, I think this might be exposed to an off-by-one error

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part numbers start at 1 actually.

$this->body->rewind();
}

$partNo = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does resume data store 1-based part numbers? Might be a mismatch during resume skip logic

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$partNo is incremented at line 388, which will unlikely cause a resumable
persist mismatch. Is there any other mismatch you are referring to here, that I
may be missing?

$chunk = $body->read(self::READ_BUFFER_SIZE);

if (fwrite($this->handle, $chunk) === false) {
throw new FileDownloadException("Failed to write data to temporary file.");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: where do these exceptions bubble up to?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If part number one fails then it will go through here, otherwise it will go through here.

Comment thread src/S3/S3Transfer/S3TransferManager.php
Comment thread src/S3/CalculatesChecksumTrait.php Outdated
Comment thread src/S3/CalculatesChecksumTrait.php Outdated
Comment thread src/S3/S3Transfer/AbstractMultipartDownloader.php Outdated
Comment thread src/S3/S3Transfer/Models/UploadRequest.php
- Add validation for when resuming an upload and the source file changed
- Add test coverage for the behavior when resuming upload and source file changed
- Address some styling issues

@stobrien89 stobrien89 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more comments. Thanks for making the changes to the CBOR tests also

Comment thread src/S3/S3Transfer/AbstractMultipartDownloader.php
Comment thread src/S3/CalculatesChecksumTrait.php Outdated
Comment thread src/S3/S3Transfer/S3TransferManager.php
Comment thread src/S3/S3Transfer/S3TransferManager.php Outdated
Comment thread tests/S3/S3Transfer/S3TransferManagerTest.php

@stobrien89 stobrien89 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, looks good- just a couple of formatting nits. Needs a changelog entry and a new integ/smoke test run before merging.

Comment thread src/S3/S3Transfer/Models/AbstractResumableTransfer.php Outdated
Comment thread src/S3/S3Transfer/S3TransferManager.php Outdated
Comment thread src/S3/S3Transfer/Utils/ResumableDownloadHandlerInterface.php Outdated
@yenfryherrerafeliz

Copy link
Copy Markdown
Contributor Author
vendor/bin/behat --format=progress --suite=smoke --tags='~@noassumerole'
...................................................................... 70
...................................................................... 140
...................................................................... 210
............

111 scenarios (111 passed)
222 steps (222 passed)
0m37.73s (96.41Mb)

vendor/bin/behat --format=progress --tags=integ
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
...................................................................... 350
...................................................................... 420
......................................................

136 scenarios (136 passed)
474 steps (474 passed)
31m49.51s (667.43Mb)

@stobrien89 stobrien89 merged commit be81a98 into aws:master Mar 30, 2026
17 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants