Describe the bug
I'm encountering an issue with the AWS SDK v2 when using FileAsyncRequestBody for S3 multipart uploads. It appears that the bufferSizeInBytes configuration is not being respected, leading to potential memory issues.
In FileAsyncRequestBodySplitHelper, when creating a multipart upload, the atomic counter numAsyncRequestBodiesInFlight is increased in the function doSendAsyncRequestBody(). Upon completion of the upload, the counter is supposed to be decreased in the function startNextRequestBody(). However, I found that startNextRequestBody() is called twice when the upload finishes. This results in two parts being created instead of one each time a part finishes uploading.
Regression Issue
Expected Behavior
The numAsyncRequestBodiesInFlight counter should only decrease once when each part completes its upload. So shouldSendMore() would limit number of inflight request body to be <= totalBufferSize / bufferPerAsyncRequestBody
Current Behavior
The numAsyncRequestBodiesInFlight counter decreases twice when each part completes its upload, causing two parts to be created instead of one. So the actual number of inflight request body would go more than totalBufferSize / bufferPerAsyncRequestBody
Reproduction Steps
- Use
S3TransferManager to upload a large file.
- Monitor the behavior of the atomic counter and the threads for inflight request bodies.
- Example code:
final Upload s3Upload = s3TransferManager.upload(UploadRequest.builder()
.putObjectRequest(putObjectRequest)
.addTransferListener(progressListener)
.requestBody(FileAsyncRequestBody.builder()
.path(file.toPath())
.chunkSizeInBytes(chunkSizeInBytes)
.build())
.build());
Possible Solution
Use private Set<Long> inflightRequestsStartPositions = Collections.synchronizedSet(new HashSet<>()); instead of private AtomicInteger numAsyncRequestBodiesInFlight = new AtomicInteger(0);, and provide startPosition as an index when add/remove instead of addAndGet/decrementAndGet.
Use inflightRequestsStartPositions.size() instead of numAsyncRequestBodiesInFlight.get() in shouldSendMore()
This will ensure even startNextRequestBody() been called twice, with the same startPosition, still maintain the correct number of inflight request body.
Additional Information/Context
No response
AWS Java SDK version used
2.29.45
JDK version used
11
Operating System and version
macOS Sequoia 15.7.1
Describe the bug
I'm encountering an issue with the AWS SDK v2 when using
FileAsyncRequestBodyfor S3 multipart uploads. It appears that thebufferSizeInBytesconfiguration is not being respected, leading to potential memory issues.In
FileAsyncRequestBodySplitHelper, when creating a multipart upload, the atomic counternumAsyncRequestBodiesInFlightis increased in the functiondoSendAsyncRequestBody(). Upon completion of the upload, the counter is supposed to be decreased in the functionstartNextRequestBody(). However, I found thatstartNextRequestBody()is called twice when the upload finishes. This results in two parts being created instead of one each time a part finishes uploading.Regression Issue
Expected Behavior
The
numAsyncRequestBodiesInFlightcounter should only decrease once when each part completes its upload. SoshouldSendMore()would limit number of inflight request body to be<= totalBufferSize / bufferPerAsyncRequestBodyCurrent Behavior
The
numAsyncRequestBodiesInFlightcounter decreases twice when each part completes its upload, causing two parts to be created instead of one. So the actual number of inflight request body would go more thantotalBufferSize / bufferPerAsyncRequestBodyReproduction Steps
S3TransferManagerto upload a large file.Possible Solution
Use
private Set<Long> inflightRequestsStartPositions = Collections.synchronizedSet(new HashSet<>());instead ofprivate AtomicInteger numAsyncRequestBodiesInFlight = new AtomicInteger(0);, and providestartPositionas an index when add/remove instead of addAndGet/decrementAndGet.Use
inflightRequestsStartPositions.size()instead ofnumAsyncRequestBodiesInFlight.get()inshouldSendMore()This will ensure even
startNextRequestBody()been called twice, with the samestartPosition, still maintain the correct number of inflight request body.Additional Information/Context
No response
AWS Java SDK version used
2.29.45
JDK version used
11
Operating System and version
macOS Sequoia 15.7.1