feat: Add presigned URL download support to S3AsyncClient and S3 Transfer Manager#7115
Open
jencymaryjoseph wants to merge 40 commits into
Open
feat: Add presigned URL download support to S3AsyncClient and S3 Transfer Manager#7115jencymaryjoseph wants to merge 40 commits into
jencymaryjoseph wants to merge 40 commits into
Conversation
…6216) * Add presigned URL request models and tests * Pre-signed URL API and Get Object Request definitions * JavaDoc changes, String to URL change * Delete services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/presignedurl/DefaultPresignedUrlManager.java * Delete services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/presignedurl/PresignedUrlGetObjectRequestMarshaller.java * Delete services/s3/src/test/java/software/amazon/awssdk/services/s3/presignedurl/model/PresignedUrlTest.java * Format PresignedUrlGetObjectRequestWrapper.java * Format PresignedUrlManager.java * Format PresignedUrlGetObjectRequest.java * Format PresignedUrlGetObjectRequestWrapperTest.java * Format PresignedUrlGetObjectRequestTest.java * sonarcube issues fixed * Delete .idea/codeStyles/Project.xml * added consumer builder, javadoc for builder, assert range value test * removing codestle from gitignore
* presigned url get object request custom marshaller * used protocolFactory and fixed sonar cube issues * removed unused test method, removed overriden method
…gned URL (#6241) * getobject implementation and tests * corrected the getObject params in test: * fixed build failure-dependency issue with Retrystrategy in test * fixed sonarcube issues * addressed sonarcube issue * edited javadocs, added invalidurl test
…ed URLs (#6255) * async presignedurlmanager api and implementation * Refactor DefaultAsyncPresignedUrlManager: move config to constructor, declare variables inline * change operation name
* async s3 client codegen changes * minimised service-2.json
…Endpoint, enhance AsyncPresignedUrlManager API (#6275) * modify operation name, null check for resolvedendpoint, add getObject APIs * added null check for resolvedendpointHeader, doc updates, test additions * fixed checkstyle issue
…iew (#6286) * surface api name changes * rename files, remove consumer builder from downloadrequest * some nitpicks * renamed variables
* Amazon DataZone Update: This release adds support for 1) highlighting relevant text in returned results for Search and SearchListings APIs and 2) returning aggregated counts of values for specified attributes for SearchListings API. * Amazon Omics Update: Add Git integration and README support for HealthOmics workflows * Updated endpoints.json and partitions.json. * Release 2.32.8. Updated CHANGELOG.md, README.md and all pom.xml. * Update to next snapshot version: 2.32.9-SNAPSHOT * Amazon AppIntegrations Service Update: Amazon AppIntegrations introduces new configuration capabilities to enable customers to manage iframe permissions, control application refresh behavior (per contact or per browser/cross-contact), and run background applications (service). * AWS Key Management Service Update: Doc only update: fixed grammatical errors. * AWS Config Update: Documentation improvements have been made to the EvaluationModel and DescribeConfigurationRecorders APIs. * AWS End User Messaging Social Update: This release introduces new WhatsApp template management APIs that enable customers to programmatically create and submit templates for approval, monitor approval status, and manage the complete template lifecycle * AWS Budgets Update: Adds IPv6 and PrivateLink support for AWS Budgets in IAD. * Amazon Simple Queue Service Update: Documentation updates for Amazon SQS fair queues feature. * AWS Elemental MediaPackage v2 Update: This release adds support for specifying a preferred input for channels using CMAF ingest. * Amazon Elastic Compute Cloud Update: Transit Gateway native integration with AWS Network Firewall. Adding new enum value for the new Transit Gateway Attachment type. * Release 2.32.9. Updated CHANGELOG.md, README.md and all pom.xml. * Update to next snapshot version: 2.32.10-SNAPSHOT * AWS IoT SiteWise Update: Add support for native anomaly detection in IoT SiteWise using new Computation Model APIs * Amazon OpenSearch Ingestion Update: Add Pipeline Role Arn as an optional parameter to the create / update pipeline APIs as an alternative to passing in the pipeline configuration body * AWS Direct Connect Update: Enable MACSec support and features on Interconnects. * Updated endpoints.json and partitions.json. * Release 2.32.10. Updated CHANGELOG.md, README.md and all pom.xml. * Update to next snapshot version: 2.32.11-SNAPSHOT * Rebase #6269: Update PresignedUrlManager to PresignedUrlExtension naming * Integ tests * Refactor AsyncPresignedUrlManager integration tests to extend S3IntegrationTestBase * updated test method names to follow testing guidelines * simplified tests * added checksum test for largeobjects * rename in test files, address nit comment, fix rebase issues --------- Co-authored-by: AWS <>
#6318) * Add multipart presigned URL download support with range-based routing * Renamed AsyncPresignedUrlExtension variable * Commented the failing test with TODO
* Transfer Manager presigned URL download request models * Fixed checkstyle issues
) * Range based multipart download subscriber * Refactor code and add tests * Fixed sonarcube issue * Address PR comments: use Long type, inline variables, allow empty files * Refactored subscriber and tests * delete testutil * Added validation tests * Fixed failing tests for IfMatch header * Refactor: split validatePartAndRequestMore into separate methods * Fixed validation tests * Remove onComplete after subscription cancel * Make validatePart boolean function
…eptorSpec null check
* Fix check-then-act race condition in parallel subscriber * address comments
* Add presigned URL download support to S3 Transfer Manager * Add presigned URL download methods to DelegatingS3TransferManager * Address PR Comments
…signed-url-getobject
…dd listener tests (#6977) * fix progress tracking for ranged requests * Add WireMock listener tests
…signed-url-getobject
…aws/aws-sdk-java-v2 into feature/master/pre-signed-url-getobject
…signed-url-getobject
…aws/aws-sdk-java-v2 into feature/master/pre-signed-url-getobject
…t subscriber (#7013) * Fix first part validation and refactor * Use String.format for exceptions
* Enable checksum validation for presigned URL downloads * Add public method to return all checksum algorithms * assert checksum validation status
…signed-url-getobject
…signed-url-getobject
… paths (#7077) * Add response rewrite infrastructure for multipart download metadata * Fix presigned URL multipart download response metadata and 416 fallback * Fix normal multipart download response metadata * Consolidate normal path integration tests with parameterized tests * Address review: Builder ctor, split(config,mapper) interface method, parameterized tests * Fix transfer manager presigned serial wrapper to use correct progress wrapper * Move responseMapper to SplittingTransformerConfiguration and simplify split wiring * Revert normal object path changes and add presigned custom transformer metadata test * Use single splitWithResponseRewrite * Clarify responseMapper Javadoc on SplittingTransformerConfiguration
…signed-url-getobject
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Adds the ability to download S3 objects using pre-signed URLs through the SDK's async client pipeline. This enables customers who receive pre-signed URLs to download objects with SDK features without needing AWS credentials configured.
Modifications
Public API
AsyncPresignedUrlExtensioninterface on S3AsyncClient (accessed vias3AsyncClient.presignedUrlExtension())PresignedUrlDownloadRequestmodel for presigned URL downloadsS3TransferManager.downloadFileWithPresignedUrl()anddownloadWithPresignedUrl()with progress tracking supportPresignedDownloadFileRequest,PresignedDownloadRequest<T>,PresignedFileDownloadTransfer Manager modelsCore SDK
SKIP_ENDPOINT_RESOLUTIONexecution attribute inSdkInternalExecutionAttributeskipEndpointResolution()utility method inAwsEndpointProviderUtilsSplittingTransformerConfiguration.responseMapper()for metadata rewriting in multipart downloadsDefaultChecksumAlgorithm.values()for dynamic checksum algorithm registrationEndpointResolutionStageupdated to respectSKIP_ENDPOINT_RESOLUTIONS3 Internal Implementation
DefaultAsyncPresignedUrlExtension— executes GetObject via presigned URL with NoOpSignerPresignedUrlDownloadRequestMarshaller— custom marshaller using the presigned URL directlyParallelPresignedUrlMultipartDownloaderSubscriber— parallel Range-header multipart downloadsPresignedUrlMultipartDownloaderSubscriber— sequential multipart downloaderMultipartAsyncPresignedUrlExtension— multipart-enabled variantCode Generation
presignedUrlExtensionSupportedcustomization flag to generatepresignedUrlExtension()on async clientsTesting
Screenshots (if appropriate)
Types of changes
Checklist
mvn installsucceedsscripts/new-changescript and following the instructions. Commit the new file created by the script in.changes/next-releasewith your changes.License