Describe the bug
S3 supports both standard endpoints, e.g. s3.REGION.amazonaws.com, as well as a dual-stack endpoint variant, which allows clients to resolve either IPv4/v6, e.g. s3.dualstack.REGION.amazonaws.com
To Reproduce
The following unit test using the DefaultObjectStoreRegistry fails to properly resolve a store for a dualstack endpoint:
#[test]
fn test_resolve_s3_dualstack_endpoint() {
let registry = DefaultObjectStoreRegistry::new();
let url = Url::parse(
"https://s3.dualstack.us-east-1.amazonaws.com/static.customizedgirl.com/images/design/3747d20127af39238a380d4fface92b1_1639568_0_bigger.jpg",
)
.unwrap();
let (_store, path) = registry
.resolve(&url)
.expect("registry should resolve S3 dual-stack endpoints");
// The bucket (`static.customizedgirl.com`) is the first path segment and
// should be stripped, leaving the object key as the trailing path.
assert_eq!(
path.as_ref(),
"images/design/3747d20127af39238a380d4fface92b1_1639568_0_bigger.jpg"
);
}
Expected behavior
The registry should return an AmazonS3 store with the dualstack endpoint configured and the us-east-1 region set.
Additional context
I noticed this while trying to write some code that downloads data from the PIXMO dataset on HuggingFace, which is a collection of image URLs and descriptions.
The repro above uses the URL which failed for me.
Describe the bug
S3 supports both standard endpoints, e.g.
s3.REGION.amazonaws.com, as well as a dual-stack endpoint variant, which allows clients to resolve either IPv4/v6, e.g.s3.dualstack.REGION.amazonaws.comTo Reproduce
The following unit test using the
DefaultObjectStoreRegistryfails to properly resolve a store for a dualstack endpoint:Expected behavior
The registry should return an
AmazonS3store with the dualstack endpoint configured and theus-east-1region set.Additional context
I noticed this while trying to write some code that downloads data from the PIXMO dataset on HuggingFace, which is a collection of image URLs and descriptions.
The repro above uses the URL which failed for me.