Add apply_endpoint_url_override helper parsing canonical AWS_ENDPOINT_URL env

[vulkoingim]

Adds helper to parse AWS_ENDPOINT_URL_<SERVICE> / AWS_ENDPOINT_URL env vars matching Go SDKs ResolveServiceBaseEndpoint implementation.

aws-codegen PR

As a disclosure I have used a LLM to generate the code, but I have tested and verified that the generated erlang lib compiles and works against:

• DynamoDB Local with AWS_ENDPOINT_URL_DYNAMODB
• Real DynamoDB, no endpoint override
• RustFS with AWS_ENDPOINT_URL_S3
• Real S3

Generates do_request in the form:

 do_request(Client, Action, Input0, Options) ->
     Client1 = Client#{service => <<"dynamodb">>},
-    Host = build_host(<<"dynamodb">>, Client1),
-    URL = build_url(Host, Client1),
+    DefaultHost = build_host(<<"dynamodb">>, Client1),
+    {URL, Host} = aws_util:apply_endpoint_url_override(
+        build_url(DefaultHost, Client1), DefaultHost, <<"/">>,
+        <<"AWS_ENDPOINT_URL_DYNAMODB">>),
     Headers = [
         {<<"Host">>, Host},
         {<<"Content-Type">>, <<"application/x-amz-json-1.0">>},

Tested after generating the code, and patching the aws_util in aws-erlang.

cd aws-erlang
rebar3 compile
./test.escript
# and targeting local instances
export AWS_ENDPOINT_URL_S3=http://loclahost:9000
export AWS_ENDPOINT_URL_DYNAMODB=http://localhost:9001

#!/usr/bin/env escript
main(_) ->
    [code:add_pathz(E) || E <- filelib:wildcard("_build/default/lib/*/ebin")],
    {ok, _} = application:ensure_all_started(hackney),
    Env = fun(K) -> case os:getenv(K) of false -> halt(1); V -> list_to_binary(V) end end,
    Client = aws_client:make_client(Env("AWS_ACCESS_KEY_ID"), Env("AWS_SECRET_ACCESS_KEY"), Env("AWS_REGION")),
    io:format("DDB:  ~p~n", [aws_dynamodb:list_tables(Client, #{}, [])]),
    io:format("S3:   ~p~n", [aws_s3:list_buckets(Client, #{}, #{}, [])]).

[onno-vos-dev]

What problem are you trying to solve here that cannot be solved with make_local_client?

The reason for approving is mostly since I'm having to do something quite similar to this for: aws-beam/aws-erlang#175 so this kinda comes at "the right time" 😄

I'm fine accepting this anyway due to this but it feels like you're simply using the wrong function here 😓

[vulkoingim]

What problem are you trying to solve here that cannot be solved with make_local_client?

The reason for approving is mostly since I'm having to do something quite similar to this for: aws-beam/aws-erlang#175 so this kinda comes at "the right time" 😄

I'm fine accepting this anyway due to this but it feels like you're simply using the wrong function here 😓

We have many small services that make use of aws-erlang, and I wanted to avoid duplicating the same code for branching out to make_local_client + parsing URLs, or putting it in a different lib. Also I'm used to having those vars available through other languages' SDKs as they are standard - so I thought a natural place for it to be is this lib as well.

Glad to hear that it is going to be useful for your other work 😄

[onno-vos-dev]

What problem are you trying to solve here that cannot be solved with make_local_client?

The reason for approving is mostly since I'm having to do something quite similar to this for: aws-beam/aws-erlang#175 so this kinda comes at "the right time" 😄

I'm fine accepting this anyway due to this but it feels like you're simply using the wrong function here 😓

We have many small services that make use of aws-erlang, and I wanted to avoid duplicating the same code for branching out to make_local_client + parsing URLs, or putting it in a different lib. Also I'm used to having those vars available through other languages' SDKs as they are standard - so I thought a natural place for it to be is this lib as well.

Glad to hear that it is going to be useful for your other work 😄

Thanks for the explanation and its a reasonable thought process! 👍😄 Will deal with the merge dance tonight (CEST time) 👍

[onno-vos-dev]

Tagged as: 1.2.0. Will prepare the aws-erlang PR shortly