az storage account create can incorrectly report failure under load / throttling conditions

[MattGal]

Describe the bug

Current Behavior:

0. The Azure subscription in question is receiving lots of programmatic MICROSOFT.STORAGE calls... say from a bunch of machines running the Az CLI.
1. The user calls az storage account create ... with correct values.
2. The SDK calls PUT/SUBSCRIPTIONS/RESOURCEGROUPS/PROVIDERS/MICROSOFT.STORAGE/STORAGEACCOUNTS
3. The Azure API responds to the SDK with a 429 or 503, but creates it anyway.
4. Given the typically-retry-able nature of 429 / 503, the SDK sleeps until the retry-after period elapses and tries the API again; this time it gets an HTTP 409 because the storage account already exists
5. The SDK translates the HTTP 409 in account creation to "An operation is currently performing on this storage account that requires exclusive access" which is not very helpful
6. The CLI sees a non-continuable 409 / conflict in the response, and bubbles the above string to the user

Related command

az storage account create

Errors

"An operation is currently performing on this storage account that requires exclusive access"

Issue script & Debug output

N/A - This was taken from logs.

I can provide console logs and (more importantly) queries to show all ARM logging of this behavior happening via reaching out to me on internal email or Teams.

Expected behavior

Desired behavior:

• The Storage API should either ensure a 202/200 response when accounts are created successfully, and only return 429/ 503 in cases where the change was not even attempted.
• The Storage Management Client SDK should provide more useful error messages so users hitting this understand that it was on a retry, possibly even detecting that the original request succeeded.
• The Azure CLI should make the above problem not a problem for its users, either by detecting that the original request succeeded and changing from PUT to GET or other means.

Environment Summary

azure cli 2.53.1

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Hi @MattGal , can you please try the latest CLI version here? https://learn.microsoft.com/en-us/cli/azure/install-azure-cli
This seems like service issue, if this is repeatedly happening, please let us know. Thank.

[MattGal]

Hi @MattGal , can you please try the latest CLI version here? https://learn.microsoft.com/en-us/cli/azure/install-azure-cli This seems like service issue, if this is repeatedly happening, please let us know. Thank.

I already agree it at its root, this is a service issue (in which case, why do you think updating the CLI would matter?). A Service issue... in the service endpoint that the az cli calls using its own choice of client and then interprets and responds to the user with. The end result is incorrect and misleading output to the console by the Azure CLI app. This is why I have filed this issue; applications that call web APIs need to handle error scenarios such as this one. If the service can also be made better such as to obviate working around its limitations, I support that but that's not what this specific issue is about.

I am not blocked and do not need support, so if this is not interesting to fix feel free to close / ignore.

[calvinhzy]

@MattGal since we have updated to newer versions of api calls to service side, it is not sure if old versions are as well supported or have issues on the service side. So I was suggesting to update to the latest version to see if it has since been fixed or was a transient issue.
I agree the client side should provide accurate info the customer but the sdk just outputs the response code from service so if service is not providing the correct one, the CLI cannot guess based on the response message itself. Thanks for bring this up to us though.