feat(azure_blob sink): Expand support for Azure authentication types

[jlaundry]

Summary

As mentioned in #24492 (comment), now that #22912 has landed, we can make the AzureAuthentication config generic, so that the other Azure authentication types can be re-supported by azure_blob (and eventually azure_data_explorer #24633, and azure_event_hub #24659).

This currently includes Azure CLI, Managed Identity, Workload Identity, as well as a special chained Managed Identity Client Assertion. I'm happy to add others that people believe they have a use-case for, I just didn't want to add code that was unlikely to be used.

Todo list

• Migrate existing AzureAuthentication config type
• Add additional requested authentication types
  • ClientCertificateCredential: feat(azure_blob sink): Expand support for Azure authentication types #24729 (comment)
• Remove block_in_place (int tests fail with thread 'sinks::azure_blob::test::azure_blob_build_config_with_client_id_and_secret' (1977) panicked at src/sinks/azure_common/config.rs:380:43: can call blocking only when running on the multi-threaded runtime)
• Integration tests with Azurite and real and Mock tokens

Vector configuration

For example:

sinks:
  blob:
    type: azure_blob
    inputs:
      - stdin

    connection_string: AccountName=teststorage
    container_name: vectorlogs

    encoding:
      codec: json
      json:
        pretty: true

    auth:
      azure_client_id: ${AZURE_CLIENT_ID}
      azure_client_secret: ${AZURE_CLIENT_SECRET}
      azure_tenant_id: ${AZURE_TENANT_ID}

How did you test this PR?

Currently testing in my lab environment; I've got WIP for running the integration test suite, but it's failing to pick up the integration test CA (#24729 (review))

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: Enhance Azure Identity Integration in Vector: Ability to Utilize Passwordless Connections #20625
• Closes: Support Azure Active Directory tokens for the Azure Blob Store Sink #24167
• Closes: Vector versions >= 0.50.0 no longer support Azure token auth via environment variables #24491
• Closes: fix(azure_blob sink): Re-add support for Azure token auth via environment variables #24492
• Closes: fix(azure_blob sink): restore non-connection string auth #24624
• Related: feat(new sink): Azure Data Explorer Sink #24633
• Related: feat(new source/sink) Added Azure Event Hub source/sink #24659

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[jlaundry]

.

[zapdos26]

Hey @jlaundry thanks for doing this! Would be possible to add ClientCertificateCredential as well? We would certainly use it!

[jlaundry]

#25124 opened for the Check Spelling issue

[pront]

Disclaimer: this comment was generated and posted by Codex.

I think there is still one azure_blob issue worth addressing before merge:

Configs that set account_name or blob_endpoint without auth currently pass config build, and build_client then falls back to anonymous access. That means a configuration which cannot successfully write blobs can still look valid at startup, only to fail later in healthcheck/runtime.

It seems safer to reject this combination up front with a config error, so that account_name / blob_endpoint imply "OAuth-style auth path" and therefore require auth to be present.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 04f91f5431

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jlaundry]

Configs that set account_name or blob_endpoint without auth currently pass config build, and build_client then falls back to anonymous access. That means a configuration which cannot successfully write blobs can still look valid at startup, only to fail later in healthcheck/runtime.

It seems safer to reject this combination up front with a config error, so that account_name / blob_endpoint imply "OAuth-style auth path" and therefore require auth to be present.

Easy, done: abff085

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: abff085801

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[pront]

Thank you @jlaundry, this is an awesome contribution. And special thanks to @LarryOsterman for the thorough reviews.

[pront]

@jlaundry I will enqueue this PR after all checks pass (needs some small fixes)

[jlaundry]

And thank you @pront and the team for the work you do!

(and sorry for the fmt issues... one of these days I'll figure out why the pre-push hook doesn't seem to work in a vscode remote)

[pront]

@jlaundry this got merged Can you recap the followups you had planned?