[ISSUE] Issue with `databricks_storage_credential` resource

[johnmckenna]

Using azure-devops-oidc authentication doesn't seem to work with this resource. TerraformTask@5 is being used in an Azure DevOps pipeline SYSTEM_ACCESSTOKEN is being passed in via the pipeline as well. We are using the same workload identity for workspace creation as well as the data plane creation.

A federation policy has already been created at the account level as well.

Configuration

provider "databricks" {
  auth_type = "azure-devops-oidc"
  client_id = data.azurerm_client_config.current.client_id
  host      = module.data_bricks.databricks_url
}

resource "databricks_storage_credential" "external_mi" {
  name = local.databricks_storage_credential
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.databricks_access_connector.id
    managed_identity_id = azurerm_user_assigned_identity.databricks_uai.id
  }
  isolation_mode  = "ISOLATION_MODE_ISOLATED"
  comment         = "Managed identity credential managed by TF"
  depends_on      = [azurerm_databricks_access_connector.databricks_access_connector]
}

Expected Behavior

The storage credential should have been created.

Actual Behavior

│ Error: cannot create storage credential: Missing validation token for service principal. Please provide a valid ARM-scoped Entra ID token in the 'X-Databricks-Azure-SP-Management-Token' request header and retry. For details, check https://docs.databricks.com/api/workspace/storagecredentials
│
│ with module.dbw_data_plane.databricks_storage_credential.external_mi,
│ on modules/dbw_data_plane/main.tf line 223, in resource "databricks_storage_credential" "external_mi":
│ 223: resource "databricks_storage_credential" "external_mi" {

Steps to Reproduce

Terraform and provider versions

terraform 1.15.6
databricks provider 1.118.0

Is it a regression?

Didn't seem to work from 1.113.0 onwards at least.

Debug Output

Important Factoids

Would you like to implement a fix?