Latest CLI v0.264.2 Error: cache: token not found when running CLI from Terraform plan

[pjonkhhnk]

Describe the issue

Since CLI v0.264.2 terraform plan fails when interacting with Databricks. It fails during a terraform plan command with the following error:

Error: cannot read metastores: cannot read data metastores: failed during request visitor: default auth: databricks-cli: cannot get access token: Error: cache: token not found.

This suddenly happened after upgrade from v0.262.0 which worked fine.

Steps to reproduce the behavior

1. The error van be reproduced from both Azure DevOps pipeline or locally
2. Init terraform using a storage account back-end configuration (init command is working fine)

RESOURCE_GROUP_NAME="rg-tfstate"
STORAGE_ACCOUNT_NAME=""
SUBSCRIPTION_ID=""
CONTAINER_NAME="tfstate"
STATE_KEY="terraform.tfstate"

execute az login to your tenant

az account set -s "$SUBSCRIPTION_ID"

Initialize Terraform

terraform init -reconfigure
-backend-config="resource_group_name=$RESOURCE_GROUP_NAME"
-backend-config="storage_account_name=$STORAGE_ACCOUNT_NAME"
-backend-config="container_name=$CONTAINER_NAME"
-backend-config="key=$STATE_KEY"

3. Execute terraform plan command
   terraform plan -out test.tfplan

4. See error### Expected Behavior
   Clear and concise description of what should have happened

Error: cannot read metastores: cannot read data metastores: failed during request visitor: default auth: databricks-cli: cannot get access token: Error: cache: token not found.

The error was resolved by adding this env variable:
export DATABRICKS_AUTH_TYPE=azure-cli

Actual Behavior

Clear and concise description of what actually happened

OS and CLI version

OS: Ubuntu 24.04.2 LTS
CLI v0.264.2

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?
Yes works in previous version (env variable is not needed).

[pietern]

Thanks for the report. We're looking into it.

[pietern]

Which version of the TF provider are you using?

[CaspervdKerk]

Can confirm this same issue pops up for permissions, jobs, notebooks, workspace files, and dbfs files.

Permissions

Error: cannot read permissions: failed during request visitor: default auth: databricks-cli: cannot get access token: Error: cache: token not found. Try logging in again with `databricks auth login --host https://adb-REDACTED.azuredatabricks.net`/ before retrying. If this fails, please report this issue to the Databricks CLI maintainers at https://github.com/databricks/cli/issues/new

Job

Error: cannot read job: failed during request visitor: default auth: databricks-cli: cannot get access token: Error: cache: token not found. Try logging in again with `databricks auth login --host https://adb-REDACTED.azuredatabricks.net`/ before retrying. If this fails, please report this issue to the Databricks CLI maintainers at https://github.com/databricks/cli/issues/new

Notebook

Error: cannot read notebook: failed during request visitor: default auth: databricks-cli: cannot get access token: Error: cache: token not found. Try logging in again with `databricks auth login --host https://adb-REDACTED.azuredatabricks.net`/ before retrying. If this fails, please report this issue to the Databricks CLI maintainers at https://github.com/databricks/cli/issues/new

Workspace file

Error: cannot read workspace file: failed during request visitor: default auth: databricks-cli: cannot get access token: Error: cache: token not found. Try logging in again with `databricks auth login --host https://adb-REDACTED.azuredatabricks.net`/ before retrying. If this fails, please report this issue to the Databricks CLI maintainers at https://github.com/databricks/cli/issues/new

DBFS file

Error: cannot read dbfs file: failed during request visitor: default auth: databricks-cli: cannot get access token: Error: cache: token not found. Try logging in again with `databricks auth login --host https://adb-REDACTED.azuredatabricks.net`/ before retrying. If this fails, please report this issue to the Databricks CLI maintainers at https://github.com/databricks/cli/issues/new

Issue show with databricks provider 1.53.0, and terraform version 1.12.2

[CaspervdKerk]

After writing the above comment we realised that we are on a relatively old provider version. When testing locally on the latest version (1.87.1) the error disappears, going back to 1.53.0 reintroduces the error.
So it is clear that our fix is to update the provider version. However, it is strange that functionality suddenly breaks when using a provider version that is "only" 10 months old.

[pietern]

The older TF provider release depends on the CLI to return an OAuth token. The newer CLI depends on the Go SDK to perform this logic, and the latest Go SDK has a change in error handling that surfaces this error in this case.

The latest TF provider depends on the SDK directly to retrieve the OAuth token, bypassing this error mode.

[pietern]

I have a fix in #3447. It will be included in the next CLI release.

[pjonkhhnk]

Thanks Pieter for the prompt reply and fix. The Databricks provider version we use is 1.62.1. Sounds like a good idea to upgrade this as well.

[pietern]

CLI v0.265.0 was released this morning. Please give it a try if you're still on the older provider version.