direct: Fix spurious recreate of schemas and volumes with mixed-case names (#5531)

The direct engine saves local config to state, so when Unity Catalog
lowercases a schema/volume identifier, the next deploy sees the original
mixed-case value drift against the normalized remote value and
recreates/renames the resource — on every second deploy with no config
changes. This PR fixes that.

The fix is expressed declaratively via a new `resources.yml` key,
`normalize_case`. The same mechanism also absorbs the existing
trailing-slash `storage_location` suppression (#5145) as a second key,
`normalize_slash` — a refactor of behavior that already lived in
per-resource `OverrideChangeDesc` methods, not a new behavior.

Author: denik

NEXT_CHANGELOG.md

@@ -18,6 +18,7 @@
 * Mark vector search index index_subtype as backend_default to prevent drift after deployment ([#5454](https://github.com/databricks/cli/pull/5454)).
 * `bundle deployment migrate`: handle resources added to or removed from `databricks.yml` since the last Terraform deploy ([#5463](https://github.com/databricks/cli/pull/5463)).
 * Add the `genie_spaces` bundle resource for managing Databricks Genie spaces as code, plus `bundle generate genie-space` to import an existing space. Direct deployment engine only ([#5282](https://github.com/databricks/cli/pull/5282)).
+* Fix spurious recreate of schemas and volumes whose names use mixed case ([#5531](https://github.com/databricks/cli/pull/5531)).
 
 ### Dependency updates
 

acceptance/bundle/invariant/configs/schema_uppercase_name.yml.tmpl

@@ -0,0 +1,14 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+# Reproduce: UC API normalizes identifier names to lowercase, causing
+# false drift on the second deploy when the config uses mixed-case names.
+# Volumes are covered by acceptance/bundle/resources/volumes/uppercase-name
+# instead: the Terraform provider rejects an uppercase volume schema_name
+# ("inconsistent final plan"), and the migrate invariant always deploys via
+# Terraform first, so an uppercase volume cannot live in this shared config.
+resources:
+  schemas:
+    foo:
+      catalog_name: main
+      name: MySCHEMA-$UNIQUE_NAME

acceptance/bundle/invariant/configs/volume_uppercase_name.yml.tmpl

@@ -0,0 +1,22 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+# Reproduce: the UC API lowercases identifier names, causing false drift on the
+# second deploy when the config uses mixed-case names. This covers a volume's
+# catalog_name, schema_name and name on the direct engine.
+#
+# Excluded from the migrate invariant: that test deploys via Terraform first,
+# and the TF provider rejects an uppercase volume schema_name ("inconsistent
+# final plan").
+resources:
+  schemas:
+    foo:
+      catalog_name: main
+      name: MySCHEMA-$UNIQUE_NAME
+
+  volumes:
+    bar:
+      catalog_name: main
+      schema_name: ${resources.schemas.foo.name}
+      name: MyVOL-$UNIQUE_NAME
+      volume_type: MANAGED

acceptance/bundle/invariant/continue_293/out.test.toml

@@ -26,3 +26,7 @@ EnvMatrixExclude.no_sql_warehouse = ["INPUT_CONFIG=sql_warehouse.yml.tmpl"]
 # The 1000-task scale case is covered by no_drift. Running it here adds ~1.5 min
 # per variant (deploy + migrate + plan at 1000 tasks) without incremental coverage.
 EnvMatrixExclude.no_pydabs_1000_tasks = ["INPUT_CONFIG=job_pydabs_1000_tasks.yml.tmpl"]
+
+# migrate deploys via Terraform first, and the TF provider rejects an uppercase
+# volume schema_name ("inconsistent final plan"). Covered by no_drift on direct.
+EnvMatrixExclude.no_volume_uppercase = ["INPUT_CONFIG=volume_uppercase_name.yml.tmpl"]

acceptance/bundle/invariant/migrate/out.test.toml

@@ -52,6 +52,7 @@ EnvMatrix.INPUT_CONFIG = [
     "registered_model.yml.tmpl",
     "schema.yml.tmpl",
     "schema_grant_ref.yml.tmpl",
+    "schema_uppercase_name.yml.tmpl",
     "schema_with_grants.yml.tmpl",
     "secret_scope.yml.tmpl",
     "secret_scope_default_backend_type.yml.tmpl",
@@ -62,6 +63,7 @@ EnvMatrix.INPUT_CONFIG = [
     "vector_search_index.yml.tmpl",
     "volume.yml.tmpl",
     "volume_external.yml.tmpl",
+    "volume_uppercase_name.yml.tmpl",
 ]
 
 [EnvMatrixExclude]

acceptance/bundle/invariant/migrate/test.toml

@@ -0,0 +1,15 @@
+bundle:
+  name: test-bundle
+
+resources:
+  schemas:
+    schema1:
+      catalog_name: main
+      name: MySCHEMA
+
+  volumes:
+    vol1:
+      catalog_name: main
+      schema_name: MySCHEMA
+      name: MyVOLUME
+      volume_type: MANAGED

acceptance/bundle/invariant/no_drift/out.test.toml

@@ -0,0 +1,61 @@
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
+{
+  "method": "POST",
+  "path": "/api/2.1/unity-catalog/schemas",
+  "body": {
+    "catalog_name": "main",
+    "name": "MySCHEMA"
+  }
+}
+{
+  "method": "POST",
+  "path": "/api/2.1/unity-catalog/volumes",
+  "body": {
+    "catalog_name": "main",
+    "name": "MyVOLUME",
+    "schema_name": "MySCHEMA",
+    "volume_type": "MANAGED"
+  }
+}
+=== Bundle summary shows lowercase IDs after UC normalizes names
+{
+  "schema_id": "main.myschema",
+  "volume_id": "main.myschema.myvolume"
+}
+=== Plan on no-op redeploy: mixed-case names are skipped, not recreated
+[
+  {
+    "resource": "resources.schemas.schema1",
+    "changes": {
+      "name": {
+        "action": "skip",
+        "reason": "uc_case"
+      }
+    }
+  },
+  {
+    "resource": "resources.volumes.vol1",
+    "changes": {
+      "name": {
+        "action": "skip",
+        "reason": "uc_case"
+      },
+      "schema_name": {
+        "action": "skip",
+        "reason": "uc_case"
+      },
+      "storage_location": {
+        "action": "skip",
+        "reason": "backend_default"
+      }
+    }
+  }
+]
+=== Redeploy without changes - should be a no-op
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!