engine/direct: recover from a failed Create during Recreate (#5173)

## Changes
* `bundle/direct/apply.go`: `Recreate` now drops the deployment state
entry (`db.DeleteState`) between `DoDelete` and the follow-up `Create`,
instead of `db.SaveState(key, "", nil, nil)`.
* `bundle/direct/bundle_plan.go`: Treat an existing state entry whose
`__id__` is empty as missing, so the next plan re-plans `Create` instead
of erroring with `invalid state: empty id`. This covers state files
written by pre-fix CLIs.
*
`acceptance/bundle/resources/vector_search_endpoints/recreate/create-fails/`:
New test that triggers the failure path end-to-end by renaming
`my_endpoint` onto a sibling endpoint's name and switching its
`endpoint_type`. The first Recreate's `Create` 409s on the conflict; the
next `bundle plan` recovers cleanly.

## Why
A direct-engine `Recreate` was a `DoDelete` → `SaveState(key, "", nil,
nil)` → `Create` sequence. If the follow-up `Create` failed for any
reason (in our reproducer: a name collision against another bundle
resource), `Finalize` persisted a state row with `__id__ == ""`. Every
subsequent `bundle plan` then refused to proceed (`invalid state: empty
id`) and `bundle destroy` couldn't recover either, leaving the bundle in
a broken state until the user hand-edited `resources.json`.

Dropping the state entry up front means a failed `Create` simply looks
like "no state for this resource" on the next plan, which is the natural
recovery path. The planner-side tolerance handles state files already
written by older CLIs.

## Tests
* New acceptance test
`bundle/resources/vector_search_endpoints/recreate/create-fails`
exercises the full path: initial deploy, Recreate triggered by
`endpoint_type` change, `Create` 409 from a name collision with
`blocker_endpoint`, then `bundle plan` showing `create my_endpoint` and
`bundle destroy` cleaning up.
* `go test ./bundle/...` passes.
* `./task lint` passes.
* `./task test` had unrelated local failures (Python
`databricks-bundles` module not installed in the fresh worktree's venv,
surfacing in pydabs/invariant tests); CI should not hit that.

_PR description drafted with Claude Code._

Author: janniklasrose

NEXT_CHANGELOG.md

@@ -8,6 +8,7 @@
 * JSON output for single objects now uses standard `"key": "value"` spacing (matching list output and `encoding/json` defaults).
 
 ### Bundles
+* engine/direct: Drop the deployment state entry on a recreate before the follow-up `Create`, so a `Create` failure no longer leaves a broken state with `invalid state: empty id` on the next `bundle plan` ([#5173](https://github.com/databricks/cli/pull/5173)).
 
 ### Dependency updates
 

acceptance/bundle/resources/vector_search_endpoints/recreate/create-fails/databricks.yml.tmpl

@@ -0,0 +1,14 @@
+bundle:
+  name: recreate-create-fails-$UNIQUE_NAME
+
+sync:
+  paths: []
+
+resources:
+  vector_search_endpoints:
+    my_endpoint:
+      name: vs-endpoint-a-$UNIQUE_NAME
+      endpoint_type: STANDARD
+    blocker_endpoint:
+      name: vs-endpoint-b-$UNIQUE_NAME
+      endpoint_type: STORAGE_OPTIMIZED

acceptance/bundle/resources/vector_search_endpoints/recreate/create-fails/out.test.toml

@@ -0,0 +1,42 @@
+
+=== Initial deploy creates two endpoints with distinct names
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/recreate-create-fails-[UNIQUE_NAME]/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
+
+=== Edit my_endpoint: rename onto blocker_endpoint's name and switch endpoint_type to trigger Recreate
+>>> update_file.py databricks.yml vs-endpoint-a-[UNIQUE_NAME] vs-endpoint-b-[UNIQUE_NAME]
+
+>>> update_file.py databricks.yml       endpoint_type: STANDARD       endpoint_type: STORAGE_OPTIMIZED
+
+=== Deploy: Recreate of my_endpoint runs Delete (ok) then Create (409, name taken by blocker)
+>>> [CLI] bundle deploy --auto-approve
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/recreate-create-fails-[UNIQUE_NAME]/default/files...
+Deploying resources...
+Error: cannot recreate resources.vector_search_endpoints.my_endpoint: Vector search endpoint with name vs-endpoint-b-[UNIQUE_NAME] already exists (409 RESOURCE_ALREADY_EXISTS)
+
+Endpoint: POST [DATABRICKS_URL]/api/2.0/vector-search/endpoints
+HTTP Status: 409 Conflict
+API error_code: RESOURCE_ALREADY_EXISTS
+API message: Vector search endpoint with name vs-endpoint-b-[UNIQUE_NAME] already exists
+
+Updating deployment state...
+
+Exit code: 1
+
+=== Subsequent plan recovers: my_endpoint state was dropped, replan as Create
+>>> [CLI] bundle plan
+create vector_search_endpoints.my_endpoint
+
+Plan: 1 to add, 0 to change, 0 to delete, 1 unchanged
+
+>>> [CLI] bundle destroy --auto-approve
+The following resources will be deleted:
+  delete resources.vector_search_endpoints.blocker_endpoint
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/recreate-create-fails-[UNIQUE_NAME]/default
+
+Deleting files...
+Destroy complete!

acceptance/bundle/resources/vector_search_endpoints/recreate/create-fails/output.txt

@@ -0,0 +1,20 @@
+envsubst < databricks.yml.tmpl > databricks.yml
+
+cleanup() {
+    trace $CLI bundle destroy --auto-approve || true
+    rm -f out.requests.txt
+}
+trap cleanup EXIT
+
+title "Initial deploy creates two endpoints with distinct names"
+trace $CLI bundle deploy
+
+title "Edit my_endpoint: rename onto blocker_endpoint's name and switch endpoint_type to trigger Recreate"
+trace update_file.py databricks.yml "vs-endpoint-a-$UNIQUE_NAME" "vs-endpoint-b-$UNIQUE_NAME"
+trace update_file.py databricks.yml "      endpoint_type: STANDARD" "      endpoint_type: STORAGE_OPTIMIZED"
+
+title "Deploy: Recreate of my_endpoint runs Delete (ok) then Create (409, name taken by blocker)"
+errcode trace $CLI bundle deploy --auto-approve
+
+title "Subsequent plan recovers: my_endpoint state was dropped, replan as Create"
+trace $CLI bundle plan

acceptance/bundle/resources/vector_search_endpoints/recreate/create-fails/script

@@ -0,0 +1 @@
+Cloud = false

acceptance/bundle/resources/vector_search_endpoints/recreate/create-fails/test.toml

@@ -86,7 +86,9 @@ func (d *DeploymentUnit) Recreate(ctx context.Context, db *dstate.DeploymentStat
 		return fmt.Errorf("deleting old id=%s: %w", oldID, err)
 	}
 
-	err = db.SaveState(d.ResourceKey, "", nil, nil)
+	// Drop the state entry so a subsequent failure of Create leaves no malformed
+	// (empty-id) entry behind. The next plan will see "no state" and retry as Create.
+	err = db.DeleteState(d.ResourceKey)
 	if err != nil {
 		return fmt.Errorf("deleting state: %w", err)
 	}

bundle/direct/apply.go

@@ -181,16 +181,15 @@ func (b *DeploymentBundle) CalculatePlan(ctx context.Context, client *databricks
 		}
 
 		dbentry, hasEntry := b.StateDB.GetResourceEntry(resourceKey)
-		if !hasEntry {
+		// Tolerate empty-id entries from older partial-recreate failures
+		// (apply.Recreate now deletes state on the way through, but pre-fix
+		// state files may still carry a malformed entry). Treat as missing
+		// and let the resource be re-created on this plan.
+		if !hasEntry || dbentry.ID == "" {
 			entry.Action = deployplan.Create
 			return true
 		}
 
-		if dbentry.ID == "" {
-			logdiag.LogError(ctx, fmt.Errorf("%s: invalid state: empty id", errorPrefix))
-			return false
-		}
-
 		savedState, err := parseState(adapter.StateType(), dbentry.State)
 		if err != nil {
 			logdiag.LogError(ctx, fmt.Errorf("%s: interpreting state: %w", errorPrefix, err))