Default to direct deployment engine for new deployments (#5468)

## Summary

The direct deployment engine is now **Generally Available**. This
switches the default engine for *new* deployments from `terraform` to
`direct`.

Existing deployments are unaffected: the engine is read from recorded
state, and the new default only applies when there is no state and no
explicit engine set (neither `bundle.engine` config nor
`DATABRICKS_BUNDLE_ENGINE`).

To opt out of the direct engine, set `engine: terraform` under `bundle`
in `databricks.yml`, or set `DATABRICKS_BUNDLE_ENGINE=terraform`.

## Changes

- `bundle/config/engine/engine.go`: `Default` changed from
`EngineTerraform` to `EngineDirect`.
- `docs/direct.md`: status updated from Public Preview to Generally
Available (GA).
- `NEXT_CHANGELOG.md`: added a Notable Changes entry with the opt-out
path and a link to the migration docs.
- `acceptance/bundle/state/engine_default/`: new acceptance test that
deploys with the engine unset and no existing state, asserting the
direct engine is used (via the request User-Agent).

## Testing

- Unit tests for `bundle/config/...`, `bundle/statemgmt/...`,
`bundle/phases/...`, `cmd/bundle/...` pass.
- New acceptance test passes on both engine matrix variants.

This pull request and its description were written by Isaac.

Author: denik

.gitignore

@@ -73,6 +73,7 @@ go.work.sum
 .codegen/openapi.json
 
 .claude/settings.local.json
+.claude/scheduled_tasks.lock
 .cursor/cli.json
 tools/gofumpt
 .claude/worktrees/

NEXT_CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Release v1.3.0
 
 ### Notable Changes
+* The `direct` deployment engine is now Generally Available and the default for new deployments. To opt out, set `engine: terraform` under `bundle` in your `databricks.yml` or set `DATABRICKS_BUNDLE_ENGINE=terraform`. Existing deployments keep their current engine; see https://docs.databricks.com/aws/en/dev-tools/bundles/direct to migrate.
 
 ### CLI
 * Added the `databricks quickstart` command, a short introduction to the CLI that prints a human-friendly guide interactively and an agent-oriented version when run non-interactively ([#5464](https://github.com/databricks/cli/pull/5464)).

acceptance/bundle/migrate/basic/output.txt

@@ -117,7 +117,7 @@ Updating deployment state...
 Deployment complete!
 
 === Different target, still on terraform
->>> DATABRICKS_BUNDLE_ENGINE= [CLI] bundle deploy -t prod
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy -t prod
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/prod/files...
 Deploying resources...
 Updating deployment state...

acceptance/bundle/migrate/basic/script

@@ -47,7 +47,7 @@ trace DATABRICKS_BUNDLE_ENGINE= $CLI bundle plan -o json > out.plan_update.json
 trace DATABRICKS_BUNDLE_ENGINE= $CLI bundle deploy
 
 title "Different target, still on terraform"
-trace DATABRICKS_BUNDLE_ENGINE= $CLI bundle deploy -t prod
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy -t prod
 trace DATABRICKS_BUNDLE_ENGINE= $CLI bundle debug states -t prod
 trace DATABRICKS_BUNDLE_ENGINE= $CLI bundle debug states -t prod --force-pull
 

acceptance/bundle/state/engine_default/databricks.yml

@@ -0,0 +1,8 @@
+bundle:
+  name: test-engine-default
+
+resources:
+  schemas:
+    foo:
+      catalog_name: bar
+      name: foo

acceptance/bundle/state/engine_default/out.test.toml

@@ -0,0 +1,11 @@
+
+=== No engine configured and no state: deploy defaults to the direct engine
+
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-engine-default/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
+
+>>> print_requests.py //api/2.1/unity-catalog/schemas
+"cli/[DEV_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_deploy cmd-exec-id/[UUID] interactive/none engine/direct auth/pat"

acceptance/bundle/state/engine_default/output.txt

@@ -0,0 +1,4 @@
+title 'No engine configured and no state: deploy defaults to the direct engine\n'
+unset DATABRICKS_BUNDLE_ENGINE
+trace $CLI bundle deploy
+trace print_requests.py //api/2.1/unity-catalog/schemas | jq '.headers["User-Agent"][0]' | contains.py 'engine/direct' '!terraform' '!tf-provider'

acceptance/bundle/state/engine_default/script

@@ -0,0 +1,8 @@
+RecordRequests = true
+IncludeRequestHeaders = ["User-Agent"]
+Ignore = [".databricks"]
+
+# Run once (not per-engine); the script unsets DATABRICKS_BUNDLE_ENGINE to
+# exercise the default. "direct" (rather than []) keeps the test in the
+# engine-filtered CI runs.
+EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]

acceptance/bundle/state/engine_default/test.toml

@@ -18,7 +18,7 @@ const (
 )
 
 // Default is used for new bundles if user has not set the value
-const Default = EngineTerraform
+const Default = EngineDirect
 
 // Parse returns EngineType from string
 func Parse(engine string) (EngineType, bool) {