Support terraform references in direct engine (#5392)

## Changes
- New package bundle/terraform_dabs_map that contains generator for a
mapping between terraform and DABs schema as well as utilities to map
from terraform and DABs resource references.
- In direct engine, we now use this mapper to support terraform
references. We also explicitly error when a reference is using
terraform-only field that has no equivalent.
- In terraform, we also use this mapper to warn about references that
are not going to work on direct engine. When we encounter such
reference, we also set a flag in telemetry.
- The package also contains a function that maps DABs references to
Terraform, that is not used currently but will be needed for new state
migration implementation.

## Why
Enables seamless and unattended migration to direct engine.

## Tests
New acceptance and unit tests.,

Author: denik

Taskfile.yml

@@ -53,6 +53,7 @@ tasks:
       # committed). Run `./task generate` explicitly when codegen inputs change.
       - task: generate-refschema
       - task: generate-schema
+      - task: generate-schema-map
       - task: generate-schema-docs
       - task: generate-validation
       - task: generate-docs
@@ -854,6 +855,19 @@ tasks:
     cmds:
       - "sh -c 'go run ./bundle/internal/validation/. && gofmt -w -s ./bundle/internal/validation/generated'"
 
+  generate-schema-map:
+    desc: Regenerate bundle/terraform_dabs_map/generated.go
+    sources:
+      - "**/*.go"
+      - exclude: bundle/terraform_dabs_map/generated.go
+      - go.mod
+      - go.sum
+      - "{{.EMBED_SOURCES}}"
+    generates:
+      - bundle/terraform_dabs_map/generated.go
+    cmds:
+      - go test ./bundle/terraform_dabs_map -run TestGenerateSchemaMap -update -v
+
   generate-direct:
     desc: Generate direct engine config (apitypes + resources)
     deps: ['generate-direct-apitypes', 'generate-direct-resources']

acceptance/bin/print_requests.py

@@ -166,7 +166,7 @@ def main():
         requests_file = Path(args.fname)
 
     if not requests_file.exists():
-        sys.exit(f"File {requests_file} not found")
+        sys.exit(f"File {requests_file.as_posix()} not found")
 
     with open(requests_file) as fobj:
         data = fobj.read()

acceptance/bundle/bundle_tag/url_ref/out.deploy.direct.txt

@@ -1,7 +1,5 @@
-Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
-Error: cannot plan resources.jobs.bar: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in jobs.JobSettings; url: field "url" not found in dresources.JobRemote
-
-Error: planning failed
+Error: "resources.jobs.foo.url": Terraform-only field; cross-resource references to Terraform-only fields are not supported by the direct engine
+  in databricks.yml:11:12
 
 
 Exit code: 1

acceptance/bundle/bundle_tag/url_ref/out.deploy.terraform.txt

@@ -1,3 +1,6 @@
+Warning: "resources.jobs.foo.url": Terraform-only field; this reference will not work with the direct engine
+  in databricks.yml:11:12
+
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
 Deploying resources...
 Updating deployment state...

acceptance/bundle/bundle_tag/url_ref/out.destroy-requests.direct.txt

@@ -0,0 +1,3 @@
+File [TEST_TMP_DIR]/out.requests.txt not found
+
+Exit code: 1

acceptance/bundle/bundle_tag/url_ref/out.destroy.direct.txt

@@ -1,4 +1,5 @@
-All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
+Error: "resources.jobs.foo.url": Terraform-only field; cross-resource references to Terraform-only fields are not supported by the direct engine
+  in databricks.yml:11:12
 
-Deleting files...
-Destroy complete!
+
+Exit code: 1

acceptance/bundle/bundle_tag/url_ref/out.destroy.terraform.txt

@@ -1,3 +1,6 @@
+Warning: "resources.jobs.foo.url": Terraform-only field; this reference will not work with the direct engine
+  in databricks.yml:11:12
+
 The following resources will be deleted:
   delete resources.jobs.bar
   delete resources.jobs.foo

acceptance/bundle/bundle_tag/url_ref/out.plan.direct.txt

@@ -1,6 +1,5 @@
-Error: cannot plan resources.jobs.bar: cannot resolve "${resources.jobs.foo.url}": schema mismatch: url: field "url" not found in jobs.JobSettings; url: field "url" not found in dresources.JobRemote
-
-Error: planning failed
+Error: "resources.jobs.foo.url": Terraform-only field; cross-resource references to Terraform-only fields are not supported by the direct engine
+  in databricks.yml:11:12
 
 
 Exit code: 1

acceptance/bundle/bundle_tag/url_ref/out.plan.terraform.txt

@@ -1,3 +1,6 @@
+Warning: "resources.jobs.foo.url": Terraform-only field; this reference will not work with the direct engine
+  in databricks.yml:11:12
+
 create jobs.bar
 create jobs.foo
 

acceptance/bundle/bundle_tag/url_ref/out.summary.direct.txt

@@ -1,13 +1,5 @@
-Name: test-bundle
-Target: default
-Workspace:
-  User: [USERNAME]
-  Path: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
-Resources:
-  Jobs:
-    bar:
-      Name: Untitled
-      URL:  ${resources.jobs.foo.url}
-    foo:
-      Name: Untitled
-      URL:  my_job_url
+Error: "resources.jobs.foo.url": Terraform-only field; cross-resource references to Terraform-only fields are not supported by the direct engine
+  in databricks.yml:11:12
+
+
+Exit code: 1