Send zero-value photon/serverless/continuous/development for pipelines

[denik]

Changes

These optional boolean fields have meaningful false values, but the Go SDK omits them unless they are listed in ForceSendFields. As a result an explicit photon = false (or serverless/continuous/development) in the config was dropped from the create/update request and the platform applied its own default instead.

Call SetForceSendFields for these fields in Create and Update so that a user-specified false value is sent to the platform.

Tests

Tested with DABs: #5806 (comment)

• make test run locally
• relevant change in docs/ folder
• covered with integration tests in internal/acceptance
• using Go SDK
• using TF Plugin Framework
• has entry in NEXT_CHANGELOG.md file

[denik]

Tested with relevant acceptance test in DABs (databricks/cli#5462)

On main test records buggy behaviour with terraform:

~/work/cli % go test ./acceptance  -run 'TestAccept/bundle/resources/pipelines/zero-value-fields/DATABRICKS_BUNDLE_ENGINE=terraform'
ok      github.com/databricks/cli/acceptance    6.984s

With provider built from this branch test fails because requests now includes zero-valued fields:

~/work/cli % rm -f acceptance/build/darwin_arm64/.terraformrc
~/work/cli % ls /tmp/custom-provider
terraform-provider-databricks
~/work/cli % DATABRICKS_TF_PROVIDER_DEV_OVERRIDE=/tmp/custom-provider go test ./acceptance  -run 'TestAccept/bundle/resources/pipelines/zero-value-fields/DATABRICKS_BUNDLE_ENGINE=terraform'
                                                                                                                                                            --- FAIL: TestAccept (1.54s)
    acceptance_test.go:1406: [python3 /Users/denis.bilenko/work/cli/acceptance/install_terraform.py --targetdir /Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64] took 51.352167ms
    acceptance_test.go:1410: [python3 /Users/denis.bilenko/work/cli/acceptance/install_terraform.py --targetdir /Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64] output: Read version 1.117.0 from /Users/denis.bilenko/work/cli/bundle/internal/tf/codegen/schema/version.go
        Writing /Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64/.terraformrc:
        # Set these env variables before running databricks cli:
        # export DATABRICKS_TF_CLI_CONFIG_FILE=/Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64/.terraformrc
        # export DATABRICKS_TF_EXEC_PATH=/Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64/terraform

        provider_installation {
            dev_overrides {
                "databricks/databricks" = "/tmp/custom-provider"
            }
            filesystem_mirror {
                path = "/Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64/tfplugins"
                include = ["registry.terraform.io/databricks/databricks"]
            }
        }

    acceptance_test.go:1406: [uv build -q --wheel --no-index --out-dir /Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64] took 193.199167ms
    acceptance_test.go:1406: [go build -o /Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64/databricks] took 681.931583ms
    acceptance_test.go:1406: [go tool -modfile=tools/task/go.mod task build-yamlfmt] took 450.037834ms
    acceptance_test.go:1410: [go tool -modfile=tools/task/go.mod task build-yamlfmt] output: task: Task "build-yamlfmt" is up to date
    acceptance_test.go:488: Summary (dirs): 2/2/945 run/selected/total, 0 skipped
    --- FAIL: TestAccept/bundle/resources/pipelines/zero-value-fields (0.00s)
        --- FAIL: TestAccept/bundle/resources/pipelines/zero-value-fields/DATABRICKS_BUNDLE_ENGINE=terraform (2.83s)
            acceptance_test.go:992:
                        Error Trace:    /Users/denis.bilenko/work/cli/libs/testdiff/testdiff.go:22
                                                                /Users/denis.bilenko/work/cli/acceptance/acceptance_test.go:992
                                                                /Users/denis.bilenko/work/cli/acceptance/acceptance_test.go:824
                                                                /Users/denis.bilenko/work/cli/acceptance/acceptance_test.go:474
                        Error:          Not equal:
                                        expected: "{\n  \"method\": \"POST\",\n  \"path\": \"/api/2.0/pipelines\",\n  \"body\": {\n    \"channel\": \"CURRENT\",\n    \"deployment\": {\n      \"kind\": \"BUNDLE\",\n      \"metadata_file_path\": \"/Workspace/Users/[USERNAME]/.bundle/zero-value-fields/default/state/metadata.json\"\n    },\n    \"edition\": \"ADVANCED\",\n    \"libraries\": [\n      {\n        \"file\": {\n          \"path\": \"/Workspace/Users/[USERNAME]/.bundle/zero-value-fields/default/files/foo.py\"\n        }\n      }\n    ],\n    \"name\": \"test-pipeline\"\n  }\n}\n"
                                        actual  : "{\n  \"method\": \"POST\",\n  \"path\": \"/api/2.0/pipelines\",\n  \"body\": {\n    \"channel\": \"CURRENT\",\n    \"continuous\": false,\n    \"deployment\": {\n      \"kind\": \"BUNDLE\",\n      \"metadata_file_path\": \"/Workspace/Users/[USERNAME]/.bundle/zero-value-fields/default/state/metadata.json\"\n    },\n    \"development\": false,\n    \"edition\": \"ADVANCED\",\n    \"libraries\": [\n      {\n        \"file\": {\n          \"path\": \"/Workspace/Users/[USERNAME]/.bundle/zero-value-fields/default/files/foo.py\"\n        }\n      }\n    ],\n    \"name\": \"test-pipeline\",\n    \"photon\": false,\n    \"serverless\": false\n  }\n}\n"

                                        Diff:
                                        --- Expected
                                        +++ Actual
                                        @@ -5,2 +5,3 @@
                                             "channel": "CURRENT",
                                        +    "continuous": false,
                                             "deployment": {
                                        @@ -9,2 +10,3 @@
                                             },
                                        +    "development": false,
                                             "edition": "ADVANCED",
                                        @@ -17,3 +19,5 @@
                                             ],
                                        -    "name": "test-pipeline"
                                        +    "name": "test-pipeline",
                                        +    "photon": false,
                                        +    "serverless": false
                                           }
                        Test:           TestAccept/bundle/resources/pipelines/zero-value-fields/DATABRICKS_BUNDLE_ENGINE=terraform
                        Messages:       bundle/resources/pipelines/zero-value-fields/out.requests.terraform.txt vs /var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAcceptbundleresourcespipelineszero-value-fieldsDATABRIC1905286007/001/out.requests.terraform.txt

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/terraform

Inputs:

• PR number: 5806
• Commit SHA: bb97865ce2f4c8932f615d9e34c220011aef9372

Checks will be approved automatically on success.

[alexott]

@denik I think that we attempted to do this before, but there are some things that should be taken into account. I.e., if we set serverless = true and we send photon = false, then it generated an error (at least as I remember). Plus some other combinations. The correct solution is port this resource to plugin framework that distinguishes between set/not set and zero values.

[hectorcast-db]

... set to false are sent in the create/update request if specified

Terraform is unable to determine whether those fields are specified by the user or not.
hashicorp/terraform-plugin-sdk#817

Using ForceSendFields for booleans causes the resource to send false on Updates even when not specified by the users, which causes server side errors or a resource which does not work as the user expects.

As Alex said, the solution is a full rewrite of the resource using the Terraform Plugin Framework.