Skip to content

Commit 9ca01e7

Browse files
committed
migrate: fix TF list-block unmarshaling, restore --noplancheck no-op, update outputs
TF state stores single-block fields (e.g. continuous, deployment) as single-element arrays [{}], not plain objects. json.Unmarshal into the generated Go schema structs fails with type mismatch. Switch LookupTFField to navigate via map[string]any + custom navigateTFState that auto-unwraps single-element lists when a string-key step follows. Also: - Restore --noplancheck as a no-op flag (backward compat; used by invariant tests for job_with_depends_on config). - Remove the plan-check lines from acceptance test output.txt files. - Update help output to include --noplancheck. Co-authored-by: Isaac
1 parent c2c96f4 commit 9ca01e7

11 files changed

Lines changed: 61 additions & 47 deletions

File tree

acceptance/bundle/help/bundle-deployment-migrate/output.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,8 @@ Usage:
1212
databricks bundle deployment migrate [flags]
1313

1414
Flags:
15-
-h, --help help for migrate
15+
-h, --help help for migrate
16+
--noplancheck No-op (kept for compatibility).
1617

1718
Global Flags:
1819
--debug enable debug logging

acceptance/bundle/migrate/basic/output.txt

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,6 @@ Updating deployment state...
1111
Deployment complete!
1212

1313
>>> [CLI] bundle deployment migrate
14-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
15-
Plan: 0 to add, 0 to change, 0 to delete, 3 unchanged
1614
Success! Migrated 3 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json
1715

1816
Validate the migration by running "databricks bundle plan", there should be no actions planned.

acceptance/bundle/migrate/dashboards/output.txt

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Updating deployment state...
66
Deployment complete!
77

88
>>> [CLI] bundle deployment migrate
9-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
10-
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
119
Success! Migrated 1 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/default/resources.json
1210

1311
Validate the migration by running "databricks bundle plan", there should be no actions planned.

acceptance/bundle/migrate/default-python/output.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,6 @@ Deployment complete!
2424

2525
>>> musterr [CLI] bundle deployment migrate
2626
Building python_artifact...
27-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
2827
Building python_artifact...
2928
update jobs.sample_job
3029

acceptance/bundle/migrate/grants/output.txt

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Updating deployment state...
66
Deployment complete!
77

88
>>> [CLI] bundle deployment migrate
9-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
10-
Plan: 0 to add, 0 to change, 0 to delete, 6 unchanged
119
Success! Migrated 6 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/default/resources.json
1210

1311
Validate the migration by running "databricks bundle plan", there should be no actions planned.

acceptance/bundle/migrate/permissions/output.txt

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Updating deployment state...
66
Deployment complete!
77

88
>>> [CLI] bundle deployment migrate
9-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
10-
Plan: 0 to add, 0 to change, 0 to delete, 4 unchanged
119
Success! Migrated 4 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/default/resources.json
1210

1311
Validate the migration by running "databricks bundle plan", there should be no actions planned.

acceptance/bundle/migrate/profile_arg/output.txt

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Updating deployment state...
66
Deployment complete!
77

88
>>> [CLI] bundle deployment migrate -p non_existent321
9-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
10-
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
119
Success! Migrated 1 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json
1210

1311
Validate the migration by running "databricks bundle plan -p non_existent321", there should be no actions planned.
@@ -24,8 +22,6 @@ Updating deployment state...
2422
Deployment complete!
2523

2624
>>> [CLI] bundle deployment migrate -p non_existent321 -t prod
27-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
28-
Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
2925
Success! Migrated 2 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/prod/resources.json
3026

3127
Validate the migration by running "databricks bundle plan -t prod -p non_existent321", there should be no actions planned.

acceptance/bundle/migrate/runas/output.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,6 @@ Consider using a adding a top-level permissions section such as the following:
8181
See https://docs.databricks.com/dev-tools/bundles/permissions.html to learn more about permission configuration.
8282
in databricks.yml:5:3
8383

84-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
8584
Recommendation: permissions section should explicitly include the current deployment identity '[USERNAME]' or one of its groups
8685
If it is not included, CAN_MANAGE permissions are only applied if the present identity is used to deploy.
8786

acceptance/bundle/migrate/var_arg/output.txt

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ Updating deployment state...
66
Deployment complete!
77

88
>>> [CLI] bundle deployment migrate --var=job_name=Custom Job Name
9-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
10-
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
119
Success! Migrated 1 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json
1210

1311
Validate the migration by running "databricks bundle plan --var 'job_name=Custom Job Name'", there should be no actions planned.
@@ -39,8 +37,6 @@ Updating deployment state...
3937
Deployment complete!
4038

4139
>>> [CLI] bundle deployment migrate --var job_name=Custom Job Name
42-
Note: Migration should be done after a full deploy. Running plan now to verify that deployment was done:
43-
Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
4440
Success! Migrated 1 resources to direct engine state file: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json
4541

4642
Validate the migration by running "databricks bundle plan --var 'job_name=Custom Job Name'", there should be no actions planned.

bundle/migrate/tf_state.go

Lines changed: 55 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,9 @@ import (
44
"encoding/json"
55
"fmt"
66
"os"
7-
"reflect"
8-
"strings"
9-
"sync"
107

118
"github.com/databricks/cli/bundle/deploy/terraform"
12-
tfschema "github.com/databricks/cli/bundle/internal/tf/schema"
139
"github.com/databricks/cli/bundle/terraform_dabs_map"
14-
"github.com/databricks/cli/libs/structs/structaccess"
1510
"github.com/databricks/cli/libs/structs/structpath"
1611
tfjson "github.com/hashicorp/terraform-json"
1712
)
@@ -55,23 +50,9 @@ func ParseTFStateAttrs(path string) (TFStateAttrs, error) {
5550
return result, nil
5651
}
5752

58-
// tfSchemaTypeMap maps TF resource type name → schema struct type (via AllResources json tags).
59-
var tfSchemaTypeMap = sync.OnceValue(func() map[string]reflect.Type {
60-
t := reflect.TypeFor[tfschema.AllResources]()
61-
m := make(map[string]reflect.Type, t.NumField())
62-
for f := range t.Fields() {
63-
tag := strings.Split(f.Tag.Get("json"), ",")[0]
64-
if tag != "" && tag != "-" {
65-
m[tag] = f.Type
66-
}
67-
}
68-
return m
69-
})
70-
7153
// LookupTFField looks up a field from TF state attributes for a bundle resource.
7254
// group is the DABs group (e.g. "pipelines"), name is the resource name.
7355
// fieldPath is the path to the field (may be in DABs or TF naming; both handled by DABsPathToTerraform).
74-
// Returns (nil, nil) for empty/zero fields, error if the resource or field is not found.
7556
func LookupTFField(state TFStateAttrs, group, name string, fieldPath *structpath.PathNode) (any, error) {
7657
tfType, ok := terraform.GroupToTerraformName[group]
7758
if !ok {
@@ -91,16 +72,62 @@ func LookupTFField(state TFStateAttrs, group, name string, fieldPath *structpath
9172
return nil, fmt.Errorf("%s.%s not found in TF state", tfType, name)
9273
}
9374

94-
schemaType, ok := tfSchemaTypeMap()[tfType]
95-
if !ok {
96-
return nil, fmt.Errorf("no schema type registered for %q", tfType)
97-
}
98-
99-
// Unmarshal attributes into a new instance of the schema struct.
100-
ptr := reflect.New(schemaType)
101-
if err := json.Unmarshal(attrsJSON, ptr.Interface()); err != nil {
75+
// Unmarshal into map[string]any to handle TF list-blocks: in TF state, single-block
76+
// fields are stored as single-element arrays [{"field": "value"}], not as plain objects.
77+
// Navigating via map avoids the json.Unmarshal type mismatch between []T in JSON and
78+
// struct-typed schema fields.
79+
var attrs map[string]any
80+
if err := json.Unmarshal(attrsJSON, &attrs); err != nil {
10281
return nil, fmt.Errorf("cannot parse TF state for %s.%s: %w", tfType, name, err)
10382
}
10483

105-
return structaccess.Get(ptr.Interface(), tfFieldPath)
84+
return navigateTFState(attrs, tfFieldPath)
85+
}
86+
87+
// navigateTFState walks the TF state map using the given path.
88+
// TF stores single-block fields as single-element arrays ([{…}]). When a string-key
89+
// step encounters a []any, it auto-descends into element [0] so callers can use plain
90+
// paths like "continuous.pause_status" even though TF stores them as [{"pause_status":…}].
91+
func navigateTFState(data map[string]any, path *structpath.PathNode) (any, error) {
92+
var current any = data
93+
for _, node := range path.AsSlice() {
94+
if current == nil {
95+
return nil, nil
96+
}
97+
98+
if key, ok := node.StringKey(); ok {
99+
// Auto-unwrap TF list-blocks: if the current value is a single-element
100+
// array and the next step wants a map key, descend into element 0.
101+
if arr, isArr := current.([]any); isArr {
102+
if len(arr) == 0 {
103+
return nil, nil
104+
}
105+
current = arr[0]
106+
}
107+
m, ok := current.(map[string]any)
108+
if !ok {
109+
return nil, fmt.Errorf("expected map at %q, got %T", key, current)
110+
}
111+
val, ok := m[key]
112+
if !ok {
113+
return nil, fmt.Errorf("%q: key not found", key)
114+
}
115+
current = val
116+
} else if idx, ok := node.Index(); ok {
117+
switch v := current.(type) {
118+
case []any:
119+
if idx < 0 || idx >= len(v) {
120+
return nil, fmt.Errorf("index %d out of range (len %d)", idx, len(v))
121+
}
122+
current = v[idx]
123+
default:
124+
// TF [0] on a non-slice (already unwrapped) is a no-op.
125+
if idx == 0 {
126+
continue
127+
}
128+
return nil, fmt.Errorf("index %d: not a slice (%T)", idx, current)
129+
}
130+
}
131+
}
132+
return current, nil
106133
}

0 commit comments

Comments
 (0)