Restore variables in config-remote-sync (databricks#5053)

ilyakuz-db · TanishqDatabricks · commit 3d39641dcc11 · 2026-05-22T17:27:44.000+02:00
## Changes
&lt;!-- Brief summary of your changes that is easy to understand --&gt;

When config-remote-sync patches YAML files with remote changes, it now
restores variable references rather than always hardcoding values. This
prevents configs from losing `${var.X}` references after UI edits.

Note: There is a lot of ambiguity in how to restore the variables,
because it's not always clear what the user's intent is, and this is
hard to describe with a simple heuristic. In this PR, we make a
best-effort deterministic attempt as a first step.

Supported use cases:
1. If any field that contains a variable was changed, and new values
match this variable, we restore it. Safe guard
2. string template `/bundle/${bundle.target}/{var.foo}` case is also
supported
3. new list item of the same type is added (job param / job task) -&gt; if
variables are used in existing items , and the field value matches the
variable, we use this variable

## Why

Improve the config-remote-sync experience for customers. We have
received feedback that customers usually have their job parameteres
defined as variables, and this PR aims to address some gaps there

## Tests
&lt;!-- How have you tested the changes? --&gt;

Added acceptance tests + tested integration in the workspace

&lt;!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. --&gt;
diff --git a/acceptance/bundle/config-remote-sync/config_edits/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/config_edits/databricks.yml.tmpl
@@ -15,6 +15,7 @@ resources:
 
 targets:
   default:
+    mode: development
     resources:
       jobs:
         my_job:
diff --git a/acceptance/bundle/config-remote-sync/config_edits/output.txt b/acceptance/bundle/config-remote-sync/config_edits/output.txt
@@ -35,14 +35,14 @@ Resource: resources.jobs.my_job
 >>> diff.py databricks.yml.backup databricks.yml
 --- databricks.yml.backup
 +++ databricks.yml
-@@ -24,5 +24,5 @@
+@@ -25,5 +25,5 @@
                - success@example.com
              on_failure:
 -              - config-failure@example.com
 +              - remote-failure@example.com
            parameters:
              - name: catalog
-@@ -35,8 +35,6 @@
+@@ -36,8 +36,6 @@
                unit: DAYS
            tags:
 -            env: config-production
diff --git a/acceptance/bundle/config-remote-sync/flushed_cache/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/flushed_cache/databricks.yml.tmpl
@@ -13,3 +13,7 @@ resources:
             spark_version: $DEFAULT_SPARK_VERSION
             node_type_id: $NODE_TYPE_ID
             num_workers: 1
+
+targets:
+  default:
+    mode: development
diff --git a/acceptance/bundle/config-remote-sync/formatting_preserved/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/formatting_preserved/databricks.yml.tmpl
@@ -39,3 +39,7 @@ resources:
       parameters:
         - {name: catalog, default: main}
         - {name: schema, default: dev}
+
+targets:
+  default:
+    mode: development
diff --git a/acceptance/bundle/config-remote-sync/formatting_preserved/output.txt b/acceptance/bundle/config-remote-sync/formatting_preserved/output.txt
@@ -42,11 +42,13 @@ Resource: resources.jobs.my_job
 +            Main processing task that runs the notebook.
            notebook_task:
              notebook_path: /Users/{{workspace_user_name}}/notebook
-@@ -40,2 +39,3 @@
+@@ -40,4 +39,5 @@
          - {name: catalog, default: main}
          - {name: schema, default: dev}
 +      timeout_seconds: 3600
 
+ targets:
+
 >>> [CLI] bundle destroy --auto-approve
 The following resources will be deleted:
   delete resources.jobs.my_job
diff --git a/acceptance/bundle/config-remote-sync/job_multiple_tasks/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/job_multiple_tasks/databricks.yml.tmpl
@@ -76,3 +76,7 @@ resources:
             spark_version: $DEFAULT_SPARK_VERSION
             node_type_id: $NODE_TYPE_ID
             num_workers: 1
+
+targets:
+  default:
+    mode: development
diff --git a/acceptance/bundle/config-remote-sync/job_multiple_tasks/output.txt b/acceptance/bundle/config-remote-sync/job_multiple_tasks/output.txt
@@ -125,7 +125,7 @@ Resource: resources.jobs.rename_task_job
 +            - task_key: b_task_renamed
            notebook_task:
              notebook_path: /Users/{{workspace_user_name}}/c_task
-@@ -79,7 +79,14 @@
+@@ -79,9 +79,16 @@
          - task_key: a_task
            notebook_task:
 -            notebook_path: /Users/{{workspace_user_name}}/a_task
@@ -142,6 +142,8 @@ Resource: resources.jobs.rename_task_job
 +            notebook_path: ./synced_notebook.py
 +          task_key: synced_task
 
+ targets:
+
 >>> [CLI] bundle destroy --auto-approve
 The following resources will be deleted:
   delete resources.jobs.my_job
diff --git a/acceptance/bundle/config-remote-sync/job_pipeline_task/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/job_pipeline_task/databricks.yml.tmpl
@@ -17,3 +17,7 @@ resources:
           pipeline_task:
             pipeline_id: ${resources.pipelines.my_pipeline.id}
             full_refresh: false
+
+targets:
+  default:
+    mode: development
diff --git a/acceptance/bundle/config-remote-sync/job_pipeline_task/output.txt b/acceptance/bundle/config-remote-sync/job_pipeline_task/output.txt
@@ -4,15 +4,15 @@ Updating deployment state...
 Deployment complete!
 
 === Modify pipeline_task full_refresh to True
-=== Modify pipeline development to True
+=== Modify pipeline continuous to True
 === Detect and save changes
 Detected changes in 2 resource(s):
 
 Resource: resources.jobs.my_job
   tasks[task_key='run_pipeline'].pipeline_task.full_refresh: replace
 
 Resource: resources.pipelines.my_pipeline
-  development: replace
+  continuous: add
 
 
 
@@ -21,19 +21,20 @@ Resource: resources.pipelines.my_pipeline
 >>> diff.py databricks.yml.backup databricks.yml
 --- databricks.yml.backup
 +++ databricks.yml
-@@ -6,5 +6,5 @@
-     my_pipeline:
-       name: test-pipeline-[UNIQUE_NAME]
--      development: false
-+      development: true
-       libraries:
-         - notebook:
-@@ -17,3 +17,3 @@
+@@ -11,4 +11,5 @@
+             path: /Users/{{workspace_user_name}}/notebook
+
++      continuous: true
+   jobs:
+     my_job:
+@@ -17,5 +18,5 @@
            pipeline_task:
              pipeline_id: ${resources.pipelines.my_pipeline.id}
 -            full_refresh: false
 +            full_refresh: true
 
+ targets:
+
 >>> [CLI] bundle destroy --auto-approve
 The following resources will be deleted:
   delete resources.jobs.my_job
diff --git a/acceptance/bundle/config-remote-sync/job_pipeline_task/script b/acceptance/bundle/config-remote-sync/job_pipeline_task/script
@@ -17,9 +17,12 @@ edit_resource.py jobs $job_id <<EOF
 r["tasks"][0]["pipeline_task"]["full_refresh"] = True
 EOF
 
-title "Modify pipeline development to True"
+title "Modify pipeline continuous to True"
+# Dev mode auto-sets pipelines.development=true, so flipping it would be a
+# no-op. Use continuous instead — it's not affected by dev mode and exercises
+# the same Replace path.
 edit_resource.py pipelines $pipeline_id <<EOF
-r["development"] = True
+r["continuous"] = True
 EOF
 
 
diff --git a/acceptance/bundle/config-remote-sync/multiple_files/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/multiple_files/databricks.yml.tmpl
@@ -6,6 +6,7 @@ include:
 
 targets:
   dev:
+    mode: development
     resources:
       jobs:
         job_one:
diff --git a/acceptance/bundle/config-remote-sync/multiple_resources/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/multiple_resources/databricks.yml.tmpl
@@ -53,3 +53,7 @@ resources:
     test_warehouse:
       name: warehouse-$UNIQUE_NAME
       cluster_size: 2X-Small
+
+targets:
+  default:
+    mode: development
diff --git a/acceptance/bundle/config-remote-sync/multiple_resources/output.txt b/acceptance/bundle/config-remote-sync/multiple_resources/output.txt
@@ -9,11 +9,11 @@ Detected changes in 2 resource(s):
 
 Resource: resources.jobs.job_one
   max_concurrent_runs: replace
-  tags: add
+  tags['team']: add
 
 Resource: resources.jobs.job_two
   max_concurrent_runs: replace
-  tags: add
+  tags['team']: add
 
 
 
diff --git a/acceptance/bundle/config-remote-sync/multiple_resources/script b/acceptance/bundle/config-remote-sync/multiple_resources/script
@@ -23,12 +23,13 @@ add_repl.py "$($CLI metastores current | jq -r .metastore_id)" "METASTORE_ID"
 title "Modify both jobs"
 edit_resource.py jobs $job_one_id <<EOF
 r["max_concurrent_runs"] = 5
-r["tags"] = {"team": "data"}
+# Add tag without replacing the map (dev mode adds its own "dev" tag).
+r.setdefault("tags", {})["team"] = "data"
 EOF
 
 edit_resource.py jobs $job_two_id <<EOF
 r["max_concurrent_runs"] = 10
-r["tags"] = {"team": "ml"}
+r.setdefault("tags", {})["team"] = "ml"
 EOF
 
 title "Detect and save changes"
diff --git a/acceptance/bundle/config-remote-sync/output_json/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/output_json/databricks.yml.tmpl
@@ -13,3 +13,7 @@ resources:
             spark_version: $DEFAULT_SPARK_VERSION
             node_type_id: $NODE_TYPE_ID
             num_workers: 1
+
+targets:
+  default:
+    mode: development
diff --git a/acceptance/bundle/config-remote-sync/output_json/output.txt b/acceptance/bundle/config-remote-sync/output_json/output.txt
@@ -7,8 +7,8 @@ Deployment complete!
   "files": [
     {
       "path": "[TEST_TMP_DIR]/databricks.yml",
-      "originalContent": "bundle:\n  name: test-bundle-[UNIQUE_NAME]\n\nresources:\n  jobs:\n    test_job:\n      max_concurrent_runs: 1\n      tasks:\n        - task_key: main\n          notebook_task:\n            notebook_path: /Users/{{workspace_user_name}}/notebook\n          new_cluster:\n            spark_version: 13.3.x-snapshot-scala2.12\n            node_type_id: [NODE_TYPE_ID]\n            num_workers: 1\n",
-      "modifiedContent": "bundle:\n  name: test-bundle-[UNIQUE_NAME]\n\nresources:\n  jobs:\n    test_job:\n      max_concurrent_runs: 3\n      tasks:\n        - task_key: main\n          notebook_task:\n            notebook_path: /Users/{{workspace_user_name}}/notebook\n          new_cluster:\n            spark_version: 13.3.x-snapshot-scala2.12\n            node_type_id: [NODE_TYPE_ID]\n            num_workers: 1\n      tags:\n        env: test\n"
+      "originalContent": "bundle:\n  name: test-bundle-[UNIQUE_NAME]\n\nresources:\n  jobs:\n    test_job:\n      max_concurrent_runs: 1\n      tasks:\n        - task_key: main\n          notebook_task:\n            notebook_path: /Users/{{workspace_user_name}}/notebook\n          new_cluster:\n            spark_version: 13.3.x-snapshot-scala2.12\n            node_type_id: [NODE_TYPE_ID]\n            num_workers: 1\n\ntargets:\n  default:\n    mode: development\n",
+      "modifiedContent": "bundle:\n  name: test-bundle-[UNIQUE_NAME]\n\nresources:\n  jobs:\n    test_job:\n      max_concurrent_runs: 3\n      tasks:\n        - task_key: main\n          notebook_task:\n            notebook_path: /Users/{{workspace_user_name}}/notebook\n          new_cluster:\n            spark_version: 13.3.x-snapshot-scala2.12\n            node_type_id: [NODE_TYPE_ID]\n            num_workers: 1\n      tags:\n        env: test\n\ntargets:\n  default:\n    mode: development\n"
     }
   ],
   "changes": {
@@ -17,11 +17,9 @@ Deployment complete!
         "operation": "replace",
         "value": 3
       },
-      "tags": {
+      "tags['env']": {
         "operation": "add",
-        "value": {
-          "env": "test"
-        }
+        "value": "test"
       }
     }
   }
diff --git a/acceptance/bundle/config-remote-sync/output_json/script b/acceptance/bundle/config-remote-sync/output_json/script
@@ -13,7 +13,9 @@ job_id="$(read_id.py test_job)"
 
 edit_resource.py jobs $job_id <<EOF
 r["max_concurrent_runs"] = 3
-r["tags"] = {"env": "test"}
+# Add tag without replacing the map — dev mode auto-adds a "dev" tag and we
+# don't want the diff to show that as removed.
+r.setdefault("tags", {})["env"] = "test"
 EOF
 
 title "JSON output format"
diff --git a/acceptance/bundle/config-remote-sync/output_no_changes/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/output_no_changes/databricks.yml.tmpl
@@ -13,3 +13,7 @@ resources:
             spark_version: $DEFAULT_SPARK_VERSION
             node_type_id: $NODE_TYPE_ID
             num_workers: 1
+
+targets:
+  default:
+    mode: development
diff --git a/acceptance/bundle/config-remote-sync/resolve_variables/.databricks/bundle/default/variable-overrides.json b/acceptance/bundle/config-remote-sync/resolve_variables/.databricks/bundle/default/variable-overrides.json
@@ -0,0 +1,3 @@
+{
+  "file_override_var": "from-overrides-file"
+}
diff --git a/acceptance/bundle/config-remote-sync/resolve_variables/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/resolve_variables/databricks.yml.tmpl
@@ -0,0 +1,96 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+variables:
+  my_catalog:
+    default: main
+  # Both resolve to same value to test ambiguity (when we have multiple matches → skip)
+  landing_schema:
+    default: raw_data
+  curated_schema:
+    default: raw_data
+  # Variable with value set only in target (no default)
+  target_env:
+    description: "Environment name"
+  # Variable set via .databricks/bundle/<target>/variable-overrides.json
+  file_override_var:
+    description: "Set from variable-overrides.json"
+  # Used by job environments below. Resolves to a pip-style spec; the same
+  # spec is then targeted by both an Add (new environment_key) and a Replace
+  # (appending another dep to the existing environment).
+  my_env_dep:
+    default: nonexistent-test-pkg==1.2.3
+  # Complex variable to verify no panics
+  cluster_config:
+    type: complex
+    default:
+      node_type_id: Standard_DS3_v2
+      num_workers: 2
+
+resources:
+  pipelines:
+    my_pipeline:
+      name: test-pipeline-$UNIQUE_NAME
+      development: false
+      libraries:
+        - notebook:
+            path: /Users/{{workspace_user_name}}/notebook
+      # Three deps cover the compound-interpolation cases:
+      #   [0] pure ${var.X}, untouched in the test → stays as ${var.my_env_dep}
+      #   [1] compound with ${workspace.file_path} → trimmed/extended in the
+      #       script; substring substitution must preserve the variable
+      #   [2] hardcoded literal that happens to equal ${var.my_env_dep}'s
+      #       resolved value → must stay literal (no false-positive promotion)
+      environment:
+        dependencies:
+          - ${var.my_env_dep}
+          - --editable ${workspace.file_path}
+          - nonexistent-test-pkg==1.2.3
+
+  jobs:
+    my_job:
+      parameters:
+        - name: catalog
+          default: ${var.my_catalog}
+        - name: env
+          default: ${var.target_env}
+        - name: file_val
+          default: ${var.file_override_var}
+        # Bundle reference: ${bundle.target} resolves to "default".
+        - name: target_name
+          default: ${bundle.target}
+        # Both use variables that resolve to the same value ("raw_data").
+        # Tests disambiguation: original reference is preserved on Replace.
+        - name: landing
+          default: ${var.landing_schema}
+        - name: curated
+          default: ${var.curated_schema}
+      tasks:
+        - task_key: main
+          notebook_task:
+            notebook_path: /Users/{{workspace_user_name}}/notebook
+            base_parameters:
+              # Compound interpolation: mixes ${var.X} and ${bundle.X} refs.
+              source_path: /mnt/${var.my_catalog}/${bundle.target}/raw/landing
+        # Resource reference: ${resources.pipelines.my_pipeline.id} resolves to
+        # the deployed pipeline's ID. Used to test that resource refs are
+        # treated the same as other reference kinds by the sync logic.
+        - task_key: run_pipeline
+          pipeline_task:
+            pipeline_id: ${resources.pipelines.my_pipeline.id}
+            full_refresh: false
+      # Job environments use ${var.my_env_dep} via a pure variable reference.
+      # The script adds a separate environments entry and modifies the existing
+      # one to verify the variable survives both code paths.
+      environments:
+        - environment_key: default
+          spec:
+            environment_version: "4"
+            dependencies:
+              - ${var.my_env_dep}
+
+targets:
+  default:
+    mode: development
+    variables:
+      target_env: production
diff --git a/acceptance/bundle/config-remote-sync/resolve_variables/out.test.toml b/acceptance/bundle/config-remote-sync/resolve_variables/out.test.toml
diff --git a/acceptance/bundle/config-remote-sync/resolve_variables/output.txt b/acceptance/bundle/config-remote-sync/resolve_variables/output.txt
diff --git a/acceptance/bundle/config-remote-sync/resolve_variables/script b/acceptance/bundle/config-remote-sync/resolve_variables/script
diff --git a/acceptance/bundle/config-remote-sync/resolve_variables/test.toml b/acceptance/bundle/config-remote-sync/resolve_variables/test.toml
diff --git a/acceptance/bundle/config-remote-sync/target_override/databricks.yml.tmpl b/acceptance/bundle/config-remote-sync/target_override/databricks.yml.tmpl
diff --git a/acceptance/bundle/config-remote-sync/target_override/output.txt b/acceptance/bundle/config-remote-sync/target_override/output.txt
diff --git a/bundle/configsync/variables.go b/bundle/configsync/variables.go
diff --git a/bundle/configsync/variables_test.go b/bundle/configsync/variables_test.go
diff --git a/cmd/bundle/config_remote_sync.go b/cmd/bundle/config_remote_sync.go
