Remove escape sequence support from interpolation parser

Escape sequences (\$ and \\) are incompatible with multi-round variable
resolution: escapes consumed in round N produce bare ${...} text that
round N+1 incorrectly resolves as a real variable reference. Remove
escape support entirely and defer to a follow-up if needed.

Co-authored-by: Isaac

Author: shreyas-goenka

NEXT_CHANGELOG.md

@@ -12,7 +12,7 @@
 * Deduplicate grant entries with duplicate principals or privileges during initialization ([#4801](https://github.com/databricks/cli/pull/4801))
 * engine/direct: Fix unwanted recreation of secret scopes when scope_backend_type is not set ([#4834](https://github.com/databricks/cli/pull/4834))
 
-* Replace regex-based variable interpolation with a character scanner. Escape sequences `\$` and `\\` are now supported ([#4747](https://github.com/databricks/cli/pull/4747)).
+* Replace regex-based variable interpolation with a character scanner ([#4747](https://github.com/databricks/cli/pull/4747)).
 
 ### Dependency updates
 

acceptance/bundle/variables/escape_sequences/databricks.yml

@@ -9,8 +9,7 @@ DABs variable interpolation (`${...}`) was regex-based. This caused:
 
 1. **Silent failures** — `${foo.bar-}` silently treated as literal text with no warning.
 2. **No suggestions** — `${bundle.nme}` produces "reference does not exist" with no hint.
-3. **No escape mechanism** — no way to produce a literal `${` in output.
-4. **No extensibility** — cannot support structured path features like key-value references `tasks[task_key="x"]` that exist in `libs/structs/structpath`.
+3. **No extensibility** — cannot support structured path features like key-value references `tasks[task_key="x"]` that exist in `libs/structs/structpath`.
 
 ## Background: How Other Systems Parse `${...}`
 
@@ -25,7 +24,7 @@ DABs variable interpolation (`${...}`) was regex-based. This caused:
 For a syntax as simple as `${path.to.var[0]}` (no nesting, no functions, no
 operators), a full recursive descent parser is overkill. A **two-mode character
 scanner** — the same core pattern used by Go's `text/template` and HCL — gives
-proper error reporting and escape support without the complexity.
+proper error reporting without the complexity.
 
 ## Design Decisions
 
@@ -36,22 +35,26 @@ No AST, no recursive descent. Easy to port to the Python implementation.
 
 See `libs/interpolation/parse.go`.
 
-### Nested `${` rejection
+### Nested `${` support
 
 Nested `${...}` inside a reference (e.g., `${var.foo_${var.tail}}`) is
-rejected as an error. This construct is ambiguous and was never intentionally
-supported — the old regex happened to match only the innermost pair by
-coincidence.
-
-### `\$` escape sequence
-
-`\$` produces a literal `$`, and `\\` produces a literal `\`. This follows
-the same convention used by Bash for escaping `$` and is the least
-surprising option for users working in shell environments.
-
-A standalone `\` before any character other than `$` or `\` is passed
-through as a literal backslash, so existing configurations that happen to
-contain backslashes are not affected.
+supported. When the parser encounters a nested `${` inside an outer
+reference, it treats the outer `${...` prefix as literal text so the inner
+reference is resolved first. Multi-round resolution (up to 11 rounds in
+`resolve_variable_references.go`) then progressively resolves from inside
+out. For example, `${a_${b_${c}}}` resolves in 3 rounds.
+
+### No escape sequences
+
+Escape sequences (e.g. `\$` → `$`) are intentionally omitted. Variable
+resolution uses multi-round processing (up to 11 rounds) where each round
+re-parses all string values. Escape sequences consumed in round N would
+produce bare `${...}` text that round N+1 would incorrectly resolve as a
+real variable reference. A safe escape mechanism would require either
+deferred consumption (escapes survive all rounds and are consumed in a
+final pass) or sentinel characters, adding significant complexity. Since
+there is no existing user demand for literal `${` in output, we defer
+escape support to a follow-up if needed.
 
 ### Malformed reference warnings
 
@@ -68,5 +71,9 @@ loop. Each `TokenRef` maps 1:1 to a resolved value, eliminating the ambiguity.
 
 ## Python sync
 
-The Python regex in `python/databricks/bundles/core/_transform.py` needs a
-corresponding update in a follow-up PR.
+PyDABs uses a regex in `python/databricks/bundles/core/_transform.py` to
+detect pure variable references (entire string is a single `${...}`). This
+regex must stay in sync with the Go parser's key/path validation. Shared
+test cases in `libs/interpolation/testdata/variable_references.json` are
+consumed by both Go (`TestParsePureVariableReferences`) and Python
+(`test_pure_variable_reference`) to verify agreement.

acceptance/bundle/variables/escape_sequences/out.test.toml

@@ -57,8 +57,7 @@ func newRef(v dyn.Value) (Ref, bool, diag.Diagnostics) {
 		}}
 	}
 
-	// Check if any token is a variable reference or if escape sequences
-	// were processed (the reconstructed string differs from the original).
+	// Check if any token is a variable reference.
 	hasRef := false
 	for _, t := range tokens {
 		if t.Kind == interpolation.TokenRef {
@@ -68,11 +67,7 @@ func newRef(v dyn.Value) (Ref, bool, diag.Diagnostics) {
 	}
 
 	if !hasRef {
-		// Even without references, if escape sequences were processed we need
-		// to return a Ref so the resolver can reconstruct the literal string.
-		if !hasEscapes(s, tokens) {
-			return Ref{}, false, nil
-		}
+		return Ref{}, false, nil
 	}
 
 	return Ref{
@@ -100,18 +95,6 @@ func (v Ref) References() []string {
 	return out
 }
 
-// hasEscapes checks whether escape sequences were processed during parsing.
-// Escape processing shortens the output (e.g., `\$` becomes `$`), so if the
-// sum of token value lengths is less than the original string length, escapes
-// were present.
-func hasEscapes(original string, tokens []interpolation.Token) bool {
-	n := 0
-	for _, t := range tokens {
-		n += len(t.Value)
-	}
-	return n < len(original)
-}
-
 // IsPureVariableReference returns true if s is a single variable reference
 // with no surrounding text.
 func IsPureVariableReference(s string) bool {

acceptance/bundle/variables/escape_sequences/output.txt

@@ -423,15 +423,3 @@ func TestResolveThreeLevelNestedVariableReference(t *testing.T) {
 	assert.Equal(t, "${a_${b_z}}", getByPath(t, out, "final").MustString())
 }
 
-func TestResolveEscapedRef(t *testing.T) {
-	in := dyn.V(map[string]dyn.Value{
-		"a": dyn.V("a"),
-		"b": dyn.V(`\${a}`),
-	})
-
-	out, err := dynvar.Resolve(in, dynvar.DefaultLookup(in))
-	require.NoError(t, err)
-
-	// Escaped reference should produce literal ${a}.
-	assert.Equal(t, "${a}", getByPath(t, out, "b").MustString())
-}

acceptance/bundle/variables/escape_sequences/script

@@ -36,10 +36,9 @@ type Token struct {
 }
 
 const (
-	dollarChar    = '$'
-	openBrace     = '{'
-	closeBrace    = '}'
-	backslashChar = '\\'
+	dollarChar = '$'
+	openBrace  = '{'
+	closeBrace = '}'
 )
 
 // keyPattern validates a single key segment in a variable path.
@@ -50,8 +49,8 @@ const (
 // (python/databricks/bundles/core/_transform.py). The patterns must stay in
 // sync. Cross-language test cases live in testdata/variable_references.json
 // and are run by both Go (TestParsePureVariableReferences) and Python
-// (test_pure_variable_reference). When changing key/index/path validation,
-// add cases to that file so both languages are tested.
+// (test_pure_variable_reference). When changing key/index/path validation
+// or reference syntax, add cases to that file so both languages are tested.
 var keyPattern = regexp.MustCompile(`^[a-zA-Z]+([-_]*[a-zA-Z0-9]+)*$`)
 
 // indexPattern matches one or more [N] index suffixes.
@@ -60,10 +59,6 @@ var indexPattern = regexp.MustCompile(`^(\[[0-9]+\])+$`)
 // Parse parses a string that may contain ${...} variable references.
 // It returns a slice of tokens representing literal text and variable references.
 //
-// Escape sequences:
-//   - "\$" produces a literal "$"
-//   - "\\" produces a literal "\"
-//
 // Nested references like "${a.${b}}" are supported by treating the outer
 // "${a." as literal text so that inner references are resolved first.
 // After resolution the resulting string (e.g. "${a.x}") is re-parsed.
@@ -72,7 +67,6 @@ var indexPattern = regexp.MustCompile(`^(\[[0-9]+\])+$`)
 //   - "hello"            -> [Literal("hello")]
 //   - "${a.b}"           -> [Ref("a.b")]
 //   - "pre ${a.b} post"  -> [Literal("pre "), Ref("a.b"), Literal(" post")]
-//   - "\${a.b}"          -> [Literal("${a.b}")]
 //   - "${a.${b}}"        -> [Literal("${a."), Ref("b"), Literal("}")]
 func Parse(s string) ([]Token, error) {
 	if len(s) == 0 {
@@ -98,30 +92,6 @@ func Parse(s string) ([]Token, error) {
 
 	for i < len(s) {
 		switch s[i] {
-		case backslashChar:
-			// Handle escape sequences: \$ -> $, \\ -> \
-			if buf.Len() == 0 {
-				litStart = i
-			}
-			if i+1 < len(s) {
-				switch s[i+1] {
-				case dollarChar:
-					buf.WriteByte(dollarChar)
-					i += 2
-				case backslashChar:
-					buf.WriteByte(backslashChar)
-					i += 2
-				default:
-					// Not a recognized escape; treat backslash as literal.
-					buf.WriteByte(backslashChar)
-					i++
-				}
-			} else {
-				// Trailing backslash at end of string: treat as literal.
-				buf.WriteByte(backslashChar)
-				i++
-			}
-
 		case dollarChar:
 			// We see '$'. Look ahead.
 			if i+1 >= len(s) {

design/interpolation-parser.md

@@ -41,64 +41,6 @@ func TestParseValidPaths(t *testing.T) {
 	}
 }
 
-func TestParseEscapeSequences(t *testing.T) {
-	tests := []struct {
-		name   string
-		input  string
-		tokens []Token
-	}{
-		{
-			"escaped_dollar",
-			`\$`,
-			[]Token{{Kind: TokenLiteral, Value: "$", Start: 0, End: 2}},
-		},
-		{
-			"escaped_ref",
-			`\${a}`,
-			[]Token{{Kind: TokenLiteral, Value: "${a}", Start: 0, End: 5}},
-		},
-		{
-			"escaped_backslash",
-			`\\`,
-			[]Token{{Kind: TokenLiteral, Value: `\`, Start: 0, End: 2}},
-		},
-		{
-			"double_escaped_backslash",
-			`\\\\`,
-			[]Token{{Kind: TokenLiteral, Value: `\\`, Start: 0, End: 4}},
-		},
-		{
-			"escaped_backslash_then_ref",
-			`\\${a.b}`,
-			[]Token{
-				{Kind: TokenLiteral, Value: `\`, Start: 0, End: 2},
-				{Kind: TokenRef, Value: "a.b", Start: 2, End: 8},
-			},
-		},
-		{
-			"backslash_before_non_special",
-			`\n`,
-			[]Token{{Kind: TokenLiteral, Value: `\n`, Start: 0, End: 2}},
-		},
-		{
-			"escaped_dollar_then_ref",
-			`\$\$${a.b}`,
-			[]Token{
-				{Kind: TokenLiteral, Value: "$$", Start: 0, End: 4},
-				{Kind: TokenRef, Value: "a.b", Start: 4, End: 10},
-			},
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			tokens, err := Parse(tt.input)
-			require.NoError(t, err)
-			assert.Equal(t, tt.tokens, tokens)
-		})
-	}
-}
-
 func TestParse(t *testing.T) {
 	tests := []struct {
 		name   string
@@ -166,20 +108,6 @@ func TestParse(t *testing.T) {
 			`abc\`,
 			[]Token{{Kind: TokenLiteral, Value: `abc\`, Start: 0, End: 4}},
 		},
-		{
-			"escaped_ref",
-			`\${a}`,
-			[]Token{{Kind: TokenLiteral, Value: "${a}", Start: 0, End: 5}},
-		},
-		{
-			"escape_between_refs",
-			`${a}\$${b}`,
-			[]Token{
-				{Kind: TokenRef, Value: "a", Start: 0, End: 4},
-				{Kind: TokenLiteral, Value: "$", Start: 4, End: 6},
-				{Kind: TokenRef, Value: "b", Start: 6, End: 10},
-			},
-		},
 		{
 			"nested_ref",
 			"${var.foo_${var.tail}}",
@@ -256,9 +184,9 @@ func TestParseErrors(t *testing.T) {
 // (python/databricks_tests/core/test_variable_references.py) to
 // verify that the Python regex stays in sync with the Go parser.
 //
-// When modifying the parser (e.g. adding new key patterns, escape
-// sequences, or reference syntax), add test cases to the JSON file
-// so both Go and Python are validated.
+// When modifying the parser (e.g. adding new key patterns or
+// reference syntax), add test cases to the JSON file so both Go
+// and Python are validated.
 func TestParsePureVariableReferences(t *testing.T) {
 	data, err := os.ReadFile("testdata/variable_references.json")
 	require.NoError(t, err)