Drop opt-out flag, switch to template-based backtick quoting

The earlier bindparam_string override only intercepted the primary
render path. It missed post-compile expansion used by IN clauses:
SQLAlchemy's _literal_execute_expanding_parameter builds expanded
markers (e.g. :col-name_1, :col-name_2) directly from
self.bindtemplate, bypassing bindparam_string entirely. Comprehensive
warehouse testing caught this — SELECT WHERE col IN (...) with a
hyphenated column still failed with UNBOUND_SQL_PARAMETER.

Switch to overriding bindtemplate and compilation_bindtemplate
themselves so every render path (normal bindparam_string, post-compile
expansion, render_bind_cast wrappers) gets backticked uniformly.
Using property descriptors with a no-op setter forces our template to
stick regardless of when super's __init__ assigns from BIND_TEMPLATES.

Also drop the quote_bind_params / ?quote_bind_params=false opt-out
flag — the dialect has no precedent for behavioral URL flags (only
routing: http_path, catalog, schema), and we have strong empirical
evidence the fix is safe on current platforms.

Expand unit and integration coverage: hyphen, dot, bracket, colon,
percent, slash, ?, #, +, *, @, $, &, |, <>, unicode (prénom, 姓名,
Straße), reserved words, leading digits, long names, col-name +
col_name collision, SELECT WHERE, UPDATE, DELETE, IN, multi-row
INSERT, NULL values — all 29 verified end-to-end against a live
warehouse.

Co-authored-by: Isaac

Author: msrathore-db

src/databricks/sqlalchemy/_ddl.py

@@ -84,52 +84,55 @@ def get_column_specification(self, column, **kwargs):
 
 
 class DatabricksStatementCompiler(compiler.SQLCompiler):
-    def bindparam_string(self, name, **kw):
-        """Render a bind parameter marker wrapped in backticks.
-
-        Databricks named parameter markers accept two identifier forms
-        (per ``SqlBaseParser.g4``): a bare ``IDENTIFIER``
-        (``[A-Za-z_][A-Za-z0-9_]*``) or a ``quotedIdentifier`` wrapped in
-        backticks. DataFrame-origin column names frequently contain
-        hyphens (e.g. ``col-with-hyphen``), which SQLAlchemy would
-        otherwise pass through verbatim and produce an invalid marker
-        ``:col-with-hyphen`` — the parser splits on ``-`` and reports
-        UNBOUND_SQL_PARAMETER.
-
-        Backticks are valid for *every* identifier (plain names included),
-        verified empirically against a Databricks SQL warehouse, so we
-        wrap unconditionally. This mirrors Oracle's ``:"name"`` approach
-        to the same grammar constraint and eliminates the collision risk
-        that any single-character escape map would carry (e.g. ``col-name``
-        vs ``col_name`` both mapping to ``:col_name``).
-
-        The backticks are SQL-side *quoting* only: the parameter's
-        logical name is still the text between them, so the params dict
-        sent to the driver keeps the original unquoted key. We therefore
-        leave ``escaped_bind_names`` untouched — ``construct_params``
-        passes keys through unchanged.
-
-        Gated by ``DatabricksDialect.quote_bind_params``. Set
-        ``?quote_bind_params=false`` on the SQLAlchemy URL to fall back
-        to stock bind-name rendering.
-        """
-        if (
-            kw.get("post_compile", False)
-            or kw.get("escaped_from")
-            or not getattr(self.dialect, "quote_bind_params", True)
-        ):
-            return super().bindparam_string(name, **kw)
-
-        accumulate = kw.get("accumulate_bind_names")
-        if accumulate is not None:
-            accumulate.add(name)
-        visited = kw.get("visited_bindparam")
-        if visited is not None:
-            visited.append(name)
-        quoted = f"`{name}`"
-        if self.state is compiler.CompilerState.COMPILING:
-            return self.compilation_bindtemplate % {"name": quoted}
-        return self.bindtemplate % {"name": quoted}
+    # Override the rendered marker format so every bind parameter is
+    # wrapped in backticks (`` :`name` ``) at render time. Databricks
+    # named parameter markers accept two identifier forms per
+    # ``SqlBaseParser.g4``: a bare ``IDENTIFIER`` (``[A-Za-z_][A-Za-z0-9_]*``)
+    # or a ``quotedIdentifier`` wrapped in backticks. DataFrame-origin
+    # column names frequently contain hyphens (e.g. ``col-with-hyphen``),
+    # which SQLAlchemy would otherwise render verbatim as an invalid bare
+    # marker ``:col-with-hyphen`` — the parser splits on ``-`` and reports
+    # UNBOUND_SQL_PARAMETER.
+    #
+    # Backticks are valid for *every* identifier (plain names included),
+    # verified empirically against a Databricks SQL warehouse. Setting the
+    # template here rather than overriding ``bindparam_string`` ensures the
+    # quoting applies uniformly across every rendering path — the normal
+    # bindparam_string, the escape-from path, and crucially the
+    # ``_literal_execute_expanding_parameter`` path used for IN clauses,
+    # which builds its own expanded markers directly from this template.
+    #
+    # The backticks are SQL-side *quoting* only: the parameter's logical
+    # name is still the text between them, so the params dict passed to
+    # the driver keeps the original unquoted key — ``escaped_bind_names``
+    # is left empty and ``construct_params`` passes keys through unchanged.
+
+    # Fixed template for this dialect. We use properties (with a setter
+    # that ignores the incoming value) because SQLAlchemy's SQLCompiler
+    # assigns ``self.bindtemplate`` / ``self.compilation_bindtemplate``
+    # from ``BIND_TEMPLATES[dialect.paramstyle]`` inside its own
+    # ``__init__`` — which is also where statement compilation runs. A
+    # subclass override in ``__init__`` runs too late, and a class-level
+    # attribute is shadowed by super's instance assignment. A property
+    # descriptor intercepts both the read (forcing our value) and the
+    # write (no-op), so the template is fixed regardless of order.
+    _BACKTICKED_BIND_TEMPLATE = ":`%(name)s`"
+
+    @property
+    def bindtemplate(self):
+        return self._BACKTICKED_BIND_TEMPLATE
+
+    @bindtemplate.setter
+    def bindtemplate(self, _value):
+        pass
+
+    @property
+    def compilation_bindtemplate(self):
+        return self._BACKTICKED_BIND_TEMPLATE
+
+    @compilation_bindtemplate.setter
+    def compilation_bindtemplate(self, _value):
+        pass
 
     def limit_clause(self, select, **kw):
         """Identical to the default implementation of SQLCompiler.limit_clause except it writes LIMIT ALL instead of LIMIT -1,

src/databricks/sqlalchemy/base.py

@@ -66,15 +66,6 @@ class DatabricksDialect(default.DefaultDialect):
     supports_sequences: bool = False
     supports_native_boolean: bool = True
 
-    # When True (default), every named bind parameter is rendered wrapped in
-    # backticks (`` :`name` ``) so that column names containing characters
-    # which are illegal in bare Databricks parameter identifiers (hyphens,
-    # spaces, dots, leading digits, etc.) work transparently. Set to False
-    # via the URL query string — ``?quote_bind_params=false`` — to fall back
-    # to stock SQLAlchemy bind-name rendering if this quoting causes an
-    # unexpected regression.
-    quote_bind_params: bool = True
-
     colspecs = {
         sqlalchemy.types.DateTime: dialect_type_impl.TIMESTAMP_NTZ,
         sqlalchemy.types.Time: dialect_type_impl.DatabricksTimeType,
@@ -126,10 +117,6 @@ def create_connect_args(self, url):
         self.schema = kwargs["schema"]
         self.catalog = kwargs["catalog"]
 
-        raw_quote_flag = url.query.get("quote_bind_params")
-        if raw_quote_flag is not None:
-            self.quote_bind_params = raw_quote_flag.lower() not in ("false", "0", "no")
-
         self._force_paramstyle_to_native_mode()
 
         return [], kwargs

tests/test_local/test_ddl.py

@@ -203,61 +203,153 @@ def test_plain_identifier_bind_names_are_also_backticked(self):
         assert ":`id`" in sql
         assert ":`name`" in sql
 
-    def test_space_and_dot_in_column_name_are_backticked(self):
+
+    def test_leading_digit_column_is_backticked(self):
+        """Databricks bind names cannot start with a digit bare."""
+        metadata = MetaData()
+        table = Table("t", metadata, Column("1col", String()))
+        compiled = self._compile_insert(table, {"1col": "x"})
+        assert ":`1col`" in str(compiled)
+
+    def test_many_special_characters_in_column_names(self):
+        """Column names containing characters that Delta allows (hyphens,
+        slashes, question marks, hash, plus, star, at, dollar, amp, pipe,
+        lt/gt) should render as valid backtick-quoted bind markers. We
+        intentionally exclude characters Delta rejects at DDL time
+        (space, parens, comma, equals) — those never land in a real
+        Databricks table, so never reach the bind-name path.
+        """
+        # Each of these survives a CREATE TABLE in Delta (verified empirically)
+        # and appears verbatim inside the backtick-quoted bind name — the
+        # default SQLAlchemy escape map does not translate any of them.
+        pass_through = [
+            "col-hyphen",
+            "col/slash",
+            "col?question",
+            "col#hash",
+            "col+plus",
+            "col*star",
+            "col@at",
+            "col$dollar",
+            "col&amp",
+            "col|pipe",
+            "col<lt>gt",
+        ]
+        metadata = MetaData()
+        columns = [Column(n, String()) for n in pass_through]
+        table = Table("t", metadata, *columns)
+        values = {n: f"v-{i}" for i, n in enumerate(pass_through)}
+        compiled = self._compile_insert(table, values)
+        sql = str(compiled)
+        params = compiled.construct_params()
+        for n in pass_through:
+            assert f":`{n}`" in sql, f"bind marker missing for {n!r}"
+            assert params[n] == values[n]
+
+    def test_sqlalchemy_escape_map_chars_still_work(self):
+        """SQLAlchemy's default ``bindname_escape_characters`` translates
+        a few chars (``.`` → ``_``, ``[`` → ``_``, ``]`` → ``_``, ``:`` →
+        ``C``, ``%`` → ``P``) before our backtick wrapping applies. That's
+        fine: the translated bind name is still backtick-quoted, and
+        ``escaped_bind_names`` translates the params dict key to match.
+        Verified end-to-end against a live warehouse.
+        """
         metadata = MetaData()
         table = Table(
             "t",
             metadata,
-            Column("col with space", String()),
             Column("col.with.dot", String()),
+            Column("col[bracket]", String()),
+            Column("col:colon", String()),
+            Column("col%percent", String()),
         )
         compiled = self._compile_insert(
-            table, {"col with space": "s", "col.with.dot": "d"}
+            table,
+            {
+                "col.with.dot": "d",
+                "col[bracket]": "b",
+                "col:colon": "c",
+                "col%percent": "p",
+            },
         )
         sql = str(compiled)
-        assert ":`col with space`" in sql
-        assert ":`col.with.dot`" in sql
-
+        # The bind name is translated by the escape map, then backticked
+        assert ":`col_with_dot`" in sql
+        assert ":`col_bracket_`" in sql
+        assert ":`colCcolon`" in sql
+        assert ":`colPpercent`" in sql
+
+        # The driver receives translated keys (escaped_bind_names tells
+        # construct_params how to rewrite the incoming dict).
         params = compiled.construct_params()
-        assert params["col with space"] == "s"
-        assert params["col.with.dot"] == "d"
+        assert params["col_with_dot"] == "d"
+        assert params["colCcolon"] == "c"
 
-    def test_quote_bind_params_can_be_disabled(self):
-        """Setting ``quote_bind_params=False`` on the dialect reverts to
-        stock SQLAlchemy bind-name rendering (the pre-fix behavior).
+    def test_unicode_column_names(self):
+        """Databricks allows arbitrary Unicode inside backtick-quoted
+        identifiers. Bind parameter quoting must handle Unicode names too.
         """
-        from databricks.sqlalchemy.base import DatabricksDialect
-
-        dialect = DatabricksDialect()
-        dialect.paramstyle = "named"
-        dialect.quote_bind_params = False
+        names = ["prénom", "姓名", "Straße"]
+        metadata = MetaData()
+        table = Table("t", metadata, *(Column(n, String()) for n in names))
+        values = {n: f"v{i}" for i, n in enumerate(names)}
+        compiled = self._compile_insert(table, values)
+        sql = str(compiled)
+        for n in names:
+            assert f":`{n}`" in sql
+        params = compiled.construct_params()
+        for n in names:
+            assert params[n] == values[n]
 
+    def test_sql_reserved_word_as_column_name(self):
+        """Reserved words used as column names must work as bind params too."""
         metadata = MetaData()
-        table = Table("t", metadata, Column("id", String()))
-        compiled = insert(table).values({"id": "1"}).compile(dialect=dialect)
+        table = Table("t", metadata, Column("select", String()), Column("from", String()))
+        compiled = self._compile_insert(table, {"select": "s", "from": "f"})
         sql = str(compiled)
-        assert ":id" in sql
-        assert ":`id`" not in sql
+        assert ":`select`" in sql
+        assert ":`from`" in sql
 
-    def test_url_query_string_disables_quoting(self):
-        """The URL query parameter ``?quote_bind_params=false`` turns the
-        flag off on the dialect.
+    def test_where_clause_with_hyphenated_column(self):
+        """The quoting must also apply when the hyphenated column appears in
+        a WHERE clause (SELECT / UPDATE / DELETE all share this path).
         """
-        from sqlalchemy import create_engine
-
-        engine = create_engine(
-            "databricks://token:****@****?http_path=****&catalog=****"
-            "&schema=****&quote_bind_params=false"
-        )
-        # create_engine lazy-initializes; force the dialect to process the URL
-        engine.dialect.create_connect_args(engine.url)
-        assert engine.dialect.quote_bind_params is False
+        from sqlalchemy import select
 
-    def test_url_query_string_defaults_to_quoting(self):
-        from sqlalchemy import create_engine
+        metadata = MetaData()
+        table = Table("t", metadata, Column("col-name", String()))
+        stmt = select(table).where(table.c["col-name"] == "x")
+        compiled = stmt.compile(bind=self.engine)
+        # SQLAlchemy anonymizes the bind as ``<column>_<n>`` — the hyphen
+        # survives into the bind name, so it must still be backtick-quoted.
+        assert ":`col-name_1`" in str(compiled)
+
+    def test_multivalues_insert_disambiguates_with_backticked_markers(self):
+        """Multi-row INSERT generates per-row suffixed bind names. Each
+        suffixed name must still render backtick-quoted correctly.
+        """
+        metadata = MetaData()
+        table = Table("t", metadata, Column("col-name", String()))
+        stmt = insert(table).values([{"col-name": "a"}, {"col-name": "b"}])
+        compiled = stmt.compile(bind=self.engine)
+        sql = str(compiled)
+        # SQLAlchemy emits e.g. `col-name_m0`, `col-name_m1` for row-level params
+        assert ":`col-name_m0`" in sql
+        assert ":`col-name_m1`" in sql
+
+    def test_in_clause_with_hyphenated_column_falls_through_to_postcompile(self):
+        """IN clauses use ``post_compile`` params which our override skips
+        (the rendered ``__[POSTCOMPILE_...]`` marker is not a bind name).
+        The anonymized bind SQLAlchemy assigns to the IN parameter does
+        still get backticked because it contains a hyphen (``col_name_1``
+        would be fine, but the column name slug can leak hyphens).
+        """
+        from sqlalchemy import select
 
-        engine = create_engine(
-            "databricks://token:****@****?http_path=****&catalog=****&schema=****"
-        )
-        engine.dialect.create_connect_args(engine.url)
-        assert engine.dialect.quote_bind_params is True
+        metadata = MetaData()
+        table = Table("t", metadata, Column("col-name", String()))
+        stmt = select(table).where(table.c["col-name"].in_(["a", "b"]))
+        compiled = stmt.compile(bind=self.engine)
+        # The POSTCOMPILE marker goes through super() — just make sure we
+        # didn't crash and the SQL is well-formed.
+        assert "POSTCOMPILE" in str(compiled) or "IN (" in str(compiled)