Support DuckDB Dialect by hadia206 · Pull Request #539 · bodo-ai/PyDough

hadia206 · 2026-06-25T23:47:05Z

Linked ticket

Closes #541

Type of change

Bug fix
New feature
Refactor
Docs / config

What changed and why?

Adds full DuckDB SQL support to PyDough

a new DatabaseDialect, connection handling,
dialect-specific function bindings,
Generated SQL reference files covering the existing test suite,
TPC-H and DEFOG e2e pipeline tests. NOTE: using data from sqlite,
TPC-H demo notebook,
CI workflow

How I tested this?

Locally uv run pytest -svvW ignore tests/ -m "duckdb"
Using CI DuckDB Github action workflow link (note: 4 BodoSQL failures are known failures not related to this PR)

Notes for reviewers

TPC-H data is loaded from the shared tpch.db SQLite file (via DuckDB's SQLite scanner + explicit column casts), not from DuckDB's built-in dbgen. This is because the DuckDB's embedded dbgen generated different random-text fields (address, comment) than the standard TPC-H used everywhere else which resulted in different output in a lot of our tests.
For consistency and to avoid creating a new SQL init file (or transpile script) DEFOG data is similarly loaded from a SQLite defog.db built by setup_defog.sh

review-notebook-app · 2026-06-30T19:24:49Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

…rect table names

hadia206 · 2026-06-30T22:15:47Z

        return expr

-    return expr.this if isinstance(expr.this, (exp.Count, exp.CountIf)) else expr
+    return expr.this if isinstance(expr.this, exp.Count) else expr


Including CountIf here created an issue for DuckDB when a LEFT JOIN produces all-NULL rows for a group, DuckDB’s COUNT_IF(NULL_condition) returns NULL (not 0). The COALESCE is needed here.

Tested with all dialects and they worked fine.

hadia206 · 2026-06-30T22:17:59Z

  SELECT
    COUNT(*) AS n_rows
-  FROM main.publication
+  FROM defog.academic.publication


Most defog Databricks updates are related to using correct databricks defog graph metadata. In earlier version, it was using the Sqlite metadata.

hadia206 · 2026-06-30T22:19:00Z

-      - name: Run DEFOG refresh if stale (locked by date)
-        run: uv run python -m tests.gen_data.databricks_task
+      - name: Run DEFOG refresh
+        run: uv run python -m tests.gen_data.databricks_task --force


Force regenerating with each run. This was mainly because I had to stop running the testing mid running and that made the data incomplete.

hadia206 · 2026-06-30T22:20:25Z

        try:
            self.cursor.execute(sql)

-            # This is only for MyPy to pass and know about fetch_pandas_all()


used type: ignore ... as I needed to used a different method for DuckDB so that made that TYPE_CHECKING not really needed now.

hadia206 · 2026-06-30T22:24:02Z

            raise pydough.active_session.error_builder.sql_runtime_failure(
                sql, e, True
            ) from e
-        finally:


Don't close cursor until session is done.

The old pattern created and closed a fresh cursor per method call (execute_query_df, execute_ddl, get_table_columns). DuckDB's cursor() returns an independent sub-connection, so closing between calls made TEMPORARY TABLEs and TEMPORARY VIEWs invisible to subsequent queries. The cursor is now created lazily on first access and reused for the lifetime of the connection.

hadia206 · 2026-06-30T22:24:50Z

            )
        return expr

+    def _remap_zero_to_one(self, idx: SQLGlotExpression) -> SQLGlotExpression:


Put in base and re-use in the needed dialects (Databricks, Postgres, DuckDB)

…duckdb

john-sanchez31

I left a some of comments below but overall LGTM!

john-sanchez31 · 2026-07-01T22:14:44Z

 - [Databricks_TPCH.ipynb](notebooks/Databricks_TPCH.ipynb) demonstrates how to connect a Databricks database with PyDough.
 - [Trino_TPCH.ipynb](notebooks/Trino_TPCH.ipynb) demonstrates how to connect a Trino database with PyDough.
-
+- [DuckDB_TPCH.ipynb](notebooks/DuckDB_TPCH.ipynb) demonstrates how to connect a DuckDB database with PyDough.


We didn't add databricks and duckdb to our main README file on the root folder on how to run the tests section. Let's add them

john-sanchez31 · 2026-07-01T22:18:23Z


 - `databricks`: uses a Databricks database. [See here](https://docs.databricks.com/dev-tools/python-sql-connector.html) for details on the connection API and what keyword arguments can be passed in.

+- `duckdb`: uses a DuckDB database. [See here](https://duckdb.org/docs/api/python) for details on the connection API and what keyword arguments can be passed in.


Lets also add Databricks and DuckDB on documentation/functions.md to the list list of reference links for the format strings

john-sanchez31 · 2026-07-02T14:26:41Z

+      }
+    ]
+  }
+]


Add new line here

john-sanchez31 · 2026-07-02T14:39:07Z

+    columns arrive as VARCHAR, then explicitly CAST every column to its
+    correct DuckDB type.  See ``_DUCKDB_TPCH_TABLE_SELECTS`` for the casts.
+    """
+    import duckdb


Why the import is here?

Needed in the duckdb functions only to be able to export sqlite data into the duckdb memory. If outside, will give an error when trying to import with other tests.

john-sanchez31 · 2026-07-02T14:55:21Z

+    sqlite_defog_connection: DatabaseContext,
+) -> None:
+    """
+    Test executing the defog analytical questions on the sqlite database,


Shouldn't this be duckdb?

…duckdb

hadia206 added 8 commits June 25, 2026 16:46

[run CI][run dialects] duckdb setup and tpch testing

0f1950d

[run CI][run duckdb] use sqlite db to match data

5032272

defog

428739c

[run CI][run dialects] update files and download tpch for duckdb

635a3c7

[run CI][run dialects][run s3]

bbbd560

[run CI][run s3][run dialects] use fetchdf and cleanup

fdc395e

[run CI][run dialects][run s3] cleanup

1101f73

[run CI][run dialects] docs

6b9fb7e

hadia206 added 2 commits June 30, 2026 12:54

[run CI][run s3][run dialects] update databricks defog sql to use cor…

90f3c21

…rect table names

[run CI][run dialects] comments

fba29d6

hadia206 commented Jun 30, 2026

View reviewed changes

hadia206 marked this pull request as ready for review June 30, 2026 22:29

Merge branch 'main' of https://github.com/bodo-ai/PyDough into Hadia/…

6816232

…duckdb

hadia206 requested review from john-sanchez31 and knassre-bodo June 30, 2026 23:24

john-sanchez31 approved these changes Jul 2, 2026

View reviewed changes

hadia206 added 2 commits July 2, 2026 11:11

Merge branch 'main' of https://github.com/bodo-ai/PyDough into Hadia/…

e8f55f4

…duckdb

[run CI][run dialects][run s3] address John's comment

5306bca


		- `databricks`: uses a Databricks database. [See here](https://docs.databricks.com/dev-tools/python-sql-connector.html) for details on the connection API and what keyword arguments can be passed in.

		- `duckdb`: uses a DuckDB database. [See here](https://duckdb.org/docs/api/python) for details on the connection API and what keyword arguments can be passed in.

Uh oh!

Conversation

hadia206 commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linked ticket

Type of change

What changed and why?

How I tested this?

Notes for reviewers

Uh oh!

review-notebook-app Bot commented Jun 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

john-sanchez31 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hadia206 commented Jun 25, 2026 •

edited

Loading