Skip to content

Python wheel should export DuckDB C API symbols (or provide a path to them) #404

@nelson2005

Description

@nelson2005

What happens?

Prior to 1.4.1, the DuckDB Python wheel exported all C API symbols on every platform. Starting with 1.4.1, duckdb-python#81 restricted exports to only PyInit__duckdb and duckdb_adbc_init via linker flags in CMakeLists.txt L83-L110.

On macOS, -exported_symbol is a hard allowlist — all C API symbols are gone. On Linux, --export-dynamic-symbol is additive, so all symbols remain visible as a side effect. This means the C API works on Linux today but is broken on macOS — and could break on Linux too if a version script is added in the future.

Version macOS C API symbols Linux C API symbols
1.3.x ~6,300 ✅ ~6,300 ✅
1.4.0 ~6,300 ✅ ~5,900 ✅
1.4.1+ 0 ❌ ~5,900 ✅ (by accident)
1.5.x 0 ❌ ~6,300 ✅ (by accident)

This is a breaking change for projects that load the C API from the Python wheel.

Use case

numbduck wraps 160 DuckDB C API functions for use inside numba @njit compiled code. It loads the shared library from the installed duckdb Python package and resolves C API symbols at runtime. This worked on all platforms through duckdb 1.4.0, but breaks on macOS starting with 1.4.1.

The standalone libduckdb downloads have all symbols, but requiring a separate install (brew install duckdb) alongside pip install duckdb is a friction point — users expect the Python package to be self-contained.

Possible solutions

  1. Export all duckdb_* C API symbols from the Python wheel on all platforms. The symbols are already compiled into the binary — they're just hidden by the linker flags.
  2. Add a ctypes-friendly entry point — e.g. a function that returns a pointer to a struct of C API function pointers (similar to how duckdb_adbc_init works for ADBC).
  3. Bundle libduckdb.dylib/.so alongside the Python extension in the wheel, so ctypes.CDLL can load it directly.
  4. Document the limitation — if the C API is intentionally not part of the Python wheel's public interface, document that C API consumers should use the standalone libduckdb downloads.

Option 1 is the smallest change — just remove the linker flags. The PR #81 commit message says these are "the only two [symbols] that are ever needed," but that assumes all consumers go through Python or ADBC.

Environment

  • duckdb 1.4.1 through 1.5.1 (macOS broken; Linux works by accident)
  • macOS ARM64 and x86_64
  • Verified with nm -gD / nm -g on installed wheel .so files

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions