DeltaLake integration not working (Python) (FFI Table providers not working)

[riziles]

Describe the bug

After upgrading to deltalake (Python) 0.25.1, this basic example fails. Was working fine before.

from deltalake import DeltaTable, write_deltalake
import pyarrow as pa
from datafusion import SessionContext

arrow_tbl = pa.Table.from_pydict({"x": [1, 2], "y": [2.0, 4.0]})

write_deltalake("./test/scratch", arrow_tbl, mode="overwrite")

dt = DeltaTable("./test/scratch")

ctx = SessionContext()

ctx.register_table_provider("tbl", dt)

print(ctx.table("tbl"))

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[riziles]

cross posted here: delta-io/delta-rs#3221

[ion-elgreco]

What error are you seeing?

[riziles]

Nothing. It just silently crashes on the register_table_provider step. Can't debug or anything. Never seen that in Python before.

[mag1cfrog]

not sure really relevant, but I'm also having some compilation issue when trying to use deltalake with datafusion in Rust.

Here's what I get:

   Compiling datafusion v44.0.0
   Compiling datafusion-proto v44.0.0
   Compiling deltalake-core v0.24.0
error[E0308]: mismatched types
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deltalake-core-0.24.0/src/kernel/models/actions.rs:567:9
    |
566 |     fn try_from(value: &TableFeatures) -> Result<Self, Self::Error> {
    |                                           ------------------------- expected `Result<ReaderFeatures, strum::ParseError>` because of return type
567 |         ReaderFeatures::try_from(value.as_ref())
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `strum::ParseError`, found a different `strum::ParseError`
    |
    = note: `strum::ParseError` and `strum::ParseError` have similar names, but are actually distinct types
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum-0.26.3/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/git/checkouts/strum-222d21164b266718/cc240c3/strum/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `strum` are being used?

error[E0308]: mismatched types
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deltalake-core-0.24.0/src/kernel/models/actions.rs:575:9
    |
574 |     fn try_from(value: &TableFeatures) -> Result<Self, Self::Error> {
    |                                           ------------------------- expected `Result<WriterFeatures, strum::ParseError>` because of return type
575 |         WriterFeatures::try_from(value.as_ref())
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `strum::ParseError`, found a different `strum::ParseError`
    |
    = note: `strum::ParseError` and `strum::ParseError` have similar names, but are actually distinct types
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum-0.26.3/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/git/checkouts/strum-222d21164b266718/cc240c3/strum/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `strum` are being used?

For more information about this error, try `rustc --explain E0308`.
error: could not compile `deltalake-core` (lib) due to 2 previous errors

[alamb]

Since this seems to be related to datafusion-python moving to that repo to begin triage

[alamb]

FYI @timsaucer

[ion-elgreco]

@alamb it's not entirely clear if the issue is in Datafusion FFI or the python binding though

[timsaucer]

Thanks. I suspect the culprit was a change made in the ffi signature which is when we also added the ability to get the major version for compatibility checking. I am surprised it silently fails. I’ll try to reproduce.

[timsaucer]

Which version of datafusion-python are you using? If you have 44.0.0 can you please try upgrading to 45? If that solves it, I know what the problem is. (updated based on the version of delta you specified)

[riziles]

@timsaucer , I upgraded deltalake to 0.25.4 and datafusion to 45.2.0, and the example above now works fine. Thank you!