When writing data from a PyArrow DataFrame, how should we handle 'null' Fields?

[ldsantos0911]

Question

import pyarrow as pa

# table created with the below pyarrow schema
schema = pa.schema(
    [
        pa.field("col1", pa.string(), nullable=True),
    ]
)

df = pa.Table.from_pylist(
    [
        {"col1": None}
    ]
)

table.overwrite(df)

In the above example, we encounter an error like this UnsupportedPyArrowTypeException: Column 'col1' has an unsupported type: null, with underlying cause

in _ConvertToIceberg.primitive(self, primitive)
   1211     return FixedType(primitive.byte_width)
-> 1213 raise TypeError(f"Unsupported type: {primitive}")

TypeError: Unsupported type: null

Is there any reason we wouldn't want to support the case where pyarrow has marked a Field as null? As a workaround/fix, I was thinking that we could exclude pa.null() Fields in visit_pyarrow(obj: pa.StructType, visitor: PyArrowSchemaVisitor[T]). This way, the column would effectively be missing and any required/nullable enforcement would be performed accordingly. Would this have any undesired consequences?

[kevinjqliu]

heres a workaround #2155 (review)

we should document pyarrow null type. I've encountered this before :)

[kris-gaudel]

Can I be assigned this?

[kevinjqliu]

@kris-gaudel done

[Fokko]

@kris-gaudel Gentle ping, are you still working on this?

[kris-gaudel]

@Fokko Sorry for the wait, still working on it but got caught up with work. PR up this week!

[kris-gaudel]

PR up. @ldsantos0911 already handled the tests and the core functionality. I made the error more descriptive so developers know the expected behaviour

cc @Fokko @kevinjqliu