Skip to content

DataFusion INSERT fills omitted columns with NULL instead of the column's write-default #2803

Description

@moomindani

Apache Iceberg Rust version

0.6.0 (latest version)

Describe the bug

The spec requires writers to use a column's write-default when the writer does not supply a value for the column. IcebergTableProvider does not implement DataFusion's TableProvider::get_column_default, so DataFusion's insert planner falls back to NULL for columns omitted from an INSERT. Rows written through DataFusion into a table whose schema declares write-default values silently get NULL instead of the default.

For comparison, iceberg-java delegates write-default application to the engine in the same way: its Spark integration exposes write-default as the engine-level column default (TypeToSparkType), and Spark materializes the value during INSERT planning. The DataFusion integration is the corresponding place in iceberg-rust.

To Reproduce

-- table schema: id int (required), category string with write-default 'general'
INSERT INTO t (id) VALUES (1);
SELECT category FROM t;  -- returns NULL, expected 'general'

Expected behavior

IcebergTableProvider implements get_column_default, mapping each top-level column's write-default literal to a DataFusion expression, so the insert planner fills omitted columns with the declared default.

Part of #2411 (Default Values).

Willingness to contribute

I can contribute a fix for this bug independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions