Skip to content

Reading v3 metadata with multi-argument transform fields (source-ids) fails #2801

Description

@moomindani

Apache Iceberg Rust version

0.6.0 (latest version)

Describe the bug

The V3 spec serializes partition fields and sort fields whose transform takes multiple arguments with a source-ids list instead of the single source-id. PartitionField and SortField only know source-id, so deserializing table metadata that contains any multi-argument transform field fails with missing field source-id — the whole v3 table becomes unreadable.

Per the spec, all v3 readers are required to read tables with unknown transforms, ignoring them.

For comparison, PyIceberg implemented this read tolerance in apache/iceberg-python#3630.

To Reproduce

use iceberg::spec::PartitionField;

let field: Result<PartitionField, _> = serde_json::from_str(
    r#"{"source-ids": [1, 2], "field-id": 1000, "name": "m", "transform": "bucket[4]"}"#,
);
// Err: missing field `source-id` — expected the field to load with Transform::Unknown

Loading v3 table metadata containing such a partition field or sort field fails the same way.

Expected behavior

Partition and sort fields with source-ids deserialize: a single-element list normalizes onto source-id; a multi-element list keeps the ids and treats the transform as Transform::Unknown. Serialization round-trips per the spec's writer rules (single-argument transforms write only source-id, multi-argument transforms write only source-ids).

Part of #2411 (Multi-arg Transforms → metadata read).

Willingness to contribute

I can contribute a fix for this bug independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions