Skip to content

fix(datafusion): honor projection in metadata table scan#2820

Open
gmhelmold wants to merge 1 commit into
apache:mainfrom
gmhelmold:fix/metadata-table-projection
Open

fix(datafusion): honor projection in metadata table scan#2820
gmhelmold wants to merge 1 commit into
apache:mainfrom
gmhelmold:fix/metadata-table-projection

Conversation

@gmhelmold

Copy link
Copy Markdown

What

IcebergMetadataTableProvider::scan() ignored the projection argument: IcebergMetadataScan always reported the full metadata-table schema and emitted unprojected batches.

Closes #2819.

Why

On any populated metadata table, aggregate queries crash and projected queries silently return every column:

INSERT INTO default.default.test_partitioned_table VALUES (1, 'a', 'b');

SELECT count(*) FROM default.default.test_partitioned_table$snapshots;
-- Internal error: Physical input schema should be the same as the one converted from
-- logical input schema. Differences: - Different number of fields: (physical) 6 vs (logical) 0.

SELECT operation FROM default.default.test_partitioned_table$snapshots;
-- returns all 6 snapshot columns

Same class as #828, which fixed this for IcebergTableScan; the metadata provider path was left behind because existing tests only exercise empty metadata tables.

Fix

IcebergMetadataScan is now projection-aware, mirroring IcebergTableScan: it carries the projection indices, reports the projected schema in its PlanProperties, and projects each emitted RecordBatch. scan() passes the projection through.

Out of scope (behavior unchanged, correctness preserved because DataFusion applies both on top of the scan): filters never reach scan() (supports_filters_pushdown is not overridden), and limit remains a missed optimization only. EXPLAIN output for the metadata scan does not yet render the projection (pre-existing) — happy to add that in a follow-up if wanted.

Tests

  • New test_metadata_table_projection integration test: aggregate on a populated $snapshots (count(*) — the empty-projection edge) plus a single-column projection.
  • New sqllogictest schedule df_inspect_snapshots_manifests: first populated-metadata-table SLT coverage (deterministic columns only), doubling as regression coverage for both failure modes.
  • Red→green: reverting the two source files to main fails the new SLT with the unprojected 12-column $manifests row; with the fix the full sqllogictest suite passes (run twice to confirm determinism).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IcebergMetadataTableProvider ignores projection in scan(): projected/aggregate queries on populated metadata tables fail

1 participant