fix: gate manifests table file counts by manifest content type#2813
Open
gmhelmold wants to merge 1 commit into
Open
fix: gate manifests table file counts by manifest content type#2813gmhelmold wants to merge 1 commit into
gmhelmold wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
The
manifestsmetadata table double-counted file entries. Every manifest's added/existing/deleted counts were written into both the*_data_files_countand the*_delete_files_countcolumns, regardless of the manifest's content type — so a data manifest reported phantom delete-file counts (and a delete manifest would report phantom data-file counts).This gates each of the six count columns by manifest content type, matching Iceberg's reference implementation (
ManifestsTable.java,manifestFileToRow):A data manifest now populates only the
*_data_files_countcolumns (delete columns = 0), and a delete manifest only the*_delete_files_countcolumns (data columns = 0). No other column is content-type dependent in the reference (partition_summariesand the rest stay unconditional), so nothing else changes.Files:
crates/iceberg/src/inspect/manifests.rs(the fix),crates/iceberg/src/scan/mod.rs(test fixture).Are these changes tested?
Yes — unit tests in
inspect::manifests::tests.test_manifests_table_data_and_delete_manifestdrives the fix with a new fixture (setup_manifest_files_with_delete) that writes one data manifest and one v2 deletes manifest (a single addedPositionDeletesentry) into the same snapshot's manifest list. The golden pins both directions: the data-manifest row reports*_delete_files_count = 0, and the delete-manifest row reports*_data_files_count = 0(withadded_delete_files_count = 1). The test fails if either half of the gating regresses.test_manifests_tablegolden was updated to drop the bug it had baked in: a single data manifest previously assertedadded/existing/deleted_delete_files_count = 1,1,1; it now correctly asserts0,0,0.Red→green verified locally: with the six gated sites reverted to the old unconditional writes, both tests fail (the expect_test diff shows the delete-manifest row's
*_data_files_countand the data-manifest row's*_delete_files_counteach expected 0 but got 1); with the fix, both pass.cargo fmt,cargo clippy --all-targets --all-features --workspace -- -D warnings, the full inspect/datafusion-integration/sqllogictest suites all pass.User-facing change
Bug fix. The
manifestsmetadata table now reports correct per-content-type file counts.