Skip to content

feat: Honor write.metadata.path for metadata file locations#2776

Open
zakariya-s wants to merge 4 commits into
apache:mainfrom
zakariya-s:feat/write-metadata-path
Open

feat: Honor write.metadata.path for metadata file locations#2776
zakariya-s wants to merge 4 commits into
apache:mainfrom
zakariya-s:feat/write-metadata-path

Conversation

@zakariya-s

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

Metadata files were always written under <table-location>/metadata, ignoring the write.metadata.path table property. Iceberg Java honors it (via BaseMetastoreTableOperations.metadataFileLocation); this brings iceberg-rust in line and mirrors the existing write.data.path handling for data files.

  • Add TableMetadata::metadata_location_root(), returning write.metadata.path (trailing slash trimmed) when set, otherwise <location>/metadata.
  • Use it in SnapshotProducer for the manifest and manifest-list paths, and in MetadataLocation for the table metadata JSON.
  • MetadataLocation now stores the resolved metadata directory rather than the table location. On commit, the directory is re-derived from the metadata being committed (the version is still parsed from the previous file name), matching Iceberg Java. As a result MetadataLocation::from_str no longer requires the file to live under a /metadata directory (a custom write.metadata.path does not have to); file-name validation is unchanged.

Are these changes tested?

Unit tests:

  • TableMetadata::metadata_location_root(): default, configured write.metadata.path, and trailing-slash trimming.
  • MetadataLocation: create (new_with_metadata) and commit (with_new_metadata) honor write.metadata.path; from_str round-trip cases updated for arbitrary metadata dirs.
  • A fast_append transaction test asserting the manifest list and manifests are written under a configured write.metadata.path.
  • Updated catalog::tests::test_table_commit to reflect the new metadata file following the updated table location.

Metadata files (manifest lists, manifests, and the table metadata JSON) were always written under `<table-location>/metadata`, ignoring the `write.metadata.path` table property. iceberg-java honors it via `metadataFileLocation`; this brings iceberg-rust in line, mirroring how `write.data.path` is already honored for data files.

Add `TableMetadata::metadata_location_root()`, returning `write.metadata.path` (trailing slash trimmed) when set, otherwise `<location>/metadata`. Use it in `SnapshotProducer` for the manifest and manifest-list paths, and in `MetadataLocation` for the metadata JSON.

`MetadataLocation` now stores the resolved metadata directory rather than the table location. On commit the directory is re-derived from the metadata being committed (the version is still parsed from the previous file name), matching iceberg-java. As a result `MetadataLocation::from_str` no longer requires the file to live under a `/metadata` directory; file-name validation is unchanged.
Comment thread crates/iceberg/src/spec/table_metadata.rs Outdated
Comment thread crates/iceberg/src/catalog/metadata_location.rs Outdated
Comment thread crates/iceberg/src/catalog/mod.rs Outdated
@zakariya-s

Copy link
Copy Markdown
Author

Thank you for the review @andybradshaw! I think I've addressed them so far, please do let me know if there are any other flags you can see

@zakariya-s zakariya-s requested a review from andybradshaw July 9, 2026 22:10

@andybradshaw andybradshaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine to me, but will need to get another review/approval from a maintainer.

@xanderbailey xanderbailey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, just one comment I think

Comment thread crates/iceberg/src/spec/table_metadata.rs Outdated
Comment thread crates/iceberg/src/catalog/metadata_location.rs
@zakariya-s zakariya-s requested a review from xanderbailey July 15, 2026 12:44
@zakariya-s

Copy link
Copy Markdown
Author

Thanks @xanderbailey! I think I've addressed the comment. I did notice though that write.data.path isn't fetched from TableProperties:

let configured_data_location = prop
.get(WRITE_DATA_LOCATION)
.or(prop.get(WRITE_FOLDER_STORAGE_LOCATION));

Might create an issue for this later if you think it's worth it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Honor write.metadata.path table property for metadata file locations

3 participants