Spark: enforce Hive layout for LOCATION and disable iceberg table rename#1
Open
anandnalya wants to merge 3 commits into
Open
Spark: enforce Hive layout for LOCATION and disable iceberg table rename#1anandnalya wants to merge 3 commits into
anandnalya wants to merge 3 commits into
Conversation
For tables of the form `<db>.<table>`, validate that any user-supplied LOCATION ends with `<db>.db/<table>`. Validation runs on CREATE TABLE, CTAS, RTAS, stage variants, and ALTER TABLE SET LOCATION; null/empty pass through so the catalog default applies. Also disable iceberg table renames in SparkSessionCatalog since renames invalidate the layout invariant. Applied to spark/v3.4, v3.5, v4.0.
Accept '<db>.db/<table>_new' alongside '<db>.db/<table>' to support rebuild/swap workflows where a replacement table is staged under a sibling directory before being promoted. Applied to spark/v3.4, v3.5, v4.0.
…nLayoutValidator Strip a trailing _new suffix from the identifier name to derive a canonical base, then accept locations ending in <db>.db/<base> or <db>.db/<base>_new. This supports swap-via-rename rebuild workflows where the staging table is named <table>_new but already points at the canonical <table> location before being renamed in the catalog. Existing Hive layout and sibling-directory rebuild allowances are unchanged.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LOCATIONonCREATE TABLE, CTAS, RTAS, stage variants, andALTER TABLE SET LOCATION: for table<db>.<table>the path must end with<db>.db/<base>or<db>.db/<base>_new, where<base>is the identifier name with any trailing_newsuffix stripped. Null/empty pass through so the catalog default applies._newallowance supports swap-via-rename rebuild workflows in both directions: a sibling<table>_newstaging directory next to<table>, or an Iceberg table named<table>_newthat already points at the canonical<table>location and is later renamed in the catalog (the pattern used by Proteus'sIcebergFullReplacePartitioned.handleSchemaEvolution).SparkSessionCatalogsince renames would invalidate the layout invariant.spark/v3.4,v3.5,v4.0. Based on apache-iceberg-1.10.1.Test plan
TestLocationLayoutValidator(17 unit tests, including 3 covering the canonical/_new/_new_newcases for_new-suffixed identifiers) passes on v3.4/v3.5/v4.0TestSparkCatalogLocationLayout(6 integration tests covering CREATE/CTAS/RTAS/ALTER reject paths, the_new-identifier reject message, and null pass-through) passes on v3.5/v4.0TestSparkSessionCatalogRename(1 integration test for rename rejection) passes on v3.5/v4.0DirectByteBuffer.<init>(long, int)) that affects everyTestBase-derived test in v3.4 — unrelated to these changes; runs cleanly on JDK 11/17.