feat(catalog): hadoop table and namespace CRUD operations#969
Open
tanmayrauth wants to merge 5 commits intoapache:mainfrom
Open
feat(catalog): hadoop table and namespace CRUD operations#969tanmayrauth wants to merge 5 commits intoapache:mainfrom
tanmayrauth wants to merge 5 commits intoapache:mainfrom
Conversation
tanmayrauth
force-pushed
the
feat/hadoop-table-crud
branch
from
3305012 to
fb3dd76
Compare
Contributor
Author
|
@laskoviymishka @zeroshade can you please review this PR? |
zeroshade
reviewed
May 4, 2026
zeroshade
requested changes
May 4, 2026
Member
zeroshade
left a comment
zeroshade
left a comment
There was a problem hiding this comment.
LGTM just update the docstrings for NewCatalog/Catalog to specify that this only supports local filesystem for now
Contributor
Author
|
Updated the docstring. |
Member
|
looks good, just need to resolve the conflicts! |
tanmayrauth
force-pushed
the
feat/hadoop-table-crud
branch
from
bd4feb8 to
82061f7
Compare
Contributor
Author
|
Resolved the conflicts. |
tanmayrauth
force-pushed
the
feat/hadoop-table-crud
branch
from
32d2407 to
37a60d8
Compare
Contributor
Author
|
@zeroshade I've rebased the branch. |
…p catalog Implement the three core table operations: - CreateTable: validates namespace exists, rejects custom locations, writes v1.metadata.json via temp-file+rename, updates version hint - LoadTable: uses findVersion with three-tier fallback, delegates to table.NewFromLocation for metadata parsing - CheckTableExists: delegates to isTableDir Relates to apache#798 Depends-on: PR 2 (version-hint), PR 3 (namespace-ops) Depended-on-by: PR 6 (CommitTable)
Add cross-compatibility integration tests verifying CreateTable, LoadTable, and CheckTableExists work between Go and Spark Hadoop catalogs. Pre-create the hadoop-warehouse directory before Docker compose to ensure runner ownership in CI.
Update Catalog and NewCatalog docstrings to note that only local filesystem paths are currently supported.
The test was writing to <repo>/internal/recipe/hadoop-warehouse/ while Spark inside Docker used /tmp/iceberg-hadoop-warehouse, causing all cross-engine table visibility tests to fail.
tanmayrauth
force-pushed
the
feat/hadoop-table-crud
branch
from
37a60d8 to
86cbac2
Compare
Contributor
Author
|
Resolved the conflicts again 🥲 |
Contributor
Author
|
@zeroshade can you please merge this PR? 🙏 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
4: CreateTable + LoadTable + CheckTableExists
Implement the three core table operations. CreateTable validates the namespace exists, rejects custom locations, builds metadata via table.NewMetadata, writes v1.metadata.json through a temp-file-plus-rename pattern, and does a best-effort version-hint write. LoadTable calls findVersion to get the current version, builds the metadata path, and delegates to table.NewFromLocation. CheckTableExists delegates to isTableDir. Tests cover create-and-load round-trip, create with partition spec / sort order / properties, reject custom location, create in non-existent namespace, create duplicate, load non-existent, load with stale hint, and check exists true/false.
Depends on #968 #963
Relates to #798