Core: Add metadata table for discovering metadata tables by yangshangqing95 · Pull Request #17134 · apache/iceberg

yangshangqing95 · 2026-07-07T20:26:26Z

Summary

This PR adds a new Iceberg metadata table, metadata_tables, to make available metadata tables discoverable through the existing metadata table mechanism.

For example, in Spark users can query:

SELECT * FROM catalog.db.table.metadata_tables;

Result:

entries
files
data_files
delete_files
history
metadata_log_entries
snapshots
refs
manifests
partitions
all_data_files
all_delete_files
all_files
all_manifests
all_entries
position_deletes
metadata_tables

The new metadata table returns the supported metadata table names

This avoids requiring users to know all metadata table names ahead of time from documentation or source code.

Also, replace selected hard-coded metadata table suffixes in tests and metadata table implementations with MetadataTableType#tableName, enhance code robustness.

gaborkaszab

Hi @yangshangqing95 ,
I took a Quick Look and left some comments wrt refactors and the new metadata scan implementation.

A general note: querying static data such as metadata table names through SELECT * FROM catalog.db.table.metadata_tables seems somewhat odd to me. We'd expect the same answer regardless the table we use for the query, right?
Anyway, I recall for Iceberg V4 there are consideration to not allow some of the current metadata table and have new ones because the physical representation of a table changes. So this might make sense to have different answers based on the table's version.
Let's see what others think!

gaborkaszab · 2026-07-09T14:24:31Z

+            .map(metadataType -> StaticDataTask.Row.of(metadataType.tableName()))
+            .toArray(StaticDataTask.Row[]::new);
+
+    DataFile metadataFile =


I randomly checked other metadata tables and they seem to use the following InputFile:
table().io().newInputFile(table().operations().current().metadataFileLocation()),

That’s a good point. For listing the available metadata tables, at least at this stage, we don’t actually need to read any physical file.

Using a directly constructed empty DataFile has a couple of benefits here:

It avoids unnecessary file operations.

It makes the intent clearer in the code, since otherwise it may look like the metadata table information is stored in or read from the table metadata file.

So I think using an empty DataFile is more explicit for this particular metadata table.

Please follow the pattern other metadata tables are using. See my comment above for constructor vs of()

gaborkaszab · 2026-07-09T14:25:08Z

+            .withFileSizeInBytes(0)
+            .build();
+
+    return new StaticDataTask(metadataFile, schema(), scan.schema(), rows);


Other metadata tables use StaticDataTask.of()

same reason as the above one

see comment above

gaborkaszab · 2026-07-09T14:30:14Z

+  }
+
+  private DataTask task(BaseTableScan scan) {
+    StaticDataTask.Row[] rows =


Checking RefsTable instead of constructing StaticDataTask.Row[]', we can simply use Collectionwith a transform function to convert it to row viaStaticDataTask.Row.of()`

Hi @gaborkaszab same reason with the usage of empty Datafile, as constructor accept parameter Row[]

StaticDataTask( DataFile metadataFile, Schema tableSchema, Schema projectedSchema, StructLike[] rows)

I don't think we are expected to use the constructor from metadata table code. They use StaticDataTask.of() instead.

yangshangqing95 · 2026-07-09T18:57:17Z

Hi @yangshangqing95 , I took a Quick Look and left some comments wrt refactors and the new metadata scan implementation.

A general note: querying static data such as metadata table names through SELECT * FROM catalog.db.table.metadata_tables seems somewhat odd to me. We'd expect the same answer regardless the table we use for the query, right? Anyway, I recall for Iceberg V4 there are consideration to not allow some of the current metadata table and have new ones because the physical representation of a table changes. So this might make sense to have different answers based on the table's version. Let's see what others think!

Hi @gaborkaszab thanks for taking a look and leaving the comments!

Regarding the metadata table query, I did consider this initially as well. From both an implementation perspective and the current project structure, I’m still leaning toward exposing it under each table, mainly for a few reasons:

We don’t currently have a default metadata catalog to store this kind of information, and introducing one just for this metadata feels a bit heavy.
Given the current code structure, putting this in MetadataTableType keeps the change smaller and less intrusive.
There may be more metadata table types in the future, and the supported set could differ across Iceberg versions such as V1, V2, V3, and V4. Keeping this under each table leaves room for future extension, where each table version can expose the metadata tables that are meaningful for it.

So from that perspective, it may not necessarily be true that all tables should always return the exact same metadata tables.

gaborkaszab

Thanks for the changes @yangshangqing95 !
Seems way simpler now than initially. Left some further comments, but in the meantime let me try to get additional feedback on the approach, because SELECT * FROM db.tbl.metadata_tables; still feels a bit odd for me, however I get the explanation too.
Maybe @nastra would you mind taking a look?

gaborkaszab · 2026-07-13T09:40:08Z

+    return this.lower;
+  }
+
+  public static List<String> lowerNames() {


I have't found any usage for this in this PR. If I haven't missed anything, please remove.

this is for testing

gaborkaszab · 2026-07-13T09:41:09Z

+  POSITION_DELETES,
+  METADATA_TABLES;
+
+  private final String lower;


I'd leave it to the metadata table code to convert this to lowercase. I'd rather remove this and the getter method too

Thanks for the review. I was treating the lowercase table name as a property of MetadataTableType, since it is the canonical metadata table name exposed to users and used during resolution. My goal was to avoid duplicating the conversion logic and hard-coded suffixes across metadata table code.

However, I see your point that adding state and accessors to the enum may be broader than necessary for this PR. If you insist I can move the lowercase conversion into the new metadata_tables implementation and remove tableName() / tableNames() from MetadataTableType.

gaborkaszab · 2026-07-13T09:46:19Z

        return new AllEntriesTable(baseTable, metadataTableName);
      case POSITION_DELETES:
        return new PositionDeletesTable(baseTable, metadataTableName);
+      case METADATA_TABLES:


Would be nice to have some test coverage too for the whole PR. Leaving the comment here to keep track.

yes there were testings but dropped with some previous modification, re-added.

github-actions Bot added spark core docs labels Jul 7, 2026

yangshangqing95 force-pushed the feat/add-metadata-tables-table branch 6 times, most recently from 44676e5 to 4c79db9 Compare July 8, 2026 19:59

gaborkaszab reviewed Jul 9, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/AllDataFilesTable.java Outdated

gaborkaszab reviewed Jul 9, 2026

View reviewed changes

yangshangqing95 force-pushed the feat/add-metadata-tables-table branch 5 times, most recently from f3ccf81 to c178a29 Compare July 10, 2026 17:46

gaborkaszab reviewed Jul 13, 2026

View reviewed changes

yangshangqing95 force-pushed the feat/add-metadata-tables-table branch 2 times, most recently from c5344da to 9786c6e Compare July 13, 2026 15:13

Core: Add metadata table for discovering metadata tables

c13a880

yangshangqing95 force-pushed the feat/add-metadata-tables-table branch from 9786c6e to c13a880 Compare July 13, 2026 15:36

Uh oh!

Conversation

yangshangqing95 commented Jul 7, 2026

Summary

Uh oh!

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yangshangqing95 commented Jul 9, 2026

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants