Skip to content

feat: add spec for Iceberg namespace #118

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jackye1995 merged 2 commits into from
Jul 9, 2025
Merged

feat: add spec for Iceberg namespace #118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0e11724
feat: add spec for Iceberg namespace
jackye1995 Jul 8, 2025
1b02c83
add no metadata-location requirement
jackye1995 Jul 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@

**Lance Namespace** is an open specification on top of the storage-based Lance data format
to standardize access to a collection of Lance tables (a.k.a. Lance datasets).
It describes how a metadata service like Apache Hive MetaStore (HMS), Apache Gravitino, Unity Catalog, etc.
should store and use Lance tables, as well as how ML/AI tools and analytics compute engines should integrate with Lance tables.
It describes how a metadata service like Apache Hive MetaStore (HMS), Apache Iceberg REST Catalog (IRC),
Apache Gravitino, Unity Catalog, etc. should store and use Lance tables,
as well as how ML/AI tools and analytics compute engines should integrate with Lance tables.

For more details, please visit the [documentation website](https://lancedb.github.io/lance-namespace).
1 change: 1 addition & 0 deletions docs/mkdocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,5 +77,6 @@ nav:
- REST: spec/impls/rest.md
- Directory: spec/impls/dir.md
- Apache Hive MetaStore: spec/impls/hive.md
- Apache Icbeerg REST Catalog: spec/impls/iceberg.md
- Contributing: contributing.md
- Lance: https://lancedb.github.io/lance
5 changes: 3 additions & 2 deletions docs/src/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@

**Lance Namespace Specification** is an open specification on top of the storage-based Lance data format
to standardize access to a collection of Lance tables (a.k.a. Lance datasets).
It describes how a metadata service like Apache Hive MetaStore (HMS), Apache Gravitino, Unity Catalog, etc.
should store and use Lance tables, as well as how ML/AI tools and analytics compute engines should integrate with Lance tables.
It describes how a metadata service like Apache Hive MetaStore (HMS), Apache Iceberg REST Catalog (IRC),
Apache Gravitino, Unity Catalog, etc. should store and use Lance tables,

@kevinjqliu kevinjqliu Jul 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are the "Apache Gravitino, Unity Catalog" integrations via IRC or custom?

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are considering 2 approaches - (1) through HMS, since that is what have been the most mature integration route in both Unity and Gravitino, (2) through REST, which we will add another REST server in these projects.

See apache/gravitino#7358 for example.

For IRC, I am actually not sure. Maybe we could integrate through this Iceberg way, but both in Gravitino and Unity, IRC is purposely built for Iceberg tables at this moment. It would require some paradigm shift. But if we think this is a good direction, happy to push for that approach in both communities.

as well as how ML/AI tools and analytics compute engines should integrate with Lance tables.

## Why _Namespace_ not _Catalog_?

Expand Down
51 changes: 51 additions & 0 deletions docs/src/spec/impls/iceberg.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Lance Iceberg Namespace

**Lance Iceberg Namespace** is an implementation using Apache Iceberg REST Catalog (IRC).
For more details about IRC, please read the [IRC Specification](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml).

!!! note

This implementation is designed against the IRC spec as of Iceberg release version 1.9.0.

## Namespace Mapping

An IRC server can be viewed as the root Lance namespace.

@kevinjqliu kevinjqliu Jul 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: iceberg tables in IRC are addressed as (namespace, table) where namespace can be "nested"/"leveled"

its not clear to me how lance tables will be presented here

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the mapping is that

  • IRC server: Lance root namespace
  • IRC multi-level namespace: Lance multi-level namespace
  • IRC tables in a namesapce: Lance tables in a namesapce

Do you think the wording is not clear? Or did I misunderstand the comment?

The Iceberg multi-level namespaces map to the multi-level child namespaces.
Whether the namespace is leveled and the number of levels depend on the specific IRC provider.

## Table Definition

A Lance table should appear as a table object in IRC in the shape of Iceberg [TableMetadata](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2482),
with the following requirements:

1. the [`location`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2494) must point to the root location of the Lance table
2. the [`properties`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2499) must follow:
1. there is a key `table_type` set to `lance` (case insensitive)

@bryanck bryanck Jul 8, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a request comes in it seems we would need to know the type before we load the metadata?

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my mind the IRC server should be able to manipulate what is returned as the response of LoadTable, for example if it's a Lance table, it can just generate a TableMetadata JSON payload without the need to read any actual Iceberg metadata JSON file, and provide this information in the properties section. Is that feasible?

@kevinjqliu kevinjqliu Jul 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HMS with iceberg works like this. It look up the generic table object and then see what the table_type property is

this also reminds me of Polaris generic table
https://polaris.apache.org/in-dev/unreleased/generic-table/

@kevinjqliu kevinjqliu Jul 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it can just generate a TableMetadata JSON payload without the need to read any actual Iceberg metadata JSON file, and provide this information in the properties section. Is that feasible?

i think its feasible as long as the reader uses the TableMetadata and not re-read the metadata.json file (i think spark does this today).

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HMS with iceberg works like this. It look up the generic table object and then see what the table_type property is

yes exactly, this is derived from the HMS experience.

i think its feasible as long as the reader uses the TableMetadata and not re-read the metadata.json file (i think spark does this today).

Yes. Maybe I should add a line that the metadata-location must not be set.

2. there is a key `managed_by` set to either `storage` or `impl` (case insensitive). If not set, default to `storage`

@bryanck bryanck Jul 8, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this property for?

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for this: https://lancedb.github.io/lance-namespace/spec/implementations/#implementation-and-storage

@kevinjqliu kevinjqliu Jul 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the default is storage, should we add a equivalent section to "Requirement for Implementation Managed Table"

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes good point, let me do that

@jackye1995 jackye1995 Jul 9, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ended up not adding a section since there is no particular requirement I can think of except for the ones in table definition. Will add one later if there is additional points unique to storage managed tables.

3. the [`current-snapshot-id`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2535) is set to the latest numeric version number of the table. This field will only be respected if `managed_by=impl`

When a user performs a `LoadTable`(https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L923) API call to retrieve the table metadata,
the server must not return a [`metadata-location`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L3260)
in the `LoadTableResponse`(https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L4669).

## Requirement for Implementation Managed Table

An update to the implementation managed table must go through IRC [UpdateTable](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L997) API
or [CommitTransaction](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L1336) API

@bryanck bryanck Jul 8, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the plan to support all IRC metadata updates? Currently Lance doesn't have branching, partitioning, etc. Maybe you could add how incompatibilities will be handled.

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this part is still hypothetical. I think we will need to start with storage managed table, which will only leverage LoadTable and CreateTable, and all the update commits directly go against the storage.

with a requirement that the [`assert-ref-snapshot-id`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L3051) is the current Lance table version.
If the commit fails due to unresolvable concurrent commits, the IRC server must fail with `409 Conflict` according to the IRC spec.

@bryanck bryanck Jul 8, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Lance support concurrent commits or are all commits serialized using the version number?

@jackye1995 jackye1995 Jul 8, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Lance support concurrent commits or are all commits serialized using the version number?

I think it's both, it's basically the same as Iceberg. There is a rebase process among concurrent commits, and then there is the commit to next version (equivalent to Iceberg's commit to catalog), if that fails it retries rebase and commit against the new concurrent commit.


## Using Lance Table in IRC with Iceberg Tooling

In order to use the table with Iceberg tooling (e.g. `pyiceberg`), the implementation can optionally set the following
in Iceberg [TableMetadata](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2482):

1. there is at least one schema in the list of [`schemas`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2504)
1. the schema reflects the latest schema of the Lance table
2. the schema has ID `1`
3. the data type conversion follows Apache Arrow to Apache Iceberg data type conversion.
2. the [`current-schema-id`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2508C9-L2508C26) is set to `1`
3. there is at least one snapshot in the list of [`snapshots`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2529).
1. the snapshot should have [`snapshot-id`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2399) set to the latest numeric version number of the table.
4. there is at least one snapshot log in the list of [`snapshot-log`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2542)
1. the snapshot log should have [`snapshot-id`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.0/open-api/rest-catalog-open-api.yaml#L2461) set to the latest numeric version number of the table.