Skip to content

Commit ac7d31c

Browse files
jackye1995claude
jackye1995
and
claude
authored
feat: add Apache Polaris support (#236)
Co-authored-by: Claude <noreply@anthropic.com>
1 parent 02de037 commit ac7d31c

11 files changed

Lines changed: 1853 additions & 7 deletions

File tree

‎docs/src/impls/.pages‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,6 @@ nav:
44
- REST: rest
55
- Directory: dir.md
66
- Apache Hive MetaStore: hive.md
7+
- Apache Polaris: polaris.md
78
- AWS Glue: glue.md
89
- Unity Catalog: unity.md

‎docs/src/impls/polaris.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Lance Polaris Namespace
2+
3+
**Lance Polaris Namespace** is an implementation using Polaris Catalog's Generic Table API.
4+
The Polaris namespace uses the following API endpoints:
5+
- **Namespace operations**: Standard Iceberg REST API endpoints (`/namespaces`)
6+
- **Table operations**: Generic Table API endpoints (`/namespaces/{namespace}/generic-tables`)
7+
8+
For more details about Polaris Catalog, please read the [Polaris Catalog Documentation](https://polaris.apache.org).
9+
10+
## Configuration
11+
12+
The Lance Polaris namespace accepts the following configuration properties:
13+
14+
| Property | Required | Description | Default | Example |
15+
|-------------------|----------|------------------------------------------------|---------|----------------------------|
16+
| `endpoint` | Yes | Polaris server endpoint URL | | `http://localhost:8182` |
17+
| `auth_token` | No | Bearer token for authentication | | `your-auth-token` |
18+
| `connect_timeout` | No | Connection timeout in milliseconds | 10000 | `30000` |
19+
| `read_timeout` | No | Read timeout in milliseconds | 30000 | `60000` |
20+
| `max_retries` | No | Maximum number of retries for failed requests | 3 | `5` |
21+
22+
### Authentication
23+
24+
The Polaris namespace supports bearer token authentication:
25+
26+
1. **Bearer Token**: Set `auth_token` with a valid Polaris access token
27+
2. **No Authentication**: For local or unsecured Polaris deployments
28+
29+
## Namespace Mapping
30+
31+
Polaris provides a flexible namespace hierarchy:
32+
33+
- A catalog in Polaris maps to the first level Lance namespace
34+
- Nested namespaces in Polaris map to subsequent Lance namespace levels
35+
- Polaris supports arbitrary nesting depth, allowing flexible namespace organization
36+
37+
## Table Definition
38+
39+
A Lance table appears as a [Generic Table](https://github.com/polaris-catalog/polaris/blob/main/spec/polaris-catalog-apis/generic-tables-api.yaml)
40+
object in Polaris with the following requirements:
41+
42+
1. the `format` must be set to `lance` to indicate this is a Lance table
43+
2. the `base-location` must point to the root location of the Lance table
44+
3. the `doc` field can be used to store an optional table description
45+
4. the `properties` must follow:
46+
1. there is a key `table_type` set to `lance` (case insensitive)
47+
2. there is a key `managed_by` set to either `storage` or `impl` (case insensitive). If not set, default to `storage`
48+
3. there is a key `version` set to the latest numeric version number of the table. This field will only be respected if `managed_by=impl`
49+
50+
## Requirement for Implementation Managed Table
51+
52+
Updates to implementation-managed Lance tables must use Polaris's table versioning mechanism
53+
for conditional updates through the UpdateTable API. The `version` property must be updated atomically
54+
to prevent concurrent modification conflicts.

‎docs/src/impls/rest/.pages‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
title: REST
22
nav:
33
- Overview: index.md
4-
- ListNamespaces: list-namespaces.md
5-
- ListTables: list-tables.md
6-
- InsertIntoTable: insert-into-table.md
7-
- MergeInsertIntoTable: merge-insert-into-table.md
8-
- CreateTable: create-table.md
9-
- ListTableTags: list-table-tags.md
4+
- Non-Standard Operations:
5+
- ListNamespaces: list-namespaces.md
6+
- ListTables: list-tables.md
7+
- InsertIntoTable: insert-into-table.md
8+
- MergeInsertIntoTable: merge-insert-into-table.md
9+
- CreateTable: create-table.md
10+
- ListTableTags: list-table-tags.md

‎java/Makefile‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,11 +97,16 @@ build-lancedb:
9797
./mvnw spotless:apply -pl lance-namespace-lancedb -am
9898
./mvnw install -pl lance-namespace-lancedb -am
9999

100+
.PHONY: build-polaris
101+
build-polaris:
102+
./mvnw spotless:apply -pl lance-namespace-polaris -am
103+
./mvnw install -pl lance-namespace-polaris -am
104+
100105
.PHONY: clean
101106
clean: clean-apache-client clean-springboot-server
102107

103108
.PHONY: gen
104109
gen: gen-apache-client gen-springboot-server lint-apache-client lint-springboot-server
105110

106111
.PHONY: build
107-
build: build-apache-client build-springboot-server build-core build-adapter build-hive2 build-hive3 build-glue build-unity build-lancedb
112+
build: build-apache-client build-springboot-server build-core build-adapter build-hive2 build-hive3 build-glue build-unity build-lancedb build-polaris

‎java/lance-namespace-polaris/pom.xml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
~ Licensed under the Apache License, Version 2.0 (the "License");
4+
~ you may not use this file except in compliance with the License.
5+
~ You may obtain a copy of the License at
6+
~
7+
~ http://www.apache.org/licenses/LICENSE-2.0
8+
~
9+
~ Unless required by applicable law or agreed to in writing, software
10+
~ distributed under the License is distributed on an "AS IS" BASIS,
11+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
~ See the License for the specific language governing permissions and
13+
~ limitations under the License.
14+
-->
15+
<project xmlns="http://maven.apache.org/POM/4.0.0"
16+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
17+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
18+
<modelVersion>4.0.0</modelVersion>
19+
20+
<parent>
21+
<groupId>com.lancedb</groupId>
22+
<artifactId>lance-namespace-root</artifactId>
23+
<version>0.0.16</version>
24+
<relativePath>../pom.xml</relativePath>
25+
</parent>
26+
27+
<artifactId>lance-namespace-polaris</artifactId>
28+
<packaging>jar</packaging>
29+
30+
<name>Lance Namespace Polaris</name>
31+
<description>Polaris Catalog implementation for Lance namespace management</description>
32+
33+
<dependencies>
34+
<dependency>
35+
<groupId>com.lancedb</groupId>
36+
<artifactId>lance-namespace-core</artifactId>
37+
<version>${lance-namespace.version}</version>
38+
</dependency>
39+
<dependency>
40+
<groupId>com.fasterxml.jackson.core</groupId>
41+
<artifactId>jackson-databind</artifactId>
42+
</dependency>
43+
<dependency>
44+
<groupId>com.fasterxml.jackson.core</groupId>
45+
<artifactId>jackson-annotations</artifactId>
46+
</dependency>
47+
<dependency>
48+
<groupId>com.fasterxml.jackson.datatype</groupId>
49+
<artifactId>jackson-datatype-jsr310</artifactId>
50+
</dependency>
51+
<dependency>
52+
<groupId>org.slf4j</groupId>
53+
<artifactId>slf4j-api</artifactId>
54+
</dependency>
55+
56+
<!-- Test dependencies -->
57+
<dependency>
58+
<groupId>org.junit.jupiter</groupId>
59+
<artifactId>junit-jupiter</artifactId>
60+
<scope>test</scope>
61+
</dependency>
62+
<dependency>
63+
<groupId>org.mockito</groupId>
64+
<artifactId>mockito-core</artifactId>
65+
<scope>test</scope>
66+
</dependency>
67+
<dependency>
68+
<groupId>org.mockito</groupId>
69+
<artifactId>mockito-junit-jupiter</artifactId>
70+
<scope>test</scope>
71+
</dependency>
72+
<dependency>
73+
<groupId>org.assertj</groupId>
74+
<artifactId>assertj-core</artifactId>
75+
<scope>test</scope>
76+
</dependency>
77+
<dependency>
78+
<groupId>ch.qos.logback</groupId>
79+
<artifactId>logback-classic</artifactId>
80+
<scope>test</scope>
81+
</dependency>
82+
</dependencies>
83+
</project>

0 commit comments

Comments
 (0)