Skip to content

Commit 2720623

Browse files
authored
feat: add glue namespace impl spec (#146)
This PR adds a design for AWS Glue Data Catalog support in Lance namespaces.
1 parent 35fa4cd commit 2720623

1 file changed

Lines changed: 28 additions & 0 deletions

File tree

docs/src/spec/impls/glue.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Lance Glue Namespace
2+
3+
**Lance Glue Namespace** is an implementation using AWS Glue Data Catalog.
4+
For more details about AWS Glue, please read the [AWS Glue Data Catalog Documentation](https://docs.aws.amazon.com/glue/).
5+
6+
## Namespace Mapping
7+
8+
An AWS Glue Data Catalog can be viewed as the root Lance namespace.
9+
A database in Glue maps to the first level Lance namespace,
10+
to form a 2-level Lance namespace as a whole.
11+
12+
## Table Definition
13+
14+
When fully implemented, a Lance table should appear as a [Table](https://docs.aws.amazon.com/glue/latest/webapi/API_Table.html)
15+
object in AWS Glue with the following requirements:
16+
17+
1. the [`TableType`](https://docs.aws.amazon.com/glue/latest/webapi/API_Table.html#Glue-Type-Table-TableType) must be set to `EXTERNAL_TABLE` to indicate this is not a Glue managed table
18+
2. the [`StorageDescriptor.Location`](https://docs.aws.amazon.com/glue/latest/webapi/API_StorageDescriptor.html#Glue-Type-StorageDescriptor-Location) must point to the root location of the Lance table
19+
3. the [`Parameters`](https://docs.aws.amazon.com/glue/latest/webapi/API_Table.html#Glue-Type-Table-Parameters) must follow:
20+
1. there is a key `table_type` set to `lance` (case insensitive)
21+
2. there is a key `managed_by` set to either `storage` or `impl` (case insensitive). If not set, default to `storage`
22+
3. there is a key `version` set to the latest numeric version number of the table. This field will only be respected if `managed_by=impl`
23+
24+
## Requirement for Implementation Managed Table
25+
26+
Updates to implementation-managed Lance tables must use AWS Glue’s `VersionId` for conditional updates through the
27+
[UpdateTable](https://docs.aws.amazon.com/glue/latest/webapi/API_UpdateTable.html) API. If the `VersionId` does not
28+
match the expected version, the update fails to prevent concurrent modification conflicts.

0 commit comments

Comments
 (0)