spec: support multi-catalog concept

README.md

@@ -1,11 +1,10 @@
 # Lance Catalog
 
-**Lance Catalog** is an OpenAPI specification on top of the storage-based Lance format.
-It provides an integration point for catalog service like Apache Hive MetaStore (HMS), Apache Gravitino, etc. 
-to store and use Lance tables. To integrate, the catalog service implements a **Lance Catalog Adapter**, 
-which is a REST server that converts the Lance catalog requests to native requests against the catalog service.
-Different tools can integrate with Lance Catalog using the generated OpenAPI clients in various languages,
-and invoke operations in Lance Catalog to read, write and manage Lance tables in the integrated catalog services.
+**Lance Catalog** is an open specification on top of the storage-based Lance open table and data format
+to standardize access to a collection of Lance tables.
+It describes how a catalog service like Apache Hive MetaStore (HMS), Apache Gravitino, Unity Catalog, etc.
+should store and use Lance tables, as well as how ML/AI tools and analytics compute engines
+should integrate with Lance.
 
 ## Repository structure
 

java/Makefile

@@ -17,7 +17,7 @@ clean-java-apache-client:
 
 gen-java-apache-client: clean-java-apache-client
 	openapi-generator-cli generate \
-	  -i ../spec/catalog.yaml \
+	  -i ../spec/rest-catalog.yaml \
 	  -g java \
 	  -o lance-catalog-apache-client \
 	  --ignore-file-override=.apache-client-ignore \
@@ -36,7 +36,7 @@ clean-java-springboot-server:
 
 gen-java-springboot-server: clean-java-springboot-server
 	openapi-generator-cli generate \
-	  -i ../spec/catalog.yaml \
+	  -i ../spec/rest-catalog.yaml \
 	  -g spring \
       -o lance-catalog-springboot-server \
       --additional-properties=groupId=com.lancedb,artifactId=lance-catalog-springboot-server,artifactVersion=$(VERSION),parentGroupId=com.lancedb,parentArtifactId=lance-catalog-root,parentVersion=$(VERSION),parentRelativePath=pom.xml,library=spring-boot,interfaceOnly=true,useOptional=true,openApiNullable=false,java8=true,apiPackage=com.lancedb.lance.catalog.server.springboot.api,modelPackage=com.lancedb.lance.catalog.server.springboot.model,useTags=true,skipDefaultInterface=false,hideGenerationTimestamp=true,licenseName=Apache-2.0,licenseUrl=https://www.apache.org/licenses/LICENSE-2.0.txt

java/lance-catalog-apache-client/.openapi-generator/FILES

@@ -1,13 +1,13 @@
 .openapi-generator-ignore
 README.md
 api/openapi.yaml
-docs/CreateNamespaceRequest.md
-docs/CreateNamespaceResponse.md
+docs/CatalogApi.md
+docs/CreateCatalogRequest.md
+docs/CreateCatalogResponse.md
 docs/ErrorResponse.md
-docs/GetNamespaceResponse.md
+docs/GetCatalogResponse.md
 docs/GetTableResponse.md
-docs/ListNamespacesResponse.md
-docs/NamespaceApi.md
+docs/ListCatalogsResponse.md
 docs/RegisterTableRequest.md
 docs/TableApi.md
 pom.xml
@@ -22,16 +22,16 @@ src/main/java/com/lancedb/lance/catalog/client/apache/RFC3339DateFormat.java
 src/main/java/com/lancedb/lance/catalog/client/apache/ServerConfiguration.java
 src/main/java/com/lancedb/lance/catalog/client/apache/ServerVariable.java
 src/main/java/com/lancedb/lance/catalog/client/apache/StringUtil.java
-src/main/java/com/lancedb/lance/catalog/client/apache/api/NamespaceApi.java
+src/main/java/com/lancedb/lance/catalog/client/apache/api/CatalogApi.java
 src/main/java/com/lancedb/lance/catalog/client/apache/api/TableApi.java
 src/main/java/com/lancedb/lance/catalog/client/apache/auth/ApiKeyAuth.java
 src/main/java/com/lancedb/lance/catalog/client/apache/auth/Authentication.java
 src/main/java/com/lancedb/lance/catalog/client/apache/auth/HttpBasicAuth.java
 src/main/java/com/lancedb/lance/catalog/client/apache/auth/HttpBearerAuth.java
-src/main/java/com/lancedb/lance/catalog/client/apache/model/CreateNamespaceRequest.java
-src/main/java/com/lancedb/lance/catalog/client/apache/model/CreateNamespaceResponse.java
+src/main/java/com/lancedb/lance/catalog/client/apache/model/CreateCatalogRequest.java
+src/main/java/com/lancedb/lance/catalog/client/apache/model/CreateCatalogResponse.java
 src/main/java/com/lancedb/lance/catalog/client/apache/model/ErrorResponse.java
-src/main/java/com/lancedb/lance/catalog/client/apache/model/GetNamespaceResponse.java
+src/main/java/com/lancedb/lance/catalog/client/apache/model/GetCatalogResponse.java
 src/main/java/com/lancedb/lance/catalog/client/apache/model/GetTableResponse.java
-src/main/java/com/lancedb/lance/catalog/client/apache/model/ListNamespacesResponse.java
+src/main/java/com/lancedb/lance/catalog/client/apache/model/ListCatalogsResponse.java
 src/main/java/com/lancedb/lance/catalog/client/apache/model/RegisterTableRequest.java

java/lance-catalog-apache-client/README.md

@@ -1,17 +1,14 @@
 # lance-catalog-apache-client
 
-Lance Catalog REST Specification
+Lance REST Catalog Specification
 
 - API version: 0.0.1
 
 - Generator version: 7.12.0
 
-**Lance Catalog** is an OpenAPI specification on top of the storage-based Lance format.
-It provides an integration point for catalog service like Apache Hive MetaStore (HMS), Apache Gravitino, etc.
-to store and use Lance tables. To integrate, the catalog service implements a **Lance Catalog Adapter**,
-which is a REST server that converts the Lance catalog requests to native requests against the catalog service.
-Different tools can integrate with Lance Catalog using the generated OpenAPI clients in various languages,
-and invoke operations in Lance Catalog to read, write and manage Lance tables in the integrated catalog services.
+**Lance Catalog** is an open specification on top of the storage-based Lance open table and data format  to standardize access to a collection of Lance tables. It describes how a catalog service like Apache Hive MetaStore (HMS), Apache Gravitino, Unity Catalog, etc. should store and use Lance tables, as well as how ML/AI tools and analytics compute engines (will together be called _\"tools\"_ in this document) should integrate with Lance.
+A Lance catalog is a centralized repository for discovering, organizing, and managing Lance tables. It is a generalized concept that is also called namespace, metastore, database, schema in other similar systems. A Lance catalog can either contain a list of tables, or contain a list of Lance catalogs recursively.
+In an enterprise environment, typically there is a requirement to store tables in a catalog service  such as Apache Hive MetaStore, Apache Gravitino, Unity Catalog, etc.  for more advanced governance features around access control, auditing, lineage tracking, etc. **Lance REST catalog** is a standardized OpenAPI protocol to read, write and manage Lance tables.
 
 
 
@@ -83,21 +80,21 @@ Please follow the [installation](#installation) instruction and execute the foll
 import com.lancedb.lance.catalog.client.apache.*;
 import com.lancedb.lance.catalog.client.apache.auth.*;
 import com.lancedb.lance.catalog.client.apache.model.*;
-import com.lancedb.lance.catalog.client.apache.api.NamespaceApi;
+import com.lancedb.lance.catalog.client.apache.api.CatalogApi;
 
-public class NamespaceApiExample {
+public class CatalogApiExample {
 
     public static void main(String[] args) {
         ApiClient defaultClient = Configuration.getDefaultApiClient();
         defaultClient.setBasePath("http://localhost:2333");
 
-        NamespaceApi apiInstance = new NamespaceApi(defaultClient);
-        CreateNamespaceRequest createNamespaceRequest = new CreateNamespaceRequest(); // CreateNamespaceRequest | 
+        CatalogApi apiInstance = new CatalogApi(defaultClient);
+        String catalog = "catalog_example"; // String | An identifier of the catalog.
+        String catalogDelimiter = "."; // String | The delimiter used by the catalog identifier
         try {
-            CreateNamespaceResponse result = apiInstance.createNamespace(createNamespaceRequest);
-            System.out.println(result);
+            apiInstance.catalogExists(catalog, catalogDelimiter);
         } catch (ApiException e) {
-            System.err.println("Exception when calling NamespaceApi#createNamespace");
+            System.err.println("Exception when calling CatalogApi#catalogExists");
             System.err.println("Status code: " + e.getCode());
             System.err.println("Reason: " + e.getResponseBody());
             System.err.println("Response headers: " + e.getResponseHeaders());
@@ -114,24 +111,24 @@ All URIs are relative to *http://localhost:2333*
 
 Class | Method | HTTP request | Description
 ------------ | ------------- | ------------- | -------------
-*NamespaceApi* | [**createNamespace**](docs/NamespaceApi.md#createNamespace) | **POST** /v1/namespaces | Create a new namespace. A catalog can manage one or more namespaces. A namespace is used to manage one or more tables. There are three modes when trying to create a namespace:   * CREATE: Create the namespace if it does not exist. If a namespace of the same name already exists, the operation fails with 400.   * EXIST_OK: Create the namespace if it does not exist. If a namespace of the same name already exists, the operation succeeds and the existing namespace is kept.   * OVERWRITE: Create the namespace if it does not exist. If a namespace of the same name already exists, the existing namespace is dropped and a new namespace with this name with no table is created. 
-*NamespaceApi* | [**dropNamespace**](docs/NamespaceApi.md#dropNamespace) | **DELETE** /v1/namespaces/{ns} | Drop a namespace from the catalog. Namespace must be empty.
-*NamespaceApi* | [**getNamespace**](docs/NamespaceApi.md#getNamespace) | **GET** /v1/namespaces/{ns} | Get information about a namespace
-*NamespaceApi* | [**listNamespaces**](docs/NamespaceApi.md#listNamespaces) | **GET** /v1/namespaces | List all namespaces in the catalog. 
-*NamespaceApi* | [**namespaceExists**](docs/NamespaceApi.md#namespaceExists) | **HEAD** /v1/namespaces/{ns} | Check if a namespace exists
-*TableApi* | [**getTable**](docs/TableApi.md#getTable) | **GET** /v1/namespaces/{ns}/tables/{table} | Get a table from the catalog
-*TableApi* | [**registerTable**](docs/TableApi.md#registerTable) | **POST** /v1/namespaces/{ns}/register | Register a new table in the given namespace. A table represents a lance dataset.  In Lance catalog, a table must be hosted in a namespace. 
-*TableApi* | [**tableExists**](docs/TableApi.md#tableExists) | **HEAD** /v1/namespaces/{ns}/tables/{table} | Check if a table exists
+*CatalogApi* | [**catalogExists**](docs/CatalogApi.md#catalogExists) | **HEAD** /v1/catalogs/{catalog} | Check if a catalog exists
+*CatalogApi* | [**createCatalog**](docs/CatalogApi.md#createCatalog) | **POST** /v1/catalogs | Create a new catalog. A catalog can manage either a collection of child catalogs, or a collection of tables. There are three modes when trying to create a catalog to differentiate the behavior when a catalog of the same name already exists:   * CREATE: the operation fails with 400.   * EXIST_OK: the operation succeeds and the existing catalog is kept.   * OVERWRITE: the existing catalog is dropped and a new empty catalog with this name is created. 
+*CatalogApi* | [**dropCatalog**](docs/CatalogApi.md#dropCatalog) | **DELETE** /v1/catalogs/{catalog} | Drop a catalog. The catalog must be empty.
+*CatalogApi* | [**getCatalog**](docs/CatalogApi.md#getCatalog) | **GET** /v1/catalogs/{catalog} | Get information about a catalog
+*CatalogApi* | [**listCatalogs**](docs/CatalogApi.md#listCatalogs) | **GET** /v1/catalogs | List all direct child catalogs of the root catalog. 
+*TableApi* | [**getTable**](docs/TableApi.md#getTable) | **GET** /v1/tables/{table} | Get a table from the catalog
+*TableApi* | [**registerTable**](docs/TableApi.md#registerTable) | **POST** /v1/catalogs/{catalog}/register | Register an existing table in the given catalog. 
+*TableApi* | [**tableExists**](docs/TableApi.md#tableExists) | **HEAD** /v1/tables/{table} | Check if a table exists
 
 
 ## Documentation for Models
 
- - [CreateNamespaceRequest](docs/CreateNamespaceRequest.md)
- - [CreateNamespaceResponse](docs/CreateNamespaceResponse.md)
+ - [CreateCatalogRequest](docs/CreateCatalogRequest.md)
+ - [CreateCatalogResponse](docs/CreateCatalogResponse.md)
  - [ErrorResponse](docs/ErrorResponse.md)
- - [GetNamespaceResponse](docs/GetNamespaceResponse.md)
+ - [GetCatalogResponse](docs/GetCatalogResponse.md)
  - [GetTableResponse](docs/GetTableResponse.md)
- - [ListNamespacesResponse](docs/ListNamespacesResponse.md)
+ - [ListCatalogsResponse](docs/ListCatalogsResponse.md)
  - [RegisterTableRequest](docs/RegisterTableRequest.md)