feat: add support of directory namespace and upgrade to 0.0.5 (#159)

Author: jackye1995

.github/workflows/java-lancedb.yml

@@ -94,4 +94,4 @@ jobs:
           LANCEDB_API_KEY: ${{ inputs.api_key || secrets.LANCEDB_CLOUD_API_KEY }}
           LANCEDB_REGION: ${{ inputs.region }}
           LANCEDB_HOST_OVERRIDE: ${{ inputs.host_override }}
-        run: make build-java-lancedb
+        run: make build-lancedb

docs/src/spec/impls/dir.md

@@ -5,46 +5,59 @@ People can easily get started with creating and using Lance tables directly on t
 local or remote storage system with a Lance directory namespace.
 
 A directory namespace maps to a directory on storage, we call such directory a **namespace directory**.
-A Lance table corresponds to a subdirectory in the namespace directory.
-We call such a subdirectories **table directory**.
+A Lance table corresponds to a subdirectory in the namespace directory that has the format `<table_name>.lance`.
+We call such a subdirectory **table directory**.
 Consider the following example namespace directory layout:
 
 ```
 .
 └── /my/dir1/
-    ├── table1/
+    ├── table1.lance/
     │   ├── data/
     │   │   ├── 0aa36d91-8293-406b-958c-faf9e7547938.lance
     │   │   └── ed7af55d-b064-4442-bcb5-47b524e98d0e.lance
     │   ├── _versions/
     │   │   └── 9223372036854775707.manifest
-    │   ├── _indices/
-    │   │   └── 85814508-ed9a-41f2-b939-2050bb7a0ed5-fts/
-    │   │       └── index.idx
-    │   └── _deletions/
-    │       └── 75c69434-cde5-4c80-9fe1-e79a6d952fbf.bin
-    ├── table2
-    └── table3
+    │   └── _indices/
+    │       └── 85814508-ed9a-41f2-b939-2050bb7a0ed5-fts/
+    │           └── index.idx 
+    ├── table2.lance
+    └── table3.lance
 ```
 
 This describes a Lance directory namespace with the namespace directory at `/my/dir1/`.
 It contains tables `table1`, `table2`, `table3` sitting at table directories
-`/my/dirs/table1`, `/my/dirs/table2`, `/my/dirs/table3` respectively.
+`/my/dirs/table1.lance`, `/my/dirs/table2.lance`, `/my/dirs/table3.lance` respectively.
 
-## Directory Path
+## Configuration
 
-There are 3 ways to specify a directory path:
+The Lance directory namespace accepts the following configuration properties:
+
+| Property      | Required | Description                                                 | Default                   | Example                         |
+|---------------|----------|-------------------------------------------------------------|---------------------------|---------------------------------|
+| `root`        | No       | The root directory of the namespace where tables are stored | Current working directory | `/my/dir`, `s3://bucket/prefix` |
+| `storage.*`   | No       | Storage-specific configuration options                      |                           | `storage.region=us-west-2`      |
+
+### Root Path
+
+There are 3 ways to specify a root path:
 
 1. **URI**: a URI that follows the [RFC 3986 specification](https://datatracker.ietf.org/doc/html/rfc3986), e.g. `s3://mu-bucket/prefix`.
 2. **Absolute POSIX storage path**: an absolute file path in a POSIX standard storage, e.g. `/my/dir`.
 3. **Relative POSIX storage path**: a relative file path in a POSIX standard storage, e.g. `my/dir2`, `./my/dir3`.
-   The absolute path of the directory should be based on the current directory of the running process.
+   The absolute path of the root should be derived from the current working directory.
+
+### Storage Options
+
+Properties with the `storage.` prefix are passed directly to the underlying OpenDAL storage system
+after removing the prefix. For example, `storage.region` becomes `region` when passed to the storage layer.
+Please visit [Apache OpenDAL](https://opendal.apache.org) for more details.
 
 ## Table Existence
 
-A table exists in a Lance directory namespace if a table directory of the specific name exists.
-This is true even if the directory is empty or the contents in the directory does not follow the Lance table format spec.
-For such cases, an operation that lists all tables in the directory should show the specific table,
-and an operation that checks if a table exists should return true.
-However, an operation that loads the Lance table metadata should fail with error
-indicating the content in the folder is not compliant with the Lance table format spec.
+A table exists in a Lance directory namespace if a table directory of the specific name exists
+**and** contains a non-empty `_versions` subdirectory.
+
+When checking if a specific table exists or deciding if a table should be listed, 
+the operation must list objects with the `_versions/` prefix and checks if any objects exist.
+If the directory exists but there is no file in the directory, it should be treated as non-existent.

docs/src/spec/impls/rest/index.md

@@ -6,6 +6,26 @@ for more advanced governance features around access control, auditing, lineage t
 by connecting those metadata services or building a custom metadata server in a standardized way.
 The REST server definition can be found in the [OpenAPI specification](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/lancedb/lance-namespace/refs/heads/main/docs/src/spec/rest.yaml).
 
+## Configuration
+
+The Lance REST namespace accepts the following configuration properties:
+
+| Property    | Required | Description                                                            | Default | Example                           |
+|-------------|----------|------------------------------------------------------------------------|---------|-----------------------------------|
+| `uri`       | Yes      | The URI endpoint for the REST API                                      |         | `https://api.example.com/lance`   |
+| `delimiter` | No       | The delimiter used to parse object string identifiers in REST routes   | `.`     | `.`, `/`, `::`, `#`               |
+| `headers.*` | No       | Additional headers to send with every request                          |         | `headers.Authorization=Bearer...` |
+
+### Headers
+
+Properties with the `headers.` prefix are passed as HTTP headers with every request to the REST server
+after removing the prefix. For example, `headers.Authorization` becomes the `Authorization` header.
+
+Common header configurations include:
+- `headers.Authorization`: Authentication tokens (Bearer, Basic, etc.)
+- `headers.X-API-Key`: API key authentication
+- `headers.X-Request-ID`: Request tracking
+
 ## REST Routes
 
 The REST route for an operation typically follows the pattern of `POST /<version>/<object>/{id}/<action>`,
@@ -69,4 +89,4 @@ here are some criteria to consider:
    but it might not always support a specific implementation. This favors the adapter approach.
 3. **Security**: if you have security concerns about the adapter being a man-in-the-middle, you should choose an implementation
 4. **Performance**: after all, adapter adds one layer of indirection and is thus not the most performant solution.
-   If you are performance sensitive, you should choose an implementation
+   If you are performance sensitive, you should choose an implementation

docs/src/spec/operations/alter-transaction.md

@@ -9,7 +9,7 @@
 ## Request Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:2028:2045"
+--8<-- "src/spec/rest.yaml:2025:2042"
 ```
 Supporting action schemas:
 
@@ -28,22 +28,22 @@ Supporting action schemas:
 ## Response Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:2046:2057"
+--8<-- "src/spec/rest.yaml:2043:2054"
 ```
 
 ## Related Components Schema
 ### Set Status Action
 
 ```yaml
---8<-- "src/spec/rest.yaml:1966:1971"
+--8<-- "src/spec/rest.yaml:1963:1968"
 ```
 ### Set Property Action
 
 ```yaml
---8<-- "src/spec/rest.yaml:1972:1981"
+--8<-- "src/spec/rest.yaml:1969:1978"
 ```
 ### Unset Property Action
 
 ```yaml
---8<-- "src/spec/rest.yaml:1994:2001"
+--8<-- "src/spec/rest.yaml:1991:1998"
 ```

docs/src/spec/operations/count-table-rows.md

@@ -9,10 +9,10 @@
 ## Request Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1362:1380"
+--8<-- "src/spec/rest.yaml:1359:1377"
 ```
 ## Response Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1381:1388"
+--8<-- "src/spec/rest.yaml:1378:1385"
 ```

docs/src/spec/operations/create-table-index.md

@@ -9,10 +9,10 @@
 ## Request Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1643:1708"
+--8<-- "src/spec/rest.yaml:1640:1705"
 ```
 ## Response Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1709:1726"
+--8<-- "src/spec/rest.yaml:1706:1723"
 ```

docs/src/spec/operations/create-table.md

@@ -9,10 +9,10 @@
 ## Request Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1880:1895"
+--8<-- "src/spec/rest.yaml:1877:1894"
 ```
 ## Response Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1896:1921"
+--8<-- "src/spec/rest.yaml:1895:1918"
 ```

docs/src/spec/operations/delete-from-table.md

@@ -9,10 +9,10 @@
 ## Request Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1515:1531"
+--8<-- "src/spec/rest.yaml:1512:1528"
 ```
 ## Response Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1532:1542"
+--8<-- "src/spec/rest.yaml:1529:1539"
 ```

docs/src/spec/operations/deregister-table.md

@@ -9,10 +9,10 @@
 ## Request Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:2089:2098"
+--8<-- "src/spec/rest.yaml:2086:2095"
 ```
 ## Response Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:2099:2112"
+--8<-- "src/spec/rest.yaml:2096:2109"
 ```

docs/src/spec/operations/describe-table-index-stats.md

@@ -9,10 +9,10 @@
 ## Request Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1776:1792"
+--8<-- "src/spec/rest.yaml:1773:1789"
 ```
 ## Response Schema
 
 ```yaml
---8<-- "src/spec/rest.yaml:1793:1816"
+--8<-- "src/spec/rest.yaml:1790:1813"
 ```