feat: add table version management to dir spec (#311)

Author: jackye1995

docs/src/dir/catalog-spec.md

@@ -105,6 +105,8 @@ The `__manifest` table has the following schema:
 | `metadata`     | String (nullable)       | JSON-encoded metadata/properties (only for namespaces)                                                                                                                          |
 | `base_objects` | List<String> (nullable) | Reserved for future use (e.g., view dependencies)                                                                                                                               |
 
+**Primary Key**: The `object_id` column is the [unenforced primary key](https://lance.org/format/table/#unenforced-primary-key) for the manifest table. Implementation of this spec must always enforce the primary key uniqueness using features like Lance merge insert with primary key deduplication.
+
 **Schema Extensibility**: The `__manifest` table schema may include additional columns beyond those listed above. Extensions like [partitioned namespaces](../partitioning-spec.md) add columns for efficient filtering. Implementations should preserve unrecognized columns during updates.
 
 ### Root Namespace Properties
@@ -140,6 +142,48 @@ The `object_id` suffix ensures uniqueness and aids debugging.
 
 In [compatibility mode](#compatibility-mode), root namespace tables use `<table_name>.lance` naming to remain compatible with V1.
 
+
+### Table Version Management
+
+V2 optionally supports managed table versioning, where table versions are tracked in the `__manifest` table instead of relying on Lance's native version management. When enabled, the directory namespace acts as an [external manifest store](https://lance.org/format/table/transaction/#external-manifest-store). This feature must be enabled for the entire namespace.
+
+#### Enabling Table Version Management
+
+To enable table version management, store `table_version_management=true` in the `__manifest` Lance table's metadata map. Once enabled, all table version operations must use the namespace APIs (`CreateTableVersion`, `BatchCreateTableVersions`, `DescribeTableVersion`, `ListTableVersions`, `BatchDeleteTableVersions`) instead of the default single-table storage-only version management.
+
+#### Table Version Object ID
+
+Table versions are stored in the `__manifest` table with `object_id` in the format `<table_id>$<version>`. For example:
+
+- Table `users` version 1: `object_id = "users$1"`
+- Table `analytics$events` (in namespace `analytics`) version 5: `object_id = "analytics$events$5"`
+
+The `object_type` for table version entries is `"table_version"`.
+
+#### Table Version Metadata Schema
+
+The `metadata` column for table version entries contains a JSON object with the following schema:
+
+| Field           | Type                        | Required | Description                                                                                                                                                                                                                     |
+|-----------------|-----------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `manifest_path` | string                      | Yes      | Path to the manifest file for this version                                                                                                                                                                                      |
+| `manifest_size` | integer                     | No       | Size of the manifest file in bytes                                                                                                                                                                                              |
+| `e_tag`         | string                      | No       | ETag for the manifest file, useful for S3 and similar object stores                                                                                                                                                             |
+| `metadata`      | object (string -> string)   | No       | Optional key-value pairs of version metadata                                                                                                                                                                                    |
+| `naming_scheme` | string                      | No       | The [naming scheme](https://lance.org/format/table/transaction/#manifest-naming-schemes) used for manifest files                                                                                                                |
+
+Example metadata JSON:
+
+```json
+{
+    "manifest_path": "_versions/9223372036854775806.manifest",
+    "manifest_size": 4096,
+    "e_tag": "abc123",
+    "metadata": {"author": "user1", "description": "Initial schema"},
+    "naming_scheme": "V2"
+}
+```
+
 ## Compatibility Mode
 
 By default, the directory namespace operates in compatibility mode, supporting both V1 and V2 tables simultaneously. This allows gradual migration from V1 to V2 without disrupting existing workflows.
@@ -154,4 +198,4 @@ In compatibility mode:
 
 ### Migration from V1 to V2
 
-To fully migrate from V1 to V2, add all existing V1 table directory paths to the manifest table. Once all tables are registered in the manifest, compatibility mode can be disabled to use only V2 behavior.
+To fully migrate from V1 to V2, add all existing V1 table directory paths to the manifest table. Once all tables are registered in the manifest, compatibility mode can be disabled to use only V2 behavior.

docs/src/dir/impl-spec.md

@@ -62,16 +62,17 @@ A valid Lance table directory must be non-empty.
 
 This operation is only supported in V2. V1 does not support explicit namespace creation since it uses a flat directory structure.
 
-The implementation creates a new namespace by inserting a row into the manifest table:
+The implementation creates a new namespace using a merge-insert operation on the manifest table:
 
 1. Validate the parent namespace exists (if not creating at root level)
-2. Check that no namespace with the same identifier already exists
-3. Insert a new row into the manifest table with:
+2. Merge-insert a new row into the manifest table with:
      - `object_id` set to the namespace identifier (e.g., `prod$analytics`)
      - `object_type` set to `"namespace"`
      - `metadata` containing the namespace properties as JSON
      - `created_at` set to the current timestamp
 
+   Primary-key deduplication on `object_id` ensures no duplicate rows are inserted. If a namespace with the same identifier already exists, the operation fails.
+
 **Error Handling:**
 
 If a namespace with the same identifier already exists, return error code `2` (NamespaceAlreadyExists).
@@ -120,7 +121,7 @@ The implementation:
 1. Check that the namespace exists in the manifest table
 2. Query for any child namespaces or tables with identifiers starting with this namespace's prefix
 3. If any children exist, the operation fails
-4. Delete the namespace row from the manifest table
+4. Delete the namespace row from the manifest table using the `object_id` primary key
 
 **Error Handling:**
 
@@ -135,17 +136,18 @@ This operation declares a new Lance table, reserving the table name and location
 The implementation:
 
 1. Validate the parent namespace exists (in V2)
-2. Check that no table with the same identifier already exists
-3. Determine the table location:
+2. Determine the table location:
      - In V1: `<root>/<table_name>.lance`
      - In V2 with `dir_listing_enabled=true` at root level: `<root>/<table_name>.lance`
      - In V2 for child namespaces or with `dir_listing_enabled=false`: `<root>/<hash>_<object_id>/`
-4. Create a `.lance-reserved` file at the location to mark the table's existence
-5. In V2, insert a row into the manifest table with:
+3. Create a `.lance-reserved` file at the location to mark the table's existence
+4. In V2, merge-insert a row into the manifest table with:
      - `object_id` set to the table identifier
      - `object_type` set to `"table"`
      - `location` set to the table directory path
 
+   Primary-key deduplication on `object_id` ensures no duplicate rows are inserted. If a table with the same identifier already exists, the operation fails.
+
 **Error Handling:**
 
 If the parent namespace does not exist, return error code `1` (NamespaceNotFound).
@@ -204,6 +206,43 @@ If the table does not exist, return error code `4` (TableNotFound).
 
 If a specific version is requested and does not exist, return error code `11` (TableVersionNotFound).
 
+### DeregisterTable
+
+This operation deregisters a table from the namespace while preserving its data on storage. The table files remain at their storage location and can be re-registered later using RegisterTable.
+
+In **V1**:
+
+1. Locate the table by checking for the `<table_name>.lance` directory
+2. Verify the table exists and is not already deregistered
+3. Create a `.lance-deregistered` marker file inside the table directory
+4. Return the table location for reference
+
+The marker file approach ensures that:
+- Table data remains intact at its original location
+- The table is excluded from `ListTables` results
+- The table returns `TableNotFound` for `DescribeTable` and `TableExists` operations
+- The table can be re-registered by removing the marker file and calling `RegisterTable`
+- `DropTable` still works on deregistered tables (removes both data and marker file)
+
+In **V2**:
+
+1. Locate the table by querying the manifest table for the table location
+2. Remove the table row from the manifest table using the `object_id` primary key
+3. Keep the table files at the storage location
+4. Return the table location and properties for reference
+
+When **both V1 and V2 are enabled** (the default [Compatibility Mode](catalog-spec.md#compatibility-mode)),
+first check the manifest table, then fall back to checking the `.lance` directory.
+If found in manifest, follow V2 behavior; otherwise follow V1 behavior.
+
+**Error Handling:**
+
+If the parent namespace does not exist, return error code `1` (NamespaceNotFound).
+
+If the table does not exist or is already deregistered, return error code `4` (TableNotFound).
+
+## Additional Operations
+
 ### DropTable
 
 This operation removes a table and its data.
@@ -217,8 +256,8 @@ In **V1**:
 In **V2**:
 
 1. Locate the table by querying the manifest table for the table location
-2. Remove the table row from the manifest table first
-3. Delete the table directory and all its contents from storage 
+2. Remove the table row from the manifest table using the `object_id` primary key
+3. Delete the table directory and all its contents from storage
    (failure here does not affect the success of the drop since the table is no longer reachable)
 
 When **both V1 and V2 are enabled** (the default [Compatibility Mode](catalog-spec.md#compatibility-mode)), 
@@ -235,37 +274,134 @@ If there is a file system permission error, return error code `15` (PermissionDe
 
 If there is an unexpected I/O error, return error code `18` (Internal).
 
-### DeregisterTable
+### CreateTableVersion
 
-This operation deregisters a table from the namespace while preserving its data on storage. The table files remain at their storage location and can be re-registered later using RegisterTable.
+This operation creates a new version entry for a table. It supports `put_if_not_exists` semantics.
 
-In **V1**:
+When **table version management is not enabled**:
 
-1. Locate the table by checking for the `<table_name>.lance` directory
-2. Verify the table exists and is not already deregistered
-3. Create a `.lance-deregistered` marker file inside the table directory
-4. Return the table location for reference
+1. Resolve the table location
+2. Parse the staging manifest path from the request
+3. Determine the final manifest path based on the naming scheme (V1 or V2)
+4. Copy the staging manifest to the final path in the `_versions/` directory using `put_if_not_exists` semantics
+5. Delete the staging manifest file
+6. Return the created version info including the final manifest path
 
-The marker file approach ensures that:
-- Table data remains intact at its original location
-- The table is excluded from `ListTables` results
-- The table returns `TableNotFound` for `DescribeTable` and `TableExists` operations
-- The table can be re-registered by removing the marker file and calling `RegisterTable`
-- `DropTable` still works on deregistered tables (removes both data and marker file)
+When **table version management is enabled** (V2 with `table_version_management=true` in `__manifest` metadata), the directory namespace acts as an external manifest store. The commit process follows these steps:
 
-In **V2**:
+1. **Stage manifest in object storage**: The caller writes the new manifest to a staging path (e.g., `{table_location}/_versions/{version}.manifest-{uuid}`). This staged manifest is not yet visible to readers.
+2. **Atomically commit to manifest table**: Merge-insert a new row into the `__manifest` table with:
+    - `object_id` set to `<table_id>$<version>` (e.g., `users$1` or `ns1$users$1`)
+    - `object_type` set to `"table_version"`
+    - `metadata` containing the JSON-encoded version metadata including the staging manifest path
 
-1. Locate the table by querying the manifest table for the table location
-2. Remove the table row from the manifest table
-3. Keep the table files at the storage location
-4. Return the table location and properties for reference
+   Primary-key deduplication on `object_id` ensures no duplicate rows are inserted. The commit is effectively complete after this step. If this fails, another writer has already committed that version.
+3. **Finalize in object storage**: Copy the staged manifest to the standard location (`{table_location}/_versions/{version}.manifest`). This makes it discoverable by readers that do not use the manifest table.
+4. **Update manifest table pointer**: Update the `metadata` in the manifest table row to point to the finalized manifest path, synchronizing both systems.
 
-When **both V1 and V2 are enabled** (the default [Compatibility Mode](catalog-spec.md#compatibility-mode)),
-first check the manifest table, then fall back to checking the `.lance` directory.
-If found in manifest, follow V2 behavior; otherwise follow V1 behavior.
+**Error Handling:**
+
+If the table does not exist, return error code `4` (TableNotFound).
+
+If the version already exists, return error code `12` (TableVersionAlreadyExists).
+
+If there is a concurrent creation attempt, return error code `14` (ConcurrentModification).
+
+### BatchCreateTableVersions
+
+This operation atomically creates version entries for multiple tables.
+
+When **table version management is not enabled**, this operation iterates through each entry and calls `CreateTableVersion` for each one. Atomicity is not guaranteed.
+
+When **table version management is enabled**, the batch commit process follows these steps:
+
+1. **Stage manifests in object storage**: For each entry, the caller writes the new manifest to a staging path (e.g., `{table_location}/_versions/{version}.manifest-{uuid}`).
+2. **Atomically commit to manifest table**: Merge-insert all version rows into the `__manifest` table in a single atomic commit, each with:
+    - `object_id` set to `<table_id>$<version>`
+    - `object_type` set to `"table_version"`
+    - `metadata` containing the JSON-encoded version metadata including the staging manifest path
+
+   Primary-key deduplication on `object_id` ensures no duplicate rows are inserted. The commit is effectively complete after this step. If any version already exists, the entire batch fails.
+3. **Finalize in object storage**: For each entry, copy the staged manifest to the standard location.
+4. **Update manifest table pointers**: Update the `metadata` in each manifest table row to point to the finalized manifest paths.
 
 **Error Handling:**
 
-If the parent namespace does not exist, return error code `1` (NamespaceNotFound).
+If any table does not exist, return error code `4` (TableNotFound).
 
-If the table does not exist or is already deregistered, return error code `4` (TableNotFound).
+If any version already exists, return error code `12` (TableVersionAlreadyExists).
+
+If there is a concurrent modification, return error code `14` (ConcurrentModification).
+
+### ListTableVersions
+
+This operation lists version entries for a table.
+
+When **table version management is not enabled**:
+
+1. Resolve the table location
+2. List all files in the `_versions/` directory
+3. Parse version numbers from manifest filenames (handling both V1 and V2 naming schemes)
+4. Extract metadata from file attributes (size, e_tag, last_modified timestamp)
+5. Sort results by version number (descending if `descending=true`)
+6. Apply pagination using `page_token` and `limit`
+
+When **table version management is enabled**:
+
+1. Query the manifest table for rows where:
+    - `object_type = "table_version"`
+    - `object_id` starts with `<table_id>$`
+2. Parse the version number from each `object_id`
+3. Parse the `metadata` column as JSON to extract version details
+4. Sort results by version number (descending if `descending=true`)
+5. Apply pagination using `page_token` and `limit`
+
+**Error Handling:**
+
+If the table does not exist, return error code `4` (TableNotFound).
+
+### DescribeTableVersion
+
+This operation retrieves details for a specific table version.
+
+When **table version management is not enabled**:
+
+1. Resolve the table location
+2. Open the Lance dataset at the specified version
+3. Read the manifest file to extract version metadata
+4. Return the version information including manifest_path, manifest_size, e_tag, timestamp_millis, and metadata
+
+When **table version management is enabled**, the read process validates and synchronizes the manifest:
+
+1. **Query manifest table**: Retrieve the manifest path for the requested version from the row with `object_id = <table_id>$<version>`. If the path matches the expected path based on the naming scheme, synchronization is complete.
+2. **Synchronize to object storage**: If the manifest path does not match the expected path based on the naming scheme (i.e., it is a staging path), copy the staged manifest to its final location (`{table_location}/_versions/{version}.manifest`). This is an idempotent operation.
+3. **Update manifest table**: Update the `metadata` in the manifest table row to reflect the finalized path for future readers.
+4. **Return version information**: Return the version information with the finalized manifest path, or error if synchronization fails.
+
+**Error Handling:**
+
+If the table does not exist, return error code `4` (TableNotFound).
+
+If the version does not exist, return error code `11` (TableVersionNotFound).
+
+### BatchDeleteTableVersions
+
+This operation deletes multiple version entries for a table.
+
+When **table version management is not enabled**:
+
+1. Resolve the table location
+2. Delete the manifest files in the `_versions/` directory for each specified version
+3. Return the count of deleted versions
+
+When **table version management is enabled**:
+
+1. Delete the manifest files in the `_versions/` directory for each specified version
+2. Delete rows from the manifest table using the `object_id` primary key for each specified version
+3. Return the count of deleted versions
+
+**Error Handling:**
+
+If the table does not exist, return error code `4` (TableNotFound).
+
+If any specified version does not exist, the operation may either skip it silently or return error code `11` (TableVersionNotFound), depending on the `ignore_missing` parameter.