feat: add lance partitioning spec (#279)

This pr tries to add partition as an experimental spec. Discussion could
be fount at:
#272

This PR remains a draft, and we still need to hold a vote to decide
whether to introduce the partition spec.

---------

Co-authored-by: Jack Ye <yezhaoqin@gmail.com>

Author: wojiaodoubao

docs/src/.pages

@@ -4,3 +4,4 @@ nav:
   - Directory Namespace: dir
   - REST Namespace: rest
   - Catalog Integrations: integrations
+  - Partitioning Spec: partitioning-spec.md

docs/src/dir/.pages

@@ -1,3 +1,3 @@
 nav:
   - Catalog Spec: catalog-spec.md
-  - Implementation Spec: impl-spec.md
+  - Implementation Spec: impl-spec.md

docs/src/dir/catalog-spec.md

@@ -105,6 +105,14 @@ The `__manifest` table has the following schema:
 | `metadata`     | String (nullable)       | JSON-encoded metadata/properties (only for namespaces)                                                                                                                          |
 | `base_objects` | List<String> (nullable) | Reserved for future use (e.g., view dependencies)                                                                                                                               |
 
+**Schema Extensibility**: The `__manifest` table schema may include additional columns beyond those listed above. Extensions like [partitioned namespaces](../partitioning-spec.md) add columns for efficient filtering. Implementations should preserve unrecognized columns during updates.
+
+### Root Namespace Properties
+
+In V2, the root namespace is implicit and does not have a row in the `__manifest` table. Instead, root namespace properties are stored in the `__manifest` Lance table's metadata map. Properties are stored as key-value pairs where the key is the property name and the value is a UTF-8 encoded byte array.
+
+For example, a partitioned namespace stores its `partition_spec_v1`, `partition_spec_v2`, and `schema` properties in the `__manifest` table's metadata.
+
 ### Manifest Table Indexes
 
 The following indexes are created on the manifest table for query performance:

docs/src/index.md

@@ -8,7 +8,7 @@ and the operations available on these objects, enabling integration with metadat
 
 ![Namespace Overview](overview.png)
 
-The Lance Namespace spec consists of three main parts:
+The Lance Namespace spec consists of four main parts:
 
 1. **[Client Spec](client/index.md)**: A consistent abstraction that adapts to various catalog specs,
    allowing users to access and operate on a collection of tables in a multimodal lakehouse. 
@@ -31,6 +31,10 @@ The Lance Namespace spec consists of three main parts:
    are considered **integrations** - anyone can provide additional implementation specs outside Lance Namespace,
    and they can be owned by external parties without needing to go through the Lance community voting process to be adopted.
 
+4. **[Partitioning Spec](partitioning-spec.md)**: Defines a storage format for partitioned namespaces built on the Directory Namespace.
+   It enables organizing data into physically separated units (partitions) that share a common schema,
+   with support for partition evolution, pruning, and multi-partition transactions.
+
 ## How the Spec Translates to Code
 
 ![Namespace Java SDK Example](java-sdk-example.png)