feat(meta): add materialized view metadata support (#20184)

* feat(meta): add materialized view definition and source index support

Persist materialized view definitions under MV table IDs.

When commit_table_meta publishes a staged MV, read the source table ID from its TableMeta options and update the source-table-to-MV index in the same transaction. CREATE OR REPLACE replaces the previous MV index membership while retaining superseded definitions for GC.

Explicit MV drop removes its definition and source-index membership. Table GC cleans up retained definitions and source-index keys.

* refactor mget_mvs_by_source_table_id and remove get_mv_ids_by_source_table_id

* refactor mget_mvs_by_source_table_id return, add source_index_seq

* add source-table and mv consistency in src/meta/api/src/api_impl/materialized_view_api.rs; add MVDefinition::schema comment in materialized_view.proto

* feat(meta): add sync creation mode to materialized view metadata

  - add immutable MVDefinition.sync_creation
  - rename schema to logical_schema to distinguish it from TableMeta.schema

* when create mv engine table create_table(as_dropped=false) and commit_table_meta do not process mv

* split vec

* feat(meta): track materialized view source generations

Replace the EmptyProto values of SourceTableMVIdent and MVSourceBindingVersionIdent with explicit generation records.

SourceTableMVIdent must represent both a durable dependency and the source generation against which an MV was built. With EmptyProto, invalidation could only be represented by deleting the relationship key. MV-invalidating source DDL would therefore require O(N) deletes and would lose the dependency needed by SHOW, refresh, DROP, and GC.

MVSourceBinding stores bound_source_generation instead. The relationship remains present after invalidating column DDL, and validity is derived by comparing the bound generation with the current source generation. Refresh can make one MV valid again by updating its relationship to the current generation.

MVSourceBindingVersionIdent also cannot use EmptyProto with its KV sequence as the semantic generation. A missing key is observed as generation 0, but its first put receives a nonzero KV sequence. An edge atomically created with bound generation 0 would therefore be stale immediately. Initializing the key in a separate transaction would leave metadata behind when CREATE MV fails and complicate concurrent CREATE operations.

MVSourceBindingVersion stores current_source_generation explicitly. This lets the first successful CREATE publish current generation 0 and an edge bound to generation 0 in one transaction. The KV sequence remains internal to the Meta API as a CAS token.

Advance the source generation only for DDL that invalidates existing MV bindings: RENAME COLUMN, DROP COLUMN, and MODIFY COLUMN. ADD COLUMN and RENAME TABLE preserve MV binding metadata. Source DROP and UNDROP also preserve the relationship and version records for the same source table identity, while source GC removes them.

Split dependency lookup into an unfiltered API for lifecycle operations and a generation-filtered API for query and maintenance paths. Validate the expected source generation during CREATE and document the CREATE, REFRESH, source DDL, DROP, UNDROP, and GC transaction contracts.

Update the unreleased v181 protobuf records and compatibility tests.

Author: TCeason

Cargo.lock

@@ -525,7 +525,7 @@ build_exceptions! {
     IllegalUser(2218),
 }
 
-// Database and Catalog Management Errors [2301-2317, 2321-2327]
+// Database and Catalog Management Errors [2301-2317, 2321-2329]
 build_exceptions! {
     /// Database already exists
     DatabaseAlreadyExists(2301),
@@ -569,6 +569,10 @@ build_exceptions! {
     GeneralDbGcFailure(2325),
     /// Table snapshot expired
     TableSnapshotExpired(2327),
+    /// Invalid materialized view metadata or state
+    InvalidMaterializedView(2328),
+    /// Materialized view already exists
+    MaterializedViewAlreadyExists(2329),
 }
 
 // Stage and Connection Errors [2501-2505, 2510-2512]

src/common/exception/src/exception_code.rs

@@ -129,6 +129,7 @@ impl AutoIncrementApiTestSuite {
                 },
                 table_meta: drop_table_meta(created_on),
                 as_dropped: true,
+                materialized_view: None,
                 table_properties: None,
                 table_partition: None,
             };

src/meta/api/src/api_impl/auto_increment_api_test_suite.rs

@@ -38,8 +38,12 @@ use databend_common_meta_app::schema::DroppedId;
 use databend_common_meta_app::schema::GcDroppedTableReq;
 use databend_common_meta_app::schema::IndexNameIdent;
 use databend_common_meta_app::schema::ListIndexesReq;
+use databend_common_meta_app::schema::MVDefinitionIdent;
+use databend_common_meta_app::schema::MVSourceBindingVersionIdent;
 use databend_common_meta_app::schema::ObjectTagIdRef;
 use databend_common_meta_app::schema::ObjectTagIdRefIdent;
+use databend_common_meta_app::schema::SourceTableMV;
+use databend_common_meta_app::schema::SourceTableMVIdent;
 use databend_common_meta_app::schema::TableCopiedFileNameIdent;
 use databend_common_meta_app::schema::TableId;
 use databend_common_meta_app::schema::TableIdHistoryIdent;
@@ -51,6 +55,7 @@ use databend_common_meta_app::schema::TaggableObject;
 use databend_common_meta_app::schema::VacuumWatermark;
 use databend_common_meta_app::schema::index_id_ident::IndexIdIdent;
 use databend_common_meta_app::schema::index_id_to_name_ident::IndexIdToNameIdent;
+use databend_common_meta_app::schema::is_materialized_view_engine;
 use databend_common_meta_app::schema::table_niv::TableNIV;
 use databend_common_meta_app::schema::vacuum_watermark_ident::VacuumWatermarkIdent;
 use databend_common_meta_app::tenant::Tenant;
@@ -772,6 +777,45 @@ async fn remove_data_for_dropped_table(
     //     warn!("{}", err);
     //     return Ok(Err(err));
     // }
+    // DROP TABLE normally deletes the MV definition. Database GC may collect
+    // tables without first running the per-table DROP path, so keep this
+    // deletion idempotent here as well.
+    if is_materialized_view_engine(&seq_meta.data.engine) {
+        txn.if_then
+            .push(txn_del(&MVDefinitionIdent::new(tenant, table_id.table_id)));
+        match seq_meta.data.materialized_view_source_table_id() {
+            Ok(source_table_id) => {
+                txn.if_then.push(txn_del(&SourceTableMVIdent::new_generic(
+                    tenant,
+                    SourceTableMV::new(source_table_id, table_id.table_id),
+                )));
+            }
+            Err(err) => {
+                warn!(
+                    table_id = table_id.table_id,
+                    error :% = err;
+                    "GC materialized view can not remove its source relationship"
+                );
+            }
+        }
+    }
+
+    let source_mv_prefix = DirName::new(SourceTableMVIdent::new_generic(
+        tenant,
+        SourceTableMV::new(table_id.table_id, 0),
+    ));
+    let source_mvs = kv_api
+        .list_pb_vec(ListOptions::unlimited(&source_mv_prefix))
+        .await?;
+    // The source table or its database is already dropped, so no new MV can
+    // reference it. Concurrent relationship deletions are idempotent.
+    txn.if_then
+        .extend(source_mvs.into_iter().map(|(ident, _)| txn_del(&ident)));
+    txn.if_then.push(txn_del(&MVSourceBindingVersionIdent::new(
+        tenant,
+        table_id.table_id,
+    )));
+
     txn_delete_exact(txn, table_id, seq_meta.seq);
 
     // Get id -> name mapping