feat: add data lineage

[youngsofun]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR adds persistent object-level and column-level data lineage, lifecycle handling, and the Enterprise GET_LINEAGE table function.

Related PRs

• feat(query): extract query lineage from plans #20201 extracts canonical lineage from optimized query plans. It is the planner foundation for this PR and is intended to merge first. This branch currently includes the same extraction commit so the persistence and SQL workflow tests remain executable end to end; after feat(query): extract query lineage from plans #20201 merges, this PR will rebase onto main and drop that duplicate commit.
• feat(query): reject alter view definition changes #20202 rejects in-place ALTER VIEW query and output-column changes. It was split out so the behavioral change can be reviewed independently. This PR no longer contains or depends on that change.

The remaining changes stay together because the metadata schema, metadata API, persistence paths, lifecycle behavior, GET_LINEAGE, and SQL workflow tests share the same identity and column-mapping contract. Intermediate subsets would not provide a complete or meaningfully testable user workflow.

Reviewing commit by commit is recommended:

1. Extract query lineage (temporary duplicate of feat(query): extract query lineage from plans #20201 until it merges).
2. Add the lineage metadata schema and protobuf compatibility.
3. Add metadata list, merge, replace, and lifecycle APIs.
4. Add the GET_LINEAGE table function.
5. Persist structural and DML lineage.
6. Add end-to-end SQL workflow tests.

This PR persists lineage for:

• CREATE VIEW
• CREATE TABLE ... AS SELECT ... (CTAS)
• INSERT ... SELECT
• REPLACE INTO ... SELECT
• multi-table insert
• UPDATE ... FROM
• MERGE

COPY INTO persistence hooks are prepared, but extraction of Plan::CopyIntoTable sources is not included yet.

Motivation

Databend currently has no persistent metadata representation for tracing where tables and columns originate or where their data flows. This PR adds persistent bidirectional lineage metadata, column-level mappings, and a Snowflake-style GET_LINEAGE table function.

Lineage model

A lineage edge always follows the canonical direction:

upstream object/column -> downstream object/column

Objects with stable table IDs use ID-addressed lineage. Name-addressed lineage is used for name-bound dependencies, including view references and tables without stable lineage IDs.

Views form a lineage boundary: DML reading from a view records the view rather than expanding directly to its base tables. Streams behave differently: stream data columns are attributed to the base table, while stream metadata columns are ignored.

Self-referential writes do not create lineage loops. For example, INSERT INTO t SELECT * FROM t does not create a t -> t edge.

Metadata and persistence

Each edge is stored in both directions under a tenant-scoped structured key. Structured key encoding provides escaping for object names and integrates with metactl, snapshots, backups, and tenant export/filter tooling.

Structural lineage is committed atomically with object metadata for CREATE VIEW and CTAS. DML lineage is persisted after a successful table commit using a best-effort callback; lineage write failures do not change a successful DML result.

DML merge operations preserve the existing lineage kind, update the last query metadata, and merge column edges without duplicates. Lineage metadata does not currently use TTL.

GET_LINEAGE

GET_LINEAGE is implemented as a table function:

GET_LINEAGE(
    '<object_name>',
    '<object_domain>',
    '<direction>'
    [, distance]
)

Supported domains are TABLE, VIEW, COLUMN, and STAGE. Supported directions are UPSTREAM and DOWNSTREAM; distance defaults to 5 and must be between 1 and 5.

The implementation uses a distance-ordered BFS. Metadata listing and object resolution are concurrent within each level, while levels remain sequential to preserve distance semantics. Query-time resolution validates the current object and column state.

GET_LINEAGE is gated by the Enterprise Feature::Lineage license feature.

Capture configuration

Capture is controlled independently from the Enterprise license:

[query.lineage]
capture_enabled = true

The default is false. This allows deployments that need lineage to opt into its planning and metadata costs. Capture remains stable across license changes, while querying through GET_LINEAGE requires the Enterprise feature.

Object lifecycle

• Rename and undrop do not rewrite lineage metadata.
• ID-addressed lineage survives rename and resolves to the current name.
• Name-addressed lineage becomes temporarily unavailable after rename and recovers if the original name is restored.
• Dropped views and transient tables remove their lineage.
• Dropped Fuse tables are filtered during queries and cleaned during GC/vacuum.
• Temporary, MEMORY, DELTA, and unsupported engines are skipped.
• Iceberg and Hive tables use name-addressed lineage.
• Stream lineage passes through to the source table.

Current limitations

• COPY INTO source lineage extraction is not implemented yet.
• Materialized view persistence is not connected yet.
• Existing views created before capture is enabled are not backfilled.
• Stream is not exposed as a GET_LINEAGE domain because lineage passes through to its base table.
• Explicit AS MATERIALIZED CTE lineage is not included yet.
• Historical cleanup and administrative retention policies are not included.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Coverage includes metadata key encoding, protobuf compatibility, lineage API behavior, BFS traversal, column propagation, CTAS, views, DML workflows, multi-table insert, rename/drop/undrop behavior, vacuum cleanup, cross-database ID resolution, self-writes, filter-only columns, aggregate expressions, CTEs, subqueries, and stream passthrough.

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

[drmingdrmer]

@drmingdrmer reviewed 20 files and made 1 comment.
Reviewable status: 20 of 76 files reviewed, 1 unresolved discussion (waiting on youngsofun).

---

src/meta/app/src/schema/lineage.rs line 24 at r1 (raw file):

pub enum LineageDirection {
    Upstream,
    Downstream,

it's very unclear what direction is UPSTREAM and what is DOWNSTREAM. there is no common knowledge on it and there is no comment explaining it. choose better terms

Code quote:

pub enum LineageDirection {
    Upstream,
    Downstream,

[drmingdrmer]

it's too big a PR and I do not think I can review it thoroughly.

I'm gonna just approve it. And please let the LLM review it. And if LLM say yes, just merge it.

[youngsofun]

@drmingdrmer reviewed 20 files and made 1 comment.
Reviewable status: 20 of 76 files reviewed, 1 unresolved discussion (waiting on youngsofun).

src/meta/app/src/schema/lineage.rs line 24 at r1 (raw file):

pub enum LineageDirection {
    Upstream,
    Downstream,

it's very unclear what direction is UPSTREAM and what is DOWNSTREAM. there is no common knowledge on it and there is no comment explaining it. choose better terms

Code quote:

pub enum LineageDirection {
    Upstream,
    Downstream,

@drmingdrmer

Good point. I propose separating the user-facing traversal direction from the physical Meta index layout.

QueryDirection::{Upstream, Downstream} describes traversal relative to the requested object. These terms remain in GET_LINEAGE, consistent with Snowflake and OpenMetadata:

• Upstream: traverse against data flow, from targets to their sources.
• Downstream: traverse along data flow, from sources to their targets.

For the Meta keys, I propose LineageIndex::{BySource, ByTarget}:

enum LineageIndex {
    /// Stores the edge as `object = source` and `related_object = target`.
    /// Prefix scans enumerate downstream targets of a source.
    BySource,

    /// Stores the edge as `object = target` and `related_object = source`.
    /// Prefix scans enumerate upstream sources of a target.
    ByTarget,
}

Individual lineage edges and result columns will use source and target.

This makes each context explicit:

• Upstream / Downstream: traversal relative to the requested object.
• source / target: endpoints of a lineage edge.
• BySource / ByTarget: how the bidirectional Meta index is keyed.