feat(query): add history-based data lineage

[youngsofun]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Add history-based object and column lineage for successful Databend queries, with bounded graph traversal through GET_LINEAGE.

The feature has three parts:

1. Extract canonical data-flow edges from resolved query plans.
2. Emit those edges as structured logs and asynchronously aggregate them into system_history.
3. Resolve active objects and expose upstream/downstream traversal through GET_LINEAGE.

This keeps lineage outside table-meta transactions and query commit retries. A lineage logging or aggregation failure does not change the result of the DDL or DML that produced the data.

This PR is stacked on #20201, which contains the first two commits for system.columns.column_id and planner lineage extraction. After #20201 merges, this branch will be rebased and those shared commits will disappear from this diff.

Concepts and terminology

Lineage describes how data moves or is derived across objects and columns.

Every persisted edge has one fixed data-flow orientation:

source -> target

• Source: an object or column whose value contributes data.
• Target: an object or column that receives or defines that data.
• Upstream query: start from a target and walk backward toward its sources.
• Downstream query: start from a source and walk forward toward targets that consume it.

source and target therefore describe the edge itself. UPSTREAM and DOWNSTREAM describe how a user traverses the graph; they do not reverse the stored edge meaning.

The implementation records two related domains:

• Object lineage, such as table_a -> table_b or stage -> table.
• Column lineage, such as table_a.a -> table_b.x. Object lineage remains available even when a source cannot provide a stable column mapping.

Query contract

The user-facing goal is a Snowflake-compatible traversal shape:

GET_LINEAGE(
    '<object_name>',
    '<object_domain>',
    '<direction>'
    [, <distance> ]
)

• object_domain: TABLE, VIEW, STAGE, or COLUMN.
• direction: UPSTREAM or DOWNSTREAM.
• distance: an integer from 1 to 5; the default is 5.
• Column input uses a fully qualified object and column name, for example catalog.database.table.column.

The result contains:

distance
source_object_domain, source_object_name, source_column_name
target_object_domain, target_object_name, target_column_name
target_status
process

target_status is currently ACTIVE. process is JSON containing query_id, query_kind, lineage_kind, and event_time.

GET_LINEAGE is the stable SQL interface for interactive traversal. A Cloud UI or graph service that needs to load many edges at once may query the resolved system_history.lineage view directly. The directional lineage_by_source and lineage_by_target views are optimized traversal inputs, while lineage_unresolved is the ingestion table containing captured identities.

Architecture

SQL statement
  -> Binder / optimized Plan
  -> Plan::query_lineage()
  -> QueryContext
  -> successful DDL or query-finish hook
  -> databend::log::lineage structured log
  -> system_history.log_history
  -> replayable batch MERGE
  -> system_history.lineage_unresolved
  -> active-object resolution views
  -> GET_LINEAGE bounded CTE traversal / direct history query

1. Planner extraction

The planner builds symbol definitions from the optimized SExpr, then starts from each target value expression and recursively resolves referenced symbols to base relation columns.

The public model is storage-independent:

QueryLineage {
    kind,
    targets: Vec<LineageTarget>,
}

LineageTarget {
    relation: QueryLineageRelation,
    sources: Vec<LineageSource>,
}

LineageSource {
    relation: QueryLineageRelation,
    columns: Vec<QueryLineageColumnEdge>,
}

QueryLineageColumnEdge {
    source: QueryLineageColumn,
    target: QueryLineageColumn,
}

QueryLineageRelation carries catalog, database, name, optional stable ID, catalog type, and relation kind (TABLE, VIEW, or STAGE). QueryLineageColumn carries both name and column ID.

Only expressions that produce target values are traversal roots. Filter-only, join-only, WHEN-only, and ordering-only columns are excluded unless they also contribute to a target value. For example, count(col) records col, while count(*) has no source column edge.

2. Successful-query logging

When lineage history is enabled, the interpreter attaches extracted lineage to QueryContext. DML logs it only after the pipeline reports success. DDL paths log only after the object operation succeeds. Logging is best effort and never changes the SQL result.

Each LineageLogEntry records:

• query ID, event time, query kind, and lineage kind;
• source and target endpoint identity;
• a stable hash of the normalized column mapping;
• source_to_target_columns and target_to_source_columns maps.

Both column-map directions are stored because column traversal must efficiently select only columns reachable from the current frontier.

Resolved source and target endpoints are compared before logging, so an object writing to itself does not persist a self-loop. Other real sources in the same statement are preserved.

3. History aggregation

GlobalHistoryLog loads structured logs through the existing log_history batch pipeline. The lineage transform is replay-safe and merges by:

source_lineage_key
+ target_lineage_key
+ lineage_kind
+ column_lineage_hash

Within one batch, duplicate identities select the newest event before MERGE. A matching row updates its latest query/process time; a different column mapping has a different hash and remains as another physical pattern. CTAS, CREATE VIEW, and DML remain distinct because lineage_kind participates in the identity.

The raw table stores forward and reverse column maps rather than one row per column pair. This avoids repeating all object/process fields for wide relations while still allowing map_pick-style traversal.

Lineage has no retention by default. Users can configure retention for lineage_unresolved; its delete template only expires DML records. Structural CREATE VIEW and CTAS records remain permanent. Object-deletion tombstones asynchronously remove ID-addressed edges that have become permanently invalid.

4. Active-object resolution

system_history.lineage resolves captured endpoints against current metadata:

• ID-addressed physical objects must still have that object ID.
• Name-addressed View dependencies must still resolve to that name.
• Dropped default-catalog objects are hidden from active results.
• Undrop naturally makes an ID-addressed edge visible again.
• A physical column rename keeps its ID and resolves to the new name.
• Dropping and recreating a same-named column allocates a new ID, so the old column edge stays invalid.

Two directional views retain the fixed source/target columns but choose different frontier keys:

• lineage_by_target supports UPSTREAM traversal.
• lineage_by_source supports DOWNSTREAM traversal.

5. Bounded traversal

The binder expands GET_LINEAGE into one SQL query with CTE generations from level 1 through the requested distance. Each generation joins the previous frontier to the appropriate directional view, resolves the next object or column, and removes duplicate semantic edges before the next level.

The final result prefers the shortest distance. If the same semantic result is represented by multiple physical history patterns, it uses the newest process information deterministically.

Supported lineage

Statements

• CREATE VIEW
• CTAS and CREATE OR REPLACE TABLE AS SELECT
• INSERT ... SELECT
• multi-table INSERT
• INSERT from a named Stage
• REPLACE INTO ... SELECT
• UPDATE ... FROM
• MERGE
• COPY INTO from/to named Stage endpoints

The planner also follows aliases, casts, scalar expressions, aggregate/window function arguments, scalar subqueries, UNION ALL, CTEs, automatically materialized CTEs, UDFs, async functions, and set-returning expressions.

Object and engine handling

Object/catalog	Addressing and behavior
Default catalog physical table	Stable table and column IDs; rename and undrop preserve lineage
Fuse, including attached/external-location Fuse	Treated as a default-catalog physical table
View	The View output is a boundary; referenced objects are name-addressed so rename breaks the edge and renaming back restores it
Stream	Transparent wrapper: data columns resolve to the backing table; Stream metadata columns are excluded
Named Stage	Name-addressed object edge only; file fields do not create column lineage
Hive catalog	Name-addressed; captured identity is returned without live external-catalog validation in v1
Iceberg catalog	Name-addressed; captured identity is returned without live validation and is currently a terminal endpoint
Paimon catalog	Name-addressed; captured identity is returned without live external-catalog validation in v1
Temporary table	Skipped because its identity is query/session scoped
MEMORY	Skipped because it has no durable identity across server lifetime
DELTA	Skipped in v1 because its identity/resolution differs from Iceberg and default-catalog tables
Unsupported or unexpressible source	Skipped without failing the user statement

Lifecycle semantics

• Renaming a physical data-movement table or column preserves its ID-addressed lineage.
• Replacing a table uses the new table ID; old edges no longer resolve to the replacement.
• Dropping a Fuse table hides its edge; undrop restores it until permanent cleanup.
• Vacuum or explicit deletion tombstones remove permanently invalid ID-addressed edges.
• A View dependency is intentionally name-addressed. Renaming the referenced table/column breaks that dependency; renaming it back restores it.
• Truncate and INSERT OVERWRITE do not erase lineage because lineage describes data-flow operations, not current row contents.

Known limitations

• History ingestion is asynchronous, so new lineage appears after the configured history transform interval rather than in the producing transaction.
• Lineage logging is best effort. A successful user query can exist without a lineage event if logging or history ingestion fails.
• Existing Views and past DML are not backfilled when lineage is enabled.
• Direct GET_LINEAGE initialization from an external-catalog object is not supported in v1 because the start-object lookup uses Databend system metadata. External objects can still appear as returned endpoints.
• External-catalog and Stage endpoints currently skip live validity checks.
• Stage file fields have no stable schema and therefore produce only object-level lineage.
• Explicit AS MATERIALIZED CTE lineage is not supported because the query-scoped temporary table loses its original producer-column mapping. Automatically materialized CTEs are supported.
• Traversal depth is bounded at 5 and target_status currently only reports ACTIVE.
• The lineage log stores query_id, query kind, lineage kind, and event time, but not query text, stored-procedure context, task name, or full process metadata. A UI can currently join query_id to query history when those details are needed. If lineage must become independent of query-history retention, those fields should be added to the lineage log/history schema.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Unit and static SQL tests

• Planner symbol resolution and canonical edge construction.
• CTAS, CREATE VIEW, INSERT SELECT, multi-table INSERT, REPLACE, UPDATE, MERGE, Stage scans, subqueries, joins, aggregates, View boundaries, Stream transparency, automatic materialized CTEs, and self-insert candidates.
• Default/Hive/Iceberg/Paimon addressing and unsupported engine skipping.
• Stable forward/reverse column maps, mapping hash, deduplication, and writer-side self-loop filtering.
• History MERGE identity, batch deduplication, delete tombstones, and optional retention.
• Parsing generated history views and every generated GET_LINEAGE SQL shape for both directions and all distances.
• system.columns.column_id schema coverage.

End-to-end history tests

tests/logging/test-history-tables.sh starts a two-node history-enabled query deployment, prepares each scenario through sqllogictest, polls the history views until the expected batches are visible, then runs the matching check suite. It does not rely on one fixed sleep for lineage readiness.

The suites cover:

• object and column traversal in both directions;
• multi-hop paths and multiple column mapping hashes;
• View boundaries and rename-away/rename-back behavior;
• Stream pass-through;
• CTAS, INSERT, multi-table INSERT, UPDATE, MERGE, REPLACE, COPY, and Stage input;
• self-insert suppression while preserving other real source edges;
• drop, undrop, replace, rename, column drop/add, and vacuum lifecycle behavior;
• Iceberg REST catalog endpoints through dedicated Docker services.

Local validation:

• cargo fmt --all -- --check
• git diff --check
• bash -n tests/logging/test-history-tables.sh
• cargo test -p databend-common-tracing predefined_tables::history_tables::tests --lib
• cargo check -p databend-common-config
• cargo check -p databend-query --lib

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

AI assistance

• AI usage: An AI coding agent assisted with implementation, test coverage, commit organization, local validation, and drafting this PR description; the responsible human directed the design and reviewed the final behavior and diff.
• Responsible human: @youngsofun
• The responsible human has read every line of this diff and can explain each change

---

This change is 

[github-actions]

🤖 CI Job Analysis

Workflow: 30532534590

📊 Summary

• Total Jobs: 90
• Failed Jobs: 4
• Retryable: 0
• Code Issues: 4

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

• ❌ linux / test_unit: Not retryable (Code/Test)
• ❌ linux / test_logs: Not retryable (Code/Test)
• ❌ linux / sqllogic / standalone (standalone, 2c, http): Not retryable (Code/Test)
• ❌ linux / sqllogic / standalone (standalone, 2c, hybrid): Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).