DRC-3809: resolve PIVOT output columns to real source columns (CLL)

[danyelf]

Note: this is Claude being very excited about the PIVOT operation in particular. In general, it's important for both our system and our demos to show that CLL properly transitions in things like a parent model with a column change may become a child with a whole-model change, or vice versa.

Background (for a reader with no context)

Recce shows column-level lineage (CLL) — for each column of a dbt model, which upstream columns feed it. It computes this by parsing each model's compiled SQL with sqlglot and walking the query's scopes to connect every output column back to its real source column(s).

This PR fixes CLL for SQL PIVOT queries (common in Snowflake).

The problem

When a query uses PIVOT, sqlglot rewrites every projected column to reference a synthetic pivot table alias (e.g. P, or an auto-generated _q_0 / _0 depending on sqlglot version). But sqlglot never registers that alias as a real source in the query's scope — the pivot hangs off the base table node instead. Recce's resolver (recce/util/cll.py) had no handling for this, so it fell through to a "phantom" fallback that emitted a dependency on a table (P / _q_0) that doesn't exist in the DAG.

Downstream, the dbt adapter couldn't map that phantom table to any model, so it dropped the edges entirely. The result: a pivoted model lost all of its lineage — not just the pivoted columns' column-level parents, but the model-level provenance too.

Before

For the query select id, jan, feb from t1 pivot (sum(rev) for month in ('jan','feb')) as p (schema t1(id, month, rev)), CLL produced:

model-level lineage (m2c):  []                    ← all source provenance to t1 LOST
column 'ID':   passthrough ← [(P, ID)]            ← phantom: P is not a real node
column 'JAN':  passthrough ← [(P, JAN)]           ← phantom
column 'FEB':  passthrough ← [(P, FEB)]           ← phantom

In the running app the pivoted model showed no real upstream lineage — the columns pointed at a pivot alias that isn't in the graph, so the edges vanished.

After

model-level lineage (m2c):  [(T1, REV), (T1, MONTH)]     ← provenance to t1 restored
column 'JAN':  derived    ← [(T1, REV), (T1, MONTH)]     ← real pre-pivot sources
column 'FEB':  derived    ← [(T1, REV), (T1, MONTH)]
column 'ID':   passthrough ← [(T1, ID)]                  ← real pass-through source

The generated pivot value columns (JAN, FEB) resolve to the columns the pivot actually consumes — the measure (t1.rev) and the dimension (t1.month) — and unchanged pass-through columns (id) resolve straight to their base column. No phantom node anywhere.

Seen in the running app

Demonstrated on a real jaffle-shop dbt-duckdb project with a payments_by_method model = stg_payments pivoted (sum(amount) for payment_method in (credit_card, coupon, bank_transfer, gift_card)). (There is no "before" screenshot because the pre-fix state produced no resolvable lineage at all — the phantom node dropped the edges; the "before" is the empty/broken lineage described above.)

A pivot value column (credit_card) correctly resolves as derived from stg_payments.payment_method + stg_payments.amount:

A pass-through column (order_id) correctly resolves as passthrough from stg_payments.order_id:

And the model-level lineage graph is intact — raw_payments → stg_payments → payments_by_method, with no phantom pivot-alias node:

How

A small, surgical resolver in _cll_select_scope / source_column_dependency (recce/util/cll.py). Before falling through to the phantom path, it builds a per-scope pivot map from the base table's pivot node and resolves:

• value output columns → derived on the measure + dimension source columns,
• everything else → passthrough on the base table,
• and extends model-level lineage (m2c) with the consumed source columns.

Resolution defers to the existing base-source logic, so it works whether the pivoted relation is a raw table or a CTE/subquery. Crucially it keys off the pivot node's structure, never the alias string — so it's immune to the _0 vs _q_0 synthetic-alias drift between sqlglot versions (verified: ideation ran sqlglot 27.29.0 → _q_0; implementation ran 30.11.0 → _0; both pass).

Verification

• New test tests/util/test_cll.py::ColumnLevelLineageTest::test_pivot — explicit-alias and select * (no alias) cases, asserting the exact edges above with no phantom node. Genuinely test-first: reverting the resolver reddens it (m2c empty, JAN→P phantom).
• Full CLL suite 49/49; full backend suite 1548 passed (once the frontend is built). The parked DRC-3806 uncorrelated-subquery case is undisturbed.
• Running-app demo on a real jaffle-shop dbt-duckdb project: a payments_by_method PIVOT model renders in the live Recce app with its value columns resolving to stg_payments.payment_method + stg_payments.amount and order_id passing through — no phantom node. (Screenshots in the workflow evidence folder docs/spacedock/recce-lineage-bugs/_evidence/drc-3809/.)

Known boundary (not a regression — possible follow-up)

The fix covers a PIVOT hanging directly off a base table. A PIVOT wrapped in a subquery/CTE (... from (select ...) pivot ...) still falls to the old phantom path. Out of scope for this AC; worth a follow-up if subquery-pivots matter.

Part of the DRC-2891 complex-SQL-pattern audit. Sibling UNNEST/LATERAL gap tracked separately (DRC-3807/3808). Opening as draft — validated in the workflow; pending final review.

🤖 Generated with Claude Code