Processing Cross Database Lineage not supported for Mssql

[igoresz]

Hello
Is "Processing Cross Database Lineage for Mssql" in roadmap of OpenMetaData team?
Many thanks in advance.

[sqlparser]

This one sat unanswered for a while — sorry to bump it late, but the cross-database MSSQL case has a couple of sharp edges worth writing up for anyone else landing here from search.

Two things have to work for cross-database lineage to show up:

1) The parser has to recognize [DatabaseA].[dbo].[Table] as a 3-part name.

OpenMetadata's parser path for MSSQL has historically had trouble with square-bracket identifiers that span databases — this is tracked as #16424, and the root cause was a greedy regex r"\[(.*)\]" that matched across bracket pairs, so [DatabaseA].[dbo].[Customers] was being collapsed into a single mangled identifier rather than a proper 3-part FQN. Part of that has been addressed in newer sqlfluff versions, but if the same query also sits inside a stored procedure (CREATE PROCEDURE ... BEGIN ... END), the outer T-SQL wrapper still defeats the parser chain (see #16737, #17586, #25299) and lineage silently drops to zero.

2) OpenMetadata has to have both databases registered as entities.

Even when the parser returns a clean 3-part name, the lineage push (PUT /api/v1/lineage) needs an entity UUID on both ends. For cross-database lineage that means both DatabaseA and DatabaseB must have been ingested through the same MSSQL service connector (or a connector that covers both), so their tables exist in OpenMetadata with resolvable FQNs like mssql_prod.DatabaseA.dbo.Customers. If DatabaseB is only ever referenced in a query but never ingested as metadata, there's nothing to link the edge to.

---

A pragmatic path that works today:

We've been maintaining a small open companion tool — gsp-openmetadata-sidecar — that handles both of the parser-layer issues above (square brackets across databases + T-SQL procedural wrappers) and pushes lineage through the public PUT /api/v1/lineage API. It doesn't patch OpenMetadata core; it just fills in the edges the core parser drops.

Concrete example on the exact shape of SQL this discussion is asking about:

CREATE VIEW [ReportDB].[dbo].[vw_CustomerOrders] AS
SELECT
    [SalesDB].[dbo].[Customers].[CustomerID],
    [SalesDB].[dbo].[Customers].[CustomerName],
    [SalesDB].[dbo].[Orders].[OrderID],
    [SalesDB].[dbo].[Orders].[OrderDate]
FROM [SalesDB].[dbo].[Customers]
INNER JOIN [SalesDB].[dbo].[Orders]
    ON [SalesDB].[dbo].[Customers].[CustomerID] = [SalesDB].[dbo].[Orders].[CustomerID]

Dry-run output:

SalesDB.dbo.Customers --> ReportDB.dbo.vw_CustomerOrders (CustomerID, CustomerName)
SalesDB.dbo.Orders   --> ReportDB.dbo.vw_CustomerOrders (OrderID, OrderDate)

Usage:

pip install gsp-openmetadata-sidecar

gsp-openmetadata-sidecar \
  --sql-file your_crossdb_query.sql \
  --db-vendor dbvmssql \
  --dry-run

Caveat it shares with any other tool: both SalesDB and ReportDB still need to be registered in OpenMetadata as database entities for the lineage edge to have UUIDs to bind to. That's an ingestion-config answer, not a parser answer — but once both are ingested, the sidecar will resolve the cross-database FQNs correctly.

More details if useful — the cross-database bracket case, temp-table chains, and MERGE are all walked through with runnable examples:

• Why MSSQL stored-procedure lineage is silently empty in OpenMetadata (and the fix) — narrative deep-dive
• OpenMetadata MSSQL sidecar — install + CLI quickstart
• gudusoftware/gsp-openmetadata-sidecar — source and example SQL files

Happy to look at a scrubbed sample of your actual cross-database query and report back what lineage it pulls out — often the specific shape of the SQL (stored-proc-wrapped? view? synonym? linked-server?) changes which layer is actually breaking.

(Disclosure: I work at Gudu Software, which builds the SQLFlow engine behind the sidecar. Sidecar is free to use; backend has an anonymous tier and a self-hosted Docker option for teams that need data to stay inside their network.)

[harshach]

@ulixius9 @mohittilala can you answer this

[mohittilala]

Closing the discussion since "Processing Cross Database Lineage not supported for Mssql" is not relevant for mssql connector.

Please provide more details on what does cross database lineage means for mssql connector if you feel this topic needs further discussion.