Lineage writes trigger a full lineage-graph GET per edge, dominating ingestion runtime on high fan-in catalogs

[harshsoni2024]

Affected module

Ingestion Framework

Describe the bug

Affected module

Ingestion (ometa client / metadata REST sink)

Describe the bug

Every lineage edge written through the sink ends with an unconditional server GET that fetches the entire 1-up/1-down lineage graph of the source entity:

• add_lineage → get_lineage_by_id(...) (ingestion/src/metadata/ingestion/ometa/mixins/lineage_mixin.py)
• add_lineage_by_name → get_lineage_by_name(...) (same file)

The only consumers of that expensive response are the sink writers (write_lineage, write_fqn_lineage in ingestion/src/metadata/ingestion/sink/metadata_rest.py), and they read a single field — entity.fullyQualifiedName — which the caller already holds in the request it just wrote. The full graph is fetched per edge and thrown away.

On catalogs that fan in on a few heavily-referenced tables, that graph is large and grows as ingestion proceeds, so the GET gets slower with every edge written.

Observed impact (production PowerBI ingestion, ~4.9k workspaces)

Processing one set of 100 workspaces took 15h 19m. Log analysis of the run:

Segment	Time	Share
Sink HTTP GETs (Sink.GET, 28,373 calls)	~10.7 h	~70%
— of which post-write graph GETs, fat tail 3–10s each (~5,600 calls)	~9 h	—
All SQL/lineage parsing combined	~28 min	~3%

• 17,579 lineage edges written; each triggered a graph GET.
• The catalog fans in on a handful of shared BigQuery views (each referenced by hundreds of PowerBI datamodels), so get_lineage_by_id on those tables walks an ever-growing graph — hourly average GET latency drifted from ~0.8s to ~2.3s during the run.
• No retries/timeouts/429s in the logs — genuine server-side graph-traversal latency, amplified per edge

To Reproduce

1. Ingest a dashboard service (e.g. PowerBI) whose datamodels reference a small set of shared upstream tables (high fan-in).
2. Enable debug logs and observe per-edge timing: every Creating lineage between db table=X and datamodel=Y is followed by a multi-second Sink.GET.
3. Note the Sink.GET latency increasing as more edges are written to the same source tables.

Expected behavior

• Writing a lineage edge should cost the edge write (PUT/PATCH, plus the patch-check lookup when merging column lineage) — not an additional full lineage-graph traversal of the source entity per edge. Callers that don't consume the graph (the sink) should be able to skip it, while the public add_lineage contract (returning the origin node's lineage) remains intact for SDK consumers and add_lineage_by_query.

OS

No response

Python version

No response

OpenMetadata version

No response

OpenMetadata Ingestion package version

No response

Additional context

No response

Pre-submission checklist

• I searched for duplicate issues.
• I removed credentials, hostnames, emails, and other sensitive data from logs and config.