Fixes #27419: Trino cross-database lineage for case-insensitive table names by hassaansaleem28 · Pull Request #27495 · open-metadata/OpenMetadata

hassaansaleem28 · 2026-04-17T18:00:06Z

Describe your changes:

Trino lowercases identifiers in query history, but OpenMetadata was matching cross-database table names too strictly. That caused the upstream Postgres CUSTOMER table to be dropped from the lineage graph.

What I worked on

I worked on Trino lineage cross-database matching because Trino lowercases identifiers while OpenMetadata was comparing table names too strictly. I added a regression test and updated the matching logic so the Postgres CUSTOMER table now links into the Trino lineage graph instead of being dropped.

Before

The lineage graph stopped at customer.
The upstream Postgres CUSTOMER node was missing.

After

Cross-database lineage now matches table names case-insensitively.
The lookup is scoped to the correct schema, so we avoid broad database-wide scans.
The full path now appears in the UI: CUSTOMER -> customer -> customer_copy.

What changed

Added a regression test for the exact Trino/Postgres case-mismatch scenario.
Updated Trino lineage matching to resolve schema and table names case-insensitively.
Kept the fallback schema-scoped and cached to avoid extra metadata queries.

Validation

Targeted Trino lineage unit test passed.
Live ingestion verification confirmed the lineage edge appears in the UI.

Type of change:

Bug fix

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes #27419: Trino cross-database lineage for case-insensitive table names
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.

Summary by Gitar

Refactored discovery logic:
- Replaced ElasticSearch-based search queries with direct metadata.list_all_entities calls for database and table discovery to improve consistency.
- Streamlined lookup logic to optimize metadata retrieval and minimize overhead.

_{This will update automatically on new commits.}

Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

github-actions · 2026-04-17T18:00:32Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

github-actions · 2026-04-17T18:00:48Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

… schema Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

github-actions · 2026-04-18T01:16:03Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

github-actions · 2026-04-18T01:17:18Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

This PR fixes Trino cross-database lineage resolution when the upstream/source database uses case-sensitive (e.g., uppercase) identifiers, while Trino normalizes identifiers to lowercase—preventing valid lineage edges from being created.

Changes:

Updates Trino cross-database table matching to be case-insensitive for table and column names, with a schema-scoped fallback lookup and caching.
Adds a regression unit test covering case-insensitive matching and ensuring schema-level lookup is used for cross-database resolution.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`ingestion/src/metadata/ingestion/source/database/trino/lineage.py`	Implements case-insensitive matching and schema-scoped cached fallback lookup for cross-database lineage.
`ingestion/tests/unit/source/database/trino/test_lineage.py`	Adds regression tests validating case-insensitive matching and schema-scoped cross-db lookup behavior.

Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

github-actions · 2026-04-18T01:33:29Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-04-18T01:33:52Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-04-18T01:38:53Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

github-actions · 2026-04-18T02:30:20Z

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

gitar-bot · 2026-04-23T13:26:45Z

+    def list_all_entities_side_effect(entity, params=None, **_kwargs):
+        if entity is Database and params == {"service": "repro_trino"}:
+            return [trino_database]
+        if entity is Database and params == {"service": "repro_postgres"}:
+            return [source_database]
+        if entity is Table and params == {"database": "repro_trino.postgres"}:
+            return [trino_table]
+        return []


💡 Bug: Test doesn't verify trino_table has columns/databaseSchema populated

The test test_yield_cross_database_lineage_finds_uppercase_source_table mocks list_all_entities to return trino_table which is a MagicMock with explicitly set columns and databaseSchema. This masks the production bug where list_all_entities is called without fields and these attributes would be empty. Consider adding an assertion that list_all_entities was called with the expected fields parameter.

_{Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion}

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

gitar-bot · 2026-04-23T13:44:03Z

Code Review 👍 Approved with suggestions 4 resolved / 5 findings

Improves Trino cross-database lineage by resolving case-insensitive matching, redundant cache miss lookups, and schema FQN sensitivity. Ensure the test suite explicitly verifies the population of columns and databaseSchema for trino_table entities.

💡 Bug: Test doesn't verify trino_table has columns/databaseSchema populated

📄 ingestion/tests/unit/source/database/trino/test_lineage.py:90-97

The test test_yield_cross_database_lineage_finds_uppercase_source_table mocks list_all_entities to return trino_table which is a MagicMock with explicitly set columns and databaseSchema. This masks the production bug where list_all_entities is called without fields and these attributes would be empty. Consider adding an assertion that list_all_entities was called with the expected fields parameter.

✅ 4 resolved

✅ Bug: Case-insensitive fallback ignores schema, may match wrong table

📄 ingestion/src/metadata/ingestion/source/database/trino/lineage.py:125-133
_get_case_insensitive_cross_database_table iterates ALL tables in the cross-database and matches only on table name (+ columns). It does not verify the schema segment of the FQN matches. If a Postgres database has schema_a.CUSTOMER and schema_b.CUSTOMER, the fallback will return whichever it encounters first, potentially creating incorrect lineage.

The exact-match path (get_by_name) constructs the full FQN including schema, so it correctly scopes. The fallback should do the same.

✅ Performance: Fallback lists all tables per database on every cache miss

📄 ingestion/src/metadata/ingestion/source/database/trino/lineage.py:174-186 📄 ingestion/src/metadata/ingestion/source/database/trino/lineage.py:128-129
_get_case_insensitive_cross_database_table calls list_all_entities(entity=Table, params={"database": cross_database_fqn}) for each Trino table that doesn't get an exact FQN match. For databases with thousands of tables and many Trino tables without exact matches, this results in repeated full-database scans via API pagination.

The cache on line 174 prevents re-lookup for the same derived FQN, but different Trino tables produce different derived FQNs, so each unique miss triggers a new full scan of the same cross-database.

✅ Edge Case: Schema FQN lookup may still be case-sensitive

📄 ingestion/src/metadata/ingestion/source/database/trino/lineage.py:125-136
The new _get_cross_database_schema_fqn constructs the schema FQN using the Trino schema name (e.g. trino_table.databaseSchema.name.root), which Trino lowercases. If the cross-database service stores the schema with different casing (e.g. Source_Schema vs source_schema), the list_all_entities(params={"databaseSchema": ...}) call may return no results, causing the fallback to silently find nothing.

This is the same class of case-sensitivity issue that this PR fixes for table names, just one level up. It's an incremental improvement opportunity, not a regression.

✅ Performance: No negative caching for schema ES lookup causes repeated queries

📄 ingestion/src/metadata/ingestion/source/database/trino/lineage.py:151-165
In _get_cross_database_schema_fqn, when search_database_schema_from_es returns None or returns schemas that don't match the lowered trino_schema_name, the mapping for that schema name is never populated. On subsequent calls with the same schema name, the cache check at line 154-156 misses again, triggering another ES query. In contrast, _get_case_insensitive_cross_database_table correctly initializes an empty list at line 202-204 to prevent repeated lookups for the same table.

This could cause redundant ES queries if multiple Trino tables reference a cross-database schema that doesn't exist in the source system.

🤖 Prompt for agents

Code Review: Improves Trino cross-database lineage by resolving case-insensitive matching, redundant cache miss lookups, and schema FQN sensitivity. Ensure the test suite explicitly verifies the population of columns and databaseSchema for trino_table entities.

1. 💡 Bug: Test doesn't verify trino_table has columns/databaseSchema populated
   Files: ingestion/tests/unit/source/database/trino/test_lineage.py:90-97

   The test `test_yield_cross_database_lineage_finds_uppercase_source_table` mocks `list_all_entities` to return `trino_table` which is a MagicMock with explicitly set columns and databaseSchema. This masks the production bug where `list_all_entities` is called without `fields` and these attributes would be empty. Consider adding an assertion that `list_all_entities` was called with the expected `fields` parameter.

Options

Display: compact → Showing less information.

Comment with these commands to change:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

sonarqubecloud · 2026-04-23T14:46:37Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
76.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

hassaansaleem28 added 2 commits April 17, 2026 22:53

fix(ingestion): make Trino cross db lineage case insensitive

162bcb1

Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

Merge branch 'main' into fix/27419-lineage-ingestion

3a4b7cc

gitar-bot Bot reviewed Apr 17, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/database/trino/lineage.py

gitar-bot Bot reviewed Apr 17, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/database/trino/lineage.py Outdated

�[200~fix(trino-lineage): scope case-insensitive cross-db matching to…

674b6d4

… schema Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

Merge branch 'main' into fix/27419-lineage-ingestion

3edcaf7

hassaansaleem28 marked this pull request as ready for review April 18, 2026 01:17

hassaansaleem28 requested a review from a team as a code owner April 18, 2026 01:17

Copilot AI review requested due to automatic review settings April 18, 2026 01:17

Copilot started reviewing on behalf of hassaansaleem28 April 18, 2026 01:17 View session

gitar-bot Bot reviewed Apr 18, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/database/trino/lineage.py Outdated

Copilot AI reviewed Apr 18, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/database/trino/lineage.py Outdated

fix(trino-lineage): resolve cross-db schema FQN case-insensitively

8124088

Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

Update ingestion/src/metadata/ingestion/source/database/trino/lineage.py

1d5e140

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 18, 2026 01:33

Copilot started reviewing on behalf of hassaansaleem28 April 18, 2026 01:34 View session

Copilot AI reviewed Apr 18, 2026

View reviewed changes

Comment thread ingestion/tests/unit/source/database/trino/test_lineage.py Outdated

Comment thread ingestion/src/metadata/ingestion/source/database/trino/lineage.py Outdated

Update ingestion/tests/unit/source/database/trino/test_lineage.py

ef9ea9c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 18, 2026 01:38

Copilot started reviewing on behalf of hassaansaleem28 April 18, 2026 01:38 View session

Copilot AI reviewed Apr 18, 2026

View reviewed changes

refactor & formatiing issue resolved

edd702f

Signed-off-by: hassaansaleem28 <iamhassaans@gmail.com>

Merge branch 'main' into fix/27419-lineage-ingestion

55f984c

Copilot AI review requested due to automatic review settings April 23, 2026 13:23

Copilot started reviewing on behalf of hassaansaleem28 April 23, 2026 13:24 View session

gitar-bot Bot reviewed Apr 23, 2026

View reviewed changes

Copilot AI reviewed Apr 23, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/database/trino/lineage.py Outdated

Update ingestion/src/metadata/ingestion/source/database/trino/lineage.py

6dcf668

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 23, 2026 13:31

Merge branch 'main' into fix/27419-lineage-ingestion

db9d8c2

Copilot started reviewing on behalf of hassaansaleem28 April 23, 2026 13:31 View session

Copilot AI reviewed Apr 23, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/database/trino/lineage.py Outdated

hassaansaleem28 had a problem deploying to test April 23, 2026 13:42 — with GitHub Actions Error

Update ingestion/src/metadata/ingestion/source/database/trino/lineage.py

d737420

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 23, 2026 13:42

Copilot started reviewing on behalf of hassaansaleem28 April 23, 2026 13:42 View session

hassaansaleem28 temporarily deployed to test April 23, 2026 13:53 — with GitHub Actions Inactive

Copilot AI reviewed Apr 23, 2026

View reviewed changes

pmbrull approved these changes Apr 24, 2026

View reviewed changes

pmbrull merged commit 9951082 into open-metadata:main Apr 24, 2026
47 of 48 checks passed

Conversation

hassaansaleem28 commented Apr 17, 2026 • edited by gitar-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes:

What I worked on

Before

After

What changed

Validation

Type of change:

Checklist:

Summary by Gitar

Uh oh!

github-actions Bot commented Apr 17, 2026

Uh oh!

github-actions Bot commented Apr 17, 2026

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

gitar-bot Bot Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

gitar-bot Bot commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Apr 23, 2026

Quality Gate passed for 'open-metadata-ingestion'

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

hassaansaleem28 commented Apr 17, 2026 •

edited by gitar-bot Bot

Loading

gitar-bot Bot commented Apr 23, 2026 •

edited

Loading