Fixes #28065: [openlinege] Add job ownership support in ingestion pipeline

[jsingh-yelp]

Describe your changes:

Fixes #28065

Change Summary:

• This PR adds OpenLineage job ownership support for pipeline ingestion.
• OpenLineage events can now populate Pipeline owners from the job-level ownership facet. The ingestion flow resolves OpenLineage owner names to existing OpenMetadata users or group teams, supports both replace and append ownership update modes.
• Updated user/team ownership APIs to efficiently filter existing Pipeline owners.

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

High-level design:

N/A — small change.

Tests:

• Ran git diff --check
• Ran Python compile checks for the touched OpenLineage ingestion files/tests
• Ran mvn -Dmaven.repo.local=/nail/tmp/om-m2 -pl openmetadata-service spotless:apply
• Manually tested OpenLineage ownership events through Kafka using jaskaran_openlineage_test
• Verified append mode behavior by sending multiple ownership events for the same job with different teams

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
• For UI changes: I attached a screen recording and/or screenshots above.
• I have added tests (unit / integration / Playwright as applicable) and listed them above.

---

Summary by Gitar

• Ingestion logic refinement:
  • Improved OwnershipResolver to handle None values for include_owners and ownership_update_mode configuration parameters.
  • Added a defensive check in _ensure_pipeline_owner_cache using _owner_cache_loaded flag instead of relying on nullability of dictionaries.
• Entity API updates:
  • Updated EntityRepository methods to pass ListFilter into listInternal, serializeJsons, and setFieldsInBulk to improve context-aware entity processing.
• Testing enhancements:
  • Added unit tests to verify None value handling for include_owners and default replace behavior for ownership_update_mode in test_openlineage_ownership.py.

_{This will update automatically on new commits.}

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

⚠️ TypeScript Types Need Update

The generated TypeScript types are out of sync with the JSON schema changes.

Since this is a pull request from a forked repository, the types cannot be automatically committed.
Please generate and commit the types manually:

cd openmetadata-ui/src/main/resources/ui
./json2ts-generate-all.sh -l true
git add src/generated/
git commit -m "Update generated TypeScript types"
git push

After pushing the changes, this check will pass automatically.

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[sonika-shah]

@jsingh-yelp pushed a Java-side fix to this branch (b61d956).

Java bug fixed: the new ownsEntityType/directOwnsOnly list path went through fetchAndSetFieldsExcept, which skipped the batched relationship fetch — so GET /v1/teams?fields=owns,domains&ownsEntityType=pipeline returned domains == null, and relationship fields fell back to N+1. Fix: run fetchAndSetRelationshipFieldsInBulk in that helper too + added a TeamResourceIT regression test. (spotless + compile pass; IT not run locally yet.)

Two Python issues worth a look:

1. append mode drops owners — existing owners are rebuilt only from Group teams + users, so non-Group team owners and bot owners get silently dropped during merge.
2. unbounded owner cache — it paginates and loads all users + teams into memory upfront → unbounded memory + slow first-event processing on large deployments.

cc @ulixius9 please check when you get time.

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ui'

Issues
1393 New issues
0 Accepted issues

Measures
24 Security Hotspots
0.0% Coverage on New Code
3.6% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
88.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[jsingh-yelp]

@sonika-shah, @ulixius9

append mode drops owners existing owners are rebuilt only from Group teams + users, so non-Group team owners and bot owners get silently dropped during merge.

Yes, append mode rebuilds existing owners from cached Group teams + users. Non-Group team owners are excluded, but that matches current OpenMetadata validation rules code ref: https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java#L7383

it paginates and loads all users + teams into memory upfront → unbounded memory + slow first-event processing on large deployments.

This is intentional for append mode we preload user/team ownership once so existing pipeline owners can be preserved while merging owners from OpenLineage events. This avoids owner lookups on each message, in my view, it should make the overall pipeline a lot faster; we can revisit a lazy/on-demand resolver if first-event cache loading becomes too expensive.

[jsingh-yelp]

cc: @mohittilala Can you please review this PR? as I see you made some recent openlineage changes

[jsingh-yelp]

cc: @ulixius9 , @mohittilala Can I please have review on this PR?

[gitar-bot]

Code Review ✅ Approved 10 resolved / 10 findings

Integrates OpenLineage job ownership into pipeline ingestion by enabling owner resolution and configurable update modes, addressing previous build and logic issues including missing API field propagation, incorrect cache initialization, and unhandled null configurations.

✅ 10 resolved

✅ Edge Case: Owner name with colon prefix (e.g. ":jdoe") silently resolves

📄 ingestion/src/metadata/ingestion/source/pipeline/openlineage/ownership_resolver.py:90-95 📄 ingestion/src/metadata/ingestion/source/pipeline/openlineage/ownership_resolver.py:101-114
In ownership_resolver.py line 90, raw_owner_name.partition(":") on an input like :jdoe yields owner_type="", separator=":", owner_name="jdoe". Since separator is truthy, it enters the qualified branch and does owner_type = "".lower() → "". This doesn't match "team" or "user", so it falls into the else (unqualified) branch at line 106, resolving as if it were unqualified. While the end result is functional, the intent is unclear—it may be better to explicitly handle or warn about a malformed qualified identifier like :name.

✅ Performance: Owner cache loads ALL teams/users, not just pipeline-owning ones

📄 ingestion/src/metadata/ingestion/source/pipeline/openlineage/ownership_resolver.py:162-176 📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/teams/UserResource.java:319-322
In _ensure_pipeline_owner_cache, the resolver calls list_all_entities(entity=Team, ...) with params={"ownsEntityType": "pipeline"} which should filter server-side. However for users it also passes "directOwnsOnly": "true". If the server-side filtering for ownsEntityType is not implemented or returns all entities regardless (since this is new functionality being added in the same PR), the cache will load every user/team in the system into memory. For large deployments this could be significant. Ensure the API actually filters—the UserResource.java change only passes the param into ListFilter but the DAO query needs to handle it.

✅ Quality: New test file uses unittest.TestCase instead of pytest

📄 ingestion/tests/unit/topology/pipeline/test_openlineage_ownership.py:1-2 📄 ingestion/tests/unit/topology/pipeline/test_openlineage_ownership.py:42
The project's testing guidelines require using pytest with plain assert statements and no unittest.TestCase inheritance. The new test_openlineage_ownership.py file uses class OpenLineageOwnershipResolverTest(unittest.TestCase) with self.assertIsNotNone, self.assertEqual, etc. This is inconsistent with project conventions and makes it harder to leverage pytest features (fixtures, parametrize, better assertion introspection).

✅ Bug: ownsEntityType filter added unconditionally even when null

📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/teams/UserResource.java:319 📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/teams/TeamResource.java:238
In UserResource.java line 319, filter.addQueryParam("ownsEntityType", ownsEntityType) is called unconditionally—even when ownsEntityType is null. This differs from how isBot is handled (guarded by if (isBot != null)). If downstream code treats a null-valued query param differently from an absent one, this could cause unexpected behavior (e.g., nullOrEmpty(filter.getQueryParam("ownsEntityType")) returns true for null, so it's likely fine, but it's inconsistent and could break if ListFilter.addQueryParam doesn't handle null gracefully).

✅ Bug: Missing base setFieldsInBulk(Fields, List, ListFilter) causes compile error

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:2153
The change in EntityRepository.java:2153 calls setFieldsInBulk(fields, entities, filter) with 3 arguments, but the base class only defines a 2-parameter overload (setFieldsInBulk(Fields, List<T>)). While UserRepository and TeamRepository define their own 3-parameter versions, these are not overrides of a base method — they are new overloads with a different signature. Since listAll is defined in EntityRepository<T> (not in the subclasses), it calls the method on this which resolves to the base class signature at compile time. This will fail to compile.

You need to add a 3-parameter base method in EntityRepository that delegates to the 2-parameter version by default, so subclasses can override it.

...and 5 more resolved from earlier reviews

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}