Skip to content

[DOP-38389] Spark job.name=unknown should be detached from others#480

Merged
dolfinus merged 1 commit into
developfrom
improvement/DOP-38389
Jul 6, 2026
Merged

[DOP-38389] Spark job.name=unknown should be detached from others#480
dolfinus merged 1 commit into
developfrom
improvement/DOP-38389

Conversation

@dolfinus

@dolfinus dolfinus commented Jul 6, 2026

Copy link
Copy Markdown
Member

Change Summary

Due to OpenLineage/OpenLineage#3846 RunEvents from Openlineage integration for Spark may contain job.name="unknown" (eventType=STARTED), but latter RunEvents (RUNNING, SUCCESS and so on) are received with proper job name (e.g. job.name="my_session").

This creates temporary job in DataRentgen, which is constantly updated:

UPDATE job SET parent_job_id=234 WHERE job_id=123; -- from one event
UPDATE job SET parent_job_id=345 WHERE job_id=123; -- from another event
UPDATE job SET parent_job_id=456 WHERE job_id=123; -- from third event

Same for job tags and dependencies.

Now consumer for case job.name="unknown" resets parent job, dependencies and tag values, so UPDATE queries are not emitted. Added DB migration to reset those in databaset as well.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus force-pushed the improvement/DOP-38389 branch from d7cf849 to 41561a3 Compare July 6, 2026 10:56
@dolfinus dolfinus force-pushed the improvement/DOP-38389 branch from 41561a3 to 132594b Compare July 6, 2026 10:56
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

Coverage

Coverage Report •
FileStmtsMissBranchBrPartCoverMissing
data_rentgen/consumer/extractors/impl
   spark.py68116296%93->95, 113->114, 114
TOTAL7710518130621192% 

@dolfinus dolfinus marked this pull request as ready for review July 6, 2026 11:05
@dolfinus dolfinus self-assigned this Jul 6, 2026
@dolfinus dolfinus merged commit f9e8165 into develop Jul 6, 2026
11 checks passed
@dolfinus dolfinus deleted the improvement/DOP-38389 branch July 6, 2026 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants