Skip to content

[DOP-25498] Do not restrict values of Job.type#205

Merged
dolfinus merged 1 commit into
developfrom
feature/DOP-25498
Apr 28, 2025
Merged

[DOP-25498] Do not restrict values of Job.type#205
dolfinus merged 1 commit into
developfrom
feature/DOP-25498

Conversation

@dolfinus
Copy link
Copy Markdown
Member

@dolfinus dolfinus commented Apr 25, 2025

Change Summary

JobType enum has hard-coded values like SPARK_APPLICATION which prevents using DataRentgen with other OpenLineage integrations (e.g. Flink). Replace it with a small table job_type with just one unique column type: string there all the unique types are stored (in format INTEGRATION_SMTH or INTEGRATION). Column Job.type: text was converted to Job.type_id: bigint.

Also updated field types in jobType facet. processingType field is mandatory, and instead of null/None it is passed as string NONE. jobType field is actually optional.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus force-pushed the feature/DOP-25498 branch from 67a37d5 to 6810b73 Compare April 28, 2025 10:08
@dolfinus dolfinus changed the title [DOP-25498] Make operation extraction more generic [DOP-25498] Do not restrict values of Job.type Apr 28, 2025
@dolfinus dolfinus added the ci:skip-changelog Add this label to skip changelog file check label Apr 28, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 28, 2025

Codecov Report

❌ Patch coverage is 88.03419% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.83%. Comparing base (7a94ee2) to head (45bce6b).
⚠️ Report is 120 commits behind head on develop.

Files with missing lines Patch % Lines
...ata_rentgen/consumer/extractors/batch_extractor.py 60.00% 2 Missing and 2 partials ⚠️
data_rentgen/dto/job.py 20.00% 3 Missing and 1 partial ⚠️
data_rentgen/dto/operation.py 50.00% 3 Missing ⚠️
data_rentgen/dto/job_type.py 84.61% 1 Missing and 1 partial ⚠️
data_rentgen/consumer/extractors/operation.py 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #205      +/-   ##
===========================================
- Coverage    91.99%   91.83%   -0.17%     
===========================================
  Files          201      205       +4     
  Lines         5033     5092      +59     
  Branches       354      364      +10     
===========================================
+ Hits          4630     4676      +46     
- Misses         328      336       +8     
- Partials        75       80       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dolfinus dolfinus force-pushed the feature/DOP-25498 branch from 6810b73 to 2c0dee9 Compare April 28, 2025 10:37
@dolfinus dolfinus marked this pull request as ready for review April 28, 2025 10:37
@dolfinus dolfinus requested a review from TiGrib April 28, 2025 10:37
@dolfinus dolfinus merged commit 559d78b into develop Apr 28, 2025
12 of 13 checks passed
@dolfinus dolfinus deleted the feature/DOP-25498 branch April 28, 2025 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:skip-changelog Add this label to skip changelog file check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants