Skip to content

[DOP-25649] Create operations for Airflow tasks#210

Merged
dolfinus merged 1 commit into
developfrom
feature/DOP-25649
May 6, 2025
Merged

[DOP-25649] Create operations for Airflow tasks#210
dolfinus merged 1 commit into
developfrom
feature/DOP-25649

Conversation

@dolfinus
Copy link
Copy Markdown
Member

@dolfinus dolfinus commented Apr 30, 2025

Change Summary

If consumer receives Airflow Task event, it creates both Run and Operation, with same id and created_at. Previously it created only Run, and thus lineage extracted from Airflow operators was completely ignored.

Added special handling for some Airflow Task facet fields:

  • task_id is used as operation.name
  • operator_class is used as operation.description
  • map_index is used as operation.position

As test cases become more complex, I've splitted tests/test_consumer/test_extractors/test_extractors_run.py to test_extractors_run_airflow.py, test_extractors_run_spark.py and test_extractors_run_unknown.py. That's why PR is large.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Apr 30, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 30, 2025

Codecov Report

❌ Patch coverage is 97.61905% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.94%. Comparing base (91d401e) to head (a960d1d).
⚠️ Report is 124 commits behind head on develop.

Files with missing lines Patch % Lines
data_rentgen/consumer/extractors/operation.py 96.77% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #210      +/-   ##
===========================================
+ Coverage    91.93%   91.94%   +0.01%     
===========================================
  Files          205      205              
  Lines         5095     5118      +23     
  Branches       365      370       +5     
===========================================
+ Hits          4684     4706      +22     
  Misses         333      333              
- Partials        78       79       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dolfinus dolfinus requested a review from TiGrib April 30, 2025 13:47
@dolfinus dolfinus marked this pull request as ready for review April 30, 2025 13:47
@dolfinus dolfinus merged commit cc39ae3 into develop May 6, 2025
13 checks passed
@dolfinus dolfinus deleted the feature/DOP-25649 branch May 6, 2025 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants