Fix batchoperator deferrable xcom links by kakatur · Pull Request #64745 · apache/airflow

kakatur · 2026-04-05T17:22:27Z

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

When BatchOperator runs with deferrable=True, operator links (job definition, job queue, CloudWatch logs) were not being persisted as XCom values, causing ERROR-level logs when the UI tried to render them. This change: - Extracts link persistence logic into _persist_links() method - Calls _persist_links() from execute_complete() for deferrable tasks - Refactors monitor_job() to use the same method (DRY principle) Fixes XCom not found errors for: - batch_job_definition - batch_job_queue - cloudwatch_events

Added _persist_links(context) call in execute() method before deferring. This ensures operator links (batch_job_definition, batch_job_queue, cloudwatch_events) are persisted as XCom values immediately after job submission, making them available in the UI while the task is deferred.

- Persist placeholder None value for cloudwatch_events XCom when logs aren't available at submission time - Change log level from WARNING to INFO for missing CloudWatch logs - Prevents 'XCom not found' warning in UI while task is deferred - CloudWatch link will be populated when job completes and logs exist

boring-cyborg · 2026-04-05T17:22:30Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Copilot

Pull request overview

This PR aims to ensure the AWS Batch operator’s extra UI links (job definition/queue/logs) are available for deferrable runs by persisting the related XComs before/around deferral and completion.

Changes:

Persist operator extra-link XComs before deferring in execute() and again in execute_complete().
Refactor link persistence logic into a new _persist_links() helper.
Adjust CloudWatch log-link persistence and logging behavior.

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

o-nikolas · 2026-04-08T00:59:22Z

Converting this to draft since there is unaddressed copilot feedback and failing tests. change this to ready for review once all feedback is addressed and the tests are green 🙂

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

- Reduce AWS API calls by 67% to avoid throttling - Fix missing job success check when awslogs disabled - Improve CloudWatch log error visibility - Handle edge cases in unit tests This addresses code review feedback on PR apache#64745.

kakatur · 2026-04-11T03:22:35Z

I've implemented the feedback from Guthub copilot.

Reduced API calls - 67% reduction (3→1) before deferring
Context validation - Handles empty context in unit tests
Removed XCom placeholder - Respects do_xcom_push setting
Eliminated duplication in monitor_job - CloudWatch fetched once
Better error logging - WARNING level for CloudWatch failures
No duplication in execute_complete - Conditional logic based on awslogs_enabled
Bug fix - Added missing check_job_success() when awslogs_enabled=False

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

Address GitHub Copilot code review feedback: - Reduce AWS Batch API calls by 67% to prevent throttling - Remove dead code and redundant operations - Fix missing job success check when awslogs disabled - Improve CloudWatch error logging visibility - Use keyword arguments for better test compatibility This optimizes the deferrable execution path by reusing job descriptions and eliminating duplicate link persistence and CloudWatch log fetching. Addresses code review feedback on PR apache#64745. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

…hub.com/kakatur/airflow into fix-batchoperator-deferrable-xcom-links

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:233

In the deferrable fast-path, _persist_links(..., skip_cloudwatch=True) is used even when the job is already in a terminal state (e.g. SUCCEEDED). In that case the task returns immediately and the CloudWatch log link is never persisted, so the operator extra link can remain missing in the UI. Consider conditionally persisting CloudWatch links when job_status is already terminal (e.g. call _persist_links(..., skip_cloudwatch=False, job_description=job) before returning/raising).

            # Persist operator links before deferring so they're available in the UI
            # Skip CloudWatch logs (not available yet) and reuse job description to reduce API calls
            job = self._persist_links(context, skip_cloudwatch=True)
            job_status = job.get("status")
            if job_status == self.hook.SUCCESS_STATE:
                self.log.info("Job completed.")
                return self.job_id
            if job_status == self.hook.FAILURE_STATE:
                raise AirflowException(f"Error while running job: {self.job_id} is in {job_status} state")

- Fix missing CloudWatch links in deferrable fast-path - Extract _persist_cloudwatch_link() helper method - Eliminate boolean flag parameters for clarity - Improve code readability and maintainability

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

- Remove redundant wait_for_job() call in execute_complete() - Deduplicate CloudWatch link persistence logic - Reduce AWS Batch API calls after job completion - Ensure consistent CloudWatch links across all execution paths

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

- Persist CloudWatch links on deferred job failures - Skip CloudWatch API call when do_xcom_push is False - Ensure consistent link behavior across success and failure paths

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

- Skip CloudWatch API calls when context lacks task instance - Respect awslogs_enabled flag in monitor_job() - Reduce unnecessary AWS Batch API usage - Improve unit test compatibility with minimal context Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py

The awslogs_enabled flag controls log streaming during wait_for_job(), not CloudWatch link persistence in the UI. Users expect CloudWatch links to be available even when log streaming is disabled, so they can view logs after task completion. Reverting the awslogs_enabled gate in monitor_job() preserves the original behavior and avoids breaking existing unit tests that expect CloudWatch links regardless of the flag value. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

kakatur · 2026-04-12T04:10:13Z

I've fixed all issues raised by Copilot :)

o-nikolas · 2026-04-14T04:02:35Z

I see many reviews by copilot and also some comments from @kakatur saying some are addressed. @kakatur can you please resolve each individual copilot thread that you have resolved?

kakatur added 3 commits April 4, 2026 13:46

kakatur requested a review from o-nikolas as a code owner April 5, 2026 17:22

boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Apr 5, 2026

o-nikolas requested a review from Copilot April 6, 2026 19:49

Copilot started reviewing on behalf of o-nikolas April 6, 2026 19:50 View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

eladkal requested review from ferruzzi and vincbeck April 7, 2026 05:43

o-nikolas marked this pull request as draft April 8, 2026 00:58

kaxil requested a review from Copilot April 10, 2026 19:55

Copilot started reviewing on behalf of kaxil April 10, 2026 19:56 View session

Copilot AI reviewed Apr 10, 2026

View reviewed changes

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated Show resolved Hide resolved

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated Show resolved Hide resolved

kakatur and others added 3 commits April 10, 2026 18:18

Merge branch 'apache:main' into fix-batchoperator-deferrable-xcom-links

e9c3ee4

Merge branch 'apache:main' into fix-batchoperator-deferrable-xcom-links

ea5178b

kakatur marked this pull request as ready for review April 11, 2026 03:20

kakatur requested a review from Copilot April 11, 2026 04:04

Copilot started reviewing on behalf of kakatur April 11, 2026 04:05 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

kakatur and others added 3 commits April 11, 2026 07:58

Merge branch 'apache:main' into fix-batchoperator-deferrable-xcom-links

23cffcf

Merge branch 'fix-batchoperator-deferrable-xcom-links' of https://git…

f61f1d5

…hub.com/kakatur/airflow into fix-batchoperator-deferrable-xcom-links

kakatur requested a review from Copilot April 11, 2026 15:05

Copilot started reviewing on behalf of kakatur April 11, 2026 15:06 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

Add terminal state CloudWatch links and simplify code

3f17702

- Fix missing CloudWatch links in deferrable fast-path - Extract _persist_cloudwatch_link() helper method - Eliminate boolean flag parameters for clarity - Improve code readability and maintainability

kakatur requested a review from Copilot April 11, 2026 16:52

Copilot started reviewing on behalf of kakatur April 11, 2026 16:53 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

Optimize BatchOperator deferrable mode execution

0a6cbad

- Remove redundant wait_for_job() call in execute_complete() - Deduplicate CloudWatch link persistence logic - Reduce AWS Batch API calls after job completion - Ensure consistent CloudWatch links across all execution paths

kakatur requested a review from Copilot April 11, 2026 18:27

Copilot started reviewing on behalf of kakatur April 11, 2026 18:28 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Show resolved Hide resolved

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Show resolved Hide resolved

Fix CloudWatch link persistence in deferrable failure cases

3b78fc9

- Persist CloudWatch links on deferred job failures - Skip CloudWatch API call when do_xcom_push is False - Ensure consistent link behavior across success and failure paths

kakatur requested a review from Copilot April 11, 2026 18:39

Copilot started reviewing on behalf of kakatur April 11, 2026 18:40 View session

Copilot AI reviewed Apr 11, 2026

View reviewed changes

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Show resolved Hide resolved

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Show resolved Hide resolved

kakatur requested a review from Copilot April 12, 2026 01:23

Copilot started reviewing on behalf of kakatur April 12, 2026 01:24 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Show resolved Hide resolved

providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py Outdated Show resolved Hide resolved

kakatur requested a review from Copilot April 12, 2026 02:33

Copilot started reviewing on behalf of kakatur April 12, 2026 02:34 View session

Copilot AI reviewed Apr 12, 2026

View reviewed changes

Conversation

kakatur commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg bot commented Apr 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

o-nikolas commented Apr 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

kakatur commented Apr 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

kakatur commented Apr 12, 2026

Uh oh!

o-nikolas commented Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kakatur commented Apr 5, 2026 •

edited

Loading