Skip to content

Fix JsonToRow swallowing downstream errors when runners fuse transforms. - #39098

Merged
stankiewicz merged 5 commits into
apache:masterfrom
utkarshparekh:fix-json-to-row-downstream-errors-20935
Jun 25, 2026
Merged

Fix JsonToRow swallowing downstream errors when runners fuse transforms.#39098
stankiewicz merged 5 commits into
apache:masterfrom
utkarshparekh:fix-json-to-row-downstream-errors-20935

Conversation

@utkarshparekh

Copy link
Copy Markdown
Contributor

Separate JSON parsing from MultiOutputReceiver output in ParseWithError so exceptions from fused downstream consumers are not misreported as parse failures. Fixes #20935.

Please add a meaningful description for your change here

Fixes a bug in JsonToRow.withExceptionReporting() where exceptions from fused downstream transforms could be caught and misreported as JSON parse failures in the dead-letter output.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Separate JSON parsing from MultiOutputReceiver output in ParseWithError so
exceptions from fused downstream consumers are not misreported as parse
failures. Fixes apache#20935.
@github-actions github-actions Bot added the java label Jun 25, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a bug in the JsonToRow transform where exceptions originating from fused downstream consumers were being incorrectly caught and misreported as JSON parsing failures. By decoupling the parsing logic from the output emission, the fix ensures that only actual parsing errors are routed to the dead-letter output, while downstream processing failures are allowed to propagate as expected.

Highlights

  • Error Handling Logic: Refactored the processElement method in JsonToRow to separate JSON parsing from output emission, ensuring that downstream exceptions are not incorrectly caught and treated as parse errors.
  • Regression Testing: Added a new test case to verify that exceptions thrown by downstream transforms are correctly propagated and not swallowed by the error reporting mechanism.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the JsonToRow transform to prevent downstream exceptions from being caught and incorrectly handled as JSON parsing errors. This is done by moving the output emission outside of the try-catch block. A unit test is also added to verify this behavior. The reviewer identified a potential serialization issue in the new test where an anonymous inner class DoFn captures the non-serializable test class, and suggested using MapElements with a lambda expression instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/JsonToRowTest.java Outdated
Avoid anonymous DoFn capturing the non-serializable test instance so the
test is safe on runners that enforce DoFn serialization.
@utkarshparekh
utkarshparekh force-pushed the fix-json-to-row-downstream-errors-20935 branch from 2879145 to 0bad579 Compare June 25, 2026 02:59
Use a static DoFn to avoid serialization issues, set row schema on the
downstream transform, and tag the test with ValidatesRunner so it can
run via Dataflow validatesRunner tasks.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

utkarshparekh and others added 2 commits June 25, 2026 02:17
Rebase onto current master (1141 commits behind) and remove ValidatesRunner
category so the regression test only runs on DirectRunner needsRunnerTests.
Suppress UnusedVariable on ThrowingDownstreamDoFn for Error Prone.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stankiewicz
stankiewicz self-requested a review June 25, 2026 09:40
@stankiewicz

Copy link
Copy Markdown
Contributor

thanks, this looks good, will merge once tests are green.

@stankiewicz
stankiewicz merged commit 9673359 into apache:master Jun 25, 2026
32 of 33 checks passed
@utkarshparekh
utkarshparekh deleted the fix-json-to-row-downstream-errors-20935 branch June 27, 2026 17:33
ash6898 pushed a commit to ash6898/beam that referenced this pull request Jun 28, 2026
…ms. (apache#39098)

* Fix JsonToRow swallowing downstream errors when runners fuse transforms.

Separate JSON parsing from MultiOutputReceiver output in ParseWithError so
exceptions from fused downstream consumers are not misreported as parse
failures. Fixes apache#20935.

* Address review: use MapElements in JsonToRow regression test.

Avoid anonymous DoFn capturing the non-serializable test instance so the
test is safe on runners that enforce DoFn serialization.

* Fix JsonToRow regression test for runner integration suites.

Use a static DoFn to avoid serialization issues, set row schema on the
downstream transform, and tag the test with ValidatesRunner so it can
run via Dataflow validatesRunner tasks.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Merge apache/beam master and fix JsonToRowTest for CI.

Rebase onto current master (1141 commits behind) and remove ValidatesRunner
category so the regression test only runs on DirectRunner needsRunnerTests.
Suppress UnusedVariable on ThrowingDownstreamDoFn for Error Prone.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
ash6898 pushed a commit to ash6898/beam that referenced this pull request Jun 29, 2026
…ms. (apache#39098)

* Fix JsonToRow swallowing downstream errors when runners fuse transforms.

Separate JSON parsing from MultiOutputReceiver output in ParseWithError so
exceptions from fused downstream consumers are not misreported as parse
failures. Fixes apache#20935.

* Address review: use MapElements in JsonToRow regression test.

Avoid anonymous DoFn capturing the non-serializable test instance so the
test is safe on runners that enforce DoFn serialization.

* Fix JsonToRow regression test for runner integration suites.

Use a static DoFn to avoid serialization issues, set row schema on the
downstream transform, and tag the test with ValidatesRunner so it can
run via Dataflow validatesRunner tasks.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Merge apache/beam master and fix JsonToRowTest for CI.

Rebase onto current master (1141 commits behind) and remove ValidatesRunner
category so the regression test only runs on DirectRunner needsRunnerTests.
Suppress UnusedVariable on ThrowingDownstreamDoFn for Error Prone.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JsonToRow can catch errors in downstream transforms

2 participants