Update Beam Protobuf Schema (Java) by baeminbo · Pull Request #35150 · apache/beam

baeminbo · 2025-06-04T14:03:33Z

Refer to https://s.apache.org/beam-protobuf.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

github-actions · 2025-06-05T09:38:03Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

liferoad · 2025-07-11T14:30:20Z

@ahmedabu98

kennknowles · 2025-07-16T21:51:05Z

I think that the many cases of @NonNull and @Nullable applied to type variables indicates a problem. Each one really needs a comment justifying it, because it is unusual and actually a somewhat inflexible situation. Probably you can remove most of them. And some places where you have a type variable and you need to constrain what types it can be instantiated with, you would use extends @NonNull Object (the default if you do not annotate is extends @Nullable Object which allows the type variable to be instantiated with any type)

baeminbo · 2025-07-17T07:40:58Z

I've removed @NonNull and @Nullable from generic types in the ProtoBeamConverter class.

I realized I had significantly misunderstood their proper usage. Nullability annotations on generics are only for specific use cases with strong reasoning (apparently a container-like class?).

baeminbo · 2025-07-24T00:50:59Z

Run Java PreCommit

baeminbo · 2025-07-25T12:53:55Z

Run Java PreCommit

baeminbo · 2025-07-25T14:21:21Z

Run Java_Amazon-Web-Services2_IO_Direct PreCommit

baeminbo · 2025-07-28T02:37:04Z

@kennknowles could you review this pr again? Thanks

derrickaw · 2025-08-12T15:11:33Z

Hi @kennknowles, can you review this PR when you get a chance? Thanks!

jkimshop · 2025-09-05T15:41:55Z

Hello @kennknowles , are you able to review this PR when again? Thank you.

kennknowles

I've got to admit it is hard for me to be certain I've checked enough to catch any potential problem. But I think I've done the best I can at this time. Hopefully we can get this tested via some real-world use cases and confirm it isn't breaking things in a way we can't recover from.

Honestly might want a second pair of eyes, like @reuvenlax

kennknowles · 2025-09-17T16:48:48Z

+        return proto ->
+            ((List<Object>) proto)
+                .stream()
+                    .map(element -> Preconditions.checkNotNull(elementConverter.convert(element)))


In general, the SDK's checkArgumentNotNull is preferred, because it throws IllegalArgumentException so that it clearly blames the caller, as it should in this case. Using Guava's checkNotNull causes it to the NPE which a user will correctly interpret as equivalent to a segfault in terms of the coding that led to it.

You don't have to fix it in this PR but it is a useful followup to just avoid Guava's checkNotNull entirely. I should make a checkstyle rule...

kennknowles · 2025-09-19T16:57:37Z

Confirmed that both test failures look like flakes. I'm re-running them.

stankiewicz · 2025-09-25T11:24:58Z

ready to merge @kennknowles ?

yyfhust · 2025-09-25T15:47:48Z

Thanks for the fix ! Friendly ask : would you do a patch release of beam that includes this fix ?

liferoad · 2025-09-25T17:16:51Z

Thanks for the fix ! Friendly ask : would you do a patch release of beam that includes this fix ?

We usually do not do a patch release. If you need this now, you can use the SNAPSHOT version: https://github.com/apache/beam/blob/master/contributor-docs/code-change-guide.md#run-your-pipeline-with-modified-beam-code

Abacn · 2025-09-25T21:37:25Z

Probably caused https://github.com/apache/beam/actions/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml?query=event%3Aschedule failing

update: confirmed: https://github.com/apache/beam/actions/runs/18021430173

This reverts commit c3ef7f3.

baeminbo · 2025-09-26T01:51:20Z

@Abacn it still failed after the revert: https://github.com/apache/beam/actions/runs/18023411365. I doubt if this change really caused the test failing.

Abacn · 2025-09-26T02:08:29Z

https://github.com/apache/beam/actions/runs/18023411365

This run was against HEAD. The revert hasn't merged into HEAD (master branch) yet (open PR: #36288). The link is a manually triggered run on HEAD+revert this PR

This reverts commit c3ef7f3.

baeminbo · 2025-09-26T13:07:48Z

@Abacn, thanks, a bug was found with OneOfType that is also found in the failed run: https://github.com/apache/beam/actions/runs/18028890287/job/51301187348 (the error stacktrace is at [1]).

I created #36297 to fix it, but I cannot run the test on the PR. Could you let me know how I can run the test?

[1]

NFO     ExpansionService:subprocess_server.py:232 Exception in thread "main" java.util.ServiceConfigurationError: org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider: Provider org.apache.beam.sdk.extensions.sql.expansion.SqlTransformSchemaTransformProvider could not be instantiated
INFO     ExpansionService:subprocess_server.py:232 	at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
INFO     ExpansionService:subprocess_server.py:232 	at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:804)
INFO     ExpansionService:subprocess_server.py:232 	at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:722)
INFO     ExpansionService:subprocess_server.py:232 	at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1395)
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.sdk.expansion.service.ExpansionServiceSchemaTransformProvider.<init>(ExpansionServiceSchemaTransformProvider.java:53)
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.sdk.expansion.service.ExpansionServiceSchemaTransformProvider.of(ExpansionServiceSchemaTransformProvider.java:69)
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.sdk.expansion.service.ExpansionService.getRegisteredSchemaTransforms(ExpansionService.java:565)
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.sdk.expansion.service.ExpansionService.main(ExpansionService.java:844)
INFO     ExpansionService:subprocess_server.py:232 Caused by: java.lang.ExceptionInInitializerError
INFO     ExpansionService:subprocess_server.py:232 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
INFO     ExpansionService:subprocess_server.py:232 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
INFO     ExpansionService:subprocess_server.py:232 	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
INFO     ExpansionService:subprocess_server.py:232 	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
INFO     ExpansionService:subprocess_server.py:232 	at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:780)
INFO     ExpansionService:subprocess_server.py:232 	... 6 more
INFO     ExpansionService:subprocess_server.py:232 Caused by: java.lang.IllegalArgumentException: OneOf fields do not support nullable subfields.
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.sdk.schemas.logicaltypes.OneOfType.create(OneOfType.java:78)
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.sdk.schemas.logicaltypes.OneOfType.create(OneOfType.java:72)
INFO     ExpansionService:subprocess_server.py:232 	at org.apache.beam.sdk.extensions.sql.expansion.SqlTransformSchemaTransformProvider.<clinit>(SqlTransformSchemaTransformProvider.java:58)
INFO     ExpansionService:subprocess_server.py:232 	... 11 more

github-actions Bot added java build extensions protobuf labels Jun 4, 2025

baeminbo force-pushed the fix-35081 branch 4 times, most recently from 9342333 to 23875da Compare June 5, 2025 06:58

baeminbo changed the title ~~[WIP] Update Beam Protobuf Schema (Java)~~ Update Beam Protobuf Schema (Java) Jun 5, 2025

baeminbo force-pushed the fix-35081 branch from 23875da to 3c61248 Compare June 5, 2025 07:33

Change to indented structured string for toString() of Schema and Row

a978b7c

baeminbo force-pushed the fix-35081 branch 4 times, most recently from 6ae0d10 to 5e53125 Compare June 6, 2025 02:16

baeminbo added 3 commits June 5, 2025 20:40

Introduce ProtoBeamConverter

6603be4

Deprecate ProtoDynamicMessageSchema in favor of ProtoBeamConverter

0412cd4

Treat optional proto field and oneof as nullable in Beam Schema

55fcd56

baeminbo force-pushed the fix-35081 branch from 5e53125 to 29376b7 Compare June 6, 2025 03:41

github-actions Bot added io gcp labels Jun 6, 2025

baeminbo force-pushed the fix-35081 branch from 27c3576 to 29376b7 Compare June 7, 2025 15:08

github-actions Bot removed io gcp labels Jun 7, 2025

baeminbo force-pushed the fix-35081 branch from 29376b7 to 814545c Compare June 7, 2025 23:24

github-actions Bot added io gcp labels Jun 7, 2025

Fix unit test failure caused by incorrect field order in Schema

fea582f

baeminbo force-pushed the fix-35081 branch from 814545c to fea582f Compare June 8, 2025 01:21

Remove unnecessary nullability annotation on generic types.

56a0ef1

baeminbo force-pushed the fix-35081 branch from 4a053bb to 56a0ef1 Compare July 17, 2025 07:32

damccorm removed the slow-review label Jul 25, 2025

baeminbo and others added 2 commits September 16, 2025 17:32

Merge branch 'master' into fix-35081

8299564

Fix CheckerFramework failure after merging with master

521eb56

baeminbo force-pushed the fix-35081 branch from 5d59db2 to 521eb56 Compare September 17, 2025 04:49

kennknowles approved these changes Sep 19, 2025

View reviewed changes

kennknowles merged commit c3ef7f3 into apache:master Sep 25, 2025
26 of 29 checks passed

Abacn added a commit that referenced this pull request Sep 25, 2025

Revert "Update Beam Protobuf Schema (Java) (#35150)"

fde6ff2

This reverts commit c3ef7f3.

baeminbo added a commit to baeminbo/beam that referenced this pull request Sep 26, 2025

Revert "Update Beam Protobuf Schema (Java) (apache#35150)"

547b08e

This reverts commit c3ef7f3.

This was referenced Sep 26, 2025

[WIP] Fix broken PreCommit YAML Xlang Direct #36294

Closed

Fix broken PreCommit YAML Xlang Direct #36297

Merged

baeminbo deleted the fix-35081 branch March 9, 2026 07:12

Uh oh!

Conversation

baeminbo commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

github-actions Bot commented Jun 5, 2025

Uh oh!

liferoad commented Jul 11, 2025

Uh oh!

kennknowles commented Jul 16, 2025

Uh oh!

baeminbo commented Jul 17, 2025

Uh oh!

baeminbo commented Jul 24, 2025

Uh oh!

baeminbo commented Jul 25, 2025

Uh oh!

baeminbo commented Jul 25, 2025

Uh oh!

baeminbo commented Jul 28, 2025

Uh oh!

derrickaw commented Aug 12, 2025

Uh oh!

jkimshop commented Sep 5, 2025

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

kennknowles Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

kennknowles commented Sep 19, 2025

Uh oh!

stankiewicz commented Sep 25, 2025

Uh oh!

Uh oh!

yyfhust commented Sep 25, 2025

Uh oh!

liferoad commented Sep 25, 2025

Uh oh!

Abacn commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

baeminbo commented Sep 26, 2025

Uh oh!

Abacn commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

baeminbo commented Sep 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

baeminbo commented Jun 4, 2025 •

edited

Loading

Abacn commented Sep 25, 2025 •

edited

Loading

Abacn commented Sep 26, 2025 •

edited

Loading