Setup Validates runner tests for Spark4 runner#38478
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates ValidatesRunner and portable ValidatesRunner tests into the Spark 4 runner module. It ensures that structured streaming batch tests are properly executed as part of the validation suite, while temporarily excluding a specific test case that is currently failing due to an upstream issue. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
R: @tkaymak |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Spark 4 runner build configuration to include necessary JVM arguments for Java 17 and adds a test exclusion for the structured streaming runner. The review feedback suggests expanding the --add-opens flags to prevent potential IllegalAccessException issues, adopting Gradle configuration avoidance practices by using tasks.named instead of eager configuration, and verifying that all intended test suites are correctly registered as dependencies for the validatesRunner task.
| tasks.withType(Test).configureEach { | ||
| jvmArgs "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED", | ||
| "--add-opens=java.base/java.nio=ALL-UNNAMED", | ||
| "--add-opens=java.base/java.util=ALL-UNNAMED", | ||
| "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" | ||
| } |
There was a problem hiding this comment.
Spark 4 on Java 17 typically requires a more comprehensive set of --add-opens flags to handle internal reflection used by Spark's memory management and serialization (e.g., Kryo). Missing flags like java.lang, java.io, and java.net can lead to IllegalAccessException in certain workloads.
tasks.withType(Test).configureEach {
jvmArgs "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
"--add-opens=java.base/java.nio=ALL-UNNAMED",
"--add-opens=java.base/java.util=ALL-UNNAMED",
"--add-opens=java.base/java.lang=ALL-UNNAMED",
"--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
"--add-opens=java.base/java.io=ALL-UNNAMED",
"--add-opens=java.base/java.net=ALL-UNNAMED",
"--add-opens=java.base/java.util.concurrent=ALL-UNNAMED"
}
| tasks.validatesStructuredStreamingRunnerBatch { | ||
| filter { | ||
| // TODO(https://github.com/apache/beam/issues/36841): currently failing with INTERNAL_ERROR] | ||
| // Couldn't find method fromByteArray with arguments | ||
| excludeTestsMatching 'org.apache.beam.sdk.transforms.CombineTest$BasicTests.testHotKeyCombining' | ||
| } | ||
| } |
There was a problem hiding this comment.
To support Gradle configuration avoidance, it is recommended to use tasks.named(...).configure { ... } instead of eager task configuration. Additionally, fixed a minor typo in the TODO comment (missing opening bracket for [INTERNAL_ERROR]).
tasks.named("validatesStructuredStreamingRunnerBatch").configure {
filter {
// TODO(https://github.com/apache/beam/issues/36841): currently failing with [INTERNAL_ERROR]
// Couldn't find method fromByteArray with arguments
excludeTestsMatching 'org.apache.beam.sdk.transforms.CombineTest$BasicTests.testHotKeyCombining'
}
}
| } | ||
| } | ||
|
|
||
| tasks.validatesRunner.dependsOn(validatesStructuredStreamingRunnerBatch) |
There was a problem hiding this comment.
The PR description mentions adding "portable validates runner tests", but they are not included as a dependency for the validatesRunner task. If validatesPortableRunnerBatch was intended to be part of this PR, it should be added here. Also, using tasks.named and string references for dependencies supports configuration avoidance.
tasks.named("validatesRunner").configure {
dependsOn "validatesStructuredStreamingRunnerBatch"
}
There was a problem hiding this comment.
this can be a follow up for Beam repo build.gradle as a whole
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
|
Also R: @shunping upcoming release manager |
|
Sorry for my slow response, LGTM too 👍 |
… remaining review comments from #38255 (#38489) * [runners-spark] Spark 4 follow-up: address remaining review comments from #38255 Addresses two trailing review comments left on PR #38255 at the approved/merged head SHA that were never folded in before merge: - Restore the `feature Y added to Java SDK` placeholder line in CHANGES.md Highlights (lost to a rebase). Comment: #38255 (comment) - Delete two orphaned `.github/trigger_files/*.json`: - `beam_PreCommit_Java_Spark4_Versions.json`: its workflow YAML was deleted in #38255. - `beam_PostCommit_Java_ValidatesRunner_Spark4StructuredStreaming.json`: flagged as "not effective yet"; since then, #38478 renamed the associated workflow YAML to `beam_PostCommit_Java_ValidatesRunner_Spark4.yml`, so this trigger JSON is doubly orphaned. Comment: #38255 (comment) The renamed workflow YAML stays in place; per the trigger-files convention these JSONs are only added when an author wants to manually trigger a specific PostCommit. * Update CHANGES.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix CHANGES.md URL: /pull/N -> /issues/N for validateChanges * Update CHANGES.md Co-authored-by: Yi Hu <huuyyi@gmail.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Yi Hu <huuyyi@gmail.com>
3 participants
Please add a meaningful description for your change here
Follow up of #38255, added Validates runner tests and portable validates runner tests.
Tested with running the test targets locally
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.