blog: GSoC 2025 - Beam YAML, Kafka and Iceberg User Accessibility - #36233
Conversation
Summary of ChangesHello @charlespnh, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request adds a new blog post to the website, detailing the outcomes of a Google Summer of Code 2025 project. The project aimed to improve the user accessibility of the Beam YAML SDK by developing and documenting production-ready machine learning pipeline examples that leverage Kafka and Iceberg data sources. These examples demonstrate how to implement complex ML workflows using Beam's declarative YAML approach, addressing a previous lack of comprehensive use cases. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
R: @chamikaramj and @liferoad |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
|
Thanks for writing this blog!!! |
|
LGTM other than XQs comment. Thanks! |
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a blog post for the GSoC 2025 project on Beam YAML, Kafka, and Iceberg user accessibility. The post is well-structured and provides a good overview of the project's context, contributions, and challenges. I've provided a few suggestions to improve grammar and clarity, which should help make the post even more polished for publication. Overall, great work on documenting the project!
| --> | ||
|
|
||
| The relatively new Beam YAML SDK was introduced in the spirit of making data processing easy, | ||
| but it has gained little adoption for complex ML tasks and hasn’t been widely used with Managed IOs such as Kafka and Iceberg. |
There was a problem hiding this comment.
For consistency and correctness, it's better to use a standard apostrophe (') instead of a right single quotation mark (’) in contractions.
| but it has gained little adoption for complex ML tasks and hasn’t been widely used with Managed IOs such as Kafka and Iceberg. | |
| but it has gained little adoption for complex ML tasks and hasn't been widely used with Managed IOs such as Kafka and Iceberg. |
|
|
||
| Beam, as well as Kafka and Iceberg, are mainstream big data technology but they also have a learning curve. | ||
| The overall theme of the project is to help democratize data processing for scientists and analysts that traditionally | ||
| don’t have a strong background in software engineering. They can now refer to these meaningful examples as the starting point, |
There was a problem hiding this comment.
For consistency and correctness, it's better to use a standard apostrophe (') instead of a right single quotation mark (’) in contractions.
| don’t have a strong background in software engineering. They can now refer to these meaningful examples as the starting point, | |
| don't have a strong background in software engineering. They can now refer to these meaningful examples as the starting point, |
chamikaramj
left a comment
chamikaramj
left a comment
There was a problem hiding this comment.
LGTM. Thanks a lot for writing this @charlespnh.
|
|
||
| The relatively new Beam YAML SDK was introduced in the spirit of making data processing easy, | ||
| but it has gained little adoption for complex ML tasks and hasn’t been widely used with Managed IOs such as Kafka and Iceberg. | ||
| As part of Google Summer of Code 2025, new illustrative, production-ready pipeline examples of ML use cases with Kafka and Iceberg |
There was a problem hiding this comment.
Managed I/O (here and elsewhere)
Also let's add a link to https://beam.apache.org/documentation/io/managed-io/.
| ## Challenges | ||
| The main challenge of the project has been a lack of previous YAML pipeline examples and good documentation to rely on. | ||
| Unlike the Python or Java SDKs where there are already many notebooks and end-to-end examples demonstrating various use cases, | ||
| the examples for YAML SDKs only involved simple transformations such as filter, group by, etc… There were a lot of deep dives |
| ## Challenges | ||
| The main challenge of the project has been a lack of previous YAML pipeline examples and good documentation to rely on. | ||
| Unlike the Python or Java SDKs where there are already many notebooks and end-to-end examples demonstrating various use cases, | ||
| the examples for YAML SDKs only involved simple transformations such as filter, group by, etc… There were a lot of deep dives |
There was a problem hiding this comment.
You meant "There weren't a lot of deep dives" ?
There was a problem hiding this comment.
There were, yeah. These were the cases with transforms where the API reference was not really clear and the doc generated had issues with missing config fields. For example, MLTransform and ReadFromIceberg.
This was more of a challenge with documentation where at the time the doc release process had had some problems generating properly. @liferoad
4 participants
GSoC 2025 blog write-up for Beam YAML, Kafka and Iceberg User Accessibility project.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.