Python examples to use IObase IOs sinks in streaming (and batch) - #35811
Conversation
…sIO) in batch and streaming
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @razvanculea, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
I've reviewed this pull request, which introduces new Python examples to demonstrate the use of IObase IO sinks for both streaming and batch data processing within Apache Beam. The examples showcase how to write data to various formats like TextIO, AvroIO, ParquetIO, and TFRecordsIO, utilizing TestStream and PeriodicImpulse as data sources for the streaming scenarios.
Highlights
- Streaming IO Examples: Added new Python examples demonstrating how to write unbounded PCollections to various IObase IO sinks (TextIO, FileIO, ParquetIO, AvroIO, TFRecordIO) in streaming mode.
- Batch IO Examples: Included new Python examples for writing bounded PCollections to TextIO, FileIO, and ParquetIO in batch processing mode.
- Data Generation: Introduced generate_event.py to create a GenerateEvent PTransform using TestStream for simulating unbounded data streams, and test_periodicimpulse.py for using PeriodicImpulse as a source.
- IO Sink Coverage: Provided comprehensive examples covering WriteToText, WriteToFiles, WriteToParquet (including batched and windowed writes), WriteToAvro, and WriteToTFRecord for different processing paradigms.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Assigning reviewers: R: @claudevdm for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
assign to @shunping |
|
Reminder, please take a look at this pr: @claudevdm |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @damccorm for label python. Available commands:
|
damccorm
left a comment
damccorm
left a comment
There was a problem hiding this comment.
Thanks - mostly LGTM, just had some doc suggestions
|
Reminder, please take a look at this pr: @damccorm |
|
waiting on author |
github-actions
Bot
added
Next Action: Author
and removed
Next Action: Reviewers
slow-review
labels
…or users). add folder README add mode comments per damccorm comments.
|
|
||
| class GenerateEvent(beam.PTransform): | ||
| """This class simulates streaming data. | ||
| It leverages [TestStream](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/testing/TestStream.html), |
|
Running pylint... |
|
@damccorm only unrelated tests are flaky, this can be merged |
3 participants
IObased IOs now support streaming.
Some examples to use TextIO, AvroIO, ParquetIO, TFRecordsIO in streaming with TestStream or PeriodicImpulse as source.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.