Fix split thresholds for BQ sink

[reuvenlax]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @reuvenlax, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on enhancing the record splitting and flushing mechanisms within the BigQuery sink, particularly when using the Storage API. The changes introduce more intelligent and robust checks to manage batch sizes effectively, ensuring that data is written efficiently while respecting configured thresholds and gracefully handling large individual records. The overall goal is to prevent the creation of oversized batches and improve the stability and performance of the BigQuery write operations.

Highlights

• Improved BigQuery Batch Splitting: The SplittingIterable now utilizes a PeekingIterator to proactively assess if adding the next record would exceed the configured splitSize. This enhancement ensures that batches generally adhere to size limits, while still accommodating individual large records that might naturally surpass the threshold.
• Refined BigQuery Flush Logic: The StorageApiWriteUnshardedRecords component has been updated to consider the size of the next record when determining whether to trigger a flush. This prevents a single large record from either prematurely forcing a flush or inadvertently blocking a flush if it's the only record present. The flush condition now also explicitly requires more than one pending record when flushing based on byte size, allowing single large records to be processed without being held back by the byte threshold.
• Code Cleanup: An unused import statement for java.util.Objects was removed from DebeziumReadSchemaTransformProvider.java, contributing to cleaner code.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[reuvenlax]

R: @Abacn
R: @liferoad

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[Abacn]

Thank you!

[codecov]

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.19%. Comparing base (6d5b984) to head (ac2ccf3).
⚠️ Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
...he/beam/sdk/io/gcp/bigquery/SplittingIterable.java	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #36422      +/-   ##
============================================
+ Coverage     36.22%   40.19%   +3.97%     
- Complexity     1667     3387    +1720     
============================================
  Files          1058     1220     +162     
  Lines        165079   186176   +21097     
  Branches       1190     3525    +2335     
============================================
+ Hits          59795    74840   +15045     
- Misses       103108   107976    +4868     
- Partials       2176     3360    +1184     

Flag	Coverage Δ
java	70.40% <80.00%> (+2.15%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.