[Gemini] Migrate Python BigQuery off of apitools client

[jrmccluskey]

Migrates the BQ client to the supported Python library and off of the deprecated apitools client.

DO NOT MERGE - needs internal testing

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[jrmccluskey]

@gemini-code-assist review

[gemini-code-assist]

Code Review

This pull request migrates the Apache Beam Python SDK's BigQuery integration from the legacy apitools client to the official google-cloud-bigquery library. The feedback highlights several critical issues and regressions introduced during this migration. Specifically, manual base64 encoding of bytes should be removed as the new client handles this natively, and broad exception-swallowing during table and dataset deletions should be avoided to prevent hiding critical errors. Additionally, several redundant try-except blocks, a leftover debug print statement, and an ad-hoc MockJob implementation on conflict should be cleaned up.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.