Skip to content

Seperated live migration 2#3947

Open
bharadwaj-aditya wants to merge 5 commits into
GoogleCloudPlatform:mainfrom
bharadwaj-aditya:seperated_live_migration_2
Open

Seperated live migration 2#3947
bharadwaj-aditya wants to merge 5 commits into
GoogleCloudPlatform:mainfrom
bharadwaj-aditya:seperated_live_migration_2

Conversation

@bharadwaj-aditya

Copy link
Copy Markdown
Contributor

This is a part of the effort to separate source specific logic from the standard template logic for spanner migrations.
PR1 - #3942
PR 2 - Move source specific logic into connectors. This does not increase test coverage for any pre-existing code.

Following PRs will move the constants and bump up the test coverage.

@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.36585% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.04%. Comparing base (3d09e7b) to head (a307d1b).

Files with missing lines Patch % Lines
...ort/v2/templates/SpannerTransactionWriterDoFn.java 77.77% 3 Missing and 1 partial ⚠️
...ce/DatastreamToSpannerSourceConnectorRegistry.java 85.18% 2 Missing and 2 partials ⚠️
...eam/source/oracle/OracleDsToSpSourceConnector.java 77.77% 2 Missing ⚠️
...ce/postgresql/PostgresqlDsToSpSourceConnector.java 77.77% 2 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##               main    #3947       +/-   ##
=============================================
+ Coverage     38.10%   62.04%   +23.94%     
- Complexity      512     3102     +2590     
=============================================
  Files           195      545      +350     
  Lines         11745    32762    +21017     
  Branches       1173     3605     +2432     
=============================================
+ Hits           4475    20326    +15851     
- Misses         6908    11367     +4459     
- Partials        362     1069      +707     
Components Coverage Δ
spanner-templates 88.07% <85.36%> (∅)
spanner-import-export ∅ <ø> (∅)
spanner-live-forward-migration 89.76% <85.36%> (∅)
spanner-live-reverse-replication 83.27% <ø> (∅)
spanner-bulk-migration 92.54% <ø> (∅)
gcs-spanner-dv 90.34% <ø> (∅)
Files with missing lines Coverage Δ
...oud/teleport/v2/templates/DataStreamToSpanner.java 94.61% <100.00%> (ø)
...tastream/source/mysql/MySqlChangeEventContext.java 100.00% <ø> (ø)
...astream/source/mysql/MySqlChangeEventSequence.java 92.18% <ø> (ø)
...tream/source/mysql/MySqlDsToSpSourceConnector.java 100.00% <100.00%> (ø)
...stream/source/oracle/OracleChangeEventContext.java 100.00% <ø> (ø)
...tream/source/oracle/OracleChangeEventSequence.java 54.54% <ø> (ø)
...source/postgresql/PostgresChangeEventSequence.java 62.26% <ø> (ø)
...eport/v2/templates/spanner/ShadowTableCreator.java 97.72% <100.00%> (ø)
...eam/source/oracle/OracleDsToSpSourceConnector.java 77.77% <77.77%> (ø)
...ce/postgresql/PostgresqlDsToSpSourceConnector.java 77.77% <77.77%> (ø)
... and 2 more

... and 393 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bharadwaj-aditya bharadwaj-aditya force-pushed the seperated_live_migration_2 branch 2 times, most recently from ccdab08 to 270feb0 Compare June 25, 2026 08:32
@bharadwaj-aditya bharadwaj-aditya added addition New feature or request improvement Making existing code better labels Jun 25, 2026
@bharadwaj-aditya bharadwaj-aditya force-pushed the seperated_live_migration_2 branch from 270feb0 to 9c7379d Compare June 25, 2026 11:16
@bharadwaj-aditya bharadwaj-aditya marked this pull request as ready for review June 26, 2026 04:23
@bharadwaj-aditya bharadwaj-aditya requested a review from a team as a code owner June 26, 2026 04:23
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request continues the effort to decouple source-specific logic from the standard migration template. By introducing a registry of source connectors, the codebase becomes more modular and easier to extend for future source types. The changes improve maintainability by centralizing source-specific behaviors and removing redundant factory classes.

Highlights

  • Refactoring to Source Connectors: Introduced the ISourceConnector interface and SourceConnectorRegistry to encapsulate source-specific logic, replacing hardcoded checks for MySQL, Oracle, and PostgreSQL.
  • Code Cleanup: Removed the legacy ChangeEventContextFactory and ChangeEventSequenceFactory classes in favor of the new connector-based registry pattern.
  • Validation Improvements: Added explicit source type validation in SpannerTransactionWriterDoFn to ensure change events match the expected source configuration.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Datastream to Spanner template to decouple source-specific logic by introducing the ISourceConnector interface and a SourceConnectorRegistry. This replaces the static factory classes (ChangeEventContextFactory and ChangeEventSequenceFactory) with dynamic connector lookups, improving extensibility for MySQL, PostgreSQL, and Oracle. A review comment points out a discrepancy in SourceConnectorRegistry where the Javadoc claims to use SPI for discovering connectors, but the implementation statically registers them instead.

Aditya Bharadwaj added 3 commits June 26, 2026 05:15
Introduce a pluggable ISourceConnector interface to encapsulate all source-specific
logic for MySQL, PostgreSQL, and Oracle databases within the datastream-to-spanner
template.

This replaces inline source-specific logic with a clean, decoupled design:
- Created ISourceConnector interface.
- Created MySqlSourceConnector, PostgresqlSourceConnector, and OracleSourceConnector implementing the interface.
- Refactored SourceConnectorRegistry to manually register and manage these connectors (no AutoService/SPI dependency).
- Updated SpannerTransactionWriterDoFn and ShadowTableCreator to interact with sources purely via the ISourceConnector interface.
- Deleted obsolete ChangeEventContextFactory and ChangeEventSequenceFactory classes, delegating context and sequence creation to the connectors.
- Kept all source-specific constants in DatastreamConstants (like in main) to preserve compatibility.
- Removed auto-service dependency from pom.xml.
- All changes are strictly restricted to the datastream-to-spanner template.

TAG=agy
CONV=b7fec87d-3652-43db-a0d1-64773a054415
@bharadwaj-aditya bharadwaj-aditya force-pushed the seperated_live_migration_2 branch from 9c7379d to f7e7462 Compare June 26, 2026 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

addition New feature or request improvement Making existing code better size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants