Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode by chenxuesdu · Pull Request #36667 · apache/beam

chenxuesdu · 2025-10-29T16:55:57Z

Add a private variable isMutableKeyRange, which will be initialized during ChangeStreamDao constructor by reading from internal spanner table.

In function changeStreamQuery, when constructing change stream query, based on the value of isMutableKeyRange, call different TVFs when the Dialect is PostgreSQL(The googleSQL part is completed).

gemini-code-assist · 2025-10-29T16:56:13Z

Summary of Changes

Hello @chenxuesdu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the Spanner change stream connector by introducing logic to adapt query behavior for PostgreSQL dialects based on the change stream's partition mode. Specifically, it allows the system to correctly query change streams that use either MUTABLE_KEY_RANGE or IMMUTABLE_KEY_RANGE partitioning by selecting the appropriate internal Spanner Table-Valued Function, ensuring proper data retrieval and compatibility.

Highlights

Dynamic TVF Selection for PostgreSQL: The ChangeStreamDao now dynamically selects between 'read_proto_bytes_' and 'read_json_' Table-Valued Functions (TVFs) for PostgreSQL change stream queries, based on the change stream's partition mode.
isMutableKeyRange Field: A new private boolean field, 'isMutableKeyRange', has been introduced in ChangeStreamDao to store the detected partition mode.
Constructor Initialization: The 'isMutableKeyRange' field is initialized during the ChangeStreamDao constructor by querying the 'information_schema.change_stream_options' table to determine the change stream's partition mode.
New Test File: A new test file, ChangeStreamDaoTest.java, has been added to validate the logic for determining and using the 'isMutableKeyRange' property.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2025-10-30T00:29:33Z

Assigning reviewers:

R: @m-trieu for label java.
R: @nielm for label spanner.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

tianz101 · 2025-10-30T20:33:23Z

+            Statement.newBuilder(
+                    "select option_name, option_value\n"
+                        + "from information_schema.change_stream_options\n"
+                        + "where change_stream_name = $1")


Can we add option_name = 'partition_mode' here so we can get the parititon mode directly and no need to filter later?

tianz101 · 2025-10-30T20:33:36Z

+            Statement.newBuilder(
+                    "select option_name, option_value\n"
+                        + "from information_schema.change_stream_options\n"
+                        + "where change_stream_name = @changeStreamName")


same comment

tianz101 · 2025-10-30T20:36:40Z

-      query =
-          "SELECT * FROM \"spanner\".\"read_json_" + changeStreamName + "\"($1, $2, $3, $4, null)";
+      // Ensure we have determined whether change stream uses mutable key range
+      boolean isMutable = isMutableKeyRangeChangeStream();


Is this called each time we issue the query? As our query will be constantly checkpointed and canceled and re-issued, this can be called many times.

I think we just need to query the partition mode once for the whole pipeline.

Moved the logic to SpannerIO so it will only be executed once when connector starts. Please take a look. Thanks.

github-actions · 2025-11-08T12:14:24Z

Reminder, please take a look at this pr: @m-trieu @nielm

derrickaw · 2025-11-11T13:30:14Z

waiting on author

jiangzzhu · 2025-12-18T22:51:53Z

  }
+
+  @Test
+  public void testFetchPartitionModeGoogleSql() {


can we parameterized the tests to have different combination among googlesql/postgresql vs immuatable/mutable/empty string ?

Will do it in a separate PR.

tianz101 · 2025-12-19T21:51:06Z

  public void testWithDefaultCredential() {
-    // Get the default credential, without setting any credentials in the pipeline options or
+    // Get the default credential, without setting any credentials in the pipeline
+    // options or


The following 2 lines of comments can be combined.

Updated. Thanks.

tianz101 · 2025-12-20T00:14:25Z

+
+  @Test
+  public void testFetchPartitionModeGoogleSql() {
+    DatabaseClient databaseClient = Mockito.mock(DatabaseClient.class);


These lines are the same for all tests below, can they be refactored to avoid repeatition? Thanks.

Refactored. Please take a look.

tianz101 · 2025-12-20T00:22:09Z

+    ResultSet resultSet = Mockito.mock(ResultSet.class);
+
+    when(databaseClient.readOnlyTransaction()).thenReturn(transaction);
+    when(transaction.executeQuery(any(Statement.class))).thenReturn(resultSet);


If through mock, how the fetch option sql statement syntax in SpannerIO.java is tested to be correct?

Updated the test. please take a look.

tianz101 · 2025-12-20T00:23:12Z

+        SpannerIO.isMutableChangeStream(
+            databaseClient, Dialect.GOOGLE_STANDARD_SQL, TEST_CHANGE_STREAM));
+  }
 }


Also should the the failure or throw exception case of fetch option sql statement be tested as well?

Updated the test. Please take a look.

the change stream partition mode For MUTABLE_KEY_RANGE change stream, use read_proto_bytes_, else use read_json_

github-actions · 2026-01-14T12:17:56Z

Reminder, please take a look at this pr: @m-trieu @nielm

github-actions · 2026-01-19T12:19:25Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label java.
R: @nielm for label spanner.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

tianz101 · 2026-01-23T23:09:34Z

+import org.mockito.ArgumentCaptor;

-@RunWith(JUnit4.class)
+@RunWith(Enclosed.class)


why this one is Enclosed.class? How is it different from JUnit4.class?

Enclosed is used to support parameterized test.

tianz101

LGTM

apache#36667) the change stream partition mode For MUTABLE_KEY_RANGE change stream, use read_proto_bytes_, else use read_json_

github-actions Bot added java io gcp spanner labels Oct 29, 2025

chenxuesdu force-pushed the master branch from 536aa2d to de5ef5f Compare October 29, 2025 18:37

chenxuesdu changed the title ~~Update ChangeStreamDao to query differnet TVF for postgresSQL based on the change stream partition mode~~ Update ChangeStreamDao to query different TVF for postgresSQL based on the change stream partition mode Oct 29, 2025

chenxuesdu force-pushed the master branch 8 times, most recently from 473321f to 88ea37b Compare October 29, 2025 22:28

github-actions Bot added the Next Action: Reviewers label Oct 30, 2025

tianz101 reviewed Oct 30, 2025

View reviewed changes

chenxuesdu force-pushed the master branch 5 times, most recently from d5f8b77 to 9209310 Compare October 31, 2025 17:17

github-actions Bot added the slow-review label Nov 8, 2025

github-actions Bot added Next Action: Author and removed Next Action: Reviewers slow-review labels Nov 11, 2025

jiangzzhu approved these changes Dec 18, 2025

View reviewed changes

jiangzzhu reviewed Dec 18, 2025

View reviewed changes

tianz101 reviewed Dec 19, 2025

View reviewed changes

tianz101 reviewed Dec 20, 2025

View reviewed changes

Update ChangeStreamDao to query different TVF for postgresSQL based on

5ebd9b3

the change stream partition mode For MUTABLE_KEY_RANGE change stream, use read_proto_bytes_, else use read_json_

chenxuesdu force-pushed the master branch from 9209310 to 7677cf9 Compare January 6, 2026 23:08

github-actions Bot added Next Action: Reviewers and removed Next Action: Author labels Jan 6, 2026

chenxuesdu force-pushed the master branch from 7677cf9 to 5ebd9b3 Compare January 6, 2026 23:13

github-actions Bot added the slow-review label Jan 14, 2026

github-actions Bot added reassigned-reviewers and removed slow-review labels Jan 19, 2026

tianz101 mentioned this pull request Jan 22, 2026

[Spanner Change Streams] Fix potential data loss issue by ensuring to only claim timestamps that have been fully processed from the restriction tracker. #37326

Merged

3 tasks

tianz101 reviewed Jan 23, 2026

View reviewed changes

tianz101 approved these changes Jan 23, 2026

View reviewed changes

Abacn merged commit 6a399df into apache:master Jan 26, 2026
16 checks passed

Uh oh!

Conversation

chenxuesdu commented Oct 29, 2025

Uh oh!

gemini-code-assist Bot commented Oct 29, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 8, 2025

Uh oh!

derrickaw commented Nov 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 14, 2026

Uh oh!

github-actions Bot commented Jan 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tianz101 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants