Investigate Spanner Change Stream Consumer Options

[data-sync-user]

• Implement Spanner Change Stream Consumer

┆Issue is synchronized with this Jira Task

[data-sync-user]

Philip Jenvey commented from Jira:

We looked into a few different aspects:

• Building a Java change stream consumer
  ** Python's Apache Beam SDK has no native SpannerIO.readChangeStream impl – it does include Spanner support but transforms provided in this module are cross-language transforms implemented in the Beam Java SDK – it requires a JVM anyway
  ** GoogleCloudPlatform/DataflowTemplates' Cloud_Spanner_Change_Streams_to_PubSub is a fairly straightforward impl and only 200 lines of code
  ** Unfortunately we would want to modify it to ignore events not involving payload_link changes (there’s no point in polluting the pub sub with such events of which there will be many of) – this means we have to own our own dataflow template, however the modification is very small and we’ve included a raw diff of it in the repo for updating it at later dates
• Build a Python publisher for integration testing
  ** However the Java consumer can be inconvenient for local/CI testing , but we can build a Python change stream consumer that simply polls the change stream table directly via the grpc API based spanner client.
  ** The Python consumer unfortunately requires some complications (in particular handling partition splits and potential idiosyncrasies of the Spanner emulator/Python client) but we can heavily leverage Claude here for building all this, including unit tests
  ** Such a Python event publisher avoids the JVM requirement, JVM/beam startup overhead (estimated at 15-30 seconds) and a potentially large memory footprint (estimated in the multiple gb). We can also include an option for running the Java version (by having it invoked by beam’s DirectRunner which is provided for testing/local development)
• Open questions
  ** How to best manage the change stream consumer Java build. It should rarely change, does it require being built in CI for every commit to main? Or could more selectively build it when necessary, possibly versioning it separately from syncstorage?
  ** The Java version’s docker currently doesn’t build on linux/arm64 platforms (probably macos/arm64 as well), can this be fixed? (this is also another good reason to have the Python version for local testing)