Skip to content

gcp: migrate Pub/Sub input and output to the pubsub/v2 SDK#4432

Open
highlyavailable wants to merge 2 commits into
redpanda-data:mainfrom
highlyavailable:gcp-pubsub-v2-migration
Open

gcp: migrate Pub/Sub input and output to the pubsub/v2 SDK#4432
highlyavailable wants to merge 2 commits into
redpanda-data:mainfrom
highlyavailable:gcp-pubsub-v2-migration

Conversation

@highlyavailable
Copy link
Copy Markdown

@highlyavailable highlyavailable commented May 16, 2026

Google's Cloud Go SDK moved Pub/Sub to a v2 package, so this migrates the gcp_pubsub input and output over to it.

The main issue is that v2 splits the data clients from admin operations. Topic and subscription existence checks, IAM, and CreateSubscription aren't on Publisher/Subscriber anymore, so those now go through TopicAdminClient/SubscriptionAdminClient with a codes.NotFound check.

One behaviour change worth calling out: v2 dropped synchronous pull, so the input's sync field doesn't map to anything now. I left it in and marked it deprecated rather than removing it since this is a Stable connector, and the reader logs a warning if someone sets it. Can rip it out instead if you'd prefer.

Ran the unit tests and the full gcp_pubsub integration suite against the emulator, all green.

Happy to take feedback and iterate on any of this.

Fixes #4041

The v2 client splits the data plane (Publisher/Subscriber) from admin
operations: Topic.Exists, Subscription.Exists, IAM and CreateSubscription
are no longer available on the data clients, so existence and permission
checks now go through TopicAdminClient/SubscriptionAdminClient and a
codes.NotFound check.

v2 also drops synchronous pull, so the input's sync field no longer maps
to anything. It is kept and marked deprecated to avoid breaking existing
configs, and the reader logs a warning when it is set.
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 16, 2026

CLA assistant check
All committers have signed the CLA.

@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@josephwoodward
Copy link
Copy Markdown
Contributor

Thank you for this @highlyavailable!

CI started flagging two pre-existing patterns in the AWS S3 read
benchmark that are unrelated to the gcp_pubsub migration but block this
PR's golangci-lint check: a sync.WaitGroup goroutine that can use the
new WaitGroup.Go helper, and an if/assign that can use max. Both are
mechanical conversions that compile and behave the same.
@highlyavailable
Copy link
Copy Markdown
Author

Quick heads up on the most recent commit (c4528b6): two of the failing golangci-lint findings were in internal/impl/aws/s3/bench/read/redpanda-connect/main.go, which is unrelated to the gcp_pubsub migration in this PR. They look like new modernize rules picking up pre-existing patterns (a sync.WaitGroup use that can use wg.Go, and an if/assign that can use max). I included the two-line fix here just to keep this PR's CI green, but if you'd rather see those land in their own PR on main I'm happy to split them out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update Google PubSub to use v2 package

3 participants