gcp: migrate Pub/Sub input and output to the pubsub/v2 SDK#4432
gcp: migrate Pub/Sub input and output to the pubsub/v2 SDK#4432highlyavailable wants to merge 2 commits into
Conversation
The v2 client splits the data plane (Publisher/Subscriber) from admin operations: Topic.Exists, Subscription.Exists, IAM and CreateSubscription are no longer available on the data clients, so existence and permission checks now go through TopicAdminClient/SubscriptionAdminClient and a codes.NotFound check. v2 also drops synchronous pull, so the input's sync field no longer maps to anything. It is kept and marked deprecated to avoid breaking existing configs, and the reader logs a warning when it is set.
|
|
|
Thank you for this @highlyavailable! |
CI started flagging two pre-existing patterns in the AWS S3 read benchmark that are unrelated to the gcp_pubsub migration but block this PR's golangci-lint check: a sync.WaitGroup goroutine that can use the new WaitGroup.Go helper, and an if/assign that can use max. Both are mechanical conversions that compile and behave the same.
|
Quick heads up on the most recent commit (c4528b6): two of the failing golangci-lint findings were in |
Google's Cloud Go SDK moved Pub/Sub to a v2 package, so this migrates the
gcp_pubsubinput and output over to it.The main issue is that v2 splits the data clients from admin operations. Topic and subscription existence checks, IAM, and CreateSubscription aren't on
Publisher/Subscriberanymore, so those now go throughTopicAdminClient/SubscriptionAdminClientwith acodes.NotFoundcheck.One behaviour change worth calling out: v2 dropped synchronous pull, so the input's
syncfield doesn't map to anything now. I left it in and marked it deprecated rather than removing it since this is a Stable connector, and the reader logs a warning if someone sets it. Can rip it out instead if you'd prefer.Ran the unit tests and the full
gcp_pubsubintegration suite against the emulator, all green.Happy to take feedback and iterate on any of this.
Fixes #4041