We've discovered a regression when upgrading from pubsub 0.30 to 1. We use 8 workers/connections (ReceiveConfig.worker_count) to pubsub and stream messages using Subscription::receive.
We loose channels and are unable to reestablish them making our streaming grind to a halt after a couple of hours.
This was never a problem with the older version that worked perfectly for months.
From time to time we get errors like this (with some added logging):
gcloud_pubsub::subscriber: reconnect - 'Status { code: Unknown, message: "error reading a body from connection", source: Some(hyper::Error(Body, Error { kind: Reset(StreamId(1), INTERNAL_ERROR, Remote) })) }' : <subscription>
From:
https://github.com/yoshidan/google-cloud-rust/blame/5c30f410af0ecfb8b087fd4be4a96e4a1e6d58af/pubsub/src/subscriber.rs#L203
The crate tries to reestablish a connection on the next loop but the error from
|
self.inner.streaming(req, path, codec).await |
is:
Status {
code: Cancelled,
message: "operation was canceled",
source: Some(tonic::transport::Error(Transport, hyper::Error(Canceled, "connection closed")))
}
Making us match this the worker exit and we loose a connection.
Anyone else seen this?
We've discovered a regression when upgrading from pubsub 0.30 to 1. We use 8 workers/connections (
ReceiveConfig.worker_count) to pubsub and stream messages usingSubscription::receive.We loose channels and are unable to reestablish them making our streaming grind to a halt after a couple of hours.
This was never a problem with the older version that worked perfectly for months.
From time to time we get errors like this (with some added logging):
From:
https://github.com/yoshidan/google-cloud-rust/blame/5c30f410af0ecfb8b087fd4be4a96e4a1e6d58af/pubsub/src/subscriber.rs#L203
The crate tries to reestablish a connection on the next loop but the error from
google-cloud-rust/googleapis/src/google.pubsub.v1.rs
Line 3256 in 5c30f41
is:
Making us match this the worker exit and we loose a connection.
Anyone else seen this?