"The operation was cancelled" happens randomly on spanner queries/txns

[infiniteregrets]

Hi, thanks for all the effort on this library!

Been running into this error for the past few days and previously a few months ago, as I see Cancelled error code is not retried in the spanner lib and seems like its recommended to retry here: https://cloud.google.com/spanner/docs/error-codes, do you have any ideas why this could be happening?

Happy to submit a patch to add Cancelled to be retried!

[yoshidan]

I have no specific ideas. Are long-running queries or transactions running?
The reason 'Cancelled' isn't included in automatic retries is because the google-cloud-go was configured that way.
If we can reproduce the issue, we can certainly add it to the automatic retries.

As a workaround, please configure the retry codes when calling the Spanner API, referencing this code
https://github.com/yoshidan/google-cloud-rust/blob/main/foundation/gax/src/retry.rs#L59