[Failing Test]: GrpcGetDataStreamTest flaky timeout

[Abacn]

What happened?

example run (Java PreCommit): https://github.com/apache/beam/runs/51711179283

testRequestKeyedData_reconnectOnStreamErrorAfterHalfClose (org.apache.beam.runners.dataflow.worker.windmill.client.grpc.GrpcGetDataStreamTest) failed

org.junit.runners.model.TestTimedOutException: test timed out after 10 minutes
...
at java.base@11.0.24/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
	at app//org.apache.beam.runners.dataflow.worker.windmill.client.grpc.GrpcGetDataStreamTest.testRequestKeyedData_reconnectOnStreamErrorAfterHalfClose(GrpcGetDataStreamTest.java:324)

This also affects internal code import

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[scwhittle]

There is a race in GetDataStream when handling retrying pending requests on a stream that has been half-closed between the retry and the threads issuing the reads. Sending a PR to fix