feat: adding read stall retry for gRPC storage client

[raj-prince]

Extending the dynamic delay algorithm and stall retry mechanism (previously HTTP XML only) to support gRPC read operations. At high level changes includes:

• Refactor of stall timeout logic to re-use at multiple places.
• Support of read-stall feature for gRPCStorage client for both ReadObject & BidiReadObject.
• Depends on: feat: adding read/write stall injection support for gRPC storage-testbench#809

[gemini-code-assist]

Code Review

This pull request adds support for dynamic read stall timeouts to the gRPC API, bringing feature parity with the XML API. It introduces a bucketDelayManager to track latencies and dynamically adjust timeouts, and adds emulator tests to verify the behavior. The review feedback highlights two main issues: first, using time.After in request-scoped paths can cause temporary memory leaks, so it should be replaced with time.NewTimer and stopped via defer timer.Stop(). Second, the dynamic stall timeout should not be increased if the outer context was cancelled by the caller, which can be prevented by checking ctx.Err() == nil before calling increase().

[krishnamd-jkp]

Why can't this be a method on the struct only instead of passing bucketDelayManager again into this?

[raj-prince]

My main motivation for keeping executeWithReadStallTimeout as a standalone helper function is to maintain a clear separation of concerns:

• bucketDelayManager acts as a pure statistics/delay calculator.
• executeWithReadStallTimeout handles concurrency orchestration (context cancellation, timers, and callbacks).
  Keeping execution flow decoupled from data management prevents bucketDelayManager from becoming tightly coupled to gRPC/HTTP stream lifecycle management.

Happy to refactor if you still prefer having it as a method on bucketDelayManager!"