shpool hangs for several minutes but eventually works

[marstj]

AI Policy Ack Ack

What happened
Frequently when I try to reattach to existing sessions, shpool hangs for a long time, from about half a minute to over 5 minutes, but it eventually works without any restart. Established sessions work fine during that time, and if I try shpool list in one of them then that hangs too.

What I expected to happen
Reestablish the session within a second.

To Reproduce
The problem is intermittent, it happens about 1 time out of 10.

Version info
shpool 0.10.2 (cl/932836465)

Logs
The attached log shows two occurrences, I've marked the points where I experience the unresponsiveness with "/... hang .../".

shpool-hangs.log

[ethanpailes]

The hangs are happening around the time that we take some locks, so this seems like a lock contention problem. Without having dug into it yet, my first guess would be that this has something to do with channel sends hanging for a bit while they are holding a lock and then eventually timing out.

[ethanpailes]

Actually, I think this has to do with the hooks taking a long time to run. The right fix is probably not in upstream shpool, but in making the telemetry in the google-internal shpool async.

[marstj]

Thanks for looking into it so quickly! Does that mean a fix is in the works, or should I open an internal ticket instead?

[ethanpailes]

If you want to file an internal ticket and close this one, that's probably a better place for it, but you can also just leave this open if you want.

[marstj]

Ok, done.