Avoid losing shutdown RPC handle

[yuandrew]

What was changed

• Make shutdown initiation and RPC-handle publication atomic.
• Distinguish between an RPC that has not started and one whose join handle was already taken.
• Ensure concurrent shutdown() callers all wait for the same in-flight ShutdownWorker RPC.
• Serialize repeated shutdown calls because downstream shutdown routines consume one-shot handles.
• Add regression tests for concurrent shutdown and cancellation of the caller that took the RPC handle.

Why?

Graceful poll shutdown requires the ShutdownWorker RPC to complete before Core waits for local polls to drain.

Before, concurrent initiate_shutdown() and shutdown() calls could expose the shutdown state before the RPC handle was stored. Concurrent shutdown() calls could also race to take the handle, allowing one caller to begin draining polls without waiting for the RPC.

If the caller holding the join handle was cancelled, dropping the Tokio JoinHandle detached the RPC task, leaving subsequent callers without a handle to await. A shared completion signal now lets those callers wait for the original RPC.

Without this ordering, shutdown can stall until server poll timeouts and can recreate the graceful-poll shutdown deadlock this RPC is intended to prevent.

Original report: #1255 (comment)

Checklist

1. Closes

2. How was this tested:

wrote a few tests that verify that concurrent and replacement shutdown callers wait for the same single in-flight ShutdownWorker RPC, even when the caller owning its join handle is cancelled.

3. Any docs updates needed?

---

Note

Medium Risk
Touches worker shutdown ordering and graceful poll release, but behavior is narrowly scoped and covered by new concurrency/cancellation tests.

Overview
Fixes a race where concurrent initiate_shutdown and shutdown could leave no join handle to await, so shutdown() could proceed before the ShutdownWorker RPC finished and risk graceful poll drain deadlocks.

Worker shutdown replaces the single optional RPC join handle with ShutdownRpcState so initiation is recorded atomically, only one RPC task is spawned, and each shutdown() caller either awaits the handle it took or waits on a CancellationToken signaled when the RPC task completes (including when an earlier caller’s task was cancelled). A shutdown_complete guard makes repeat shutdown() calls idempotent.

Tests add blocked-RPC scenarios proving concurrent shutdown callers stay pending until the RPC completes once, and that a replacement shutdown still waits after the first caller is aborted.

^{Reviewed by Cursor Bugbot for commit 38bbd96. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jmaeagle99]

Should this short circuit be move up just under the lock acquisition so that other callers do not do the same checks and logging?