[PD] Un-blacklist mooncake sessions when probe succeeds

[kflansburg]

MooncakeKVManager adds a mooncake_session_id to self.failed_sessions the first time any KV transfer to that peer fails (one strike, no retry). The session is only removed when the decode side re-registers its KV args via the bootstrap thread. If a transient failure (network blip, peer GC pause, peer briefly unhealthy) blacklists the session but the decode side never re-registers, every subsequent prefill->decode request routed through that session fails forever with 'Decode instance could be dead' until one side restarts.

Add a daemon thread on the prefill side that periodically issues a lightweight session probe (engine.send_probe) against every blacklisted session. If the probe succeeds, the session is removed from failed_sessions and session_failures, the event is logged, and sglang:failed_session_recoveries_total is incremented.

Configuration:
SGLANG_FAILED_SESSION_PROBE_INTERVAL_S (default 30s)

The probe loop runs only in PREFILL mode (failed_sessions is prefill-only). Probe exceptions are swallowed with a warning so a single bad peer cannot kill the probe thread.

Requires engine.send_probe on the mooncake-transfer-engine wheel, which is added in kvcache-ai/Mooncake#2088 and must be released (and the bundled MOONCAKE_VERSION bumped) before this change takes effect. Without that, the probe call raises AttributeError which is caught and logged; blacklist behavior is unchanged from today.

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #25943738343
Latest PR Test (Extra): ⚠️ Not enabled -- add run-ci-extra label to opt in.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[ShangmingCai]

Overall, I think this PR has a point, and well-implemented.

But we might need to wait for a new release of mooncake. Also, should we add an env var switch for this feature since we might only need this for specific usage, and other use-cases should not pay (another background thread and cpu overhead) for what they don't need?

[ShangmingCai]

/tag-and-rerun-ci

[ShangmingCai]

CI has passed:

We plan to release a new mooncake version this week. You should be able to use this feature in the new version.