Skip to content

Commit 454502a

Browse files
pluskymimi1vx
authored andcommitted
fix(mcp): bound session disconnect so a wedged host close cannot hang teardown
_disconnect_targets submitted every per-host Target.close() to a ContextExecutor and used concurrent.futures.wait(..., timeout=45) to bound the wait -- but the `with ContextExecutor()` block's __exit__ runs shutdown(wait=True), which re-joins every worker regardless of that timeout. A paramiko close that wedges (dead peer, no RST) keeps its worker thread alive, so shutdown(wait=True) blocked forever and close() never returned; under http this hung the registry idle-sweep awaiting it and leaked the thread, contradicting the docstring's "bounded wait" promise. Drop the context-manager exit for the bound: shut the pool down explicitly with wait=False (cancel_futures=True) after the timed wait, log any host that overran the budget, and abandon its close (the worker thread leaks, but teardown always returns). The 45s bound is now a named module constant and _disconnect_targets takes a `timeout` argument so the behaviour is testable without a 45s wait. Adds a revert-verified regression test: a host whose close() blocks forever no longer stalls _disconnect_targets -- the healthy host is still closed, the stuck one is logged, and every template's host group is cleared. Resolves finding openSUSE#10 from the recent code review.
1 parent d56530c commit 454502a

3 files changed

Lines changed: 98 additions & 7 deletions

File tree

CHANGELOG.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,13 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
237237
connections — instead of silently overwriting it. Repeated `regenerate`
238238
cycles no longer leak file descriptors or leave stale arbiter claims that
239239
could block later slot re-acquisition on the same refhost pool.
240+
- `mtui-mcp` session teardown (idle-TTL sweep or explicit eviction) is now
241+
genuinely time-bounded. A wedged refhost close (a dead peer that never sends
242+
an RST can hang paramiko's disconnect forever) no longer blocks the whole
243+
disconnect: the stuck close is waited on only up to a fixed budget, then
244+
logged and abandoned so `close()` — and the http registry's idle-sweep behind
245+
it — always returns. Previously the bound was defeated by the thread pool's
246+
context-manager exit, which re-joined every worker regardless of the timeout.
240247
- An unscoped multi-template fan-out of a host-phase command (e.g. `lock`,
241248
`run`) no longer fails the whole fan-out — or, for `lock`, pretends to
242249
succeed — when a loaded template has no connected host. A host-less template

mtui/mcp/session.py

Lines changed: 41 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,14 @@
103103
#: on ``run`` / ``update`` / ``set_repo`` / ``commit`` etc.).
104104
DEFAULT_PROGRESS_INTERVAL_SECONDS: float = 10.0
105105

106+
#: Upper bound (seconds) that :meth:`McpSession._disconnect_targets` waits
107+
#: for the parallel per-host ``Target.close()`` calls to finish before it
108+
#: gives up and returns. A wedged paramiko close (dead peer, no RST) can
109+
#: block its worker thread forever; the bound guarantees session teardown
110+
#: still completes, abandoning the stuck close rather than blocking the
111+
#: whole disconnect (and, under http, the registry idle-sweep behind it).
112+
DISCONNECT_TIMEOUT_SECONDS: float = 45.0
113+
106114

107115
class _LogCaptureHandler(Handler):
108116
"""Tee ``mtui`` log records emitted *during a command* into its stdout.
@@ -553,14 +561,29 @@ async def close(self) -> None:
553561
return
554562
await asyncio.to_thread(self._disconnect_targets)
555563

556-
def _disconnect_targets(self) -> None:
564+
def _disconnect_targets(self, timeout: float = DISCONNECT_TIMEOUT_SECONDS) -> None:
557565
"""Synchronous parallel host-disconnect core for :meth:`close`.
558566
559567
Runs in a worker thread. Closes every loaded template's
560568
:class:`Target` on its own pool thread (paramiko teardown is
561-
blocking) with a bounded wait, then clears each template's host
562-
group regardless of individual outcomes. Per-host errors are
563-
logged at WARNING, never raised.
569+
blocking) with a genuinely bounded wait, then clears each
570+
template's host group regardless of individual outcomes.
571+
Per-host errors are logged at WARNING, never raised.
572+
573+
The bound is enforced by shutting the pool down with
574+
``wait=False`` after the timed :func:`concurrent.futures.wait`,
575+
*not* by the ``with`` block's exit — ``Executor.__exit__`` calls
576+
``shutdown(wait=True)``, which would re-block on a wedged close
577+
(a dead peer with no RST keeps its worker thread alive forever)
578+
and defeat the whole point of the timeout. A close that overruns
579+
``timeout`` is logged and abandoned: its worker thread leaks, but
580+
``close()`` — and, under http, the registry idle-sweep awaiting
581+
it — always returns. (That worker is a non-daemon pool thread, so
582+
a close that stays wedged forever can still delay a *clean*
583+
interpreter exit via ``concurrent.futures``' atexit join; that is
584+
strictly better than the old behaviour, which blocked ``close()``
585+
itself during steady-state operation, and is bounded in practice
586+
by the OS TCP timeout.)
564587
"""
565588
# (HostsGroup, hostname) for every host across every loaded template.
566589
work = [
@@ -575,9 +598,20 @@ def _close_one(targets, name: str) -> None:
575598
except Exception as exc: # noqa: BLE001 - best-effort teardown
576599
self.log.warning("error disconnecting host %s: %s", name, exc)
577600

578-
with ContextExecutor() as executor:
579-
futures = [executor.submit(_close_one, t, name) for t, name in work]
580-
concurrent.futures.wait(futures, timeout=45)
601+
executor = ContextExecutor()
602+
try:
603+
futures = {executor.submit(_close_one, t, name): name for t, name in work}
604+
_done, not_done = concurrent.futures.wait(futures, timeout=timeout)
605+
for future in not_done:
606+
self.log.warning(
607+
"host %s did not disconnect within %ss; abandoning its close",
608+
futures[future],
609+
timeout,
610+
)
611+
finally:
612+
# Never join a wedged close: wait=False keeps the bound above real,
613+
# cancel_futures drops any per-host close still queued behind it.
614+
executor.shutdown(wait=False, cancel_futures=True)
581615

582616
for report in self.templates.all():
583617
report.targets.clear()

tests/test_mcp_session.py

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -451,6 +451,56 @@ def test_close_disconnects_every_loaded_templates_hosts(tmp_path: Path) -> None:
451451
assert other.targets == {}
452452

453453

454+
def test_disconnect_targets_bounded_wait_survives_a_wedged_close(
455+
tmp_path: Path, caplog: pytest.LogCaptureFixture
456+
) -> None:
457+
"""A host whose ``close()`` never returns must not block teardown.
458+
459+
``Executor.__exit__`` runs ``shutdown(wait=True)``, so bounding the
460+
wait with only the ``with`` block would re-block on a wedged paramiko
461+
close and hang ``close()`` (and the http idle-sweep behind it)
462+
forever. The explicit ``shutdown(wait=False)`` keeps the timed wait
463+
real: teardown returns despite the stuck close, the healthy host is
464+
still closed, the stuck one is reported, and every template's host
465+
group is cleared.
466+
"""
467+
sess = _make_session(tmp_path)
468+
469+
release = threading.Event()
470+
wedged_host = MagicMock()
471+
wedged_host.close.side_effect = lambda: release.wait(30)
472+
good_host = MagicMock()
473+
474+
report = MagicMock()
475+
report.id = "SUSE:Maintenance:1:1"
476+
report.targets = {"wedged-host": wedged_host, "good-host": good_host}
477+
sess.templates.add(report)
478+
sess.templates.set_active("SUSE:Maintenance:1:1")
479+
480+
worker = threading.Thread(target=sess._disconnect_targets, kwargs={"timeout": 0.2})
481+
try:
482+
with caplog.at_level(logging.WARNING):
483+
worker.start()
484+
# Generous guard: the fix returns in ~0.2 s; a regression that
485+
# relies on the ``with`` exit would still be blocked here.
486+
worker.join(timeout=15)
487+
488+
assert not worker.is_alive()
489+
# The healthy host was closed even though a sibling close hung.
490+
good_host.close.assert_called_once()
491+
# The stuck host was reported, not silently dropped.
492+
assert any(
493+
"wedged-host" in r.getMessage() and "did not disconnect" in r.getMessage()
494+
for r in caplog.records
495+
)
496+
# Host groups are emptied regardless of the stuck close.
497+
assert report.targets == {}
498+
finally:
499+
# Let the abandoned worker thread finish so it does not linger.
500+
release.set()
501+
worker.join(timeout=5)
502+
503+
454504
# --------------------------------------------------------------------------- #
455505
# Fan-out dispatch honours the ``template`` parameter (Phase 4) #
456506
# --------------------------------------------------------------------------- #

0 commit comments

Comments
 (0)