Server.reboot operation incompatible with ProxyCommand

[Tbruno25]

Describe the bug

When connected to a host with ProxyCommand in its SSH config, the server.reboot operation hangs indefinitely.

The issue is likely due to L89.

• The server drops the connection as expected
• The proxy process remains alive
• Because the proxy is still running, the connection is not seen as closed
• pyinfra waits indefinitely for a return code that will never arrive

Setting paramiko.transport.set_keepalive(4) resolves the hang, but isn't a viable fix — the transport object isn't exposed by pyinfra, and the stale proxy connection causes threading/atexit errors during teardown.

To Reproduce

1. Configure a host with ProxyCommand ssh -W %h:%p <proxy_host> in its~/.ssh/config
2. Perform aserver.reboot operation

Expected behavior

The operation should detect that the server has gone away (or timeout) and proceed with reconnection.

Meta

❯ uv run pyinfra --support

    System: Darwin
      Platform: macOS-26.4.1-arm64-arm-64bit-Mach-O
      Release: 25.4.0
      Machine: arm64
    pyinfra: v3.6.1
      click: v8.3.1
      distro: v1.9.0
      gevent: v25.9.1
      jinja2: v3.1.6
      packaging: v26.0
      paramiko: v3.5.1
      pydantic: v2.12.5
      python-dateutil: v2.9.0.post0
      typeguard: v4.5.1
      typing-extensions: v4.15.0
    Python: 3.13.12 (CPython, Clang 21.1.4 )

[wowi42]

I traced this on the 3.x branch and the analysis here is correct.

What's happening

server.reboot emits StringCommand("reboot", _success_exit_codes=[0, -1]). The SSH connector then calls paramiko's recv_exit_status after the read loop. With a direct SSH connection the target's sshd dies, the TCP connection drops, paramiko returns -1, and the operation continues. With a ProxyCommand, the proxy process has no way to know the target's sshd is gone, so from paramiko's view the channel is still open and recv_exit_status blocks forever. _timeout does not help: it only covers the buffer read loop, not the exit-status call that follows.

What I changed

I have a candidate fix on a branch: I replaced the inline reboot with a backgrounded subshell whose std fds are detached from the SSH channel:

( sleep 1 && reboot ) </dev/null >/dev/null 2>&1 &

Breakdown:

• ( sleep 1 && reboot ) runs a subshell that waits one second then issues the reboot. The sleep gives the parent shell time to exit and the SSH session time to tear down before the kernel actually goes away, which makes the local close-out cleaner.
• </dev/null reopens the subshell's stdin from /dev/null, breaking the inheritance from the parent shell's stdin (which is wired to the SSH channel).
• >/dev/null 2>&1 does the same for stdout and stderr.
• & backgrounds the subshell. The parent shell returns immediately with exit status 0.

The reason this matters: an OpenSSH non-PTY exec channel only closes once every fd inherited by the remote shell is released. With the redirects in place, the parent shell holds no fds tied to the channel, so as soon as it exits, the channel closes. paramiko gets a clean 0 and the operation moves on to wait_and_reconnect. The actual reboot then fires a second later, well after pyinfra has already let go.

Caveats

I'm not sure this is the best fix. A few reasons:

• It only fixes server.reboot. Anything else that relies on the SSH channel closing after the remote process disappears (custom server.shell calls that trigger a kernel panic, network reconfig that breaks the proxy hop, etc.) would still hang. A connector-level fix (channel-side timeout on recv_exit_status, or an opt-in transport.set_keepalive for the duration of a marked command) would cover those cases too, at the cost of being more invasive.
• The 1 second sleep is a heuristic. On a slow proxy hop it might not be enough; on a fast one it's wasted. It is the simplest knob that works in practice.
• The visible command is no longer a literal reboot, which is a small change in observed behavior for anyone parsing pyinfra output.

If a connector-level fix is preferred, I'm happy to take another pass. Otherwise, I can open a PR with the change above plus the updated fixture.

[Tbruno25]

@wowi42 Thanks for digging into this.

I really like that the sub shell solution only requires minor changes but think there may be some edge cases where it's a bit fragile.

The actual reboot then fires a second later, well after pyinfra has already let go.

The default delay looks to be 10s. Consider a scenario where:

• We issue the reboot and return
• The server is under heavy load and unable to shutdown ssh within 10s
• We falsely reconnect

There's probably a missing step in here where we issue the reboot, then verify we cannot connect, before attempting to reconnect.

What do you think?

[Fizzadar]

There's probably a missing step in here where we issue the reboot, then verify we cannot connect, before attempting to reconnect.

I think that has the same issue but the other way - if the target is really fast we might check down but it's already back up and miss the drop.

I think the only guaranteed way would be to check the uptime output to confirm boot time change...

[Tbruno25]

I think the only guaranteed way would be to check the uptime output to confirm boot time change...

Agreed. This is probably the most robust way.

Seems like before #1711 gets merged, we should:

• Add Uptime fact
• Update server.reboot to query Uptime pre/post reboot.