Multiple sessions created by the same SSH client disconnected and timed out one after another at the same time.

[Moonergfp]

Version

2.15.0

Bug description

My use case is remote port forwarding . Both the client and server are implemented with Apache MINA SSHD. And multiple custom global requests are sent before sending the tcpip-forward request. I have discovered several abnormal issues:
After the SSH connection is established, the client sends 5 consecutive keepalive heartbeats to the server. However, no global keepalive packets are captured in the server's debug logs. Occasionally, the server receives 1 or 2 keepalive packets and then stops receiving any more randomly. This results in the SSH connection dropping repeatedly.
Multiple sessions created by the same client almost disconnect one after another within the same minute, and then all drop together again after a few minutes.
I implemented logic on the client side to re-establish a new connection once the session object is closed. Strangely, every reconnection succeeds on the first attempt, and none of the custom global requests get lost. The only faulty behavior occurs with the keepalive mechanism on the server side.
The server initiates a TCP connection to a random forwarded port every 20 seconds. If no keepalive response is received and the session remains open at this time, the client often fails to receive the incoming TCP connection notification.

Below is the client-side code

while(true){
try (ClientSession session =
SshUtils.getClientSession(
sameSshClientObj)) {
//send global request1
//send global request2
//send global request3
try (ExplicitPortForwardingTracker portForwardingTracker =
createPortForwardingTracker(session)) {
while(!session.isClosed()){
//sleep x seconds
}
}
}
}

Below is the server-side code that initiates a connection to a random forwarded port every 20 seconds.

try (Socket testSocket = new Socket(host, port)) {
}

Then I frequently observe debug logs showing heartbeat timeouts on the client side, alongside idle timeout logs on the server side.

Actual behavior

Frequent heartbeat timeouts on the client side, while the server fails to receive heartbeat requests.

Expected behavior

The server can receive heartbeats normally and send corresponding responses.

Relevant log output

### Below is the client-side log
2026-06-16T18:04:23.410+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: signalAuthFailure(ClientSessionImpl[classic-tx-sh
anghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) type=SshException, signal
led=false, first=false: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.411+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: exceptionCaught(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822])[state=Opened] SshException:
Got 5 heartbeat requests without reply
2026-06-16T18:04:23.412+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] o.a.s.c.session.helpers.SessionTimeoutListener []: sessionException(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) SshException: Got 5 heartbea
t requests without reply
2026-06-16T18:04:23.412+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] c.s.p.t.d.u.SshClientCreator$DialerClientConnectionService []: sendHeartBeat(ClientSessionImpl[classic-
tx-shanghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8418@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) failed (SshException
) to send heartbeat #6 request=keepalive@tunnel.sensorsdata.cn: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.419+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: signalAuthFailure(ClientSessionImpl[classic-tx-sh
anghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) type=SshException, signal
led=false, first=false: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.419+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: exceptionCaught(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822])[state=Opened] SshException:
Got 5 heartbeat requests without reply
2026-06-16T18:04:23.419+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] o.a.s.c.session.helpers.SessionTimeoutListener []: sessionException(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) SshException: Got 5 heartbea
t requests without reply
2026-06-16T18:04:23.421+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] c.s.p.t.d.u.SshClientCreator$DialerClientConnectionService []: sendHeartBeat(ClientSessionImpl[classic-
tx-shanghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8329@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) failed (SshException
) to send heartbeat #6 request=keepalive@tunnel.sensorsdata.cn: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.681+0800 DEBUG - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: signalAuthFailure(ClientSessionImpl[classic-tx-sh
anghai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8106@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822]) type=SshException, signal
led=false, first=false: Got 5 heartbeat requests without reply
2026-06-16T18:04:23.681+0800 WARN - sshd-LoadBalancedSshClient[624b523]-timer-thread-1 [] org.apache.sshd.client.session.ClientSessionImpl []: exceptionCaught(ClientSessionImpl[classic-tx-shang
hai-01.fuchuang.deployx:hybrid01.fuchuang-new.deployx.8106@sfn-tx-shanghai-01.saas.deployx/212.129.159.242:8822])[state=Opened] SshException:
Got 5 heartbeat requests without reply

 ### Below is the server-side log
2026-06-16T18:01:57.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/223.76.236.226:55728]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40266/40000 ms.
2026-06-16T18:05:07.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/113.57.11.210:47586]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40488/40000 ms.
2026-06-16T18:18:07.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/113.57.11.210:39636]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40263/40000 ms.
2026-06-16T18:27:27.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/58.48.59.226:46870]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40066/40000 ms.
2026-06-16T18:51:08.990+0800 INFO - sshd-SshServer[2a8a3ada](port=8822)-timer-thread-1 [] c.s.p.t.s.sshd.components.StationServerSessionImpl []: Disconnecting(StationServerSessionImpl[classic-tx-shanghai-01.fuchuang.deploy.sensorsdata.cloud:hybrid01.fuchuang-new.deploy.sensorsdata.cloud.8329@/58.48.59.226:55670]): SSH2_DISCONNECT_PROTOCOL_ERROR - Detected IdleTimeout after 40655/40000 ms.

Other information

No response

[Moonergfp]

#461
I originally configured HEARTBEAT_REPLY_WAIT=20s, which caused a timeout whenever a single heartbeat went unanswered. After checking related existing issues, I switched to HEARTBEAT_NO_REPLY_MAX=5, yet the problem where the server fails to receive heartbeats still persists. I have confirmed that the network between the two servers is in good condition.

[Moonergfp]

#461 I originally configured HEARTBEAT_REPLY_WAIT=20s, which caused a timeout whenever a single heartbeat went unanswered. After checking related existing issues, I switched to HEARTBEAT_NO_REPLY_MAX=5, yet the problem where the server fails to receive heartbeats still persists. I have confirmed that the network between the two servers is in good condition.

Since all other global requests get normal responses, I suspect there is an underlying bug with the heartbeat mechanism.

[tomaswolf]

Perhaps caused by your setting a custom global request name. Off-hand I would suspect that the server doesn't know that request and something goes wrong with the error reply, or that the server doesn't send back an error reply.

Please provide a small Java maven project running a client and server that demonstrates the problem.

[Moonergfp]

client-side tcpdump

This is a client-side TCPdump capture file. A key observation is that the client’s receive window size is only 38 bytes, indicating the client’s I/O thread cannot consume data transmitted by the server at a sufficient rate.
ps: 10.102.54.13 is the client's intranet IP, and 212.129.159.242 is the server's public network IP

server-side tcpdump

Server Intranet: 10.142.0.6:8822 Client Public Network: 58.48.59.226:34908

[Moonergfp]

Perhaps caused by your setting a custom global request name. Off-hand I would suspect that the server doesn't know that request and something goes wrong with the error reply, or that the server doesn't send back an error reply.

Please provide a small Java maven project running a client and server that demonstrates the problem.

The server can recognize these custom global requests. After this SSH connection has been used for a period of time, the client-to-server heartbeat (hb) requests and server-to-client probe requests will go missing. I am currently unable to reproduce this issue, and it only occurs in one customer environment. My only option is to log in remotely to capture logs or perform tcpdump.

[tomaswolf]

That's not actionable for us.