Fix Sim2Conn read hanging after peer close

[daleiz]

When a peer closed a simulated connection, the receiver coroutine would co_await Never() after stopReceive fired, leaving readers with no way to discover the connection was done. Add an incomingClosed AsyncVar that is set after all in-flight bytes have been delivered, so read() and whenReadable() can surface connection_failed instead of blocking forever.

Care is taken to switch to self->process before setting incomingClosed, since stopReceive's delay may fire on the closing peer's process.

[gxglass]

@tclinkenbeard-oai could you have a look at this? I am not sure if this is a latent problem or specifically related to coroutine conversion

@daleiz Thanks for sending this in. Do you happen to have any specific repro info, e.g. a commit hash + random seed + buggify value + test name that you have seen problems?

[tclinkenbeard-oai]

Thanks, this looks like a good fix. I'm curious what simulation failures you saw from the underlying bug. I think connectionMonitor should usually detect the peer closure at a higher level

[tclinkenbeard-oai]

Closing and reopening to trigger CI

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 720af66
• Duration 0:40:04
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 720af66
• Duration 0:43:37
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 720af66
• Duration 0:54:47
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 720af66
• Duration 0:54:52
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[daleiz]

@tclinkenbeard-oai

This was not discovered by an FDB simulator test. I hit it in one of our own simulator-based tests
built on top of FDB's simulated IConnection, without going through FlowTransport / Peer, so
connectionMonitor was not involved.

In that setup, once read() returns 0, the caller depends on onReadable() to either become ready when more
bytes arrive or throw once the connection is no longer readable. The bug in Sim2Conn was that after peer
close, it could reach a state where no more bytes could ever arrive, but onReadable() still waited forever.

[daleiz]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 720af66
• Duration 0:54:47
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

fixed

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 720af66
• Duration 2:15:36
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 546a812
• Duration 0:25:52
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 546a812
• Duration 0:44:08
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 546a812
• Duration 0:45:15
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 546a812
• Duration 0:47:39
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 546a812
• Duration 2:12:23
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[tclinkenbeard-oai]

It looks like there are some ~1/1k test failures due to incomingClosed.set(true) waking readers causing invalid Sim2Conn object access

[daleiz]

It looks like there are some ~1/1k test failures due to incomingClosed.set(true) waking readers causing invalid Sim2Conn object access

fixed

[gxglass]

@daleiz having a look. FWIW Claude review notes are attached if any of this is helpful (I have not looked closely at it).
pr12906_review.md

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: feaddc5
• Duration 0:26:49
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: feaddc5
• Duration 0:36:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: feaddc5
• Duration 0:43:17
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: feaddc5
• Duration 0:46:38
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: feaddc5
• Duration 0:56:31
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: feaddc5
• Duration 1:03:32
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: feaddc5
• Duration 2:05:25
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[daleiz]

@gxglass @tclinkenbeard-oai

I checked the latest correctness failures and fixed them: pass=9998 fail=2 and pass=9999 fail=1, all 3 failed seeds now pass locally. Please run the tests again.

I also fixed a separate lifetime bug in HTTPServer, where the response mutex used by callback actors could outlive the stack frame that created it. That mutex is now held through a ref-counted wrapper.

[gxglass]

@gxglass @tclinkenbeard-oai

I checked the latest correctness failures and fixed them: pass=9998 fail=2 and pass=9999 fail=1, all 3 failed seeds now pass locally. Please run the tests again.

I also fixed a separate lifetime bug in HTTPServer, where the response mutex used by callback actors could outlive the stack frame that created it. That mutex is now held through a ref-counted wrapper.

Thanks, I'm hoping to have a deeper look tomorrow. BTW the amount of changes here continues to grow -- I am not really sure if there is anything you can do about that, but any way to break this down into smaller PRs would be helpful. Also on a related note, I am not sure if it is possible to come up with a test case that reproduces the original issue you saw, but if there is, that would be beneficial on general principle. But not sure what's easy/possible/etc.