Make port collision detection more robust in fdb_cluster_fixture by saintstack · Pull Request #13249 · apple/foundationdb

saintstack · 2026-05-20T15:32:47Z

This ctest failed in last night's nightly run. The blob_backup_restore_tests ctest fails on busy CI instances when ports are already in use by parallel tests. The existing loop retried on port-in-use errors by grepping the output file for 'Local address in use', but both stdout and stderr were tee'd to the same output file via async process substitution, causing races where the grep misses the error string and the loop exits prematurely.

Fix: tee stderr to a separate file (output.stderr) so the grep for 'Local address in use' is reliable and not corrupted by interleaved stdout. Non-port-in-use failures still fail immediately with stderr printed for debugging.

Here is from the nightly failure (mocks3 retries a few ports and so does fdb but it only tries two ports.... before failing)


....
2026-05-20T07:30:58+00:00 Testing against MockS3Server
Starting MockS3Server on 127.0.0.1:8080 (attempt 1/10)
Starting MockS3Server on 127.0.0.1:8080
Port 8080 already in use, trying next port
Starting MockS3Server on 127.0.0.1:8081 (attempt 2/10)
Starting MockS3Server on 127.0.0.1:8081
Port 8081 already in use, trying next port
Starting MockS3Server on 127.0.0.1:8082 (attempt 3/10)
Starting MockS3Server on 127.0.0.1:8082
Port 8082 already in use, trying next port
Starting MockS3Server on 127.0.0.1:8083 (attempt 4/10)
Starting MockS3Server on 127.0.0.1:8083
Port 8083 already in use, trying next port
Starting MockS3Server on 127.0.0.1:8084 (attempt 5/10)
Starting MockS3Server on 127.0.0.1:8084
MockS3Server ready on port 8084
Starting Cluster:  test1:testdb1@127.0.0.1:1601
Error initializing networking with public address 127.0.0.1:1601 and listen address 127.0.0.1:1601 (Local address in use)
Try `/codebuild/output/src4209381410/src/github.com/apple/foundationdb/build_output/bin/fdbserver --help' for more information.
2026-05-20T07:31:07+00:00 ERROR: Failed to start server on port 1601
2026-05-20T07:31:07+00:00 ERROR: Failed to start DC1 stateless
Retrying PID extraction (attempt 1/5)...
Retrying PID extraction (attempt 2/5)...
Retrying PID extraction (attempt 3/5)...
Retrying PID extraction (attempt 4/5)...
Retrying PID extraction (attempt 5/5)...
WARNING: No FDB PIDs found for tracking
Error initializing networking with public address 127.0.0.1:1601 and listen address 127.0.0.1:1601 (Local address in use)
2026-05-20T07:31:10+00:00 Ports in use; retry cluster start but with different ports
Error initializing networking with public address 127.0.0.1:1701 and listen address 127.0.0.1:1701 (Local address in use)
Try `/codebuild/output/src4209381410/src/github.com/apple/foundationdb/build_output/bin/fdbserver --help' for more information.
Starting Cluster:  test1:testdb1@127.0.0.1:1701
2026-05-20T07:31:11+00:00 ERROR: Failed to start server on port 1701
2026-05-20T07:31:11+00:00 ERROR: Failed to start DC1 stateless
....

Copilot

Pull request overview

This PR reduces CI flakiness in blob_backup_restore_tests by allowing the fdb_cluster_fixture.sh loopback cluster startup to retry more port ranges when initial ports are already in use on busy runners.

Changes:

Add a bounded retry counter (up to 20 attempts) when restarting the loopback cluster on “Local address in use”.
Emit a failure message when no free port range is found after exhausting retries.

ploxiln · 2026-05-20T16:16:07Z

  local port_prefix=1500
+  local max_port_retries=20
+  local port_attempt=0
  while : ; do
    port_prefix="$(( port_prefix + 100 ))"
+    port_attempt="$(( port_attempt + 1 ))"
+    if (( port_attempt > max_port_retries )); then
+      err "Failed to find available ports after ${max_port_retries} attempts (tried ports ${1600}..${port_prefix})"


Yeah, you just want to replace tried ports ${1600}.. with tried ports 1600..

(I think it would make things more elegant if the port_prefix was increased just before the continue, rather than at the beginning of the loop. Not a big deal though.)

Ronitsabhaya75 · 2026-05-20T15:47:04Z

@saintstack Fix looks good to me. one bug tho

Line 124: ${1600} in the error message isn't a variable, it'll expand unexpectedly. Should be a named variable like local start_port=1600 defined alongside port_prefix, then referenced as ${start_port} in the message.

Also worth noting: when the guard fires, port_prefix has already been incremented to the next candidate that was never tried, so the upper bound in the error range is slightly off.

Otherwise the approach is solid bounded retry with a clear failure message is exactly the right fix here.

foundationdb-ci · 2026-05-20T15:58:15Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: c2909bc
Duration 0:25:14
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T16:05:43Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: c2909bc
Duration 0:32:44
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

ploxiln · 2026-05-20T16:17:33Z

Previously the loop only tried 2 ports before giving up.

I actually don't see where it would give up after 2 attempts, before?

foundationdb-ci · 2026-05-20T16:18:35Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: c2909bc
Duration 0:45:33
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T16:20:08Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: c2909bc
Duration 0:47:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T16:20:18Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: c2909bc
Duration 0:47:20
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T16:24:50Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: c2909bc
Duration 0:51:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T16:39:10Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: c2909bc
Duration 1:06:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

Ronitsabhaya75 · 2026-05-20T18:01:08Z

@saintstack looks good to me

foundationdb-ci · 2026-05-20T18:02:31Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 177c123
Duration 0:06:15
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T18:20:19Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 177c123
Duration 0:24:05
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2026-05-20T18:23:35Z

Thank you for review @Ronitsabhaya75 . Addressed.

I actually don't see where it would give up after 2 attempts, before?

@ploxiln Good one. It's been reliable up to this. It depends on this grep ' 184 if grep -a 'Local address in use' "${output}"; then' working properly. I've found since that ctest grepping outputs can be flakey/interrupted/overwritten. Thought explicit counter 'safer' but you remind me that elsewhere I've moved to clearer what we grep. Let me apply that philosophy here too.. just grep the stderr stream (issue seems to be interleaving here though not plain from review of the log -- will print string we fail on going forward.).

saintstack · 2026-05-20T18:24:06Z

This failure is not the same ctest #13249 (comment)

foundationdb-ci · 2026-05-20T18:30:07Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 177c123
Duration 0:33:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

The blob_backup_restore_tests ctest fails on busy CI instances when ports are already in use by parallel tests. The existing loop retried on port-in-use errors by grepping the output file for 'Local address in use', but both stdout and stderr were tee'd to the same output file via async process substitution, causing races where the grep misses the error string and the loop exits prematurely. Fix: tee stderr to a separate file (output.stderr) so the grep for 'Local address in use' is reliable and not corrupted by interleaved stdout. Non-port-in-use failures still fail immediately with stderr printed for debugging.

foundationdb-ci · 2026-05-20T18:38:09Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 177c123
Duration 0:41:57
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T18:41:37Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 177c123
Duration 0:45:23
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:20:10Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: beec5b7
Duration 0:44:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:23:44Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: beec5b7
Duration 0:48:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:45:44Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: beec5b7
Duration 1:09:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-05-20T20:01:24Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: beec5b7
Duration 1:25:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T20:22:56Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: beec5b7
Duration 1:47:12
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T21:36:04Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: beec5b7
Duration 3:00:18
Result: ❌ FAILED
Error: Build has timed out.
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T23:22:33Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: beec5b7
Duration 0:56:06
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T00:11:07Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: beec5b7
Duration 0:23:18
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T00:20:40Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: beec5b7
Duration 0:32:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T00:31:52Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: beec5b7
Duration 0:43:58
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T00:34:13Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: beec5b7
Duration 0:46:25
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T00:43:00Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: beec5b7
Duration 0:55:11
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T00:44:42Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: beec5b7
Duration 0:56:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T00:55:37Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: beec5b7
Duration 1:07:48
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

Replace process substitution with tee with plain file redirects. The tee was unnecessary since neither stream needs real-time terminal output: stdout is only grepped for PIDs and stderr is only checked for port collisions or dumped on failure.

foundationdb-ci · 2026-05-21T01:03:57Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: c0bc2a4
Duration 0:04:44
Result: ❌ FAILED
Error: Error while executing command: cmake -S . -B build_output -D USE_CCACHE=ON -D USE_WERROR=ON -D RUN_JUNIT_TESTS=OFF -D RUN_JAVA_INTEGRATION_TESTS=OFF -D BUILD_DOCUMENTATION=OFF -D BUILD_AWS_BACKUP=ON -G Ninja. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-05-21T01:22:38Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: c0bc2a4
Duration 0:23:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T01:45:05Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: c0bc2a4
Duration 0:45:52
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T01:55:03Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: c0bc2a4
Duration 0:55:50
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T01:58:56Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: c0bc2a4
Duration 0:59:43
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T02:04:54Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: c0bc2a4
Duration 1:05:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T02:27:59Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: c0bc2a4
Duration 1:28:45
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-21T03:03:20Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: c0bc2a4
Duration 1:07:00
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack · 2026-05-21T18:54:10Z

Thanks for the reviews @ploxiln and @Ronitsabhaya75 . In particular Pierce, the questions which made final change simpler, clearer.

saintstack requested review from Copilot and ploxiln May 20, 2026 15:32

saintstack added nightly correctness bugs nightlies Issues to address failures in the nighty runs. labels May 20, 2026

Copilot started reviewing on behalf of saintstack May 20, 2026 15:33 View session

saintstack removed the nightly correctness bugs label May 20, 2026

Copilot AI reviewed May 20, 2026

View reviewed changes

Ronitsabhaya75 approved these changes May 20, 2026

View reviewed changes

ploxiln approved these changes May 20, 2026

View reviewed changes

saintstack force-pushed the ports branch from 177c123 to beec5b7 Compare May 20, 2026 18:35

ploxiln reviewed May 20, 2026

View reviewed changes

Comment thread fdbclient/tests/fdb_cluster_fixture.sh

ploxiln approved these changes May 20, 2026

View reviewed changes

ploxiln mentioned this pull request May 20, 2026

Fix dead TLog hang with sender-side peek timeout #13248

Merged

saintstack closed this May 20, 2026

saintstack reopened this May 20, 2026

saintstack changed the title ~~Retry up to 20 times on port collision in fdb_cluster_fixture~~ Make port collision detection more robust in fdb_cluster_fixture May 21, 2026

ploxiln approved these changes May 21, 2026

View reviewed changes

saintstack merged commit 7d04d9c into apple:main May 21, 2026
7 checks passed

Uh oh!

Conversation

saintstack commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ploxiln May 20, 2026

Choose a reason for hiding this comment

Uh oh!

ploxiln May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Ronitsabhaya75 commented May 20, 2026

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

ploxiln commented May 20, 2026

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

Ronitsabhaya75 commented May 20, 2026

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

saintstack commented May 20, 2026

Uh oh!

saintstack commented May 20, 2026

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

saintstack commented May 20, 2026 •

edited

Loading