Fix dead TLog hang with sender-side peek timeout by saintstack · Pull Request #13248 · apple/foundationdb

saintstack · 2026-05-20T05:26:35Z

One fix for a category of problems that I've labelled 'dead TLog' where tests pass but fail anyways because the database never quiesces; the cluster has been damaged in simulation such it can't recover. #13227 is one and mentions a few other case. The below mentions others.

BEHAVIOR CHANGE: Non-parallel TLog peek requests now have a sender-side timeout (PEEK_REPLY_TIMEOUT knob, default 30s). Previously, a peek would wait indefinitely for a reply or an interface change. Now, if no reply arrives within PEEK_REPLY_TIMEOUT seconds, the peek is re-sent. Set PEEK_REPLY_TIMEOUT to 0 to disable and restore the old wait-forever behavior.

Problem: when a TLog is killed with data destruction (KillType=6) and reboots at the same address, FlowTransport silently drops requests to old endpoint tokens. The sender (LogRouter/SS/consistency checker) waits forever because:

The connection is healthy (same address, process is alive)
No ENDPOINT_NOT_FOUND is sent (non-stream unknown endpoints are silently discarded by scanPackets in FlowTransport.cpp)
The interface does not change (same address)

This manifests as simulation timeouts (5400s) in multi-region configs with single replication after datacenter kills. Seen in recent gcc nightlies (tests: AtomicOps.toml seed 309473476, DataLossRecovery.toml seed 3721859193, MaxGrvQueueDelay.toml seed 3023304950, TxnStateStoreCycleTest.toml seed 2843566457) and in local reproduction.

Fix: in LogSystemPeekCursor::serverPeekGetMoreImpl, the existing race(peekReply, interf->onChange()) now includes a third branch: delay(PEEK_REPLY_TIMEOUT). When the timeout fires, the loop continues (re-sends the peek request) rather than throwing.

Why retry (continue) instead of throw: throwing timed_out() propagated up and killed the storage server (SSUpdateError/StorageServerFailed), causing false failures in clog tests (ClogRemoteTLog, GcGenerations) where TLogs are intentionally unreachable for extended periods.

Why this approach over receiver-side ENDPOINT_NOT_FOUND: a receiver- side fix was attempted (sending ENDPOINT_NOT_FOUND for unknown tokens) but abandoned because the receiver cannot distinguish "dead endpoint" from "clogged endpoint" -- both appear as packets to an unknown token. This caused widespread false failures (71/6760 runs, 1%) in tests that inject network clogging (ClogRemoteTLog, DcLag, GcGenerations).

How the fix resolves the dead TLog hang:

Peek request sent to dead TLog -> silently dropped
30s timeout fires -> peek re-sent (new request, same dead endpoint)
Also silently dropped -> timeout fires again
Meanwhile, cluster controller detects the dead TLog
Interface updated -> interf->onChange() fires -> cursor gets new TLog endpoint -> recovery/consistency check proceeds

Tested: 100,000 Joshua correctness runs, 99,997 passed. 3 failures all unrelated (RocksDB file lock bug, TracedTooManyLines in long test).

20260520-001835-stack_dead_tlog-ea137360b7a91210 compressed=True data_size=41116969 duration=3731822 ended=100000 fail=3 max_runs=100000 pass=99997 priority=100 remaining=0 runtime=2:27:48 sanity=False started=100000 stopped=20260520-024623 submitted=20260520-001835 timeout=5400 username=stack_dead_tlog

RandomSeed="1774749498" SourceVersion="b1fc4d938d0e6ac09ca2a11fc11459ff31f9798e" Time="1779240835" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/slow/GcGenerations.toml"
RandomSeed="1680132275" SourceVersion="b1fc4d938d0e6ac09ca2a11fc11459ff31f9798e" Time="1779242782" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/slow/SwizzledCycleTest.toml"
RandomSeed="2248809336" SourceVersion="b1fc4d938d0e6ac09ca2a11fc11459ff31f9798e" Time="1779243800" BuggifyEnabled="0" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/fast/MaxGrvQueueDelay.toml"

Copilot

Pull request overview

Adds a sender-side retry timeout for non-parallel TLog peek requests to avoid indefinite hangs when requests are silently dropped (e.g., after a TLog reboots at the same address with new endpoint tokens). This improves simulation/cluster quiescence behavior in “dead TLog” scenarios without relying on receiver-side endpoint errors.

Changes:

Add SERVER_KNOBS->PEEK_REPLY_TIMEOUT (default 30s, 0 disables) to bound how long a non-parallel peek waits for a reply.
Extend serverPeekGetMoreImpl() to race peek reply vs interface change vs timeout, and retry on timeout.
Introduce a new trace event on timeout to aid diagnosis.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
fdbserver/logsystem/LogSystemPeekCursor.cpp	Adds timeout branch to non-parallel peek reply wait loop and retries on timeout.
fdbserver/core/ServerKnobs.cpp	Initializes new `PEEK_REPLY_TIMEOUT` knob (default 30s).
fdbserver/core/include/fdbserver/core/Knobs.h	Declares new `PEEK_REPLY_TIMEOUT` server knob.

foundationdb-ci · 2026-05-20T05:30:22Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: e6eed6a
Duration 0:03:32
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T05:31:07Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: e6eed6a
Duration 0:04:20
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T05:31:07Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: e6eed6a
Duration 0:04:21
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T05:31:09Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: e6eed6a
Duration 0:04:21
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T05:31:20Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: e6eed6a
Duration 0:04:33
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

BEHAVIOR CHANGE: Non-parallel TLog peek requests now have a sender-side timeout (PEEK_REPLY_TIMEOUT knob, default 30s). Previously, a peek would wait indefinitely for a reply or an interface change. Now, if no reply arrives within PEEK_REPLY_TIMEOUT seconds, the peek is re-sent. Set PEEK_REPLY_TIMEOUT to 0 to disable and restore the old wait-forever behavior. Problem: when a TLog is killed with data destruction (KillType=6) and reboots at the same address, FlowTransport silently drops requests to old endpoint tokens. The sender (LogRouter/SS/consistency checker) waits forever because: - The connection is healthy (same address, process is alive) - No ENDPOINT_NOT_FOUND is sent (non-stream unknown endpoints are silently discarded by scanPackets in FlowTransport.cpp) - The interface does not change (same address) This manifests as simulation timeouts (5400s) in multi-region configs with single replication after datacenter kills. Seen in recent gcc nightlies (tests: AtomicOps.toml seed 309473476, DataLossRecovery.toml seed 3721859193, MaxGrvQueueDelay.toml seed 3023304950, TxnStateStoreCycleTest.toml seed 2843566457) and in local reproduction. Fix: in LogSystemPeekCursor::serverPeekGetMoreImpl, the existing race(peekReply, interf->onChange()) now includes a third branch: delay(PEEK_REPLY_TIMEOUT). When the timeout fires, the loop continues (re-sends the peek request) rather than throwing. Why retry (continue) instead of throw: throwing timed_out() propagated up and killed the storage server (SSUpdateError/StorageServerFailed), causing false failures in clog tests (ClogRemoteTLog, GcGenerations) where TLogs are intentionally unreachable for extended periods. Why this approach over receiver-side ENDPOINT_NOT_FOUND: a receiver- side fix was attempted (sending ENDPOINT_NOT_FOUND for unknown tokens) but abandoned because the receiver cannot distinguish "dead endpoint" from "clogged endpoint" -- both appear as packets to an unknown token. This caused widespread false failures (71/6760 runs, 1%) in tests that inject network clogging (ClogRemoteTLog, DcLag, GcGenerations). How the fix resolves the dead TLog hang: 1. Peek request sent to dead TLog -> silently dropped 2. 30s timeout fires -> peek re-sent (new request, same dead endpoint) 3. Also silently dropped -> timeout fires again 4. Meanwhile, cluster controller detects the dead TLog 5. Interface updated -> interf->onChange() fires -> cursor gets new TLog endpoint -> recovery/consistency check proceeds Tested: 100,000 Joshua correctness runs, 99,997 passed. 3 failures all unrelated (RocksDB file lock bug, TracedTooManyLines in long test).

foundationdb-ci · 2026-05-20T06:00:05Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: e6eed6a
Duration 0:33:17
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T06:05:18Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: afbc652
Duration 0:24:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T06:12:57Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: e6eed6a
Duration 0:46:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T06:25:55Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: afbc652
Duration 0:44:42
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T06:27:14Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: afbc652
Duration 0:45:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T06:35:27Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: afbc652
Duration 0:54:11
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T06:37:56Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: afbc652
Duration 0:56:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T06:48:09Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: afbc652
Duration 1:06:55
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-05-20T06:50:55Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: afbc652
Duration 1:09:40
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:00:02Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 9d412f9
Duration 0:03:39
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:00:53Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 9d412f9
Duration 0:04:30
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-05-20T19:00:59Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 9d412f9
Duration 0:04:36
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:01:01Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 9d412f9
Duration 0:04:40
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:01:19Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 9d412f9
Duration 0:04:51
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:42:57Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 9d412f9
Duration 0:46:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T19:51:41Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 9d412f9
Duration 0:55:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T20:08:23Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 83e6ae6
Duration 0:03:34
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T20:09:08Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 83e6ae6
Duration 0:04:23
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T20:09:14Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 83e6ae6
Duration 0:04:26
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-05-20T20:09:17Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 83e6ae6
Duration 0:04:27
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T20:10:17Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 83e6ae6
Duration 0:05:28
Result: ❌ FAILED
Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T20:44:30Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 8afa595
Duration 0:31:29
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T20:59:16Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 8afa595
Duration 0:46:18
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T21:03:03Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 8afa595
Duration 0:50:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T21:14:17Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 8afa595
Duration 1:01:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T21:21:54Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 8afa595
Duration 1:08:57
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-05-20T21:30:02Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 8afa595
Duration 1:17:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T21:32:26Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 8afa595
Duration 1:19:28
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T22:09:40Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 83e6ae6
Duration 2:04:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-20T22:58:09Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 83e6ae6
Duration 2:53:21
Result: ❌ FAILED
Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2026-05-21T02:03:36Z

I ran 100k again:

20260520-234221-stack-98b4cd86237c9320 compressed=True data_size=37163449 duration=2754470 ended=100000 fail=2 fail_fast=10 max_runs=100000 pass=99998 priority=100 remaining=0 runtime=1:11:53 sanity=False started=100000 stopped=20260521-005414 submitted=20260520-234221 timeout=5400 username=stack

RandomSeed="3730907482" SourceVersion="8afa59566acd48a9e5c8df66b295f9585103a017" Time="1779321205" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/fast/DataLossRecovery.toml"

Looking at trace logs after rerunning the failure the startMoveKeys exhausted its 51-retry limit because buggified TLog unavailability prevented commits (transaction_too_old), which is write path — unrelated to the peek read path which this PR modifies, confirmed by zero PeekReplyTimeout events in the trace.

and

RandomSeed="2932876011" SourceVersion="8afa59566acd48a9e5c8df66b295f9585103a017" Time="1779322974" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/fast/InventoryTestAlmostReadOnly.toml"

Ditto. A dual-region MachineAttrition test ran for 22K simulated seconds with a storage server unable to reach a replica (unreachable_storage_replica, 43K events), flooding the trace log until it hit the 1M line limit and aborted — zero PeekReplyTimeout events in the trace.

* Extend PEEK_REPLY_TIMEOUT to parallel peek path (LogRouter fix) The sender-side peek timeout added in 446079c (PR #13248) only covers the non-parallel peek path (serverPeekGetMoreImpl), used by storage servers. LogRouters use the parallel peek path (serverPeekParallelGetMoreImpl) via peekLogRouter/SetPeekCursor, which still had the old wait-forever behavior. Add the same PEEK_REPLY_TIMEOUT race branch to serverPeekParallelGetMoreImpl. When the timeout fires, clear the pending futures and re-send the peek (same retry-without-throw approach as the non-parallel path). Extract peekReplyTimeout() helper used by both paths to compute the timeout future consistently. Both sites cross-reference each other in comments. This fixes the live-TLog LogRouter hang where a TLog is re-recruited with new endpoint tokens. The LogRouter's peek request is silently dropped (same root cause as the dead-TLog hang), but the parallel path never detected it. Seed 408543238 (TxnStateStoreCycleTest.toml, buggify=on) reproduces on x86_64 linux. Passes with this fix (206s vs 24000+ hang). Also fix fdb_c_wiggle_only hang on shutdown due to stats timer race. The fdb_c_api_tester process hangs on shutdown after all workloads complete, causing the test harness to SIGKILL it and report failure. Root cause: schedulePrintStatistics() re-schedules itself inside its timer callback. When the last workload finishes, workloadDone() calls statsTimer->cancel() then scheduler->stop(). But if the timer fires and the callback creates a new timer between cancel() and stop() (or just after stop()), the new timer keeps io_ctx.run() alive indefinitely since the work guard has already been released but there is still an outstanding async operation. This race is more likely to trigger now because the sender-side peek timeout -- this and #13248 -- changes recovery timing during the cluster wiggle, causing more transient transaction errors (connection_failed, commit_unknown_result, not_committed). Workloads take longer to drain their in-flight transactions after receiving STOP, giving the 5-second stats timer more opportunities to fire and race with shutdown. Fix: add an atomic flag checked by the timer callback before re-scheduling. Set the flag before cancelling the timer to guarantee no new timer is created after shutdown begins.

saintstack requested a review from Copilot May 20, 2026 05:26

saintstack added the nightlies Issues to address failures in the nighty runs. label May 20, 2026

Copilot started reviewing on behalf of saintstack May 20, 2026 05:27 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Comment thread fdbserver/logsystem/LogSystemPeekCursor.cpp Outdated

Comment thread fdbserver/logsystem/LogSystemPeekCursor.cpp Outdated

michael stack added 2 commits May 19, 2026 22:38

Formatted

afbc652

saintstack force-pushed the peek branch from e6eed6a to afbc652 Compare May 20, 2026 05:41

saintstack requested review from sbodagala and spraza May 20, 2026 05:41

ploxiln reviewed May 20, 2026

View reviewed changes

Comment thread fdbserver/core/include/fdbserver/core/Knobs.h Outdated

ploxiln reviewed May 20, 2026

View reviewed changes

Comment thread fdbclient/tests/fdb_cluster_fixture.sh Outdated

Address review feedback

83e6ae6

saintstack force-pushed the peek branch from 9d412f9 to 83e6ae6 Compare May 20, 2026 20:04

Formatting

8afa595

sbodagala approved these changes May 20, 2026

View reviewed changes

ploxiln approved these changes May 21, 2026

View reviewed changes

spraza approved these changes May 21, 2026

View reviewed changes

saintstack merged commit 446079c into apple:main May 22, 2026
7 checks passed

This was referenced May 22, 2026

Fix peekLogRouter() skip logic that breaks multi-region recovery #13228

Closed

Require minimumReplication=2 for AutomaticIdempotency test #13227

Closed

Extend PEEK_REPLY_TIMEOUT to parallel peek path (LogRouter fix) #13266

Merged

Uh oh!

Conversation

saintstack commented May 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented May 20, 2026

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented May 20, 2026