Skip to content

prevent infinity waiting on remote DC in multiregional mode when failover #12071#13295

Open
MarkSh1 wants to merge 2 commits into
apple:mainfrom
MarkSh1:fix-dd-hangon-when-multerigional-failover
Open

prevent infinity waiting on remote DC in multiregional mode when failover #12071#13295
MarkSh1 wants to merge 2 commits into
apple:mainfrom
MarkSh1:fix-dd-hangon-when-multerigional-failover

Conversation

@MarkSh1

@MarkSh1 MarkSh1 commented May 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit 1: XDB-546 — Show keys data size and moving size when DD is initializing

File: fdbserver/clustercontroller/Status.cpp

What: Moved the "DD is initializing" early-return check after parsing dataStats, so that partition count, moving data size, and other data metrics are still populated in the status JSON even during DD initialization. Previously, these fields were omitted because the function returned before parsing them.

Why: To show moving-data volume during DD initialization (e.g., after a failover) instead of waiting for initialization to complete.

Commit 2: XDB-546 — Add degraded multiregion status when recovery is stuck at accepting_commits

Files: fdbserver/clustercontroller/Status.cpp, fdbclient/Schemas.cpp, fdbcli/StatusCommand.cpp

What:

  1. Introduced a degraded_multi_region flag: set to true when the cluster is configured with usableRegions > 1 and recovery is stuck at accepting_commits. Exposed this field in the status JSON schema.
  2. When DD reports its state as initializing in a degraded multiregion scenario, the description now reads "Degraded multiregional (Re)initializing automatic data distribution" — making the nature of the degradation explicit.
  3. In fdbcli status output, when degraded_multi_region is true, the warning message "may have data loss and availability loss" is suppressed. This is replaced with a neutral header followed by the actionable message (which TLog interfaces to restart).

Why: In a multi-region cluster, when one region fails, recovery gets stuck at accepting_commits because the failed region's logs are unreachable. However, commits are already being accepted in the surviving region (and its satellite), and all committed data is safe and consistent. This is a degraded-but-not-data-losing state. Previously, fdbcli status showed a misleading warning about potential data loss. Now the status honestly reflects degraded multiregion without a false data-loss alarm.

topic in foundationdb forum about this problem : link

@gxglass

gxglass commented May 28, 2026

Copy link
Copy Markdown
Collaborator

It's a little hard for me to think through all the implications of this. Let's start with running CIs.

@gxglass gxglass closed this May 28, 2026
@gxglass gxglass reopened this May 28, 2026
@foundationdb-ci

foundationdb-ci commented May 28, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: 4a22692
  • Duration 0:03:46
  • Result: ❌ FAILED
  • Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 28, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 4a22692
  • Duration 0:04:22
  • Result: ❌ FAILED
  • Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 28, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 4a22692
  • Duration 0:04:23
  • Result: ❌ FAILED
  • Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 28, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 4a22692
  • Duration 0:04:24
  • Result: ❌ FAILED
  • Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 28, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 4a22692
  • Duration 0:04:28
  • Result: ❌ FAILED
  • Error: Error while executing command: if [[ $(git diff --shortstat 2> /dev/null | tail -n1) == "" ]]; then echo "CODE FORMAT CLEAN"; else echo "CODE FORMAT NOT CLEAN"; echo; echo "THE FOLLOWING FILES NEED TO BE FORMATTED"; echo; git ls-files -m; echo; if [[ $FDB_VERSION =~ 7\.\3. ]]; then echo skip; else exit 1; fi; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 28, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: 4a22692
  • Duration 0:29:59
  • Result: ❌ FAILED
  • Error: Error while executing command: ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ${HOME}/.ssh_key ec2-user@${MAC_EC2_HOST} /usr/local/bin/bash --login ./build_pr_macos.sh. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 28, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: 4a22692
  • Duration 0:51:52
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@MarkSh1

MarkSh1 commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

@gxglass , could you please restart CI ? I've applied the clang format to DataDistribution.cpp.

@gxglass gxglass closed this May 29, 2026
@gxglass gxglass reopened this May 29, 2026
@foundationdb-ci

foundationdb-ci commented May 29, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: b363e11
  • Duration 0:22:32
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 29, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

  • Commit ID: b363e11
  • Duration 0:34:53
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 29, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux CentOS 7

  • Commit ID: b363e11
  • Duration 0:46:03
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 29, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS Ventura 13.x

  • Commit ID: b363e11
  • Duration 0:46:10
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 29, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: b363e11
  • Duration 0:48:10
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 29, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: b363e11
  • Duration 0:50:39
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented May 29, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: b363e11
  • Duration 1:06:26
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@gxglass

gxglass commented May 29, 2026

Copy link
Copy Markdown
Collaborator

Here's what the local AI said about this. I don't know how much weight to give to this line of thinking, but here is a concern: this code change could have easily been made in the prior decade, so I'm wondering if there are other reasons not to do it.

pr13295-review.md

@MarkSh1

MarkSh1 commented Jun 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Hello @gxglass! Could you please restart CI? I've reviewed your AI recommendations and made the necessary changes.

@gxglass gxglass closed this Jun 17, 2026
@gxglass gxglass reopened this Jun 17, 2026
@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: c61c5d7
  • Duration 0:00:44
  • Result: ❌ FAILED
  • Error: Unexpected EC2 error: InsufficientFreeAddressesInSubnet
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: c61c5d7
  • Duration 0:42:56
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: c61c5d7
  • Duration 0:46:55
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: c61c5d7
  • Duration 0:18:57
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: c61c5d7
  • Duration 1:02:50
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: c61c5d7
  • Duration 1:03:28
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: c61c5d7
  • Duration 1:06:07
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: c61c5d7
  • Duration 1:18:52
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@MarkSh1 MarkSh1 force-pushed the fix-dd-hangon-when-multerigional-failover branch from c61c5d7 to 8c963a5 Compare June 22, 2026 13:33
@MarkSh1

MarkSh1 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Hello @gxglass! Could you please restart CI? I changed the code again and added new comments.

@gxglass gxglass closed this Jul 6, 2026
@gxglass gxglass reopened this Jul 6, 2026
gxglass
gxglass reviewed Jul 6, 2026
View reviewed changes

@gxglass gxglass left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick AI review attached. Working through a good sized backlog, sorry for the delays. Could you PTAL at this and address if needed or explain why the AI is off

pr13295-review.md

@foundationdb-ci

foundationdb-ci commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-ide on Linux RHEL 9

  • Commit ID: 8c963a5
  • Duration 0:22:38
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos-m1 on macOS 14.x

  • Commit ID: 8c963a5
  • Duration 0:32:30
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang-arm on Linux RHEL 9

  • Commit ID: 8c963a5
  • Duration 0:48:37
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr on Linux RHEL 9

  • Commit ID: 8c963a5
  • Duration 0:52:30
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-clang on Linux RHEL 9

  • Commit ID: 8c963a5
  • Duration 0:59:55
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-macos on macOS 14.x

  • Commit ID: 8c963a5
  • Duration 1:03:46
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci

foundationdb-ci commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

  • Commit ID: 8c963a5
  • Duration 4:00:23
  • Result: ❌ FAILED
  • Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gxglass gxglass gxglass left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@MarkSh1 @gxglass @foundationdb-ci