Add two flags to add a random delay at the top of dataDistributionRel…

[gxglass]

…ocator for restored and unrestored moves

Replace this text with your description here...

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 3dbea7d
• Duration 0:43:34
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 3dbea7d
• Duration 0:46:06
• Result: ❌ FAILED
• Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 3dbea7d
• Duration 0:47:38
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 3dbea7d
• Duration 0:54:15
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 3dbea7d
• Duration 0:57:46
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 3dbea7d
• Duration 2:10:48
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[spraza]

@gxglass is this PR still needed or can we close it for now?

[gxglass]

@gxglass is this PR still needed or can we close it for now?

It's not my favorite PR but all things considered if we find ourselves in need of this sort of control again, I'd rather have it than not have it (or have to put out another hot fix to get something we could have put in in advance). In the mean time, the cost is negligible. So I'd rather put it in. And in main and release-7.3.

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 3dbea7d
• Duration 0:38:43
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: 3dbea7d
• Duration 0:47:36
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 3dbea7d
• Duration 0:50:34
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 3dbea7d
• Duration 0:55:23
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 3dbea7d
• Duration 0:57:19
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[spraza]

It's not my favorite PR but all things considered if we find ourselves in need of this sort of control again, I'd rather have it than not have it (or have to put out another hot fix to get something we could have put in in advance). In the mean time, the cost is negligible.

SGTM

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 3dbea7d
• Duration 2:18:35
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[gxglass]

It's not my favorite PR but all things considered if we find ourselves in need of this sort of control again, I'd rather have it than not have it (or have to put out another hot fix to get something we could have put in in advance). In the mean time, the cost is negligible.

SGTM

I am seeing joshua failures on this PR. Do you have an opinion about cherrypicking 7.3 PRs into 7.4 or vice-versa? I assume a main-based PR will be impossible due to file moves. I'll just retype this into a fresh 7.3 PR since I expect to be doing development on that in coming weeks.

[spraza]

It's not my favorite PR but all things considered if we find ourselves in need of this sort of control again, I'd rather have it than not have it (or have to put out another hot fix to get something we could have put in in advance). In the mean time, the cost is negligible.

SGTM

I am seeing joshua failures on this PR. Do you have an opinion about cherrypicking 7.3 PRs into 7.4 or vice-versa? I assume a main-based PR will be impossible due to file moves. I'll just retype this into a fresh 7.3 PR since I expect to be doing development on that in coming weeks.

Yeah, yesterday I also saw 5 failures in 10K runs in CI joshua. Not sure if those were related.

In terms of order, typically our workflow is: main -> 7.4 -> 7.3

Given we're working on 7.3 patches and testing those, starting with 7.3 also makes sense, so 7.3 -> 7.4 -> main. That's what I'm planning to do with the dd getTeamByServers O(1) change as well.

So no preference in terms of order. We just need to make sure all three branches have your changes.