DD 7.3: Batch drain relocationComplete to prevent fetchKeysComplete OOM by saintstack · Pull Request #12993 · apple/foundationdb

saintstack · 2026-04-14T23:44:29Z

The DDQueue choose loop processes one event per iteration. When
dataTransferComplete and other events are frequent, relocationComplete
processing is starved — fetchKeysComplete entries (erased only in the
relocationComplete handler) accumulate without bound. On one cluster this reached
33,593 entries causing OOM.

Extract completion processing into DDQueue::processRelocationComplete()
and batch-drain all ready relocationComplete events after the first
waitNext, capped at 1000 per iteration to avoid hogging the event loop.
This keeps fetchKeysComplete bounded regardless of event interleaving.

20260415-043932-stack_centos7_all_starvation-db5246251777f6a compressed=True data_size=51146319 duration=4673620 ended=99999 fail=1 fail_fast=10 max_runs=100000 pass=99998 priority=100 remaining=0:00:00 runtime=0:52:07 sanity=False started=100000 submitted=20260415-043932 timeout=5400 username=stack_centos7_all_starvation

The failure was RandomSeed="4014465418" SourceVersion="5bfd01720280600535c8bedf5c4bd2cbd4da453d" Time="1776228920" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/fast/GetMappedRange.toml"

Copilot

Pull request overview

This PR addresses a DDQueue event-loop starvation issue where relocationComplete processing can be delayed by other frequent events (e.g. dataTransferComplete), allowing fetchKeysComplete to grow without bound and potentially cause OOM. It refactors relocation-completion handling into a helper and batch-drains ready completion events to keep fetchKeysComplete bounded.

Changes:

Extracted relocation completion bookkeeping into DDQueue::processRelocationComplete().
Added a bounded batch-drain loop to process additional ready relocationComplete events (up to a fixed cap) in one choose-branch.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`fdbserver/include/fdbserver/DDRelocationQueue.h`	Declares the new `processRelocationComplete()` helper on `DDQueue`.
`fdbserver/DDRelocationQueue.actor.cpp`	Implements `processRelocationComplete()` and adds a bounded drain loop to reduce starvation/backlog.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

foundationdb-ci · 2026-04-15T00:24:30Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 0b7a802
Duration 0:39:47
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-15T00:24:30Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 0b7a802
Duration 0:39:47
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-15T00:24:30Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 0b7a802
Duration 0:39:47
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-15T06:26:46Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 3fe923d
Duration 0:55:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-15T06:26:46Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 3fe923d
Duration 0:55:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-15T06:26:46Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 3fe923d
Duration 0:55:54
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

gxglass · 2026-04-21T01:48:45Z

Initial review:

gxglass · 2026-04-24T03:36:59Z

On the PR description, On one cluster this reached 33,593 entries causing OOM, why would this cause O(4GB) increased memory consumption as seen? That's like 100K per entry.

saintstack · 2026-04-24T06:57:34Z

Address the AI feedback at #12993 (comment) It is good. Commit has a note on how this is hard to test:

    Testing: the batch drain only triggers when multiple relocation
    completions are ready simultaneously. Simulation clusters are too small
    (~10 servers, few shards) to produce completion bursts -- verified with
    BUGGIFY_WITH_PROB(1.0), 5s delay, and Attrition workloads (CycleTest,
    CycleWithKills, MoveKeysCycle) with -b on: only 1-2 completions occur
    per run. The original OOM was at a scale simulation cannot reach without
    OOM itself. The drain loop is a simple, reviewable pattern (while
    isReady, pop, process, cap at 1000). RelocationCompleteBatchDrain
    provides production observability when the path fires.

On the PR description, On one cluster this reached 33,593 entries causing OOM, why would this cause O(4GB) increased memory consumption as seen? That's like 100K per entry.

No debugger attached, no symbols, so can't say for sure. What we do see is rise in in-flight count correlates to rise in memory. Surviving fetchKeysComplete and associates are suspect. This PR attempts servicing of in-flight decrement and fetchKeysComplete freeing.

Simulation is a bust here. Will go elsewhere to make a test.

gxglass · 2026-04-24T07:27:05Z

On the PR description, On one cluster this reached 33,593 entries causing OOM, why would this cause O(4GB) increased memory consumption as seen? That's like 100K per entry.

No debugger attached, no symbols, so can't say for sure. What we do see is rise in in-flight count correlates to rise in memory. Surviving fetchKeysComplete and associates are suspect. This PR attempts servicing of in-flight decrement and fetchKeysComplete freeing.

This isn't in production yet but shortly you should be able to have your buddy munge through our TraceEvent number stream and see if it can produce a graph of these move entities as against RSS or Arena related stats, and maybe we can line these graphs up to our satisfaction. Also maybe your buddy can figure out why the all-in overhead might really be 100K per move event. That seems hard to believe but maybe there is a bunch of misc actor state data structures laying around waiting to be freed for a whole pile of nearly dead actors who are waiting to be reaped, so to speak. (Sounds like an Oscar party.)

foundationdb-ci · 2026-04-24T07:29:05Z

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Commit ID: 2d7c0c4
Duration 0:49:00
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-24T08:04:36Z

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Commit ID: 2886089
Duration 1:26:00
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-24T08:11:26Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 2d7c0c4
Duration 1:31:23
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-24T09:39:19Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 2886089
Duration 3:00:40
Result: ❌ FAILED
Error: Build has timed out.
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-24T09:39:25Z

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Commit ID: 2886089
Duration 3:00:45
Result: ❌ FAILED
Error: Build has timed out.
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-24T09:40:44Z

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Commit ID: 2d7c0c4
Duration 3:00:40
Result: ❌ FAILED
Error: Build has timed out.
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-24T18:21:30Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 2d7c0c4
Duration 0:48:59
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-24T18:25:23Z

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Commit ID: 2d7c0c4
Duration 0:52:52
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-24T20:03:02Z

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Commit ID: 2d7c0c4
Duration 2:30:32
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

saintstack · 2026-04-27T23:53:33Z

Make the change smaller. Still working on a test.

foundationdb-ci · 2026-04-28T00:34:41Z

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Commit ID: 3f6bd3d
Duration 0:41:38
Result: ❌ FAILED
Error: Error while executing command: ctest -j ${NPROC} --no-compress-output -T test --output-on-failure. Reason: exit status 8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-28T00:58:58Z

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Commit ID: 3f6bd3d
Duration 1:05:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-28T01:00:33Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 3f6bd3d
Duration 1:07:31
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2026-04-28T05:39:24Z

Added a counter that trips every time we batch. Add a unit test. Now trying to reproduce the scenario in n integration test. Will report back.

The DDQueue choose loop processes one event per iteration. When dataTransferComplete and other events are frequent, relocationComplete processing is starved -- fetchKeysComplete entries (erased only in the relocationComplete handler) accumulate unboundedly. On p67 this reached 33,593 entries causing OOM. Fix: after processing the first relocationComplete via waitNext, batch drain all remaining ready completions (capped at 1000) before returning to the choose loop. Each drained completion gets the same processing as the original: processRelocationComplete + delay(0) -> rangesComplete for launchQueuedWork. The delay(0) -> rangesComplete -> keysToLaunchFrom path is preserved exactly as before. Changes: - Extract processRelocationComplete() for reuse in drain loop - Store FutureStream as state variable for isReady()/pop() access - Add RelocationCompleteBatchDrain trace event for observability - Add /dd/queue/relocationComplete/batchDrained SimpleCounter metric - Add unit test that sends 100 completions in a burst and verifies all are drained, fetchKeysComplete is emptied, and the counter increments

saintstack · 2026-04-28T05:40:02Z

Rebase

foundationdb-ci · 2026-04-28T05:43:24Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: cf2f20f
Duration 0:04:42
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (cf2f20fd2d4ce267230d0652d3b1ebd2b336f5d7) for primary source and source version cf2f20fd2d4ce267230d0652d3b1ebd2b336f5d7
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-28T05:43:27Z

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Commit ID: cf2f20f
Duration 0:04:49
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (cf2f20fd2d4ce267230d0652d3b1ebd2b336f5d7) for primary source and source version cf2f20fd2d4ce267230d0652d3b1ebd2b336f5d7
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-28T05:43:29Z

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Commit ID: cf2f20f
Duration 0:04:49
Result: ❌ FAILED
Error: git checkout failed with exit status 128: fatal: unable to read tree (cf2f20fd2d4ce267230d0652d3b1ebd2b336f5d7) for primary source and source version cf2f20fd2d4ce267230d0652d3b1ebd2b336f5d7
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-28T06:44:33Z

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Commit ID: 8451324
Duration 1:04:27
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-28T06:44:48Z

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Commit ID: f0a78b1
Duration 1:03:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2026-04-28T06:48:24Z

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Commit ID: f0a78b1
Duration 1:06:53
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-28T06:49:34Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: f0a78b1
Duration 1:08:03
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-28T06:49:47Z

Result of foundationdb-pr-73 on Linux RHEL 9

Commit ID: 8451324
Duration 1:09:45
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-04-28T06:49:49Z

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Commit ID: 8451324
Duration 1:09:41
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-31T03:51:54Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: f0a78b1
Duration 0:07:59
Result: ❌ FAILED
Error: Error while executing command: ninja -v -C build_output -j ${NPROC} all packages strip_targets. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2026-05-31T04:16:29Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: f0a78b1
Duration 0:33:04
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2026-07-08T04:02:06Z

Closing in favor of a #13660 (#13660 includes the helpful feedback gleaned here). The heat went off this fix -- at least for the 7.3 timeframe. Still seems like a good idea.

saintstack requested a review from Copilot April 14, 2026 23:44

Copilot started reviewing on behalf of saintstack April 14, 2026 23:45 View session

Copilot AI reviewed Apr 14, 2026

View reviewed changes

Comment thread fdbserver/DDRelocationQueue.actor.cpp Outdated

Comment thread fdbserver/DDRelocationQueue.actor.cpp Outdated

saintstack force-pushed the dd_7.3_starvation branch from 3fe923d to 2886089 Compare April 24, 2026 06:38

saintstack closed this Apr 24, 2026

saintstack reopened this Apr 24, 2026

saintstack mentioned this pull request Apr 25, 2026

Throttle isRestore data moves through canLaunchSrc on DD restart #12958

Closed

saintstack force-pushed the dd_7.3_starvation branch from 2d7c0c4 to 3f6bd3d Compare April 27, 2026 23:52

saintstack force-pushed the dd_7.3_starvation branch from 3f6bd3d to cf2f20f Compare April 28, 2026 05:38

saintstack force-pushed the dd_7.3_starvation branch from cf2f20f to 8451324 Compare April 28, 2026 05:39

Formatting

f0a78b1

saintstack mentioned this pull request Jul 8, 2026

DDQueue: batch drain relocationComplete to bound fetchKeysComplete #13660

Merged

saintstack closed this Jul 8, 2026

Uh oh!

Conversation

saintstack commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

foundationdb-ci commented Apr 15, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 15, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 15, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 15, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 15, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 15, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

gxglass commented Apr 21, 2026

Uh oh!

gxglass commented Apr 24, 2026

Uh oh!

saintstack commented Apr 24, 2026

Uh oh!

gxglass commented Apr 24, 2026

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 24, 2026

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Uh oh!

saintstack commented Apr 27, 2026

Uh oh!

foundationdb-ci commented Apr 28, 2026

Result of foundationdb-pr-clang-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 28, 2026

Result of foundationdb-pr-cluster-tests-73 on Linux RHEL 9

Uh oh!

foundationdb-ci commented Apr 28, 2026

Result of foundationdb-pr-73 on Linux RHEL 9

Uh oh!

saintstack commented Apr 28, 2026

Uh oh!

saintstack commented Apr 28, 2026

Uh oh!

saintstack commented Apr 14, 2026 •

edited

Loading