Add DD init visibility, metrics retries, shard tracking, scan progress, and team collection logging (#12913)

* Add DD init and team collection logging for diagnosing slow startups

When SHARD_ENCODE_LOCATION_METADATA=true we take new codepaths often
opaque. Add logging.

For example, DD init hung for 14-16 minutes with zero visibility into
what was stuck. The only clue was a gap between DDInitUpdatedReplicaKeys
and DDInitGotInitialDD trace events.  Diagnosing the root cause required
extensive log splunking of SS metrics to determine that a single
getRange(dataMoveKeys) read was queued on an overloaded storage server.

DDTxnProcessor.actor.cpp:
- Log elapsed time for the server list + data move read transaction
  (DDInitServerListAndDataMoveReadComplete) with NumDataMoves, NumServers
- Log elapsed time for the keyServer scan (DDInitKeyServerScanComplete)
  with NumShards
- Warn when getRange(dataMoveKeys) takes >5 seconds
  (DDInitSlowDataMoveRead)

DataDistribution.actor.cpp:
- Add NumShards and NumServers to DDInitGotInitialDD
- Promote DDInitFoundDataMove from SevDebug to SevInfo so individual
  data moves are visible in production logs
- Add DDInitResumedDataMoves summary event with ValidMoves,
  CancelledMoves, EmptyMoves counts and elapsed time

DDTeamCollection.actor.cpp:
- Add Reason and Address details to UndesiredStorageServer trace events
  to distinguish version lag, same-address, wrong-class, and exclusion
  causes without needing to correlate with other log lines

* Revert DDInitFoundDataMove to SevDebug to avoid log spam with many data moves

* Add DD startup visibility: metrics retries, shard tracking, scan progress

Additional logging to address DD operational opacity during startup,
based on past incidents where DD hung with no visibility into the cause.

NativeAPI.actor.cpp:
- Log WaitStorageMetricsRetrying every 60s when waitStorageMetrics is
  stuck retrying wrong_shard_server or all_alternatives_failed, with
  the key range, retry count, and elapsed time. Previously these retries
  were silent (SevDebug only), making it impossible to identify which
  shard was stuck or that retries were even happening.

DDShardTracker.actor.cpp:
- Log TrackInitialShardsComplete after shard tracker setup with count
- Log TrackInitialShardsMetricsComplete after changeSizes() finishes
  with elapsed time. changeSizes() waits for ALL shards to report
  metrics via getFirstSize/waitStorageMetrics -- if any shard metrics
  never arrive, this hangs silently.

DDTxnProcessor.actor.cpp:
- Log DDInitKeyServerScanProgress every 30s during the multi-transaction
  keyServer scan with current beginKey, batch count, shards scanned,
  and elapsed time. With 255K shards this scan requires many transactions
  and a stuck one was previously invisible.

DataDistribution.actor.cpp:
- Log DDInitComplete with elapsed time after DataDistributor::init()
  finishes, providing a single event showing total init duration.

* Formatting

* Try to get past gcc 13 compile faillures

* Try and fix

FAILED: flow/CMakeFiles/flow.dir/ActorCollection.actor.g.cpp.o
/usr/local/bin/ccache /usr/local/bin/clang++ -DBOOST_ATOMIC_NO_LIB -DBOOST_CONTEXT_NO_LIB -DBOOST_ERROR_CODE_HEADER_ONLY -DBOOST_FILESYSTEM_NO_LIB -DBOOST_IOSTREAMS_NO_LIB -DBOOST_SERIALIZATION_NO_LIB -DBOOST_SYSTEM_NO_DEPRECATED -DBOOST_SYSTEM_NO_LIB -DNO_INTELLISENSE -DWITH_FOLLY_MEMCPY -DCOMPILATION_UNIT=/codebuild/output/src2645037596/src/github.com/apple/foundationdb/build_output/flow/ActorCollection.actor.g.cpp -I/codebuild/output/src2645037596/src/github.com/apple/foundationdb/flow/include -I/codebuild/output/src2645037596/src/github.com/apple/foundationdb/build_output/flow/include -I/codebuild/output/src2645037596/src/github.com/apple/foundationdb/contrib/stacktrace/include -I/codebuild/output/src2645037596/src/github.com/apple/foundationdb/contrib/fmt-8.1.1/include -I/codebuild/output/src2645037596/src/github.com/apple/foundationdb/contrib/SimpleOpt/include -I/codebuild/output/src2645037596/src/github.com/apple/foundationdb/contrib/crc32/include -I/codebuild/output/src2645037596/src/github.com/apple/foundationdb/contrib/folly_memcpy -isystem /opt/boost_1_78_0_clang/include -stdlib=libc++ -O3 -DNDEBUG -std=gnu++20 -DCMAKE_BUILD -fno-omit-frame-pointer -gdwarf-4 -ggdb1 -gz -Wall -Wextra -Wredundant-move -Wpessimizing-move -Woverloaded-virtual -Wshift-sign-overflow -Wno-sign-compare -Wno-undefined-var-template -Wno-unknown-warning-option -Wno-unused-parameter -Wno-constant-logical-operand -Wno-deprecated-copy -Wno-delete-non-abstract-non-virtual-dtor -Wno-range-loop-construct -Wno-reorder-ctor -Wno-unused-command-line-argument -Wno-ambiguous-reversed-operator -Wno-register -Werror -Wno-error=format -Wunused-variable -Wno-deprecated -fvisibility=hidden -Wreturn-type -fPIC -march=armv8.2-a+crc+simd -DHAVE_OPENSSL -Wno-unused-function -MD -MT flow/CMakeFiles/flow.dir/ActorCollection.actor.g.cpp.o -MF flow/CMakeFiles/flow.dir/ActorCollection.actor.g.cpp.o.d -o flow/CMakeFiles/flow.dir/ActorCollection.actor.g.cpp.o -c /codebuild/output/src2645037596/src/github.com/apple/foundationdb/build_output/flow/ActorCollection.actor.g.cpp
In file included from /codebuild/output/src2645037596/src/github.com/apple/foundationdb/flow/ActorCollection.actor.cpp:21:
In file included from /codebuild/output/src2645037596/src/github.com/apple/foundationdb/flow/include/flow/ActorCollection.h:25:
In file included from /codebuild/output/src2645037596/src/github.com/apple/foundationdb/flow/include/flow/flow.h:23:
In file included from /codebuild/output/src2645037596/src/github.com/apple/foundationdb/flow/include/flow/Arena.h:37:
In file included from /opt/boost_1_78_0_clang/include/boost/functional/hash.hpp:6:
/opt/boost_1_78_0_clang/include/boost/container_hash/hash.hpp:132:33: error: no template named 'unary_function' in namespace 'std'; did you mean '__unary_function'?
  132 |         struct hash_base : std::unary_function<T, std::size_t> {};
      |                            ~~~~~^
/usr/local/bin/../include/c++/v1/__functional/unary_function.h:46:1: note: '__unary_function' declared here
   46 | using __unary_function = __unary_function_keep_layout_base<_Arg, _Result>;
      | ^
1 error generated.

* Address PR review feedback

- NativeAPI.actor.cpp: Move retry logging outside the error-type if block
  so all errors get keys/elapsed/retries details. Use severity upgrade
  (SevDebug -> SevWarn after 60s) on the existing WaitStorageMetricsHandleError
  event instead of a separate event name.
- DataDistribution.actor.cpp: Add periodic progress logging (every 30s) in
  resumeFromDataMoves loop so operators can watch counts go up during long
  data move recovery.
- CompileBoost.cmake: Remove BOOST_NO_CXX98_FUNCTION_BASE since 7.3 CI is
  broken independently of this change.

* More definition of what comprises DD startup

* Add explanation as to why DDStarted is where it is

* Rename traces I added to have DDInit as prefix instead of just DD

* s/DD*/DDInit*/ for DD init loggings query

* Log DD exit reason as DDExiting at SevWarn

DD exits (e.g. movekeys_conflict) were invisible because
reportErrorsExcept suppresses logging for "normal" DD errors.
Add DDExiting trace event at SevWarn with error and code so
every DD death is visible in trace logs.

* Fix comment

---------

Co-authored-by: michael stack <stack@duboce.com>

Author: saintstack

fdbclient/NativeAPI.actor.cpp

@@ -8022,6 +8022,9 @@ ACTOR Future<std::pair<Optional<StorageMetrics>, int>> waitStorageMetrics(
     int expectedShardCount,
     Optional<Reference<TransactionState>> trState) {
 	state Span span("NAPI:WaitStorageMetrics"_loc, generateSpanID(cx->transactionTracingSample));
+	state double startTime = now();
+	state double lastLogTime = 0;
+	state int retryCount = 0;
 	loop {
 		if (trState.present()) {
 			wait(trState.get()->startTransaction());
@@ -8065,7 +8068,14 @@ ACTOR Future<std::pair<Optional<StorageMetrics>, int>> waitStorageMetrics(
 				return std::make_pair(res, -1);
 			}
 		} catch (Error& e) {
-			TraceEvent(SevDebug, "WaitStorageMetricsHandleError").error(e);
+			retryCount++;
+			// Upgrade from SevDebug to SevWarn after 60 seconds of retrying
+			Severity sev = (now() - startTime > 60.0) ? SevWarn : SevDebug;
+			TraceEvent(sev, "WaitStorageMetricsHandleError")
+			    .error(e)
+			    .detail("Keys", keys)
+			    .detail("Elapsed", now() - startTime)
+			    .detail("Retries", retryCount);
 			if (e.code() == error_code_wrong_shard_server || e.code() == error_code_all_alternatives_failed) {
 				cx->invalidateCache(tenantInfo.prefix, keys);
 				wait(delay(CLIENT_KNOBS->WRONG_SHARD_SERVER_DELAY, TaskPriority::DataDistribution));

fdbserver/DDShardTracker.actor.cpp

@@ -1313,9 +1313,23 @@ ACTOR Future<Void> trackInitialShards(DataDistributionTracker* self, Reference<I
 		wait(yield(TaskPriority::DataDistribution));
 	}
 
+	TraceEvent("TrackInitialShardsComplete", self->distributorId).detail("ShardsTracked", s);
+
+	state double changeSizesStart = now();
 	Future<Void> initialSize = changeSizes(self, KeyRangeRef(allKeys.begin, allKeys.end), 0);
 	self->readyToStart.send(Void());
 	wait(initialSize);
+
+	TraceEvent("TrackInitialShardsMetricsComplete", self->distributorId)
+	    .detail("ElapsedSeconds", now() - changeSizesStart);
+
+	// DDInitDone bookends DDInitRunning — marks DD fully operational. Uses DD* prefix so
+	// the full startup sequence can be queried with Type="DD*" in trace logs.
+	// Ideally this would be in DataDistribution.actor.cpp but dataDistribution() has no
+	// callback after trackInitialShards completes — it launches all actors and waits forever.
+	// Adding a new promise to signal completion would work but risks disturbing current operation.
+	TraceEvent("DDInitDone", self->distributorId);
+
 	self->maxShardSizeUpdater = updateMaxShardSize(self->dbSizeEstimate, self->maxShardSize);
 
 	return Void();

fdbserver/DDTeamCollection.actor.cpp

@@ -1264,7 +1264,17 @@ class DDTeamCollectionImpl {
 		state Future<Void> storageMetadataTracker = self->updateStorageMetadata(server);
 		try {
 			loop {
-				status.isUndesired = !self->disableFailingLaggingServers.get() && server->ssVersionTooFarBehind.get();
+				{
+					bool versionLagUndesired =
+					    !self->disableFailingLaggingServers.get() && server->ssVersionTooFarBehind.get();
+					if (versionLagUndesired && !status.isUndesired) {
+						TraceEvent(SevWarn, "UndesiredStorageServer", self->distributorId)
+						    .detail("Server", server->getId())
+						    .detail("Address", server->getLastKnownInterface().address())
+						    .detail("Reason", "VersionLag");
+					}
+					status.isUndesired = versionLagUndesired;
+				}
 				status.isWrongConfiguration = false;
 				status.isWiggling = false;
 				hasWrongDC = !self->isCorrectDC(*server);
@@ -1298,6 +1308,7 @@ class DDTeamCollectionImpl {
 								TraceEvent(SevWarn, "UndesiredStorageServer", self->distributorId)
 								    .detail("Server", server->getId())
 								    .detail("Address", server->getLastKnownInterface().address())
+								    .detail("Reason", "SameAddress")
 								    .detail("OtherServer", i.second->getId())
 								    .detail("NumShards",
 								            self->shardsAffectedByTeamFailure->getNumberOfShards(server->getId()))
@@ -1323,6 +1334,8 @@ class DDTeamCollectionImpl {
 					if (self->optimalTeamCount > 0) {
 						TraceEvent(SevWarn, "UndesiredStorageServer", self->distributorId)
 						    .detail("Server", server->getId())
+						    .detail("Address", server->getLastKnownInterface().address())
+						    .detail("Reason", "WrongMachineClass")
 						    .detail("OptimalTeamCount", self->optimalTeamCount)
 						    .detail("Fitness", server->getLastKnownClass().machineClassFitness(ProcessClass::Storage));
 						status.isUndesired = true;
@@ -1399,9 +1412,16 @@ class DDTeamCollectionImpl {
 				}
 
 				if (worstStatus != DDTeamCollection::Status::NONE) {
+					const char* exclusionType = worstStatus == DDTeamCollection::Status::WIGGLING   ? "Wiggling"
+					                            : worstStatus == DDTeamCollection::Status::FAILED   ? "Failed"
+					                            : worstStatus == DDTeamCollection::Status::EXCLUDED ? "Excluded"
+					                                                                                : "Unknown";
 					TraceEvent(SevWarn, "UndesiredStorageServer", self->distributorId)
 					    .detail("Server", server->getId())
-					    .detail("Excluded", worstAddr.toString());
+					    .detail("Address", server->getLastKnownInterface().address())
+					    .detail("Reason", "Excluded")
+					    .detail("ExclusionType", exclusionType)
+					    .detail("ExcludedAddress", worstAddr.toString());
 					status.isUndesired = true;
 					status.isWrongConfiguration = true;
 

fdbserver/DDTxnProcessor.actor.cpp

@@ -255,6 +255,8 @@ class DDTxnProcessorImpl {
 		CODE_PROBE((bool)skipDDModeCheck, "DD Mode won't prevent read initial data distribution.");
 		// Get the server list in its own try/catch block since it modifies result.  We don't want a subsequent failure
 		// causing entries to be duplicated
+		// Phase 1: Single transaction to read server list and all persisted data moves
+		state double serverListAndDataMoveReadStart = now();
 		loop {
 			numDataMoves = 0;
 			server_dc.clear();
@@ -312,7 +314,12 @@ class DDTxnProcessorImpl {
 					}
 				}
 
+				state double dataMoveReadStart = now();
 				RangeResult dms = wait(tr.getRange(dataMoveKeys, CLIENT_KNOBS->TOO_MANY));
+				if (now() - dataMoveReadStart > 5.0) {
+					TraceEvent(SevWarn, "DDInitSlowDataMoveRead", distributorId)
+					    .detail("ElapsedSeconds", now() - dataMoveReadStart);
+				}
 				ASSERT(!dms.more && dms.size() < CLIENT_KNOBS->TOO_MANY);
 				// For each data move, find out the src or dst servers are in primary or remote DC.
 				for (int i = 0; i < dms.size(); ++i) {
@@ -360,6 +367,11 @@ class DDTxnProcessorImpl {
 
 				succeeded = true;
 
+				TraceEvent("DDInitServerListAndDataMoveReadComplete", distributorId)
+				    .detail("NumDataMoves", numDataMoves)
+				    .detail("NumServers", result->allServers.size())
+				    .detail("ElapsedSeconds", now() - serverListAndDataMoveReadStart);
+
 				break;
 			} catch (Error& e) {
 				TraceEvent("GetInitialTeamsRetry", distributorId).error(e);
@@ -371,6 +383,10 @@ class DDTxnProcessorImpl {
 
 		// If keyServers is too large to read in a single transaction, then we will have to break this process up into
 		// multiple transactions. In that case, each iteration should begin where the previous left off
+		// Scan keyServers in batches to build the shard map
+		state double keyServerScanStart = now();
+		state double lastScanLogTime = now();
+		state int scanBatchCount = 0;
 		while (beginKey < allKeys.end) {
 			CODE_PROBE(beginKey > allKeys.begin, "Multi-transactional getInitialDataDistribution");
 			loop {
@@ -457,6 +473,15 @@ class DDTxnProcessorImpl {
 
 					ASSERT_GT(keyServers.size(), 0);
 					beginKey = keyServers.end()[-1].key;
+					scanBatchCount++;
+					if (now() - lastScanLogTime >= 30.0) {
+						lastScanLogTime = now();
+						TraceEvent("DDInitKeyServerScanProgress", distributorId)
+						    .detail("BeginKey", beginKey)
+						    .detail("Batches", scanBatchCount)
+						    .detail("ShardsScanned", result->shards.size())
+						    .detail("ElapsedSeconds", now() - keyServerScanStart);
+					}
 					break;
 				} catch (Error& e) {
 					TraceEvent("GetInitialTeamsKeyServersRetry", distributorId).error(e);
@@ -473,6 +498,10 @@ class DDTxnProcessorImpl {
 		// a dummy shard at the end with no keys or servers makes life easier for trackInitialShards()
 		result->shards.push_back(DDShardInfo(allKeys.end));
 
+		TraceEvent("DDInitKeyServerScanComplete", distributorId)
+		    .detail("NumShards", result->shards.size())
+		    .detail("ElapsedSeconds", now() - keyServerScanStart);
+
 		if (SERVER_KNOBS->SHARD_ENCODE_LOCATION_METADATA && numDataMoves > 0) {
 			for (int shard = 0; shard < result->shards.size() - 1; ++shard) {
 				const DDShardInfo& iShard = result->shards[shard];

fdbserver/DataDistribution.actor.cpp

@@ -505,6 +505,31 @@ struct DataDistributor : NonCopyable, ReferenceCounted<DataDistributor> {
 	// Initialize the required internal states of DataDistributor from system metadata. It's necessary before
 	// DataDistributor start working. Doesn't include initialization of optional components, like TenantCache, DDQueue,
 	// Tracker, TeamCollection. The components should call its own ::init methods.
+	//
+	// DD Startup Progress (trace events in order):
+	//   DDInitRunning                     - DD process recruited and starting init
+	//   DDInitTakingMoveKeysLock          - Acquiring move keys lock
+	//   DDInitTookMoveKeysLock            - Lock acquired
+	//   DDInitGotConfiguration            - Database configuration loaded
+	//   DDInitUpdatedReplicaKeys          - Replica keys updated
+	//   DDInitSlowDataMoveRead            - (SevWarn) dataMoveKeys read taking >5s
+	//   DDInitServerListAndDataMoveReadComplete - Server list + data moves read: NumDataMoves, NumServers,
+	//   ElapsedSeconds
+	//   DDInitKeyServerScanProgress       - (every 30s) keyServer scan: BeginKey, Batches, ShardsScanned
+	//   DDInitKeyServerScanComplete       - keyServer scan done: NumShards, ElapsedSeconds
+	//   DDInitGotInitialDD                - Init data loaded: NumShards, NumServers
+	//   DDInitDataLoaded                  - Init data loaded, ElapsedSeconds (does NOT mean DD is fully operational)
+	//
+	// After init(), the following startup events fire from other components:
+	//   DDInitResumeDataMovesProgress     - (every 30s) data move resume: ValidMoves, CancelledMoves, EmptyMoves
+	//   DDInitResumedDataMoves            - Data move resume complete with counts
+	//   TrackInitialShards                - Shard tracker setup started with InitialShardCount
+	//   TrackInitialShardsComplete        - Shard trackers created: ShardsTracked
+	//   DDTrackerStarting                 - Teams ready (fires from DDTeamCollection after readyToStart + delay)
+	//   TrackInitialShardsMetricsComplete - All shard metrics received: ElapsedSeconds
+	//                                       WaitStorageMetricsHandleError may fire (SevWarn after 60s) if a
+	//                                       shard's metrics read is stuck retrying: Keys, Retries
+	//   DDInitDone                        - DD is fully operational with all shard sizes loaded
 	ACTOR static Future<Void> init(Reference<DataDistributor> self) {
 		loop {
 			TraceEvent("DDInitTakingMoveKeysLock", self->ddId).log();
@@ -554,13 +579,17 @@ struct DataDistributor : NonCopyable, ReferenceCounted<DataDistributor> {
 				    .detail("E", self->initData->shards.end()[-1].key)
 				    .detail("Src", describe(self->initData->shards.end()[-2].primarySrc))
 				    .detail("Dest", describe(self->initData->shards.end()[-2].primaryDest))
+				    .detail("NumShards", self->initData->shards.size())
+				    .detail("NumServers", self->initData->allServers.size())
 				    .trackLatest(self->initialDDEventHolder->trackingKey);
 			} else {
 				TraceEvent("DDInitGotInitialDD", self->ddId)
 				    .detail("B", "")
 				    .detail("E", "")
 				    .detail("Src", "[no items]")
 				    .detail("Dest", "[no items]")
+				    .detail("NumShards", self->initData->shards.size())
+				    .detail("NumServers", self->initData->allServers.size())
 				    .trackLatest(self->initialDDEventHolder->trackingKey);
 			}
 
@@ -752,13 +781,19 @@ struct DataDistributor : NonCopyable, ReferenceCounted<DataDistributor> {
 	// TODO: unit test needed
 	ACTOR static Future<Void> resumeFromDataMoves(Reference<DataDistributor> self, Future<Void> readyToStart) {
 		state KeyRangeMap<std::shared_ptr<DataMove>>::iterator it = self->initData->dataMoveMap.ranges().begin();
+		state int validMoves = 0;
+		state int cancelledMoves = 0;
+		state int emptyMoves = 0;
+		state double resumeStart = now();
+		state double lastLogTime = now();
 
 		wait(readyToStart);
 
 		for (; it != self->initData->dataMoveMap.ranges().end(); ++it) {
 			const DataMoveMetaData& meta = it.value()->meta;
 			if (meta.ranges.empty()) {
 				TraceEvent(SevInfo, "EmptyDataMoveRange", self->ddId).detail("DataMoveMetaData", meta.toString());
+				emptyMoves++;
 				continue;
 			}
 			if (it.value()->isCancelled() || (it.value()->valid && !SERVER_KNOBS->SHARD_ENCODE_LOCATION_METADATA)) {
@@ -767,6 +802,7 @@ struct DataDistributor : NonCopyable, ReferenceCounted<DataDistributor> {
 				rs.cancelled = true;
 				self->relocationProducer.send(rs);
 				TraceEvent("DDInitScheduledCancelDataMove", self->ddId).detail("DataMove", meta.toString());
+				cancelledMoves++;
 			} else if (it.value()->valid) {
 				TraceEvent(SevDebug, "DDInitFoundDataMove", self->ddId).detail("DataMove", meta.toString());
 				ASSERT(meta.ranges.front() == it.range());
@@ -790,9 +826,24 @@ struct DataDistributor : NonCopyable, ReferenceCounted<DataDistributor> {
 				self->shardsAffectedByTeamFailure->moveShard(rs.keys, teams);
 				self->relocationProducer.send(rs);
 				wait(yield(TaskPriority::DataDistribution));
+				validMoves++;
+			}
+			if (now() - lastLogTime >= 30.0) {
+				lastLogTime = now();
+				TraceEvent("DDInitResumeDataMovesProgress", self->ddId)
+				    .detail("ValidMoves", validMoves)
+				    .detail("CancelledMoves", cancelledMoves)
+				    .detail("EmptyMoves", emptyMoves)
+				    .detail("ElapsedSeconds", now() - resumeStart);
 			}
 		}
 
+		TraceEvent("DDInitResumedDataMoves", self->ddId)
+		    .detail("ValidMoves", validMoves)
+		    .detail("CancelledMoves", cancelledMoves)
+		    .detail("EmptyMoves", emptyMoves)
+		    .detail("ElapsedSeconds", now() - resumeStart);
+
 		// Trigger background cleanup for datamove tombstones
 		self->addActor.send((self->removeDataMoveTombstoneBackground(self)));
 
@@ -920,6 +971,7 @@ ACTOR Future<Void> dataDistribution(Reference<DataDistributor> self,
 
 	loop {
 		trackerCancelled = false;
+		state double ddStartTime = now();
 		// whether all initial shard are tracked
 		self->initialized = Promise<Void>();
 
@@ -930,6 +982,8 @@ ACTOR Future<Void> dataDistribution(Reference<DataDistributor> self,
 		try {
 			wait(DataDistributor::init(self));
 
+			TraceEvent("DDInitDataLoaded", self->ddId).detail("ElapsedSeconds", now() - ddStartTime);
+
 			// When/If this assertion fails, Evan owes Ben a pat on the back for his foresight
 			ASSERT(self->configuration.storageTeamSize > 0);
 
@@ -1115,6 +1169,10 @@ ACTOR Future<Void> dataDistribution(Reference<DataDistributor> self,
 			trackerCancelled = true;
 			state Error err = e;
 			TraceEvent("DataDistributorDestroyTeamCollections").error(e);
+			// Log every DD exit with the reason. movekeys_conflict is the most common
+			// non-kill cause — it's a normal internal restart, but was previously
+			// invisible because reportErrorsExcept suppresses logging for "normal" DD errors.
+			TraceEvent(SevWarn, "DDExiting", self->ddId).error(e).detail("ErrorCode", e.code());
 			state std::vector<UID> teamForDroppedRange;
 			if (removeFailedServer.getFuture().isReady() && !removeFailedServer.getFuture().isError()) {
 				// Choose a random healthy team to host the to-be-dropped range.
@@ -3298,6 +3356,9 @@ ACTOR Future<Void> dataDistributor(DataDistributorInterface di, Reference<AsyncV
 
 	try {
 		TraceEvent("DataDistributorRunning", di.id());
+		// DDInitRunning duplicates the above with DDInit* prefix so the full startup sequence
+		// can be queried with Type="DDInit*" in trace logs
+		TraceEvent("DDInitRunning", di.id());
 		self->addActor.send(waitFailureServer(di.waitFailure.getFuture()));
 		self->addActor.send(cacheServerWatcher(&cx));
 		state Future<Void> distributor = reportErrorsExcept(

flow/include/flow/IndexedSet.h

@@ -344,11 +344,11 @@ struct IndexedSet {
 	// direction 0 = left, 1 = right
 	template <int direction>
 	static void moveIterator(Node const*& node) {
-		moveIteratorImpl<direction, true>(node);
+		IndexedSet::template moveIteratorImpl<direction, true>(node);
 	}
 	template <int direction>
 	static void moveIterator(Node*& node) {
-		moveIteratorImpl<direction, false>(node);
+		IndexedSet::template moveIteratorImpl<direction, false>(node);
 	}
 
 public: // but testonly