HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up. by ArafatKhan2198 · Pull Request #10617 · apache/ozone

ArafatKhan2198 · 2026-06-26T08:26:19Z

What changes were proposed in this pull request?

When an SCM follower restarts in an HA cluster, it used to start talking to datanodes right away, even while it was still catching up on the Ratis log.

That caused problems:

Datanodes report containers the follower doesn’t know about yet → CONTAINER_NOT_FOUND
Or the follower tries to update container state and fails → NotLeaderException
In both cases, replica info gets dropped
If that SCM later becomes leader, containers can show missing or wrong replicas

The fix:

Don’t start the datanode server in HA mode during normal SCM startup.
Wait until catch-up is done, then start it from SCMStateMachine.
Don’t let followers write container state changes during report handling — only the leader should.

Why: Replica locations are rebuilt from datanode reports. Those reports must only be processed after the SCM has replayed all committed Ratis entries.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14989

How was this patch tested?

Integration tests

TestSCMFollowerCatchupWithContainerReport -

testFollowerCatchupAfterContainerClose — close-while-down (HDDS-14989 scenario)
testFollowerCatchupAfterContainerCreate — create-while-down (CONTAINER_NOT_FOUND scenario)
testFollowerCatchupOnIdleCluster — idle cluster edge case

Manual test (docker-compose `ozone-ha`)

Environment: hadoop-ozone/dist/target/ozone-2.3.0-SNAPSHOT/compose/ozone-ha

Config: RF=3, 3 datanodes, hdds.container.report.interval=1h, ozone.scm.container.size=1GB

Procedure (same for with/without fix):

Start cluster: OZONE_REPLICATION_FACTOR=3 docker compose up -d --scale datanode=3
Write 50 × 1MB keys to vol1/buck1 (containers 1–3)
Stop follower scm3
Close containers 1, 2, 3
Write 50 × 1MB keys to vol1/buck2 (creates containers 4, 5, 6 while scm3 is down)
Restart scm3
Transfer SCM leadership to scm3
Inspect scm3 logs and ozone admin container info for containers 4–6

Without the fix:

06:57:58.622  ScmDatanodeProtocol RPC server ... listening at /0.0.0.0:9861
06:57:58.837  CONTAINER_NOT_FOUND for Container #4
06:57:58.837  CONTAINER_NOT_FOUND for Container #5
06:57:58.837  CONTAINER_NOT_FOUND for Container #6
(6 errors total — 2 datanodes × 3 containers)

After leadership transfer, containers 4–6 had 1 replica each (expected 3).

With the fix:

07:24:28.377  Follower caught up with leader: lastAppliedIndex=49, leaderCommit=49
07:24:28.378  ScmDatanodeProtocol RPC server ... listening at /0.0.0.0:9861

CONTAINER_NOT_FOUND on scm3: 0
After leadership transfer, containers 4–6 each had 3 replicas from all datanodes

@afterall

…ring cluster Reduce test execution time by ~60% (2m28s → ~1m) via three targeted optimizations: 1. Share one cluster across all three test methods (@BeforeAll/@afterall instead of @BeforeEach/@AfterEach). This eliminates 2 of 3 cluster bring-up cycles, which account for ~80-90% of the test duration. Each test uses its own volume/bucket name to avoid collisions on the shared cluster. 2. Lower the datanode heartbeat interval to 1 second in the config. This speeds safe-mode exit at startup and replica re-reporting after the deferred DN-server start, both happening in ~1s instead of default timers. 3. Tighten the waitFor poll interval from 1000ms to 250ms. This allows the test to notice when async conditions (safe-mode exit, leadership transfer) are met sooner, without changing the timeout ceilings. The test essence is fully preserved: 3 SCMs, 3 DNs, 5-minute report interval, down→mutate→restart→catch-up→promote sequence all unchanged. Only the test harness (cluster setup and polling) was optimized. The regression still catches empty replicas on the unfixed code and confirms full replicas on the fix. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

ivandika3 · 2026-06-26T13:05:32Z

If some of the implementation is taken from #10059 , don't forget to add @xichen01 as the co-author.

sumitagrawl

@ArafatKhan2198 Thanks for working over this, given few comments

isStateMachineReady is the condition of the StateMachineReadyRule safe-mode rule, so it must be set once the SCM is caught up independent of the deferred datanode-server start. Reusing it as the server-start guard coupled a safe-mode-critical flag to the start path and risked hanging safe mode if the catch-up check misfired. Restore isStateMachineReady to its original semantics (set on leader change and on term catch-up in notifyTermIndexUpdated, each followed by refreshAndValidate) and introduce a separate dnServerStarted flag that guards the one-time datanode protocol server start when isLeader() || isFollowerCaughtUp(). The leaderCommitIndexOnStart capture is now gated on !dnServerStarted since isStateMachineReady flips early again. The fix is preserved: the datanode server is still deferred until catch-up, and the datanode-report safe-mode rules still hold safe mode shut until the server starts and reports arrive, so a promoted follower never serves empty replicas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…pled approach) Per review: the deferred datanode server start should simply set isStateMachineReady=true at the same point it starts the server, with no separate flag or extra restructuring. Revert the decoupling change and keep the simpler approach where tryStartDNServerAndRefreshSafeMode() sets isStateMachineReady, starts the datanode protocol server, and refreshes safe mode once the SCM is caught up (isLeader() || isFollowerCaughtUp()). This reverts commit 9e2c2b9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

sumitagrawl

@ArafatKhan2198 overall looks good with Minor comment

Per review: drop the leaderCommit >= 0 guard when capturing the catch-up target in notifyLeaderChanged. A leader change may bring a different leader whose commit index differs, so the target should always be refreshed. getLeaderCommitIndex() returns -1 only when the leader is not known yet, which isFollowerCaughtUp() already treats as uncaptured and re-reads later, so setting it unconditionally is safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

sumitagrawl

LGTM

Copilot

Pull request overview

This PR addresses SCM-HA correctness during follower restart by deferring the SCM datanode protocol server start until the follower has caught up to the leader’s committed Ratis log, preventing early container report processing against stale container state. It also ensures followers do not attempt replicated container state transitions while handling container reports, and adds integration coverage for the restart/catch-up scenarios.

Changes:

Defer ScmDatanodeProtocol server startup in HA mode and start it from SCMStateMachine once the SCM is leader or a caught-up follower.
Add follower catch-up detection based on a fixed leader committed index captured at (re)join time, and trigger DN server start/safemode refresh from state machine callbacks.
Add an integration test suite covering close-while-down, create-while-down, and idle-cluster follower catch-up scenarios.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMFollowerCatchupWithContainerReport.java	New HA integration tests validating replica rebuild correctness after follower restart/catch-up and leadership transfer.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java	Defers datanode protocol server start when SCM HA is enabled.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java	Starts DN server/safemode refresh after leader promotion or follower catch-up to a captured leader commit index.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java	Avoids replicated lifecycle state transitions on followers during container report handling.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up

df1f0f3

ArafatKhan2198 requested a review from sumitagrawl June 26, 2026 08:26

ivandika3 requested a review from xichen01 June 26, 2026 13:03

jojochuang requested a review from szetszwo July 6, 2026 16:27

sumitagrawl reviewed Jul 7, 2026

View reviewed changes

ArafatKhan2198 and others added 3 commits July 8, 2026 14:20

Fixed review comments

d33f0c9

ArafatKhan2198 requested a review from sumitagrawl July 9, 2026 15:58

sumitagrawl reviewed Jul 10, 2026

View reviewed changes

Comment thread hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java Outdated

sumitagrawl approved these changes Jul 10, 2026

View reviewed changes

sumitagrawl marked this pull request as ready for review July 10, 2026 07:01

Copilot AI review requested due to automatic review settings July 10, 2026 07:02

Copilot started reviewing on behalf of sumitagrawl July 10, 2026 07:02 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Fixed co pilot comments

107679f

adoroszlai marked this pull request as draft July 10, 2026 19:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up.#10617

HDDS-14989. Delay follower SCM DN server start until Ratis log catch-up.#10617
ArafatKhan2198 wants to merge 7 commits into
apache:masterfrom
ArafatKhan2198:follower_slow

ArafatKhan2198 commented Jun 26, 2026

Uh oh!

ivandika3 commented Jun 26, 2026

Uh oh!

sumitagrawl left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Uh oh!

Uh oh!

sumitagrawl left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ArafatKhan2198 commented Jun 26, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Integration tests

Manual test (docker-compose ozone-ha)

Uh oh!

ivandika3 commented Jun 26, 2026

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Manual test (docker-compose `ozone-ha`)