[Bug] LogSet shard pending indefinitely after rapid InPlace restarts: Raft config change deadlock

[xzxiong]

Bug Description

LogSet Pod 在短时间内多次 InPlace 重启（image 变更）后，某个 shard 的 Raft 成员变更陷入死锁，shard 永久 pending 无法自愈。

Environment

• Version: v3.0.0-b79773d55-2026-04-25
• Cluster: dev freetier-01, 3-replica LogSet
• Trigger: InPlace pod restart (image change) 2 times within ~13 minutes

Steps to Reproduce

1. 3-replica LogSet running normally (log-0, log-1, log-2)
2. Change LogSet image (InPlace update) → log-2 restarts (container kill + restart)
3. While log-2 shard 1 is still recovering, change image again → log-2 restarts again
4. After second restart, shard 1 on log-2 enters permanent pending state

Observed Behavior

log-2 continuously outputs (every second, indefinitely):

shard 1 is pending, not included into the heartbeat

log-0 (shard 1 leader) fails every L/Add attempt:

ERROR logservice/service_commands.go:80 failed to add replica
error: request rejected
  dragonboat/v4@.../request.go:79

HAKeeper keeps retrying with incrementing replicaID (5420635 → 5420781+), every ~18 seconds, all rejected.

Deleting and recreating log-2 Pod does not fix the issue — the pending config change is persisted in log-0/log-1 Raft log.

Expected Behavior

• shard 1 should eventually recover after Pod restart
• If a config change is stuck, there should be a timeout/cleanup mechanism
• At minimum, HAKeeper should detect the deadlock and take corrective action (e.g., remove the stuck pending config change before retrying L/Add)

Root Cause Analysis

dragonboat rejects AddReplica requests when there is an ongoing (pending) config change in the Raft group. The sequence:

1. log-2 restarts → HAKeeper detects shard 1 replica down → sends L/Add with new replicaID
2. log-0 starts executing the config change (AddReplica)
3. Before config change completes, log-2 restarts again
4. HAKeeper sends another L/Add with newer replicaID
5. dragonboat rejects because previous config change is still pending
6. The old config change never completes (target node restarted with different state)
7. Deadlock: old config change blocks new ones, but old one cannot complete

Impact

• shard 1 runs with only 2/3 replicas (degraded, no redundancy)
• LogSet CR status shows Ready=True, 3/3 Up — misleading, since it only checks store-level heartbeat (shard 0 / HAKeeper shard is fine)
• If one more log node fails, shard 1 loses majority → data unavailable

Suggested Fix Areas

1. dragonboat: Add timeout for pending config changes, auto-abort if not completed within threshold
2. logservice: Before retrying L/Add, check if there is a pending config change and wait/abort it first
3. HAKeeper: Detect repeated L/Add failures and escalate (e.g., L/Remove old replica first, then L/Add)
4. LogSet status: Report per-shard health, not just store-level heartbeat

Logs

Full analysis with timeline: internal doc handbooks:docs/analysis/20260426-dev-freetier01-logset-shard1-raft-deadlock.md

[xzxiong]

https://grafana.cn-dev.matrixone.tech/goto/yMR5zITDg?orgId=1

[volgariver6]

Upstream root cause: stale local membership → zombie replica on restart

Digging into a similar outage on freetier-01 (2026-04-23, ~21 min HAKeeper
unavailability during a LogSet rolling upgrade), I found what I believe is the
upstream failure for this issue. The shard 1 is pending / request rejected
deadlock described in the bug report is the recovery-time symptom; the
original trigger is further back in the timeline and lives in
pkg/logservice/store.go.

Filing the analysis here so the two layers stay linked. Fix for the upstream
trigger is on branch logservice-zombie-fix (commits 35136fdbc and
f4eb1612e).

---

What actually happens when InPlace restarts land close together

The scenario matches the reproduction steps in this issue almost exactly:
3-replica LogSet, two InPlace image updates within a short window, log-2
restarting twice. Observed concrete numbers from the 2026-04-23 instance:

• Each pod spent ~10 min 23 s between SIGTERM and Ready (image pull
  • scheduling on a constrained node). That is already longer than HAKeeper's
    LogStoreTimeout = 300s.
• HAKeeper's checkExpired (pkg/logservice/store.go:480) fires exactly at
  SIGTERM + 5min and the scheduler issues L/Remove for the expired pod.
  For log-2 this landed at raft index 146,490,273.
• log-2's local dragonboat snapshot ended at index 146,330,457 — i.e.
  before the REMOVE ConfChange. When it restarts, it replays its own
  log up to its lastIndex = 146,483,547, which is still behind the REMOVE.
  So log-2 has no local evidence that it has been kicked out.
• store.startReplicas() (pkg/logservice/store.go:233) iterates local
  metadata and unconditionally calls nh.StartReplica for each
  (shardID, replicaID). With stale replicaID=33565 still recorded
  locally, dragonboat happily re-joins the raft group. A zombie has been
  born.

The zombie then campaigns forever — preVote succeeds against itself, peers
(who correctly no longer list it in their config) drop the RequestPreVote
silently. In the production logs this looked like:

raft.go:2123  [00000:33565] t126 dropped proposal, no leader
raft.go:1011  [00000:33565] t126 became PreVote candidate
hakeeper_client.go:929  error: cannot locate ha keeper
service_commands.go:239 failed to create HAKeeper client: cannot locate ha keeper

…every ~1 s, indefinitely.

Why HAKeeper's existing cleanup path cannot save us

HAKeeper already has a designed cleanup for exactly this shape of state:
checkZombie in pkg/hakeeper/checkers/logservice/check.go:360 proposes
L/Kill against a replicaID that's in a store's heartbeat but not in the
authoritative membership. That path works — when the cluster eventually
self-healed in the 2026-04-23 incident, L/Add → L/Kill → L/Start completed
in ~8 seconds once a leader was elected.

The problem is that proposing L/Kill requires a HAKeeper shard leader.
During overlapping rolling restarts, HAKeeper shard-0 kept losing quorum:

UTC	log-0	log-1	log-2 (zombie)	HAKeeper leader?
21:36:24	up	SIGTERM	up (zombie)	loses leader within election timeout
21:36:24–21:46:58	up	down	up (zombie)	no leader (zombie's votes don't count)
21:47:05–21:57:28	SIGTERM	replaying	up (zombie)	no leader (log-0 down)
21:57:30	up	leader t127	up (zombie)	leader elected → L/Kill dispatched

So the bug amplifies itself: the upgrade creates the zombie and destroys
the mechanism that would clean it. The pending-L/Add rejection described
in this issue's original report is what you see next, once a leader is back
but membership still contains a stale reference.

Root-cause chain

1. Pods take 10+ min to restart (environmental — image pull, scheduling).
2. LogStoreTimeout = 300s < typical upgrade downtime → planned shutdown
   indistinguishable from a dead node → L/Remove fires.
3. Rolling upgrade does not gate on HAKeeper quorum health.
4. Restarted logservice re-invokes nh.StartReplica with its stale local
   replicaID without checking live membership → zombie. ← the code-level
   bug fixed here.
5. Existing checkZombie path needs a leader to propose L/Kill; zombie +
   overlapping restarts starve quorum → leaderless → cleanup blocked.

Items 1–3 are environmental / operator-layer. Item 4 is the code-level
trigger that this PR closes.

Fix: membership self-check in startReplicas()

Commit 35136fdbc adds a best-effort self-check. Before
nh.StartReplica for each locally-persisted (shardID, replicaID):

• Query live shard membership via GetShardInfo against the discovery /
  first configured service address.
• If replicaID is absent from the returned Replicas map, log WARN and
  skip StartReplica for that pair.
• Local data is preserved. The existing HAKeeper recovery flow
  (L/Add new replicaID → L/Kill stale replicaID → L/Start) remains
  authoritative for cleanup.
• RPC failure / unknown shard / no configured address → treat as
  "not a zombie" and fall through to legacy behavior so cold-start keeps
  working.

Summary of the change (pkg/logservice/store.go):

• new (*store).checkZombieReplicas queries GetShardInfo per local
  shard and returns the set of (shardID, replicaID) to skip
• getShardInfoFn package variable allows tests to stub the RPC
• startReplicas() consults the zombie set and skips matched pairs,
  preserving every existing dispatch branch (HAKeeper / normal shard,
  voting / non-voting)
• 6 unit tests cover present / absent / RPC error / unknown shard / no
  address / end-to-end skip

Local reproduction and verification

Commit f4eb1612e adds pkg/logservice/zombie_repro_test.go — a Go
integration test that drives the exact zombie starting condition in one
process:

1. Build 3-node HAKeeper cluster (replicas {1, 2, 3}) sharing shard 0.
2. Pin leader to replica 1 via RequestLeaderTransfer.
3. Close svc3 (simulate pod down).
4. Commit REMOVE(shard=0, replica=3) ConfChange on the survivors via
   nh.SyncRequestDeleteReplica (simulates L/Remove without needing
   the scheduler).
5. Verify via GetShardInfo that cluster membership is now {1, 2}.
6. Restart svc3 with the same UUID and same persistent data so
   loadMetadata() reads the stale (shard=0, replica=3) record.
7. NewService → loadMetadata → startReplicas — same path as the real pod.

Two variants:

• TestZombieRepro_WithoutFix stubs getShardInfoFn to return
  (ShardInfo{}, false, nil) → self-check treats as "unknown" → falls
  back to legacy behavior. Equivalent to disabling the fix.
• TestZombieRepro_WithFix lets the real GetShardInfo RPC run.

Observed results:

TestZombieRepro_WithoutFix:
  [00000:00003] t3 became PreVote candidate
  [00000:00003] received RequestPreVoteResp from n00003   (self-vote)
  [00000:00003] sent RequestPreVote to n00001             (no response)
  [00000:00003] sent RequestPreVote to n00002             (no response)
  ... repeats every ~60 ms indefinitely ...
  zombie symptom without fix: svc3b saw no-leader in 40/40 samples

TestZombieRepro_WithFix:
  cn-service skip starting zombie replica,
    waiting for HAKeeper to re-add {shardID: 0, replicaID: 3}
  (no preVote campaigns, no "no leader" drops, no dragonboat activity
   for shard 0 on svc3b)
  ShardInfoList on svc3b has no entry for (shard=0, replica=3)
  haKeeperReplicaID == 0   (startHAKeeperReplica not called)
--- PASS: TestZombieRepro_WithFix (2.52s)

Stability: 3/3 consecutive runs pass.

What this fix does NOT cover

The self-check is narrow and deliberate. It does not:

1. Restore HAKeeper quorum when overlapping restarts drain it. If log-2
   is a clean wait-and-heartbeat pod after the fix, but log-1 is also down
   and log-0 is alone, shard 0 still has no quorum regardless. The fix
   prevents log-2 from worsening the outage (no preVote storms, no
   cannot locate ha keeper churn in its own log), but does not shorten
   the outage window below quorum-restore time.

2. Silently falls back to legacy behavior on cold start. If no peer
   answers GetShardInfo (intended for whole-cluster cold start), every
   shard is treated as "not a zombie" and all replicas start as before.
   Correct for cold start. Trade-off: in a "network is completely down but
   cluster is warm" case (DNS glitch at startup) the pod would become a
   zombie for the duration of the partition. Accepted to keep cold-start
   working.

3. Not affected by BootstrapHAKeeper. In production upgrade scenarios
   the operator config does not set BootstrapCluster=true, so
   startHAKeeperReplica is not invoked and the self-check in
   startReplicas() is the sole gate.

4. The L/Start recovery path is unchanged. The fix keeps
   hakeeper_client / heartbeat workers alive, so when HAKeeper eventually
   sends L/Add new_replicaID → L/Kill old_replicaID → L/Start new_replicaID
   the pod still processes them normally.

5. Does not address this issue's manual-recovery observation directly.
   The request rejected from dragonboat during retried L/Add is a
   downstream consequence of a stale membership entry on the survivors.
   Preventing the pod from becoming a zombie in the first place avoids the
   state that produces those retries. If we also want to make the
   dragonboat-side pending-config-change survivable, that belongs in the
   "Suggested Fix Areas" items 1–3 from the bug report and can ship
   separately.

Layers the postmortem flagged as out of scope here

#	Issue	Type	Owner
1	Image pull / scheduling takes 10 min	environmental	k8s / registry / nodes
2	LogStoreTimeout=300s < typical upgrade downtime	config	matrixone-operator (configurable)
3	Rolling upgrade doesn't check HAKeeper health between pods	operator	matrixone-operator / Kruise
4	Restart killed after ~100 ms on one pod	unknown	probe / Kruise config
5	Zombie on restart	code	this fix
6	Recovery requires leader, but leader formation requires no-zombie	code coupling	pkg/hakeeper/checkers/logservice/check.go

Items 1–4 are environmental/operator changes. Item 5 is this PR. Item 6 is
worth a separate conversation — options include coalescing L/Remove + L/Kill, persisting pending L/Kill so it survives a leader loss, or
raising LogStoreTimeout out of the upgrade-downtime range.

Commits on logservice-zombie-fix

• d21f487ca docs(logservice): spec for zombie replica self-check on startup
• ed704255e docs(logservice): implementation plan
• 35136fdbc logservice: skip zombie replicas on startup via membership self-check
• f4eb1612e logservice: add integration test reproducing zombie replica scenario