bug: databend-meta: state-machine drift across replicas after V002→V004, despite identical Raft logs

[inviscid]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Summary

After a databend-meta cluster is upgraded through the V002→V004 transition, followers' state machines remain permanently diverged from the leader's despite identical Raft logs. Followers report roughly half the SnapshotKeyCount and LastSeq of the leader, with no automatic convergence over months of normal operation. The only known recovery is wiping a follower's raft-dir and forcing a fresh snapshot install via --join.

This is consistent with a non-deterministic state-machine migration during V002→V004 — each replica ran the migration independently and arrived at a different state from the same prior log.

Affected versions

• databend-meta images in the v1.2.7xx–v1.2.8xx range (observed on v1.2.881 and v1.2.790-nightly)
• DataVersion: V004, in clusters that were originally on V002

What's Wrong?

Evidence

A 3-replica cluster with healthy Raft replication but diverged state machines:

	meta-0 (Leader)	meta-1	meta-2
CurrentTerm	5569	5569	5569
LastLogIndex	9,282,381	9,282,381	9,282,383
LastApplied	T5569-N0.9282381	T5569-N0.9282381	T5569-N0.9282383
SnapshotKeyCount	370,804	185,123	185,123
LastSeq	22,248,393	9,786,823	9,786,823

Both followers carry the same diverged numbers, suggesting a simultaneous event (the upgrade) rather than runtime drift.

Replication reports all voters caught up to T5569-N0.9282381 — replication is healthy. Purged is well below LastApplied, so log compaction hasn't yet forced the followers into an InstallSnapshot and the divergence has stayed latent.

Pattern across multiple clusters

The same divergence shape (followers at one identical SKC/LastSeq, leader at a higher one) is observed across multiple independently-upgraded clusters. Severity varies (sub-1% to ~50% SnapshotKeyCount divergence). Clusters that have had operator-driven raft-dir resets on followers show zero drift, durable for months. Clusters that haven't carry the divergence indefinitely.

Why this looks like a non-deterministic upgrade migration

• Raft logs identical at every observable level → not a replication bug.
• Divergence is stable over time despite ongoing writes → not runtime drift.
• Both followers carry identical diverged numbers → simultaneous event.
• Emerged around V004 transition; absent on freshly-bootstrapped V004 clusters.

Most parsimonious explanation: a V002→V004 migration step ran per-replica and was non-deterministic.

Operational concern

Raft elections pick a leader based on log term/index, not state-machine content. If a drifted follower wins an election (e.g., the canonical leader's pod is preempted or its node fails), the cluster now operates against the diverged state. If the original leader is then down long enough for log compaction on the new leader to advance past its position, Raft sends InstallSnapshot from the drifted leader to the recovering pod and the canonical state machine is silently overwritten — total loss of the divergent keys.

This makes the latent divergence a real data-loss risk in any environment where a leader pod can fail and recover slowly.

What would help

1. Confirm or refute that V002→V004 migration is meant to be deterministic across replicas.
2. A safe upgrade path for future state-version transitions that doesn't risk per-replica divergence.
3. Built-in drift detection — databend-metactl status already exposes the relevant metrics; a peer-comparison check (built-in or documented pattern) would surface this kind of bug in hours instead of months.
4. Optional: a built-in databend-metactl resync-from-leader (or similar) that wipes local state and triggers snapshot install in a controlled way, so operators don't have to delete raft-dir and restart manually. The wipe has to happen before databend-meta starts — wiping a running process's storage directory races against its background compaction.

Happy to share full databend-metactl status output or any other diagnostic data that would help.

How to Reproduce?

Reproduction (best-effort)

There is no deterministic reproduction available because the specific migration step that diverged is unknown. The conditions appear to be:

1. Bootstrap a databend-meta cluster on a pre-V004 version (e.g., V002).
2. Accumulate non-trivial state (>100k keys, mixed writes/deletes/expirations/transactions).
3. Upgrade to a V004-capable version via rolling restart.
4. Compare SnapshotKeyCount and LastSeq across replicas via databend-metactl status.

The bug may be sensitive to the specific pre-upgrade workload (operations that V002 and V004 represent differently in the state machine).

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[drmingdrmer]

What's the version of the databend-meta you were upgrading from and to?

Have you upgraded the cluster by shutting down the cluster first or via a rolling upgrade (which means you upgrade a follower first, then another follower, and then the leader)?

[drmingdrmer]

Thanks for the detailed report. To investigate this, could you share more details about the actual databend-meta upgrade process?

The exact binary version matters here. V004 was introduced around v1.2.655, but direct on-disk upgrade from DataVersion::V002 was later removed/disabled around v1.2.769+. A node still on V002 should upgrade through a V003-capable version first, rather than jumping directly to a newer V004 binary.

Could you provide:

1. The exact databend-meta image tags used, in upgrade order.
2. Whether the cluster was upgraded by full stop/start or by rolling restart.
3. If rolling restart: the node upgrade order and which node was leader at that time.
4. Debug-level startup logs from each meta node during the on-disk upgrade.

The most useful log section is around lines containing:

• Working DataVersion
• On Disk Data
• Upgrade on-disk data
• From:
• To:
• snapshot id / last applied log id, if present

Debug logs during the upgrade are especially valuable because the on-disk migration is performed locally on each node. They should help identify which binary performed each V002 -> V003 -> V004 step and whether all nodes followed the same upgrade path.

[drmingdrmer]

I tried to reproduce this locally with V002 on-disk data.

In my local test, the old V002 data was generated with:

databend-meta-binaries v1.2.546-nightly-0ce48c9e92

Then I tried to open it with the latest binary I have locally:

databend-metactl v1.2.905-nightly-47408a76dc

With this latest binary, directly opening V002 data does not continue to serve requests. It panics and exits during startup:

Upgrade ondisk data if out of date: V002
    Find and clean previous unfinished upgrading

V002 is no longer supported, since 2025-07-01

So the expected behavior for an unsupported direct V002 upgrade is that the process quits, not that it starts serving with a partially upgraded or inconsistent snapshot.

The behavior you observed is still unclear to me. If the follower was able to start, catch up logs from the leader, but had a snapshot missing some keys, it may involve a specific binary version or an older upgrade path where V002 -> V003/V004 was still allowed.

Could you please provide:

• The exact databend-meta --version before and after the upgrade.
• The exact upgrade process, especially whether there was any intermediate version.
• DEBUG-level logs from the node during startup/upgrading.
• If available, databend-metactl status output from the affected node after upgrade.

The DEBUG upgrade log is especially useful here, because it should show which on-disk version was detected and which upgrade path was executed.