[Bug] MsgMigrateValidator: unbounded chain-wide store iteration under tx gas meter — large validators cannot migrate

[coinsspor]

Summary

MsgMigrateValidator performs three chain-wide store iterations under the tx gas meter, so migration gas cost scales with total chain state, not with the validator's own record count. Large validators burn 100% of any provided gas limit and cannot complete migration.

Tested on lumera-testnet-2 with a validator holding 5087 delegations (0 unbonding, 0 redelegations), after the cap raise to 7500 (prop 13).

Reproduction / evidence

Every attempt consumed the entire gas limit and failed just above it, at iteration/read cost locations:

Gas provided	Result	Location	Tx hash
20M	out of gas (gasUsed 20,000,524)	ValuePerByte	BDB4EB678AB4A68786F067F9FCAB186C06BC52A30A635F942A1697055843A31F
50M	out of gas (gasUsed 50,001,404)	WriteFlat	F3F0E6692D8DF9D2488A89D76F2B617D0BAEAE4B7EFB28781A84550BE21CDFA6
500M	out of gas (gasUsed 500,000,005)	IterNextFlat	D82E1FE3722B61F068CC42745B2D48A12CECE6CA15C78EFD8FCEF50FB446DD71
2B	out of gas (gasUsed 2,000,000,087)	ValuePerByte	7B2E8838E050285592EBC64AD920FAB2432FD885FB2C1C43E354CBE7B0034933
20B	broadcast accepted, never landed in a block — appears silently dropped from mempools	—	7110C40CE70B871F44CE63704D4F2604B61BFE602F51CAE1E90A74AD5408F85A

Side effects observed:

• Broadcasting via the local node stalls it during CheckTx (validator missed blocks, required restart).
• CheckTx times out public RPC gateways (504 from two independent providers).
• The 2B execution visibly slowed block production network-wide.

Root cause

x/evmigration/keeper/migrate_validator.go (v1.20.0):

• L158 — stakingKeeper.IterateRedelegations(...): iterates all redelegations on the chain, filters ValidatorSrcAddress/DstAddress == oldValAddr inside the Go callback.
• L244 — distributionKeeper.IterateValidatorHistoricalRewards(...): iterates all validators' historical rewards, filters val.Equals(oldValAddr) in the callback.
• L263 — distributionKeeper.IterateValidatorSlashEvents(...): same pattern.

The callbacks always return false, so the iterator walks the entire store; every record read charges ReadCostPerByte against the tx gas meter. On a long-running chain the historical-rewards store alone is millions of records, which matches the observed failure locations (IterNextFlat, ValuePerByte).

Impact

• Any validator above a modest size cannot migrate, regardless of gas.
• Cost grows with chain state over time, while the migration window is fixed (~3 months) — worst case, large validators permanently lose delegations per the announced deadline semantics.
• Mainnet state is larger, so this hits mainnet harder than testnet.
• Heavy CheckTx/execution of a single fee-free tx measurably slows the chain (mild DoS surface).

Suggested fix

• Prefix-scope the iterations to oldValAddr — the distribution store is already keyed by validator prefix, so IterateValidatorHistoricalRewards/SlashEvents can be ranged scans; redelegations can use the per-validator (src/dst) index instead of the global iterator.
• Alternatively, run migration under an infinite gas meter (migration txs are already fee-free/signature-free) — though prefix scoping is the cleaner fix.

Happy to test a patched binary on testnet.
Validator: coinsspor / legacy lumera1jfl8rf4pmf9pwzfzckl6pv8jmz0fqp2t3lk0xv / new lumera1gxauzl4ssrdtqznra9dcwn4cz9lq0qsqrdl9ka

[astrostake]

Additional Findings

Following up on the behavior of the 20B gas transaction (7110C40CE70B871F44CE63704D4F2604B61BFE602F51CAE1E90A74AD5408F85A), data from the explorer and local RPC endpoints shows that the transaction was not dropped from the mempool. Instead, it has been repeatedly included across multiple consecutive blocks hours after being broadcasted and is still actively appearing up to the current block tip.

Key Observations:

• Mempool Eviction Failure: The transaction continues to be included by block proposers despite consistently failing with a Code 11 (Out of Gas) error and consuming the full 20,000,000,088 gas.
• Gas Consumption Fluctuations: There are slight variances in the exact gas_used metric across different block heights, indicating active re-execution attempts per block.
• Validator Peer Drop: Telemetry shows a severe drop in connected peers on the validator side (falling from roughly 50 down to just 3 peers during this event), suggesting the heavy processing loop is impacting node connectivity.
• Network Impact: This continuous inclusion loop and the accompanying drop in peer connectivity have triggered a noticeable slowdown in network-wide block production times.

Block Execution Data

Below is the verified log extracted from the node's block_results and block header data showing the inclusion pattern:

Block Height	Timestamp (UTC)	Tx Index	Status Code	Gas Used
#5555946	2026-07-03 06:13:10 UTC	1	11	20000000027
#5555945	2026-07-03 06:09:17 UTC	2	11	20000000027
#5555944	2026-07-03 06:03:55 UTC	2	11	20000000088
#5555943	2026-07-03 06:00:03 UTC	1	11	20000000088
#5555942	2026-07-03 05:53:47 UTC	1	11	20000000088
#5555941	2026-07-03 05:48:48 UTC	1	11	20000000088
#5555940	2026-07-03 05:43:42 UTC	0	11	20000000088
#5555939	2026-07-03 05:38:54 UTC	0	11	20000000088
#5555938	2026-07-03 05:31:44 UTC	0	11	20000000061
#5555937	2026-07-03 05:25:56 UTC	0	11	20000000088
#5555936	2026-07-03 05:21:53 UTC	2	11	20000000088
#5555935	2026-07-03 05:21:47 UTC	0	11	20000000088
#5555932	2026-07-03 05:17:33 UTC	1	11	20000000088

[coinsspor]

Second issue: failed migration tx is not evicted from mempool — repeated re-inclusion (DoS)

Follow-up on the 20B-gas attempt (7110C40CE70B871F44CE63704D4F2604B61BFE602F51CAE1E90A74AD5408F85A). Credit to @astrostake for tracing this.

The tx was not dropped. It has been re-included across many consecutive blocks for hours, each time failing with Code 11 (out of gas) while consuming the full ~20,000,000,088 gas, and is still being included at the current tip.

Observed inclusion pattern (block / gas_used, all Code 11):

Height	Gas Used
5555946	20000000027
5555945	20000000027
5555944	20000000088
5555943	20000000088
5555942	20000000088
...	...
5555932	20000000088

(continuous from ~5555932 to tip)

Why this is a DoS vector, not just a stall:

• A fee-free migration tx that fails out-of-gas is apparently not evicted on recheck, so proposers keep selecting it.
• Every block re-executes it and burns ~20B gas for nothing, slowing network-wide block production.
• Any account can broadcast such a tx (migration is fee-free / signature-free), so this is a cheap, sustained DoS with a single message.

Combined with the O(chain-state) gas cost from the original report, a large-validator migration tx is both impossible to land AND self-perpetuating in the mempool. Both need addressing before mainnet: fix the gas/iteration cost, and ensure failed migration txs are evicted (or charge/limit them so re-inclusion stops).