Summary
A single masternode (0x22f69af9…EaEDD) flooded devnet mempools with ~4,700 zero-gas BlockSigners.sign() txs that can't mine.
Many dimensions to the problem, multiple bugs and behavior change came to cause current issue.
ref issue commit id: 7c94a61
ref mainnet behavior version: v2.7.0
Root cause
1. Special transactions was not separated from normal transactions
In current mainnet signingTX processing flows to the wrong logic branch, causing the signing window check of 1800 blocks to never activate (signingTXs are valid for all blocks). This behavior was changed in #1893, properly separates specialTXs and normalTXs.
if tx.IsSpecialTransaction() {
specialTxs = append(specialTxs, tx)
} else {
normalTxs = append(normalTxs, tx)
}
Once #1893 was applied, miner/worker.go#L1003 reject sign() for blocks older than head - epoch*2 (1800). Out-of-window sign() txs are skipped by the miner entirely — never included, never consume their nonce, never drain.
2. SigningTX nonce behavior changed from using on-chain nonce to pending nonce
#2213 generate nonce on top of pending txs, doesn't replace. So once one sign() can't mine, every subsequent tx stacks behind it at increasing nonces. This causes the flood inside mempool.
3. Block Number Extraction: fixed
#1894 this is a bugfix that corrects the extraction of the signed block number. Minor impact on our current issue.
Proposed Fix
Note
It's important to point out the Nonce replacement of special transactions is not allowed. This led to #2213 change.
func (l *list) Add(tx *types.Transaction, priceBump uint64) (bool, *types.Transaction) {
// If there's an older better transaction, abort
old := l.txs.Get(tx.Nonce())
if old.IsSpecialTransaction() {
return false, nil
}
Proposing #2363
- Allowing of replacement of specialTXs with the same nonce.
- Making sign() method refer to on-chain nonce instead of pending mempool nonce.
This will prevent potential flooding if a signing transaction was unable to be mined, it will get replaced by the next TX in 15 blocks instead of having subsequent tx stacks behind it at increasing nonces.
- Dropped the "1800 blocks acceptance window" as a safety measure to not have lingering signingTX in mempool. This doesn't affect reward and penalty calculation in any way.
Summary
A single masternode (
0x22f69af9…EaEDD) flooded devnet mempools with ~4,700 zero-gasBlockSigners.sign()txs that can't mine.Many dimensions to the problem, multiple bugs and behavior change came to cause current issue.
ref issue commit id: 7c94a61
ref mainnet behavior version: v2.7.0
Root cause
1. Special transactions was not separated from normal transactions
In current mainnet signingTX processing flows to the wrong logic branch, causing the signing window check of 1800 blocks to never activate (signingTXs are valid for all blocks). This behavior was changed in #1893, properly separates specialTXs and normalTXs.
Once #1893 was applied,
miner/worker.go#L1003reject sign() for blocks older thanhead - epoch*2(1800). Out-of-window sign() txs are skipped by the miner entirely — never included, never consume their nonce, never drain.2. SigningTX nonce behavior changed from using on-chain nonce to pending nonce
#2213 generate nonce on top of pending txs, doesn't replace. So once one sign() can't mine, every subsequent tx stacks behind it at increasing nonces. This causes the flood inside mempool.
3. Block Number Extraction: fixed
#1894 this is a bugfix that corrects the extraction of the signed block number. Minor impact on our current issue.
Proposed Fix
Note
It's important to point out the Nonce replacement of special transactions is not allowed. This led to #2213 change.
Proposing #2363
This will prevent potential flooding if a signing transaction was unable to be mined, it will get replaced by the next TX in 15 blocks instead of having subsequent tx stacks behind it at increasing nonces.