-
Notifications
You must be signed in to change notification settings - Fork 5
Add operator rollback and slashing recovery guides #127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| --- | ||
| title: Roll back a node update | ||
| description: Learn how to reinstall a previous Dusk node installer release when an update needs to be rolled back. | ||
| --- | ||
|
|
||
| Rolling back means reinstalling a previous `node-installer` release. This can be useful if a newly installed node release has a problem and the network has not yet activated a protocol change that requires it. | ||
|
|
||
| :::caution | ||
| Do not roll back across a network upgrade or activation unless the Dusk team explicitly instructs operators to do so. If the chain already requires the newer node version, an older node may stop following the network. | ||
| ::: | ||
|
|
||
| ## Before you roll back | ||
|
|
||
| Check the version currently installed: | ||
|
|
||
| ```sh | ||
| ruskquery version | ||
| ``` | ||
|
|
||
| Check whether the node is syncing: | ||
|
|
||
| ```sh | ||
| ruskquery block-height | ||
| tail -n 50 /var/log/rusk.log | ||
| ``` | ||
|
|
||
| If the node is only stuck or behind, try [fast-sync](/operator/guides/fast-sync) or [manual resync](/operator/guides/manual-resync) before rolling back. | ||
|
|
||
| ## Roll back mainnet | ||
|
|
||
| Replace `vX.Y.Z` with the installer release you want to roll back to. | ||
|
|
||
| For example, to roll back to installer release `v1.2.3`, set: | ||
|
|
||
| ```sh | ||
| INSTALLER_VERSION="v1.2.3" | ||
| ```sh | ||
| INSTALLER_VERSION="vX.Y.Z" | ||
|
|
||
| sudo service rusk stop | ||
|
|
||
| curl --proto '=https' --tlsv1.2 -sSfL \ | ||
| "https://github.com/dusk-network/node-installer/releases/download/${INSTALLER_VERSION}/node-installer.sh" \ | ||
| | sudo bash | ||
|
|
||
| sudo service rusk start | ||
| ``` | ||
|
|
||
| ## Roll back testnet | ||
|
|
||
| Use the same pinned installer release, but pass the testnet flag. | ||
|
|
||
| ```sh | ||
| INSTALLER_VERSION="vX.Y.Z" | ||
|
|
||
| sudo service rusk stop | ||
|
|
||
| curl --proto '=https' --tlsv1.2 -sSfL \ | ||
| "https://github.com/dusk-network/node-installer/releases/download/${INSTALLER_VERSION}/node-installer.sh" \ | ||
| | sudo bash -s -- --network testnet | ||
|
|
||
| sudo service rusk start | ||
| ``` | ||
|
|
||
| ## Verify the rollback | ||
|
|
||
| Confirm the installed version: | ||
|
|
||
| ```sh | ||
| ruskquery version | ||
| ``` | ||
|
|
||
| Check the service: | ||
|
|
||
| ```sh | ||
| service rusk status | ||
| ``` | ||
|
|
||
| Check whether the node is progressing: | ||
|
|
||
| ```sh | ||
| ruskquery block-height | ||
| tail -F /var/log/rusk.log | ||
| ``` | ||
|
|
||
| If block height does not progress, compare your height with the explorer and consider [fast-syncing the node](/operator/guides/fast-sync). | ||
|
|
||
| ## Notes | ||
|
|
||
| - Use a pinned installer release URL. Do not use `latest` for rollback. | ||
| - Rollback changes the installed node software and service configuration. It does not automatically restore an older chain state. | ||
| - Archive nodes may have database migrations that are not safe to downgrade. If an archive node fails after rollback, update back to the supported version or resync the archive state. | ||
| - If you operate a provisioner, monitor the node after rollback. Downtime or running an incompatible version can affect consensus participation. See [Slashing prevention and recovery](/operator/guides/slashing-recovery). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| --- | ||
| title: Slashing prevention and recovery | ||
| description: Learn how Dusk soft slashing affects provisioners and what to check after a slashing event. | ||
| --- | ||
|
|
||
| Dusk uses **soft slashing** to discourage repeated faults and long downtime. Stake is not burned, but a provisioner can lose eligibility or effective participation, which reduces rewards. | ||
|
|
||
| Soft slashing can happen when a provisioner repeatedly fails to participate correctly in consensus. Common operational causes include: | ||
|
|
||
| - Running an outdated or incompatible node version. | ||
| - Being offline for too long. | ||
| - Falling behind the network tip. | ||
| - Network or firewall problems that prevent consensus messages from being sent or received. | ||
| - Consensus key or node configuration issues. | ||
|
|
||
| ## If your provisioner was slashed | ||
|
|
||
| Start by restoring healthy node operation. Slashing is a symptom; the first priority is to make sure the node is on the right chain, on the right version, and progressing. | ||
|
|
||
| ### 1. Check the installed version | ||
|
|
||
| ```sh | ||
| ruskquery version | ||
| ``` | ||
|
|
||
| If the network has a required release, upgrade to it: | ||
|
|
||
| ```sh | ||
| curl --proto '=https' --tlsv1.2 -sSfL https://github.com/dusk-network/node-installer/releases/latest/download/node-installer.sh | sudo bash | ||
| sudo service rusk start | ||
| ``` | ||
|
|
||
| For testnet: | ||
|
|
||
| ```sh | ||
| curl --proto '=https' --tlsv1.2 -sSfL https://github.com/dusk-network/node-installer/releases/latest/download/node-installer.sh | sudo bash -s -- --network testnet | ||
| sudo service rusk start | ||
| ``` | ||
|
|
||
| If a newly installed release is known to be problematic and the network can still run the previous version, follow [Rollback a node update](/operator/guides/rollback-node-update). | ||
|
|
||
| ### 2. Check sync status | ||
|
|
||
| ```sh | ||
| ruskquery block-height | ||
| ``` | ||
|
|
||
| Compare the height with the explorer. If the node is stuck or far behind, use [fast-sync](/operator/guides/fast-sync): | ||
|
|
||
| ```sh | ||
| download_state | ||
| sudo service rusk start | ||
| ``` | ||
|
|
||
| ### 3. Check service status and logs | ||
|
|
||
| ```sh | ||
| service rusk status | ||
| tail -n 100 /var/log/rusk.log | ||
| ``` | ||
|
|
||
| Look for errors related to: | ||
|
|
||
| - wrong network or chain mismatch | ||
| - consensus key loading | ||
| - peer discovery | ||
| - Kadcast address or UDP connectivity | ||
| - state/database errors | ||
|
|
||
| ### 4. Check staking status | ||
|
|
||
| ```sh | ||
| rusk-wallet stake-info | ||
| ``` | ||
|
|
||
| Confirm that the expected stake is still present and inspect the reported stake state. If the wallet cannot connect, fix node connectivity first. | ||
|
|
||
| ### 5. Monitor after recovery | ||
|
|
||
| After the node is updated, synced, and running, keep the logs open and check the block height repeatedly: | ||
|
|
||
| ```sh | ||
| tail -F /var/log/rusk.log | ||
| ruskquery block-height | ||
| ``` | ||
|
|
||
| The height should continue to progress. If the node falls behind again, treat it as an unresolved infrastructure or networking issue. | ||
|
|
||
| ### 6. Unstake and restake if the node is healthy | ||
|
|
||
| If the node is on the right version, fully synced, and operating normally, unstake and restake to restore normal provisioner participation. | ||
|
|
||
| Slashing can be caused by temporary operational issues, such as cloud provider downtime, network disruption, or downtime during a live node update. If the underlying issue is gone, restaking is the recovery step. | ||
|
|
||
| ```sh | ||
| rusk-wallet unstake | ||
| rusk-wallet stake --amt <amount> | ||
| ``` | ||
|
|
||
| Replace `<amount>` with the amount you want to stake. The new stake must mature before it starts participating again. | ||
|
|
||
| ## Prevention checklist | ||
|
|
||
| - Keep the node updated during announced network upgrades. | ||
| - Monitor `ruskquery block-height` against the explorer. | ||
| - Alert on service downtime and repeated restart loops. | ||
| - Keep UDP `9000` reachable for Kadcast. | ||
| - Keep consensus keys backed up and readable by the Rusk service. | ||
| - Avoid running experimental or mismatched binaries on a staked provisioner. | ||
| - Use [fast-sync](/operator/guides/fast-sync) when a node falls behind instead of waiting for a long resync from genesis. | ||
|
|
||
| ## Related guides | ||
|
|
||
| - [Upgrade a node](/operator/guides/upgrade-node) | ||
| - [Roll back a node update](/operator/guides/rollback-node-update) | ||
| - [Fast-sync a node](/operator/guides/fast-sync) | ||
| - [Troubleshooting](/operator/troubleshooting) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.