Skip to content

Commit 884e5b8

Browse files
filipdjokicfilipdjokic
committed
Additional details & minor format changes
1 parent c8fbadc commit 884e5b8

1 file changed

Lines changed: 80 additions & 36 deletions

File tree

‎docs/validator-guide/reenable-pruning.md‎

Lines changed: 80 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,44 @@
1-
# Guide for re-enabling pruning and recovering node db
2-
This guide is specifically made for the the validators/node operators affected by the pruning issue which caused a mainnet halt on September 6th 2025.
1+
# Guide for re-enabling pruning and recovering node db
32

4-
Re enabling pruning (to `default`/`custom` from `nothing`) on the affected nodes will cause the nodes to halt again as the database pruning resumes its operation. To avoid this it is recommended to reset your node db and recover it using state sync or a db snapshot. The nodes that weren't affected by the pruning issue or that had recovered by the time the `nothing` pruning fix was rolled out do not have to undergo this procedure.
3+
This guide is specifically made for the the validators/node operators affected by the pruning issue encountered following our v4.x upgrade. This issue required some validators/node operators having to disable pruning entirely.
54

6-
## State Sync:-
7-
Stop the systemd service of the node.
5+
Re-enabling pruning (to `default`/`custom` from `nothing`) on the affected nodes will cause the nodes to halt again as the database pruning resumes its operation. To avoid this it is recommended to reset your node db and recover it using state sync or a db snapshot. The nodes that weren't affected by the pruning issue or that had recovered by the time the pruning fix was rolled out do not have to undergo this procedure.
6+
7+
Following this procedure will significantly reduce the disk space required for your node’s regular operations, thereby lowering operational costs (on the nodes we manage, we observed storage usage drop from 700+ GB to under 10 GB). Additionally, running a node with less disk usage will likely improve performance.
8+
9+
You have two options here:
10+
11+
1) Reset via State Sync
12+
2) Reset by using DB snapshot
13+
14+
## State Sync
15+
16+
Stop the systemd service of the node.
817
`sudo systemctl stop cheqd-cosmovisor.service`
918

10-
Take a backup of the priv_validator_state.json. **This step is very Important for validator nodes.**
11-
`cp ~/.cheqdnode/data/priv_validator_state.json ~/priv_validator_state.json`
19+
Take a backup of the priv_validator_state.json. **This step is very Important for validator nodes:**
20+
21+
```bash
22+
cp ~/.cheqdnode/data/priv_validator_state.json ~/priv_validator_state.json
23+
```
1224

1325
Turn the pruning strategy to default or custom based on your preference in the `~/.cheqdnode/config/app.toml` file.
1426

15-
Reset the database.
16-
`cheqd-noded tendermint unsafe-reset-all --home ~/.cheqdnode/ --keep-addr-book`
27+
Reset the database:
1728

18-
Restore the priv_validator_state.json . **This step is very Important for validator nodes**
19-
`cp ~/priv_validator_state.json ~/.cheqdnode/data/priv_validator_state.json`
29+
```bash
30+
cheqd-noded tendermint unsafe-reset-all --home ~/.cheqdnode/ --keep-addr-book
31+
```
2032

21-
Enable statesync on the node and provide the required variables.
33+
Restore the priv_validator_state.json. **This step is very Important for validator nodes:**
34+
35+
```bash
36+
cp ~/priv_validator_state.json ~/.cheqdnode/data/priv_validator_state.json
2237
```
38+
39+
Enable statesync on the node and provide the required variables:
40+
41+
```bash
2342
STATESYNC_RPC="https://rpc.cheqd.net:443"
2443
LATEST_HEIGHT=$(curl -s $STATESYNC_RPC/block | jq -r .result.block.header.height)
2544
BLOCK_HEIGHT=$((LATEST_HEIGHT - 2000))
@@ -30,31 +49,56 @@ s|^(trust_height[[:space:]]+=[[:space:]]+).*$|\1$BLOCK_HEIGHT| ; \
3049
s|^(trust_hash[[:space:]]+=[[:space:]]+).*$|\1\"$TRUST_HASH\"|" ~/.cheqdnode/config/config.toml
3150
```
3251

33-
Start the node
34-
`sudo systemctl restart cheqd-cosmovisor.service`
52+
Start the node:
3553

36-
The node should start looking for statesync chunks from it's peers and begin the restoration process in a few minutes.
54+
```bash
55+
sudo systemctl restart cheqd-cosmovisor.service
56+
```
57+
58+
The node should start looking for statesync chunks from it's peers and begin the restoration process in a few minutes. After some time, it should catch up with the network and continue siging blocks.
59+
60+
## Snapshot
61+
62+
Stop the systemd service of the node:
63+
64+
```bash
65+
sudo systemctl stop cheqd-cosmovisor.service`
66+
```
67+
68+
Take a backup of the priv_validator_state.json. **This step is very Important for validator nodes:**
69+
70+
```bash
71+
cp ~/.cheqdnode/data/priv_validator_state.json ~/priv_validator_state.json
72+
```
3773

38-
## Snapshot:-
39-
Stop the systemd service of the node.
40-
`sudo systemctl stop cheqd-cosmovisor.service`
41-
 
42-
Take a backup of the priv_validator_state.json. **This step is very Important for validator nodes.**
43-
`cp ~/.cheqdnode/data/priv_validator_state.json ~/priv_validator_state.json`
44-
 
4574
Turn the pruning strategy to default or custom based on your preference in the `~/.cheqdnode/config/app.toml` file.
46-
 
47-
Reset the database.
48-
`cheqd-noded tendermint unsafe-reset-all --home ~/.cheqdnode/ --keep-addr-book`
49-
 
50-
Restore the priv_validator_state.json. **This step is very Important for validator nodes**
51-
`cp ~/priv_validator_state.json ~/.cheqdnode/data/priv_validator_state.json`
52-
53-
Download the latest lz4 tar archive for mainnet from this link https://snapshots.cheqd.net/#mainnet/
54-
`wget https://cheqd-node-backups.ams3.digitaloceanspaces.com/mainnet/<timestamp>/cheqd-mainnet-1_<timestamp>.tar.lz4`
55-
56-
Unpack the tar archive and restore the db
57-
`lz4 -c -d cheqd-mainnet-1_<timestamp>.tar.lz4 | tar -x -C ~/.cheqdnode`
58-
 
75+
76+
Reset the database:
77+
78+
```bash
79+
cheqd-noded tendermint unsafe-reset-all --home ~/.cheqdnode/ --keep-addr-book`
80+
```
81+
82+
Restore the priv_validator_state.json. **This step is very Important for validator nodes:**
83+
84+
```bash
85+
cp ~/priv_validator_state.json ~/.cheqdnode/data/priv_validator_state.json
86+
```
87+
88+
Download the latest lz4 tar archive for mainnet from [our snapshots page](https://snapshots.cheqd.net/#mainnet/)
89+
90+
```bash
91+
wget https://cheqd-node-backups.ams3.digitaloceanspaces.com/mainnet/<timestamp>/cheqd-mainnet-1_<timestamp>.tar.lz4
92+
```
93+
94+
Unpack the tar archive and restore the db:
95+
96+
```bash
97+
lz4 -c -d cheqd-mainnet-1_<timestamp>.tar.lz4 | tar -x -C ~/.cheqdnode`
98+
```
99+
59100
Start the node
60-
`sudo systemctl restart cheqd-cosmovisor.service`
101+
102+
```bash
103+
sudo systemctl restart cheqd-cosmovisor.service
104+
```

0 commit comments

Comments
 (0)