You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Scenario 4: One node disappears from the cluster
49
49
50
-
This is the case when one node becomes unavailable due to power outage, hardware failure, kernel panic, mysqld crash, **kill -9** on mysqld pid, etc.
50
+
This is the case when one node becomes unavailable due to power outage, hardware failure, kernel panic, mysqld crash, kill -9 on mysqld pid, etc.
51
51
52
52
Two remaining nodes notice the connection to node A is down and start trying to re-connect to it. After several timeouts, node A is removed from the cluster. The quorum is saved (two out of three nodes are up), so no service disruption happens. After it is restarted, node A joins automatically (as described in [Scenario 1: Node A is gracefully stopped](#scenario-1-node-a-is-gracefully-stopped)).
53
53
54
54
## Scenario 5: Two nodes disappear from the cluster
55
55
56
-
Two nodes are not available and the remaining node (node C) is not able to form the quorum alone. The cluster has to switch to a non-primary mode, where MySQL refuses to serve any SQL queries. In this state, the **mysqld** process on node C is still running and can be connected to but any statement related to data fails with an error.
56
+
Two nodes are not available and the remaining node (node C) is not able to form the quorum alone. The cluster has to switch to a non-primary mode, where MySQL refuses to serve any SQL queries. In this state, the mysqld process on node C is still running and can be connected to but any statement related to data fails with an error.
In this case, you cannot be sure that all nodes are consistent with each other. We cannot use `safe_to_bootstrap` variable to determine the node that has the last transaction committed as this variable is set to **0** for each node.
112
+
In this case, you cannot be sure that all nodes are consistent with each other. The `safe_to_bootstrap` variable is set to 0 on every node and cannot be used to identify which node has the last transaction committed.
113
113
114
-
An attempt to bootstrap from such a node will fail unless you start `mysqld` with the `--wsrep-recover` option:
114
+
!!! warning "Risk of split-brain"
115
+
116
+
Setting `safe_to_bootstrap: 1` on a node without first confirming that node has the highest recovered position can cause split-brain and data loss. Always run the validation step below on every node and bootstrap only from the node with the highest seqno.
117
+
118
+
### Validation step: recover and record position on every node
119
+
120
+
On each node that was part of the cluster, run `mysqld` with the `--wsrep-recover` option so that the server prints the recovered position and exits (the server does not stay running):
115
121
116
122
```shell
117
123
mysqld --wsrep-recover
118
124
```
119
125
120
-
Search the output for the line that reports the recovered position after the node UUID (**1122** in this case):
126
+
In the output, find the line that reports the recovered position in the form `UUID:seqno`:
The node where the recovered position is marked by the greatest number is the best bootstrap candidate. In its `grastate.dat` file, set the safe_to_bootstrap variable to **1**. Then, bootstrap from this node:
136
+
Run the command on every node and record the UUID and seqno from each. Use a table like the following so that you can compare and choose the correct bootstrap candidate:
137
+
138
+
| Node (hostname or label) | UUID | seqno |
139
+
|--------------------------|------|-------|
140
+
| node1 |||
141
+
| node2 |||
142
+
| node3 |||
143
+
144
+
!!! warning "When highest seqno is not safe to use"
145
+
146
+
The procedure below assumes you have access to every node that was in the cluster and that the recovered positions are trustworthy. If either is false, bootstrapping from the node with the highest seqno can permanently destroy data.
147
+
148
+
* Access to all nodes: If a node is unreachable (for example, in another datacenter or still down), you cannot assume the highest seqno you see is the true cluster state. The missing node may have had a higher seqno. Bootstrap only after you have run `mysqld --wsrep-recover` on every member and recorded the result.
149
+
150
+
* Trustworthiness of the "highest" node: A node can report a higher seqno but have corrupt or incomplete data—for example, after a partition (it was in a minority and applied writes that were never committed cluster-wide), a write-ahead or disk failure (it reported a seqno that was not fully persisted), or an unclean shutdown. Bootstrapping from that node forces the rest of the cluster to sync to that state. The cluster will then permanently drop or overwrite the transactions that existed only on the other nodes. If you suspect the "highest" node was partitioned, had storage or write-ahead issues, or you cannot verify its history, do not bootstrap from it without expert guidance or a verified backup strategy. Prefer [Get help from Percona](get-help.md) or your support channel when in doubt.
151
+
152
+
If you have verified all nodes and trust the node with the greatest seqno, that node is the intended bootstrap candidate. If two nodes show the same UUID and seqno, either can be used.
153
+
154
+
### Bootstrap step: set safe_to_bootstrap and start the first node
155
+
156
+
Only on the node that has the highest seqno from the validation step (and only after the caveats above are satisfied), set `safe_to_bootstrap` to 1 in that node’s `grastate.dat` file, then bootstrap from that node:
131
157
132
158
```shell
159
+
# On the chosen node only: edit grastate.dat and set safe_to_bootstrap: 1, then:
133
160
systemctl start mysql@bootstrap.service
134
161
```
135
162
136
-
After a shutdown, you can bootstrap from the node which is marked as safe in the `grastate.dat` file.
137
-
138
-
```{.text .no-copy}
139
-
...
140
-
safe_to_bootstrap: 1
141
-
...
142
-
```
163
+
After a clean shutdown in the future, you can bootstrap from the node which is marked as safe in the `grastate.dat` file (where `safe_to_bootstrap: 1`).
143
164
144
165
In recent Galera versions, the option [`pc.recovery`](wsrep-provider-index.md#pcrecovery) (enabled by default) saves the cluster state into a file named `gvwstate.dat` on each member node. As the name of this option suggests (pc – primary component), it saves only a cluster being in the PRIMARY state. An example content of the file may look like this:
145
166
@@ -184,7 +205,7 @@ After this, you are able to work on the manually restored part of the cluster, a
184
205
185
206
Then, as the Galera replication model truly cares about data consistency: once the inconsistency is detected, nodes that cannot execute row change statement due to a data difference – an emergency shutdown will be performed and the only way to bring the nodes back to the cluster is via the full [SST](glossary.md#sst)
186
207
187
-
**Based on material from Percona Database Performance Blog**
208
+
Based on material from Percona Database Performance Blog
188
209
189
210
This article is based on the blog post [Galera replication - how to recover a PXC cluster by *Przemysław Malkowski* :octicons-link-external-16:]: https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/
# Emergency quorum recovery (when nodes are up but traffic is blocked)
2
+
3
+
Nodes are running and accept connections, but MySQL refuses writes (and often reads) with errors such as `WSREP has not yet prepared node for application use`. The cluster has lost quorum or the *primary component*.
4
+
5
+
Why this happens: Each node has one vote; the cluster needs a majority to form a *primary component*. Only a primary accepts SQL. If one node leaves cleanly, votes decrease and the remaining nodes can form a primary—but if a node crashes (power loss, kill, kernel panic), the others still expect its vote until they time out. In a 3-node cluster, two nodes left after a crash often cannot form quorum (they expect three votes). Likewise, with two nodes left after a planned restart, a network flicker between those two can cause both to drop to non-primary; the cluster is then "online" but refuses every query.
6
+
7
+
Recovery options:
8
+
9
+
1. Restore connectivity so the remaining nodes can see each other and re-form a primary.
10
+
2. Bring the missing node(s) back so the cluster can form quorum again.
11
+
3. Emergency override (when you have confirmed the other nodes are really down): force one node to form a new primary so it can serve traffic; then start the other nodes so they rejoin.
12
+
13
+
## Emergency override: force a primary when traffic is blocked
14
+
15
+
Run the following on one node that is still up (connected as a user with sufficient privileges):
16
+
17
+
```sql
18
+
SET GLOBAL wsrep_provider_options='pc.bootstrap=YES';
19
+
```
20
+
21
+
The [`pc.bootstrap`](wsrep-provider-index.md#pcbootstrap) option makes that node form a new primary component. After the command runs, the node accepts writes; the cluster is effectively that one node until the others are started and rejoin (via IST or SST). Then start the other nodes so they join this primary.
22
+
23
+
!!! warning "Only when the other nodes are down"
24
+
25
+
Run this override only when you have confirmed that the other nodes are actually down or unreachable. If another node is still primary elsewhere (for example, in another datacenter after a split), setting `pc.bootstrap=YES` on a second node creates two separate clusters with diverging data (split-brain). See [Scenario 5: Two nodes disappear from the cluster](crash-recovery.md#scenario-5-two-nodes-disappear-from-the-cluster) and [Scenario 7: Split brain](crash-recovery.md#scenario-7-the-cluster-loses-its-primary-state-due-to-split-brain) in Crash recovery.
26
+
27
+
For more support options, see [Get help from Percona](get-help.md).
These often prevent nodes from joining and do not show up in the MySQL error log. If a node refuses to join, work through this section. See [Restart the cluster nodes](restarting-nodes.md) for the full recovery index.
4
+
5
+
## Security context (AppArmor, SELinux): the "SST jail"
6
+
7
+
On Ubuntu, the default AppArmor profile does not know about XtraBackup or socat and can block the donor from sending data and the joiner from running the SST method. The failure is often silent. Without fixing this, no amount of PID file removal or grastate.dat editing will make the node join.
8
+
9
+
Option A — Temporary bridge (confirm AppArmor is the cause): Put both profiles in complain mode and restart MySQL on the joiner. If the node joins, add proper rules and put the profiles back in enforce mode. Run as root or with `sudo`:
10
+
11
+
```shell
12
+
aa-complain /usr/sbin/mysqld
13
+
aa-complain /usr/bin/wsrep_sst_xtrabackup-v2
14
+
systemctl restart mysql
15
+
```
16
+
17
+
Option B — Keep enforce mode, allow SST: Add rules to allow the SST script to run xtrabackup and socat. Add to both `/etc/apparmor.d/usr.sbin.mysqld` and `/etc/apparmor.d/usr.bin.wsrep_sst_xtrabackup-v2` (adjust paths for your datadir and binaries):
18
+
19
+
```text
20
+
/usr/bin/xtrabackup rix,
21
+
/usr/bin/xbstream rix,
22
+
/usr/bin/socat rix,
23
+
/var/lib/mysql/ r,
24
+
/var/lib/mysql/** rwk,
25
+
```
26
+
27
+
Then `systemctl reload apparmor` and `systemctl start mysql`. If the node joined after Option A, add rules like Option B (see [Enable AppArmor](apparmor.md)), then:
28
+
29
+
```shell
30
+
aa-enforce /usr/sbin/mysqld
31
+
aa-enforce /usr/bin/wsrep_sst_xtrabackup-v2
32
+
```
33
+
34
+
SELinux (RHEL / Rocky / Alma): Ensure the SST script and ports are allowed; see [SELinux](selinux.md).
35
+
36
+
## Systemd timeout: the "silent killer"
37
+
38
+
On Ubuntu 22.04, systemd manages MySQL. If an SST takes longer than the start timeout (often 90 seconds), systemd kills the process. The user thinks the node "refuses to join," but it was killed by the OS mid-SST. Check `journalctl -u mysql` for timeout or unit failure.
39
+
40
+
The fix: Create a drop-in so the mysql unit has no start timeout. Run as root or with `sudo`:
41
+
42
+
```shell
43
+
systemctl edit mysql.service
44
+
```
45
+
46
+
(On RHEL use `systemctl edit mysqld.service`.) In the editor, add and save:
47
+
48
+
```ini
49
+
[Service]
50
+
TimeoutStartSec=0
51
+
```
52
+
53
+
Then:
54
+
55
+
```shell
56
+
systemctl daemon-reload
57
+
systemctl start mysql
58
+
```
59
+
60
+
`TimeoutStartSec=0` disables the start timeout so systemd waits indefinitely for SST and recovery. If the node was already killed mid-SST, you may need to clear or restore the data directory and retry after increasing the timeout.
61
+
62
+
## Firewalls and network
63
+
64
+
Ensure the SST port (default 4444 for Clone, or the port used by your SST method) and the cluster communication ports are open between joiner and donor. Blocked ports cause silent join failures. See [Secure the network](secure-network.md) for cluster and client connectivity.
65
+
66
+
For more support options, see [Get help from Percona](get-help.md).
To restart a cluster node, shut down MySQL and restart the service. The node leaves the cluster, reducing the total vote count for quorum.
3
+
This page is the entry point for restarts and recovery when nodes are managed directly (systemd, bare metal, or VMs). In PXC 8.4, Clone SST (`wsrep_sst_method=clone`) is the default rejoin path—set it once and most restarts are "clear port/process, start the service, let Clone run." For Kubernetes, use the [Percona Operator for MySQL](https://docs.percona.com/percona-operator-for-mysql/pxc/) (PXC-based); it automates bootstrap and recovery. For full-cluster down (all nodes stopped, bootstrap from scratch), see [Crash recovery](crash-recovery.md).
4
4
5
-
The quorum refers to the minimum number of votes required for the cluster to operate effectively and make decisions. Each node in the cluster typically represents one vote. When a node leaves the cluster, the total number of votes decreases, affecting the cluster's ability to achieve quorum. If the cluster does not maintain quorum, it may become unable to process transactions or make changes, potentially leading to a split-brain scenario where different parts of the cluster operate independently.
5
+
Which problem do you have?
6
6
7
-
Upon rejoining, the node synchronizes using IST (Incremental State Transfer). IST allows the node to catch up with the current state of the cluster by transferring only the changes that occurred while the node was offline. If the necessary changes for IST do not exist in the `gcache` file on any other node within the cluster, the process will perform SST (State Snapshot Transfer) instead. SST involves transferring a complete database snapshot to the node, which can be more time-consuming but ensures that the node receives all data. This approach makes restarting cluster nodes for rolling configuration changes, or software upgrades straightforward from the cluster’s perspective.
7
+
| Scenario | Topic |
8
+
|----------|--------|
9
+
| Nodes are up but traffic is blocked — Cluster accepts connections but refuses every SQL query (`WSREP has not yet prepared node for application use`). Quorum or primary component lost. |[Emergency quorum recovery](emergency-quorum-recovery.md)|
10
+
| Nodes refuse to join — Joiner won't sync, SST/Clone fails, or node never reaches `Synced`. |[SST/Clone failure recovery](sst-clone-failure-recovery.md)|
11
+
| Environmental blockers — AppArmor, systemd killing the process mid-SST, or firewalls blocking SST/GCOMM. |[Environmental blockers](environmental-blockers.md)|
8
12
9
-
If a node restarts with an invalid configuration change that prevents MySQL from loading, Galera drops the node’s state and forces an SST for that node.
10
-
11
-
In the event of a MySQL failure, the system does not remove the PID file because the system deletes this file only during a clean shutdown. As a result, the server does not restart if an existing PID file is present. When MySQL encounters a failure, check the log records for details. You must remove the PID file manually.
12
-
13
-
Use the `rm` command in a Unix/Linux shell to do this:
14
-
15
-
```shell
16
-
bash rm /path/to/mysql.pid
17
-
```
18
-
19
-
Replace `/path/to/mysql.pid` with the actual path to your MySQL PID file. The default location for the PID file is often `/var/run/mysqld/mysqld.pid` or `/var/lib/mysql/mysql.pid`, but this can vary based on your configuration. Before executing this command, ensure that MySQL is not running, as removing the PID file while the server is active can lead to issues.
13
+
Each topic is a separate page with step-by-step procedures. Start with the one that matches your situation; the pages cross-link where needed (for example, SST/Clone failure recovery points to Environmental blockers when the cause is AppArmor or systemd).
0 commit comments