PXC-5113 [DOCS] - add troubleshooting steps for a stalled SST 8.4

	modified:   docs/restarting-nodes.md

Author: patrickbirch

docs/_static/scenario-1.png

@@ -6,6 +6,8 @@ However, there are scenarios where the database service can stop with no node be
 
 ## Scenario 1: Node A is gracefully stopped
 
+![Scenario 1: Node A gracefully stopped in a three-node cluster](_static/scenario-1.png)
+
 In a three node cluster (node A, Node B, node C), one node (node A, for example) is gracefully stopped: for the purpose of maintenance, configuration change, etc.
 
 In this case, the other nodes receive a “good bye” message from the stopped node and the cluster size is reduced; some properties like quorum calculation or auto increment are automatically changed. As soon as node A is started again, it joins the cluster based on its [`wsrep_cluster_address`](wsrep-system-index.md#wsrep_cluster_address) variable in `my.cnf`.
@@ -15,6 +17,8 @@ If the writeset cache ([`gcache.size`](wsrep-provider-index.md#gcachesize)) on n
 
 ## Scenario 2: Two nodes are gracefully stopped
 
+![Scenario 2: Two nodes gracefully stopped; one node remains in the cluster](_static/scenario-2.png)
+
 Similar to [Scenario 1: Node A is gracefully stopped](#scenario-1-node-a-is-gracefully-stopped), the cluster size is reduced to one — even the single remaining node C forms the primary component and is able to serve client requests. To get the nodes back into the cluster, you just need to start them.
 
 However, when a new node joins the cluster, node C will be switched to the “Donor/Desynced” state as it has to provide the state transfer at least to the first joining node. It is still possible to read/write to it during that process, but it may be much slower, which depends on how large amount of data should be sent during the state transfer. Also, some load balancers may consider the donor node as not operational and remove it from the pool. So, it is best to avoid the situation when only one node is up.
@@ -31,6 +35,8 @@ systemctl start mysql
 
 ## Scenario 3: All three nodes are gracefully stopped
 
+![Scenario 3: All three cluster nodes gracefully stopped](_static/scenario-3.png)
+
 The cluster is completely stopped and the problem is to initialize it again. It is important that a PXC node writes its last executed position to the `grastate.dat` file.
 
 By comparing the seqno number in this file, you can see which is the most advanced node (most likely the last stopped). The cluster must be bootstrapped using this node, otherwise nodes that had a more advanced position will have to perform the full [SST](glossary.md#sst) to join the cluster initialized from the less advanced one. As a result, some transactions will be lost). To bootstrap the first node, invoke the startup script like this:
@@ -47,16 +53,22 @@ systemctl start mysql@bootstrap.service
 
 ## Scenario 4: One node disappears from the cluster
 
-This is the case when one node becomes unavailable due to power outage, hardware failure, kernel panic, mysqld crash, **kill -9** on mysqld pid, etc.
+![Scenario 4: One node disappears from the cluster](_static/scenario-4.png)
+
+This is the case when one node becomes unavailable due to power outage, hardware failure, kernel panic, mysqld crash, kill -9 on mysqld pid, etc.
 
 Two remaining nodes notice the connection to node A is down and start trying to re-connect to it. After several timeouts, node A is removed from the cluster. The quorum is saved (two out of three nodes are up), so no service disruption happens. After it is restarted, node A joins automatically (as described in [Scenario 1: Node A is gracefully stopped](#scenario-1-node-a-is-gracefully-stopped)).
 
 ## Scenario 5: Two nodes disappear from the cluster
 
-Two nodes are not available and the remaining node (node C) is not able to form the quorum alone. The cluster has to switch to a non-primary mode, where MySQL refuses to serve any SQL queries. In this state, the **mysqld** process on node C is still running and can be connected to but any statement related to data fails with an error.
+![Scenario 5: Two nodes disappear from the cluster](_static/scenario-5.png)
+
+Two nodes are not available and the remaining node (node C) is not able to form the quorum alone. The cluster has to switch to a non-primary mode. While node C is still deciding whether it can reach the other nodes, reads may still work and new writes are usually refused. Once node C gives up and the component is non-primary, `wsrep_ready` is `OFF` and normal client queries—including trivial selects—fail.
+
+For example:
 
 ```sql
-SELECT * FROM test.sbtest1;
+SELECT 1 FROM DUAL;
 ```
 
 ??? example "The error message"
@@ -65,7 +77,7 @@ SELECT * FROM test.sbtest1;
     ERROR 1047 (08S01): WSREP has not yet prepared node for application use
     ```
 
-Reads are possible until node C decides that it cannot access node A and node B. New writes are forbidden.
+The SQLSTATE is `08S01`; some builds or code paths may show `ERROR 1047 (08S01): Unknown Command` instead of the longer WSREP text—both are the same class of failure when the node is not prepared for application use.
 
 As soon as the other nodes become available, the cluster is formed again automatically. If node B and node C were just network-severed from node A, but they can still reach each other, they will keep functioning as they still form the quorum.
 
@@ -83,6 +95,8 @@ This approach only works if the other nodes are down before doing that! Otherwis
 
 ## Scenario 6: All nodes went down without a proper shutdown procedure
 
+![Scenario 6: All nodes went down without a proper shutdown](_static/scenario-6.png)
+
 This scenario is possible in the following cases:
 
 * Data center power failure
@@ -109,37 +123,58 @@ cat /var/lib/mysql/grastate.dat
     safe_to_bootstrap: 0
     ```
 
-In this case, you cannot be sure that all nodes are consistent with each other. We cannot use `safe_to_bootstrap` variable to determine the node that has the last transaction committed as this variable is set to **0** for each node. 
+In this case, you cannot be sure that all nodes are consistent with each other. The `safe_to_bootstrap` variable is set to 0 on every node and cannot be used to identify which node has the last transaction committed.
+
+!!! warning "Risk of split-brain"
+
+    Setting `safe_to_bootstrap: 1` on a node without first confirming that node has the highest recovered position can cause split-brain and data loss. Always run the validation step below on every node and bootstrap only from the node with the highest seqno.
 
-An attempt to bootstrap from such a node will fail unless you start `mysqld` with the `--wsrep-recover` option:
+### Validation step: recover and record position on every node
+
+On each node that was part of the cluster, run `mysqld` with the `--wsrep-recover` option so that the server prints the recovered position and exits (the server does not stay running):
 
 ```shell
 mysqld --wsrep-recover
 ```
 
-Search the output for the line that reports the recovered position after the node UUID (**1122** in this case):
+In the output, find the line that reports the recovered position in the form `UUID:seqno`:
 
-??? example "Expected output"
+??? example "Example output"
 
     ```{.text .no-copy}
     ...
     ... [Note] WSREP: Recovered position: 220dcdcb-1629-11e4-add3-aec059ad3734:1122
     ...
     ```
 
-The node where the recovered position is marked by the greatest number is the best bootstrap candidate. In its `grastate.dat` file, set the safe_to_bootstrap variable to **1**. Then, bootstrap from this node:
+Run the command on every node and record the UUID and seqno from each. Use a table like the following so that you can compare and choose the correct bootstrap candidate:
+
+| Node (hostname or label) | UUID | seqno |
+|--------------------------|------|-------|
+| node1                    |      |       |
+| node2                    |      |       |
+| node3                    |      |       |
+
+!!! warning "When highest seqno is not safe to use"
+
+    The procedure below assumes you have access to every node that was in the cluster and that the recovered positions are trustworthy. If either is false, bootstrapping from the node with the highest seqno can permanently destroy data.
+
+    * Access to all nodes: If a node is unreachable (for example, in another datacenter or still down), you cannot assume the highest seqno you see is the true cluster state. The missing node may have had a higher seqno. Bootstrap only after you have run `mysqld --wsrep-recover` on every member and recorded the result.
+
+    * Trustworthiness of the "highest" node: A node can report a higher seqno but have corrupt or incomplete data—for example, after a partition (it was in a minority and applied writes that were never committed cluster-wide), a write-ahead or disk failure (it reported a seqno that was not fully persisted), or an unclean shutdown. Bootstrapping from that node forces the rest of the cluster to sync to that state. The cluster will then permanently drop or overwrite the transactions that existed only on the other nodes. If you suspect the "highest" node was partitioned, had storage or write-ahead issues, or you cannot verify its history, do not bootstrap from it without expert guidance or a verified backup strategy. Prefer [Get help from Percona](get-help.md) or your support channel when in doubt.
+
+If you have verified all nodes and trust the node with the greatest seqno, that node is the intended bootstrap candidate. If two nodes show the same UUID and seqno, either can be used.
+
+### Bootstrap step: set safe_to_bootstrap and start the first node
+
+Only on the node that has the highest seqno from the validation step (and only after the caveats above are satisfied), set `safe_to_bootstrap` to 1 in that node’s `grastate.dat` file, then bootstrap from that node:
 
 ```shell
+# On the chosen node only: edit grastate.dat and set safe_to_bootstrap: 1, then:
 systemctl start mysql@bootstrap.service
 ```
 
-After a shutdown, you can bootstrap from the node which is marked as safe in the `grastate.dat` file.
-
-```{.text .no-copy}
-...
-safe_to_bootstrap: 1
-...
-```
+After a clean shutdown in the future, you can bootstrap from the node which is marked as safe in the `grastate.dat` file (where `safe_to_bootstrap: 1`).
 
 In recent Galera versions, the option [`pc.recovery`](wsrep-provider-index.md#pcrecovery) (enabled by default) saves the cluster state into a file named `gvwstate.dat` on each member node. As the name of this option suggests (pc – primary component), it saves only a cluster being in the PRIMARY state. An example content of the file may look like this:
 
@@ -168,6 +203,8 @@ The log file shows recovery completion:
 
 ## Scenario 7: The cluster loses its primary state due to split brain
 
+![Scenario 7: Split brain; cluster loses primary state](_static/scenario-7.png)
+
 For the purpose of this example, let’s assume we have a cluster that consists of an even number of nodes: six, for example. Three of them are in one location while the other three are in another location and they lose network connectivity.  It is best practice to avoid such topology: if you cannot have an odd number of real nodes, you can use an additional arbitrator (garbd) node or set a higher pc.weight to some nodes. But when the split brain happens any way, none of the separated groups can maintain the quorum: all nodes must stop serving requests and both parts of the cluster will be continuously trying to re-connect.
 
 If you want to restore the service even before the network link is restored, you can make one of the groups primary again using the same command as described in [Scenario 5: Two nodes disappear from the cluster](#scenario-5-two-nodes-disappear-from-the-cluster)
@@ -184,7 +221,7 @@ After this, you are able to work on the manually restored part of the cluster, a
 
     Then, as the Galera replication model truly cares about data consistency: once the inconsistency is detected, nodes that cannot execute row change statement due to a data difference – an emergency shutdown will be performed and the only way to bring the nodes back to the cluster is via the full [SST](glossary.md#sst)
 
-**Based on material from Percona Database Performance Blog**
+Based on material from Percona Database Performance Blog
 
 This article is based on the blog post [Galera replication - how to recover a PXC cluster by *Przemysław Malkowski* :octicons-link-external-16:]: https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/ 
 

docs/_static/scenario-2.png

@@ -0,0 +1,27 @@
+# Emergency quorum recovery (when nodes are up but traffic is blocked)
+
+Nodes are running and accept connections, but MySQL refuses writes (and often reads) with errors such as `WSREP has not yet prepared node for application use`. The cluster has lost quorum or the *primary component*.
+
+Why this happens: Each node has one vote; the cluster needs a majority to form a *primary component*. Only a primary accepts SQL. If one node leaves cleanly, votes decrease and the remaining nodes can form a primary—but if a node crashes (power loss, kill, kernel panic), the others still expect its vote until they time out. In a 3-node cluster, two nodes left after a crash often cannot form quorum (they expect three votes). Likewise, with two nodes left after a planned restart, a network flicker between those two can cause both to drop to non-primary; the cluster is then "online" but refuses every query.
+
+Recovery options:
+
+1. Restore connectivity so the remaining nodes can see each other and re-form a primary.
+2. Bring the missing node(s) back so the cluster can form quorum again.
+3. Emergency override (when you have confirmed the other nodes are really down): force one node to form a new primary so it can serve traffic; then start the other nodes so they rejoin.
+
+## Emergency override: force a primary when traffic is blocked
+
+Run the following on one node that is still up (connected as a user with sufficient privileges):
+
+```sql
+SET GLOBAL wsrep_provider_options='pc.bootstrap=YES';
+```
+
+The [`pc.bootstrap`](wsrep-provider-index.md#pcbootstrap) option makes that node form a new primary component. After the command runs, the node accepts writes; the cluster is effectively that one node until the others are started and rejoin (via IST or SST). Then start the other nodes so they join this primary.
+
+!!! warning "Only when the other nodes are down"
+
+    Run this override only when you have confirmed that the other nodes are actually down or unreachable. If another node is still primary elsewhere (for example, in another datacenter after a split), setting `pc.bootstrap=YES` on a second node creates two separate clusters with diverging data (split-brain). See [Scenario 5: Two nodes disappear from the cluster](crash-recovery.md#scenario-5-two-nodes-disappear-from-the-cluster) and [Scenario 7: Split brain](crash-recovery.md#scenario-7-the-cluster-loses-its-primary-state-due-to-split-brain) in Crash recovery.
+
+For more support options, see [Get help from Percona](get-help.md).

docs/_static/scenario-3.png

@@ -0,0 +1,66 @@
+# Environmental blockers (AppArmor, systemd, firewalls)
+
+These often prevent nodes from joining and do not show up in the MySQL error log. If a node refuses to join, work through this section. See [Restart the cluster nodes](restarting-nodes.md) for the full recovery index.
+
+## Security context (AppArmor, SELinux): the "SST jail"
+
+On Ubuntu, the default AppArmor profile does not know about XtraBackup or socat and can block the donor from sending data and the joiner from running the SST method. The failure is often silent. Without fixing this, no amount of PID file removal or grastate.dat editing will make the node join.
+
+Option A — Temporary bridge (confirm AppArmor is the cause): Put both profiles in complain mode and restart MySQL on the joiner. If the node joins, add proper rules and put the profiles back in enforce mode. Run as root or with `sudo`:
+
+```shell
+aa-complain /usr/sbin/mysqld
+aa-complain /usr/bin/wsrep_sst_xtrabackup-v2
+systemctl restart mysql
+```
+
+Option B — Keep enforce mode, allow SST: Add rules to allow the SST script to run xtrabackup and socat. Add to both `/etc/apparmor.d/usr.sbin.mysqld` and `/etc/apparmor.d/usr.bin.wsrep_sst_xtrabackup-v2` (adjust paths for your datadir and binaries):
+
+```text
+  /usr/bin/xtrabackup rix,
+  /usr/bin/xbstream rix,
+  /usr/bin/socat rix,
+  /var/lib/mysql/ r,
+  /var/lib/mysql/** rwk,
+```
+
+Then `systemctl reload apparmor` and `systemctl start mysql`. If the node joined after Option A, add rules like Option B (see [Enable AppArmor](apparmor.md)), then:
+
+```shell
+aa-enforce /usr/sbin/mysqld
+aa-enforce /usr/bin/wsrep_sst_xtrabackup-v2
+```
+
+SELinux (RHEL / Rocky / Alma): Ensure the SST script and ports are allowed; see [SELinux](selinux.md).
+
+## Systemd timeout: the "silent killer"
+
+On Ubuntu 22.04, systemd manages MySQL. If an SST takes longer than the start timeout (often 90 seconds), systemd kills the process. The user thinks the node "refuses to join," but it was killed by the OS mid-SST. Check `journalctl -u mysql` for timeout or unit failure.
+
+The fix: Create a drop-in so the mysql unit has no start timeout. Run as root or with `sudo`:
+
+```shell
+systemctl edit mysql.service
+```
+
+(On RHEL use `systemctl edit mysqld.service`.) In the editor, add and save:
+
+```ini
+[Service]
+TimeoutStartSec=0
+```
+
+Then:
+
+```shell
+systemctl daemon-reload
+systemctl start mysql
+```
+
+`TimeoutStartSec=0` disables the start timeout so systemd waits indefinitely for SST and recovery. If the node was already killed mid-SST, you may need to clear or restore the data directory and retry after increasing the timeout.
+
+## Firewalls and network
+
+Ensure the SST port (default 4444 for Clone, or the port used by your SST method) and the cluster communication ports are open between joiner and donor. Blocked ports cause silent join failures. See [Secure the network](secure-network.md) for cluster and client connectivity.
+
+For more support options, see [Get help from Percona](get-help.md).