PXC-5113 [DOCS] - add troubleshooting steps for a stalled SST 8.4

	modified:   docs/restarting-nodes.md

Author: patrickbirch

docs/crash-recovery.md

@@ -47,13 +47,13 @@ systemctl start mysql@bootstrap.service
 
 ## Scenario 4: One node disappears from the cluster
 
-This is the case when one node becomes unavailable due to power outage, hardware failure, kernel panic, mysqld crash, **kill -9** on mysqld pid, etc.
+This is the case when one node becomes unavailable due to power outage, hardware failure, kernel panic, mysqld crash, kill -9 on mysqld pid, etc.
 
 Two remaining nodes notice the connection to node A is down and start trying to re-connect to it. After several timeouts, node A is removed from the cluster. The quorum is saved (two out of three nodes are up), so no service disruption happens. After it is restarted, node A joins automatically (as described in [Scenario 1: Node A is gracefully stopped](#scenario-1-node-a-is-gracefully-stopped)).
 
 ## Scenario 5: Two nodes disappear from the cluster
 
-Two nodes are not available and the remaining node (node C) is not able to form the quorum alone. The cluster has to switch to a non-primary mode, where MySQL refuses to serve any SQL queries. In this state, the **mysqld** process on node C is still running and can be connected to but any statement related to data fails with an error.
+Two nodes are not available and the remaining node (node C) is not able to form the quorum alone. The cluster has to switch to a non-primary mode, where MySQL refuses to serve any SQL queries. In this state, the mysqld process on node C is still running and can be connected to but any statement related to data fails with an error.
 
 ```sql
 SELECT * FROM test.sbtest1;
@@ -109,37 +109,58 @@ cat /var/lib/mysql/grastate.dat
     safe_to_bootstrap: 0
     ```
 
-In this case, you cannot be sure that all nodes are consistent with each other. We cannot use `safe_to_bootstrap` variable to determine the node that has the last transaction committed as this variable is set to **0** for each node. 
+In this case, you cannot be sure that all nodes are consistent with each other. The `safe_to_bootstrap` variable is set to 0 on every node and cannot be used to identify which node has the last transaction committed.
 
-An attempt to bootstrap from such a node will fail unless you start `mysqld` with the `--wsrep-recover` option:
+!!! warning "Risk of split-brain"
+
+    Setting `safe_to_bootstrap: 1` on a node without first confirming that node has the highest recovered position can cause split-brain and data loss. Always run the validation step below on every node and bootstrap only from the node with the highest seqno.
+
+### Validation step: recover and record position on every node
+
+On each node that was part of the cluster, run `mysqld` with the `--wsrep-recover` option so that the server prints the recovered position and exits (the server does not stay running):
 
 ```shell
 mysqld --wsrep-recover
 ```
 
-Search the output for the line that reports the recovered position after the node UUID (**1122** in this case):
+In the output, find the line that reports the recovered position in the form `UUID:seqno`:
 
-??? example "Expected output"
+??? example "Example output"
 
     ```{.text .no-copy}
     ...
     ... [Note] WSREP: Recovered position: 220dcdcb-1629-11e4-add3-aec059ad3734:1122
     ...
     ```
 
-The node where the recovered position is marked by the greatest number is the best bootstrap candidate. In its `grastate.dat` file, set the safe_to_bootstrap variable to **1**. Then, bootstrap from this node:
+Run the command on every node and record the UUID and seqno from each. Use a table like the following so that you can compare and choose the correct bootstrap candidate:
+
+| Node (hostname or label) | UUID | seqno |
+|--------------------------|------|-------|
+| node1                    |      |       |
+| node2                    |      |       |
+| node3                    |      |       |
+
+!!! warning "When highest seqno is not safe to use"
+
+    The procedure below assumes you have access to every node that was in the cluster and that the recovered positions are trustworthy. If either is false, bootstrapping from the node with the highest seqno can permanently destroy data.
+
+    * Access to all nodes: If a node is unreachable (for example, in another datacenter or still down), you cannot assume the highest seqno you see is the true cluster state. The missing node may have had a higher seqno. Bootstrap only after you have run `mysqld --wsrep-recover` on every member and recorded the result.
+
+    * Trustworthiness of the "highest" node: A node can report a higher seqno but have corrupt or incomplete data—for example, after a partition (it was in a minority and applied writes that were never committed cluster-wide), a write-ahead or disk failure (it reported a seqno that was not fully persisted), or an unclean shutdown. Bootstrapping from that node forces the rest of the cluster to sync to that state. The cluster will then permanently drop or overwrite the transactions that existed only on the other nodes. If you suspect the "highest" node was partitioned, had storage or write-ahead issues, or you cannot verify its history, do not bootstrap from it without expert guidance or a verified backup strategy. Prefer [Get help from Percona](get-help.md) or your support channel when in doubt.
+
+If you have verified all nodes and trust the node with the greatest seqno, that node is the intended bootstrap candidate. If two nodes show the same UUID and seqno, either can be used.
+
+### Bootstrap step: set safe_to_bootstrap and start the first node
+
+Only on the node that has the highest seqno from the validation step (and only after the caveats above are satisfied), set `safe_to_bootstrap` to 1 in that node’s `grastate.dat` file, then bootstrap from that node:
 
 ```shell
+# On the chosen node only: edit grastate.dat and set safe_to_bootstrap: 1, then:
 systemctl start mysql@bootstrap.service
 ```
 
-After a shutdown, you can bootstrap from the node which is marked as safe in the `grastate.dat` file.
-
-```{.text .no-copy}
-...
-safe_to_bootstrap: 1
-...
-```
+After a clean shutdown in the future, you can bootstrap from the node which is marked as safe in the `grastate.dat` file (where `safe_to_bootstrap: 1`).
 
 In recent Galera versions, the option [`pc.recovery`](wsrep-provider-index.md#pcrecovery) (enabled by default) saves the cluster state into a file named `gvwstate.dat` on each member node. As the name of this option suggests (pc – primary component), it saves only a cluster being in the PRIMARY state. An example content of the file may look like this:
 
@@ -184,7 +205,7 @@ After this, you are able to work on the manually restored part of the cluster, a
 
     Then, as the Galera replication model truly cares about data consistency: once the inconsistency is detected, nodes that cannot execute row change statement due to a data difference – an emergency shutdown will be performed and the only way to bring the nodes back to the cluster is via the full [SST](glossary.md#sst)
 
-**Based on material from Percona Database Performance Blog**
+Based on material from Percona Database Performance Blog
 
 This article is based on the blog post [Galera replication - how to recover a PXC cluster by *Przemysław Malkowski* :octicons-link-external-16:]: https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/