Correct HA stateless node removal guidance (#62264)

Co-authored-by: Andrey Bazhutkin <andrba@github.com>

Author: steph-sabotasan

content/admin/administering-your-instance/administering-your-instance-from-the-command-line/command-line-utilities.md

@@ -1159,27 +1159,32 @@ ghe-dpages evacuate pages-server-UUID
 
 ### ghe-remove-node
 
-This utility removes a node from a cluster. If you're replacing a node, after you've set up a replacement node, you can use this command to take the old node offline. For more information, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/configuring-clustering/replacing-a-cluster-node).
+This utility removes a node from a cluster{% ifversion ghes > 3.17 %} or an additional node from a high availability (HA) configuration{% endif %}. For a planned replacement of a functional cluster node, set up the replacement node before using this command to remove the old node. For more information, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-a-functional-node).{% ifversion ghes > 3.17 %} For the required HA checks and verification steps, see [Removing an additional node](/admin/monitoring-and-managing-your-instance/additional-nodes/configuring-additional-nodes#removing-an-additional-node).{% endif %}
 
-You must run this command from the primary MySQL node in your cluster, which is typically the node designated as `mysql-master` in your cluster configuration file (`cluster.conf`). You can use this command to remove any node, with the exception of the `mysql-master` or `redis-master` node. For more information, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/configuring-clustering/initializing-the-cluster#about-the-cluster-configuration-file).
+Before using this command for a planned removal, install the latest patch release for your feature release on every node. Every node must run the same exact release. Wait for any upgrade or configuration run to finish before starting removal. For emergency replacement of an unavailable cluster node, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/configuring-clustering/replacing-a-cluster-node#replacing-a-node-in-an-emergency).
+
+You must run this command from the primary MySQL node, which is typically the node designated as `mysql-master` in the cluster configuration file (`cluster.conf`).{% ifversion ghes > 3.17 %} In an HA configuration, run the command from the HA primary.{% endif %} You cannot remove the `mysql-master` or `redis-master` node. For more information, see [AUTOTITLE](/admin/monitoring-and-managing-your-instance/configuring-clustering/initializing-the-cluster#about-the-cluster-configuration-file).
 
 ```shell
 ghe-remove-node HOSTNAME
 ```
 
 The command does the following things:
-* Evacuates data from any data services running on the node, so that the remaining nodes in your cluster contain copies of the data
-* Marks the node as offline in your configuration, applies this change to the rest of the nodes in the cluster, and stops traffic being routed to the node
+
+* Evacuates data from any data services running on the node, so that the remaining nodes contain copies of the data
+* Drains workloads from the node
+* Removes the node from the configuration.{% ifversion ghes > 3.17 %} If another non-primary node remains, the command runs `ghe-config-apply` and stops routing traffic to the removed node. If no non-primary node remains, the command removes cluster metadata and converts the primary to a standalone instance without running `ghe-config-apply`.{% else %} The command runs `ghe-config-apply` and stops routing traffic to the removed node.{% endif %}
 
 You can run the command with the following flags.
 
 Flag | Description
 ---- | ----------
-`-ne/--no-evacuate` | Skips evacuation of data services (warning: may result in data loss).
+`-ne/--no-evacuate` | Marks the node offline in the configuration instead of removing it, and skips evacuation of data services (warning: may result in data loss).
 `-v/--verbose` | Prints additional information to the console.
 
 > [!NOTE]
-> * This command can only be used to remove a node from a cluster configuration. It cannot be used to remove a node from a high availability configuration.
+> {% ifversion ghes > 3.17 %}* In an HA configuration, you can use this command to remove an additional node. You cannot use it to remove the HA primary or a replica.{% endif %}
+> * The target node must report `ready` in `nomad node status` to complete removal. The `--no-evacuate` flag does not remove an offline node from the configuration.
 > * This command does not support parallel execution. To remove multiple nodes, you must wait until this command has finished before running it for another node.
 
 ### ghe-spokesctl

content/admin/monitoring-and-managing-your-instance/additional-nodes/configuring-additional-nodes.md

@@ -64,15 +64,39 @@ We recommend a maintenance window to add stateless nodes.
 
 ## Removing an additional node
 
-To remove a node, run `ghe-remove-node` from the node you want to remove. Then, on the primary node, you must run:
+Before removing an additional node, install the same latest patch for your feature release on every node in the HA deployment and schedule a maintenance window. Wait for any upgrade or configuration run to finish before starting removal.
 
-``` shell copy
-ghe-config-apply
-```
+1. On the HA primary, check the status of every node in the HA deployment.
+
+    ```shell copy
+    ghe-cluster-nodes
+    ghe-cluster-nodes --offline
+    nomad node status
+    ghe-cluster-status --extended --verbose
+    ```
+
+    Confirm that both `ghe-cluster-nodes` commands list the same hostnames and include the hostname of the node you plan to remove. Confirm that every node has a Nomad status of `ready`, and `connect-ssh` and `enterprise-version` are `ok` for every node. Confirm that stateful services on the primary and any replicas are healthy. If no replica remains, a warning that no MySQL replica was found is expected. Failures limited to web, job, or memcache workloads on the target do not block removal. If any other node-level or stateful-service check fails, contact {% data variables.contact.github_support %} before removal.
+
+1. On the HA primary, remove the additional node. Replace `HOSTNAME` with the hostname of the additional node.
+
+    ```shell copy
+    ghe-remove-node --verbose HOSTNAME
+    ```
+
+    If another non-primary node remains, the command drains the target, removes it from the HA configuration, and runs `ghe-config-apply`. If no non-primary node remains, the command removes the cluster metadata and converts the primary to a standalone instance without running `ghe-config-apply`. Do not run `ghe-config-apply` separately in either case.
+
+1. Verify the removal.
+
+    If another non-primary node remains, run the following commands on the HA primary. Confirm that the hostname is absent and that the HA configuration is healthy.
+
+    ```shell copy
+    ghe-cluster-nodes --offline
+    ghe-cluster-status --extended --verbose
+    ```
 
-The `ghe-config-apply` command is a requirement to remove stateless nodes.
+    If no non-primary node remains, do not run the cluster-only commands. Confirm that the removal output contains `Cluster artifacts removed; now standalone.`, then confirm that the primary serves user traffic and processes web and job workloads.
 
-We recommend a maintenance window to remove stateless nodes.
+If an additional node is offline, unreachable, or on a different version, or if `ghe-remove-node` or a verification check fails, contact {% data variables.contact.github_support %}. Do not edit `cluster.conf` manually.
 
 ## Reprovisioning a node that previously hosted {% data variables.product.prodname_ghe_server %}
 

content/admin/monitoring-and-managing-your-instance/configuring-clustering/replacing-a-cluster-node.md

@@ -1,6 +1,6 @@
 ---
 title: Replacing a cluster node
-intro: If a node fails in a {% data variables.product.prodname_ghe_server %} cluster, or if you want to add a new node with more resources, mark any nodes to replace as offline, then add the new node.
+intro: Replace a functional or failed node in a {% data variables.product.prodname_ghe_server %} cluster while preserving the services that the node provides.
 product: '{% data reusables.gated-features.cluster %}'
 redirect_from:
   - /enterprise/admin/clustering/replacing-a-cluster-node
@@ -28,7 +28,9 @@ After you replace a node, {% data variables.location.product_location %} does no
 
 You can replace an existing, functional node in your cluster. For example, you may want to provide a virtual machine (VM) with additional CPU, memory, or storage resources.
 
-To replace a functional node, install the {% data variables.product.prodname_ghe_server %} appliance on a new VM, configure an IP address, add the new node to the cluster configuration file, initialize the cluster and apply the configuration, then take the node you replaced offline.
+To replace a functional node, install the {% data variables.product.prodname_ghe_server %} appliance on a new VM, configure an IP address, add the new node to the cluster configuration file, initialize the cluster and apply the configuration, then remove the node you replaced.
+
+Before starting the replacement, install the latest patch release for your feature release on every cluster node, including the replacement node. Every node must run the same exact release. Wait for any upgrade or configuration run to finish before starting replacement.
 
 > [!NOTE]
 > If you're replacing the primary database node, see [Replacing the primary database node](#replacing-the-primary-database-node-mysql-or-mysql-and-mssql).
@@ -38,13 +40,13 @@ To replace a functional node, install the {% data variables.product.prodname_ghe
 {% data reusables.enterprise_clustering.replacing-a-cluster-node-modify-cluster-conf %}
 {% data reusables.enterprise_clustering.replacing-a-cluster-node-initialize-new-node %}
 {% data reusables.enterprise_clustering.replacing-a-cluster-node-config-node %}
-1. To take the node you're replacing offline, from the primary MySQL node of your cluster, run the following command.
+1. To remove the node you're replacing, from the primary MySQL node of your cluster, run the following command.
 
    ```shell
    ghe-remove-node NODE-HOSTNAME
    ```
 
-   This command will evacuate data from any data services running on the node, mark the node as offline in your configuration, and stop traffic being routed to the node. For more information, see [AUTOTITLE](/admin/administering-your-instance/administering-your-instance-from-the-command-line/command-line-utilities#ghe-remove-node).
+   This command evacuates data from any data services running on the node, drains its workloads, removes the node from the cluster configuration, applies the change, and stops traffic from being routed to the node. For more information, see [AUTOTITLE](/admin/administering-your-instance/administering-your-instance-from-the-command-line/command-line-utilities#ghe-remove-node).
 
 ## Replacing a node in an emergency
 
@@ -55,6 +57,8 @@ You can replace a failed node in your cluster. For example, a software or hardwa
 
 To replace a node in an emergency, you'll take the failed node offline, add your replacement node to the cluster, then run commands to remove references to data services on the removed node.
 
+Before starting the replacement, confirm that every available node that will remain in the cluster already runs the latest patch release for your feature release. Install that exact release on the replacement node. If the remaining nodes are not already on that release, contact {% data variables.contact.github_support %} before continuing. Wait for any active upgrade or configuration run to finish.
+
 1. To remove the node that is experiencing issues from the cluster, from the primary MySQL node of your cluster, run the following command. Replace NODE-HOSTNAME with the hostname of the node you're taking offline.
 
    ```shell

data/reusables/enterprise_clustering/replacing-a-cluster-node-modify-cluster-conf.md

@@ -1,4 +1,4 @@
-1. To add the newly provisioned replacement node, on any node, modify the `cluster.conf` file to remove the failed node and add the replacement node. For example, this modified `cluster.conf` file replaces `ghe-data-node-3` with the newly provisioned node, `ghe-replacement-data-node-3`:
+1. To add the newly provisioned replacement node, on any node, modify the `cluster.conf` file to add the replacement node. Keep the node you're replacing in `cluster.conf` until you run `ghe-remove-node` later in this procedure. For example, this modified `cluster.conf` file adds the newly provisioned node, `ghe-replacement-data-node-3`:
 
    <pre>
    [cluster "<em>ghe-replacement-data-node-3</em>"]