Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 22 additions & 5 deletions modules/graceful-restart.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
[id="graceful-restart_{context}"]
= Restarting the cluster

[role="_abstract"]
You can restart your cluster after it has been shut down gracefully.

.Prerequisites
Expand Down Expand Up @@ -127,7 +128,7 @@ ip-10-0-182-134.ec2.internal Ready worker 64m v1.35.0
ip-10-0-250-100.ec2.internal Ready worker 64m v1.35.0
----

. If the compute nodes are _not_ ready, then check whether there are any pending certificate signing requests (CSRs) that must be approved.
. If the compute nodes are _not_ ready, then check whether there are any pending CSRs that must be approved.

.. Get the list of current CSRs:
+
Expand Down Expand Up @@ -184,9 +185,9 @@ $ oc adm certificate approve <csr_name>
$ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm uncordon ${node} ; done
----

. Verify that the cluster started properly.
.Verification

.. Check that there are no degraded cluster Operators.
. Check that there are no degraded cluster Operators.
+
[source,terminal]
----
Expand All @@ -209,7 +210,7 @@ etcd {product-version}.0 True Fa
...
----

.. Check that all nodes are in the `Ready` state:
. Check that all nodes are in the `Ready` state:
+
[source,terminal]
----
Expand All @@ -229,4 +230,20 @@ ip-10-0-211-16.ec2.internal Ready control-plane,master 82m v1.35.0
ip-10-0-250-100.ec2.internal Ready worker 69m v1.35.0
----
+
If the cluster did not start properly, you might need to restore your cluster using an etcd backup. For more information, see "Restoring to a previous cluster state".
If the cluster did not start properly, you might need to restore your cluster by using an etcd backup. For more information, see "Restoring to a previous cluster state".

. If a CSR is not approved after the restart, you can investigate possible issues by checking the logs:
+
.. Check the Machine Approver logs by entering the following command:
+
[source,terminal]
----
$ oc logs -n openshift-cluster-machine-approver -l app=machine-approver
----
+
.. Check the Kube Controller Manager logs by entering the following command:
+
[source,terminal]
----
$ oc logs -n openshift-kube-controller-manager -l app=kube-controller-manager
----
Comment on lines +235 to +249

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might make sense for us to be a bit more explicit as to the scenarios in which one might want to check which components logs.

Based on the README of https://github.com/openshift/cluster-machine-approver , it looks like machine-approver would be more for the scenario where CSRs for new worker nodes attempting to join the cluster have not been approved.

It looks like kube-controller-manager seems like it would be generally useful to look at logs for in CSR error scenarios as it looks like it is largely responsible for most of the CSR operations.