Skip to content

Commit fe9f114

Browse files
author
Michael Burke
committed
OSDOCS 18143 Document how the boot image skew management feature works
1 parent 4bd0ef1 commit fe9f114

8 files changed

Lines changed: 326 additions & 0 deletions

‎_topic_maps/_topic_map.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2463,6 +2463,8 @@ Topics:
24632463
File: machine-config-pin-preload-images-about
24642464
- Name: Boot image management
24652465
File: mco-update-boot-images
2466+
- Name: Boot image skew enforcement
2467+
File: mco-update-boot-skew-mgmt
24662468
- Name: Manually updating the boot image
24672469
File: mco-update-boot-images-manual
24682470
- Name: Managing unused rendered machine configs

‎machine_configuration/mco-update-boot-skew-mgmt.adoc‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
:context: mco-update-boot-skew-mgmt
3+
include::_attributes/common-attributes.adoc[]
4+
[id="mco-update-boot-skew-mgmt"]
5+
= Boot image skew enforcement
6+
7+
toc::[]
8+
9+
[role="_abstract"]
10+
You can use boot image skew enforcement to help ensure that the boot images in a cluster are up-to-date with the {product-title} and {op-system} version being used in the cluster. Using an older boot image could cause issues when scaling new nodes. If the images are older than a predetermined version, the MCO disables cluster upgrades until it deems the boot images to be compliant.
11+
12+
include::modules/mco-update-boot-skew-mgmt-about.adoc[leveloffset=+1]
13+
14+
include::modules/mco-update-boot-skew-mgmt-modes.adoc[leveloffset=+2]
15+
16+
include::modules/mco-update-boot-skew-mgmt-configuring.adoc[leveloffset=+1]
17+
18+
include::modules/mco-update-boot-skew-mgmt-updating.adoc[leveloffset=+1]
19+
20+
[role="_additional-resources"]
21+
[id="additional-resources_{context}"]
22+
== Additional resources
23+
24+
* xref:../machine_configuration/mco-update-boot-images.adoc#mco-update-boot-images[Boot image management]
25+
* xref:../machine_configuration/mco-update-boot-images-manual.adoc#mco-update-boot-images-manual[Manually updating the boot image]

‎modules/mco-update-boot-images-configuring.adoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ Because the boot image management feature for worker nodes is default for the {g
2424

2525
Enabling the feature updates the boot image to the {op-system-first} boot image version appropriate for your cluster. If the cluster is again updated to a new {product-title} version in the future, the boot image is updated again. New nodes created after enabling the feature use the updated boot image. This feature has no effect on existing nodes.
2626

27+
When boot image management is enabled, the MCO automatically enables boot image skew enforcement to ensure that the boot image version is compliant for your cluster. For more information, see "Boot image skew enforcement".
28+
2729
.Prerequisites
2830

2931
* If you are enabling boot image management for control plane machine sets, you enabled the required Technology Preview features for your cluster by editing the `FeatureGate` CR named `cluster`.

‎modules/mco-update-boot-images-disable.adoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ include::snippets/mco-update-boot-images-intro.adoc[]
1919

2020
Disabling this feature does not rollback the nodes or machine sets to the originally-installed boot image. The machine sets retain the boot image version that was present when the feature was disabled and is not updated if the cluster is upgraded to a new {product-title} version in the future. This feature has no effect on existing nodes.
2121

22+
If boot image management is disabled, you must update the boot image version that is used by the boot image skew enforcement feature to ensure that the boot image is current for your cluster. For more information, see "Boot image skew enforcement".
23+
2224
After disabling the feature, you can re-enable the feature at any time. For more information, see "Enabling updated boot images".
2325

2426
.Procedure

‎modules/mco-update-boot-skew-mgmt-about.adoc‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine-configuration/mco-update-boot-skew-mgmt.adoc
4+
5+
:_mod-docs-content-type: CONCEPT
6+
[id="mco-update-boot-skew-mgmt-about_{context}"]
7+
= About boot image skew enforcement
8+
9+
[role="_abstract"]
10+
Using boot image skew enforcement, you can ensure that the boot images in a cluster are up-to-date with the {product-title} and {op-system} version being used in the cluster. Making sure that your boot images are current can help you avoid the problems associated with running older images.
11+
12+
When boot image skew enforcement is active in a cluster, the Machine Config Operator (MCO) examines the boot image version reported in the `MachineConfiguration` object to determine if that boot image is appropriate for the cluster. If the boot image version is too old, the Operator reports that _boot image version skew_ is detected and blocks cluster updates until you manually update the boot image or disable boot image skew enforcement by setting the `None` mode, as described in this section.
13+
14+
The limit for boot image version skew is set within the MCO and cannot be modified.
15+
16+
For information on manually configuring the boot image in your cluster, see "Manually updating the boot image".

‎modules/mco-update-boot-skew-mgmt-configuring.adoc‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine-configuration/mco-update-boot-skew-mgmt.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="mco-update-boot-skew-mgmt-configuring_{context}"]
7+
= Configuring boot image skew enforcement
8+
9+
[role="_abstract"]
10+
You can configure the current boot image skew enforcement mode that the Machine Config Operator (MCO) uses. By configuring the boot image skew enforcement mode, you can determine if the boot image version in the `MachineConfiguration` object is updated automatically or manually.
11+
12+
Alternatively, you can disable boot image skew enforcement by setting the `mode` to `None`. When disabled, the MCO does not monitor for boot image skew, and older boot images could be used, possibly introducing issues when scaling new nodes.
13+
14+
In {product-title} 4.22, the automatic mode is available only for {aws-short}, {gcp-short}, {azure-short}, and {vmw-short} clusters and is the default for these platforms. If you modify a cluster from the automatic mode to the manual or none mode, you can revert a cluster back to automatic mode only by removing the `bootImageSkewEnforcement` stanza from the `MachineConfiguration` object.
15+
16+
All other platforms default to manual mode with the {product-title} version set as the boot image version in the `MachineConfiguration` object. In manual mode, you are expected to manually update the `MachineConfiguration` object with new boot image version whenever you update the boot image.
17+
18+
.Procedure
19+
20+
. For manual mode, you can obtain the current boot image on a node by using the following command:
21+
+
22+
[source,terminal]
23+
----
24+
$ oc debug node/<new-node> -- chroot /host cat /sysroot/.coreos-aleph-version.json
25+
----
26+
+
27+
.Example output
28+
[source,terminal]
29+
----
30+
# ...
31+
"ref": "docker://ostree-image-signed:oci-archive:/rhcos-9.6.20251023-0-ostree.x86_64.ociarchive",
32+
"version": "9.6.20251023-0"
33+
----
34+
+
35+
You should use the newest node on the cluster, because the boot image might have been updated after the older nodes were created. Ideally, test the newest node from each machine set and use the oldest boot image among them.
36+
37+
. Specify the boot image skew enforcement mode and set the boot image version as needed:
38+
+
39+
[source,yaml]
40+
----
41+
apiVersion: operator.openshift.io/v1
42+
kind: MachineConfiguration
43+
metadata:
44+
name: cluster
45+
spec:
46+
# ...
47+
bootImageSkewEnforcement:
48+
mode: Manual
49+
manual:
50+
mode: RHCOSVersion
51+
rhcosVersion: 9.6.20251023-0
52+
# ...
53+
----
54+
where:
55+
56+
`spec.bootImageSkewEnforcement.mode`:: Specifies the boot image enforcement mode, one of the following values:
57+
+
58+
--
59+
* `Manual`. Specifies that boot image skew management is in manual mode. You must specify the `spec.bootImageSkewEnforcement.manual` parameters.
60+
* `None`. Specifies that boot image skew management is disabled. You do not need to specify the `spec.bootImageSkewEnforcement.manual` parameters.
61+
--
62+
63+
`spec.bootImageSkewEnforcement.manual.mode`:: Specifies the version you want to represent the current boot image, either `OCPVersion` or `RHCOSVersion`. You must include one of the following parameters:
64+
+
65+
--
66+
** For `RHCOSVersion`, use `spec.bootImageSkewEnforcement.manual.rhcosVersion` to specify the {op-system} version that is being used as a boot image in the `[major].[minor].[datestamp(YYYYMMDD)]-[buildnumber]` or `[major].[minor].[timestamp(YYYYMMDDHHmm)]-[buildnumber]` format. This field must be between 14 and 21 characters.
67+
** For `OCPVersion`, use `spec.bootImageSkewEnforcement.manual.ocpVersion` to specify the {product-title} version associated with the boot image that is being used in the `x.y.z` format. This field must be between 5 and 10 characters.
68+
--

‎modules/mco-update-boot-skew-mgmt-modes.adoc‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine-configuration/mco-update-boot-skew-mgmt-modes.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="mco-update-boot-skew-mgmt-about-modes_{context}"]
7+
= About boot image skew enforcement modes
8+
9+
[role="_abstract"]
10+
Review the following information to learn about the boot image skew enforcement modes. Use the information to determine the best method for your cluster.
11+
12+
Boot image skew enforcement operates in one of the following modes:
13+
14+
Automatic::
15+
When set to `Automatic`, with boot image management also enabled, if the cluster is updated from one {product-title} version to the next, the MCO automatically updates the boot image version in the `MachineConfiguration` object and tests the boot image version for skew.
16+
+
17+
[NOTE]
18+
====
19+
In {product-title} 4.22, the automatic mode is available only for {aws-short}, {gcp-short}, {azure-short}, and {vmw-short} clusters and is the default for these platforms.
20+
====
21+
+
22+
The MCO automatically configures this mode when the following conditions are met:
23+
+
24+
--
25+
* Boot image management is available for the platform that your cluster uses. Currently boot image management is available for only {aws-short}, {gcp-short}, {azure-short}, and {vmw-short} clusters.
26+
* You have enabled boot image management for compute machine sets.
27+
* You have not set skew enforcement to the manual or none mode.
28+
--
29+
+
30+
For information on boot image management, see "Boot image management".
31+
+
32+
.Example `MachineConfiguration` object with automatic skew enforcement
33+
[source,yaml]
34+
----
35+
apiVersion: operator.openshift.io/v1
36+
kind: MachineConfiguration
37+
metadata:
38+
name: cluster
39+
status:
40+
# ...
41+
bootImageSkewEnforcementStatus:
42+
automatic:
43+
ocpVersion: 4.22.0
44+
mode: Automatic
45+
----
46+
+
47+
The MCO examines the boot image reported in the `ocpVersion` parameter to determine if the cluster is violating the boot image version skew limits.
48+
49+
Manual::
50+
When set to `Manual`, if the boot image version is updated, a cluster administrator is responsible for manually updating the `MachineConfiguration` object with the {op-system} version of the new boot image or the {product-title} version associated with the new boot image. The MCO then tests the boot image version for skew.
51+
+
52+
.Example `MachineConfiguration` object with skew enforcement based on an {op-system} version
53+
[source,yaml]
54+
----
55+
apiVersion: operator.openshift.io/v1
56+
kind: MachineConfiguration
57+
metadata:
58+
name: cluster
59+
# ...
60+
spec:
61+
bootImageSkewEnforcement:
62+
mode: Manual
63+
manual:
64+
mode: RHCOSVersion
65+
rhcosVersion: 9.2.20251023-0
66+
# ...
67+
status:
68+
bootImageSkewEnforcementStatus:
69+
manual:
70+
mode: RHCOSVersion
71+
rhcosVersion: 9.2.20251023-0
72+
mode: Manual
73+
----
74+
+
75+
.Example `MachineConfiguration` object with skew enforcement based on an {product-title} version
76+
[source,yaml]
77+
----
78+
apiVersion: operator.openshift.io/v1
79+
kind: MachineConfiguration
80+
metadata:
81+
name: cluster
82+
# ...
83+
spec:
84+
bootImageSkewEnforcement:
85+
manual:
86+
mode: OCPVersion
87+
ocpVersion: 4.22.0
88+
mode: Manual
89+
# ...
90+
status:
91+
bootImageSkewEnforcementStatus:
92+
manual:
93+
mode: OCPVersion
94+
ocpVersion: 4.22.0
95+
mode: Manual
96+
----
97+
+
98+
The MCO examines the boot image reported in the `rhcosVersion` or `ocpVersion` parameter to determine if the cluster is violating the boot image version skew limits.
99+
100+
None::
101+
When set to `None`, boot image skew enforcement is disabled. When disabled, the MCO does not monitor for boot image skew and does not report if new nodes are provisioned with older boot images, which could introduce issues when scaling new nodes.
102+
+
103+
.Example `MachineConfiguration` object with skew enforcement disabled
104+
[source,yaml]
105+
----
106+
apiVersion: operator.openshift.io/v1
107+
kind: MachineConfiguration
108+
metadata:
109+
name: cluster
110+
# ...
111+
spec:
112+
bootImageSkewEnforcement:
113+
mode: None
114+
# ...
115+
status:
116+
bootImageSkewEnforcementStatus:
117+
mode: None
118+
----
119+
+
120+
When in the none mode, the MCO reports a Prometheus alert that skew enforcement is disabled and that scale-up might run into issues due to old boot images. The alert does not cause any functional issues for the cluster.
121+
+
122+
{sno-caps} clusters default to the none mode regardless of platform, because they do not scale. The skew enforcement Prometheus alert is not reported for {sno} clusters.
123+
+
124+
Bare-metal clusters running {product-title} version 4.10 and later do not use the MCO to keep their boot images up-to-date. Skew enforcement defaults to the none mode and the skew enforcement Prometheus alert mentioned is not reported. For bare-metal clusters running {product-title} version 4.9 and earlier, you need to perform a one-time action to migrate to the 4.10 system, this is explained further in the bare metal boot image update docs. For information, see "Manually updating the boot image".

‎modules/mco-update-boot-skew-mgmt-updating.adoc‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * machine-configuration/mco-update-boot-skew-mgmt.adoc
4+
5+
6+
:_mod-docs-content-type: PROCEDURE
7+
[id="mco-update-boot-skew-mgmt-updating.adoc_{context}"]
8+
= Updating the boot image skew enforcement version
9+
10+
[role="_abstract"]
11+
If you are running boot image skew enforcement in the manual mode, you must manually update the boot image version in the `MachineConfiguration` object each time you update the boot image in your cluster. With the boot image updated in the `MachineConfiguration` object, the Machine Config Operator (MCO) can properly perform boot image skew enforcement to ensure that your nodes are up-to-date.
12+
13+
.Procedure
14+
15+
. If necessary, obtain the {op-system} or {product-title} version of the current boot image on an updated node by using one of the following commands:
16+
17+
* Obtain the {op-system} version by running the following command:
18+
+
19+
[source,terminal]
20+
----
21+
$ oc debug node/<new-node> -- chroot /host cat /sysroot/.coreos-aleph-version.json
22+
----
23+
+
24+
.Example output
25+
[source,terminal]
26+
----
27+
# ...
28+
"ref": "docker://ostree-image-signed:oci-archive:/rhcos-9.6.20251023-0-ostree.x86_64.ociarchive",
29+
"version": "9.6.20251023-0"
30+
----
31+
32+
* Obtain the {product-title} version by running the following command:
33+
+
34+
[source,terminal]
35+
----
36+
$ openshift-install version
37+
----
38+
+
39+
Ensure that you use the same `openshift-install` binary that you used when updating the boot image.
40+
+
41+
.Example output
42+
[source,terminal]
43+
----
44+
openshift-install 4.22.0
45+
----
46+
47+
. Specify the boot image version in the `MachineConfiguration` object with either the {op-system} or {product-title} version:
48+
+
49+
* Update the `MachineConfiguration` object with the {op-system} version:
50+
+
51+
[source,yaml]
52+
----
53+
apiVersion: operator.openshift.io/v1
54+
kind: MachineConfiguration
55+
metadata:
56+
name: cluster
57+
# ...
58+
spec:
59+
bootImageSkewEnforcement:
60+
mode: Manual
61+
manual:
62+
mode: RHCOSVersion
63+
rhcosVersion: 9.2.20251023-0
64+
# ...
65+
----
66+
+
67+
If the `spec.bootImageSkewEnforcement.manual.mode` is `RHCOSVersion`, specify the {op-system} version of the boot image with the `rhcosVersion` parameter, as shown in the example.
68+
69+
* Update the `MachineConfiguration` object with the {product-title} version
70+
+
71+
[source,yaml]
72+
----
73+
apiVersion: operator.openshift.io/v1
74+
kind: MachineConfiguration
75+
metadata:
76+
name: cluster
77+
# ...
78+
spec:
79+
bootImageSkewEnforcement:
80+
mode: Manual
81+
manual:
82+
mode: OCPVersion
83+
ocpVersion: 4.22.0
84+
# ...
85+
----
86+
+
87+
If the `spec.bootImageSkewEnforcement.manual.mode` is `OCPVersion`, specify the {product-title} version of the boot image with the `ocpVersion` parameter, as shown in the example.

0 commit comments

Comments
 (0)