Skip to content

Commit 4a18938

Browse files
ciecierskiciecierski
committed
Add minor-update target-stage gate annotation
Introduce the core.openstack.org/minor-update-target-stage annotation on OpenStackVersion. When set, the minor update controller completes all stages up to and including the named stage, marks the next stage as blocked (FalseCondition/RequestedReason), and pauses reconciliation. Removing the annotation or advancing it to a later stage resumes the update. Includes stage-name constants, the gated-message format string, controller logic for all seven stages, functional tests for block/resume/ advance scenarios, and updated operator documentation. AI-assisted: Cursor (Claude Sonnet 4.6 by Anthropic)
1 parent 5c74e7f commit 4a18938

5 files changed

Lines changed: 648 additions & 0 deletions

File tree

‎api/core/v1beta1/conditions.go‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -596,4 +596,7 @@ const (
596596

597597
// OpenStackVersionMinorUpdateAvailableMessage
598598
OpenStackVersionMinorUpdateAvailableMessage = "update available"
599+
600+
// OpenStackVersionMinorUpdateReadyGatedMessage - format string; arg is the target stage name
601+
OpenStackVersionMinorUpdateReadyGatedMessage = "Minor update progression stopped after stage: %s . Set annotation to next stage after %s to resume OpenStack update"
599602
)

‎api/core/v1beta1/openstackversion_types.go‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,28 @@ const (
3434
MinorUpdateControlPlane string = "Minor Update Controlplane In Progress"
3535
// MinorUpdateComplete -
3636
MinorUpdateComplete string = "Complete"
37+
38+
// MinorUpdateTargetStageAnnotation - specifies the update stage after which the minor update
39+
// should pause. All stages up to and including the named stage will be completed; subsequent
40+
// stages will be blocked until the annotation is removed or updated to a later stage.
41+
// Valid values: "ovn-controlplane", "ovn-dataplane", "rabbitmq", "mariadb", "memcached",
42+
// "keystone", "controlplane". Remove the annotation to let the update proceed to completion.
43+
MinorUpdateTargetStageAnnotation string = "core.openstack.org/minor-update-target-stage"
44+
45+
// MinorUpdateStageOVNControlplane - stage name for OVN controlplane update
46+
MinorUpdateStageOVNControlplane string = "ovn-controlplane"
47+
// MinorUpdateStageOVNDataplane - stage name for OVN dataplane update
48+
MinorUpdateStageOVNDataplane string = "ovn-dataplane"
49+
// MinorUpdateStageRabbitMQ - stage name for RabbitMQ update
50+
MinorUpdateStageRabbitMQ string = "rabbitmq"
51+
// MinorUpdateStageMariaDB - stage name for MariaDB update
52+
MinorUpdateStageMariaDB string = "mariadb"
53+
// MinorUpdateStageMemcached - stage name for Memcached update
54+
MinorUpdateStageMemcached string = "memcached"
55+
// MinorUpdateStageKeystone - stage name for Keystone update
56+
MinorUpdateStageKeystone string = "keystone"
57+
// MinorUpdateStageControlplane - stage name for full controlplane update
58+
MinorUpdateStageControlplane string = "controlplane"
3759
)
3860

3961
// OpenStackVersionSpec - defines the desired state of OpenStackVersion

‎docs/assemblies/proc_minor-update-staged-rollout.md‎

Lines changed: 272 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,272 @@
1+
# Performing a Staged Minor Update of OpenStack
2+
3+
A minor update of OpenStack environment in a fixed sequence of stages. By default the
4+
update runs all stages automatically. The `core.openstack.org/minor-update-target-stage`
5+
annotation lets you pause the update after any stage so you can validate the environment,
6+
coordinate maintenance windows, or simply advance one stage at a time.
7+
8+
## Examples to use staged rollouts
9+
10+
- You want to verify OVN networking is healthy before allowing the rest of the update to
11+
proceed.
12+
- Your organisation requires a sign-off after each major component is updated.
13+
- You are performing the update in phases across a maintenance window and need to stop at a
14+
known safe point.
15+
16+
## Understanding the update pipeline
17+
18+
The update always runs stages in this order. Each stage must complete before the next one
19+
starts.
20+
21+
| Stage | What gets updated | Requires manual action? |
22+
|--------------------|-----------------------------------------|---------------------------------------------------------|
23+
| `ovn-controlplane` | OVN control plane images | No |
24+
| `ovn-dataplane` | OVN controller data plane images on compute nodes | **Yes** — create an OVN `OpenStackDataPlaneDeployment` |
25+
| `rabbitmq` | RabbitMQ images | No |
26+
| `mariadb` | MariaDB/Galera images | No |
27+
| `memcached` | Memcached images | No |
28+
| `keystone` | Keystone API images | No |
29+
| `controlplane` | All remaining control-plane services | No |
30+
| *(completion)* | Data-plane services on compute nodes | **Yes** — create a full `OpenStackDataPlaneDeployment` |
31+
32+
> **Note:** Two stages require you to create an `OpenStackDataPlaneDeployment` manually.
33+
> The `ovn-dataplane` stage and the final data-plane completion step do not self-drive —
34+
> the controller waits for the corresponding deployment to finish before advancing.
35+
> See [Required manual deployments](#required-manual-deployments) below.
36+
37+
## Prerequisites
38+
39+
- A running cluster with a deployed OpenStack environment.
40+
- `OpenStackControlPlane` and `OpenStackVersion` are both `Ready`.
41+
- `status.deployedVersion` is set on the `OpenStackVersion` CR.
42+
- A newer version is available: `status.availableVersion` differs from
43+
`status.deployedVersion`.
44+
45+
The examples below use:
46+
- Namespace: `openstack`
47+
- `OpenStackVersion` CR name: `openstack`
48+
49+
---
50+
51+
## Performing a fully staged update
52+
53+
The recommended approach is to set the annotation to the first stage before bumping
54+
`targetVersion`, then advance the annotation one stage at a time after you have validated
55+
each step.
56+
57+
### Step 1 — Confirm an update is available
58+
59+
```bash
60+
oc get openstackversion openstack -n openstack \
61+
-o jsonpath='Available: {.status.availableVersion} Deployed: {.status.deployedVersion}{"\n"}'
62+
```
63+
64+
Note the `availableVersion` value — this is `<new-version>` in the commands below.
65+
66+
### Step 2 — Set the initial pause point
67+
68+
Choose the stage after which you want the first pause. To pause after OVN control-plane:
69+
70+
```bash
71+
oc annotate openstackversion openstack \
72+
core.openstack.org/minor-update-target-stage=ovn-controlplane \
73+
-n openstack
74+
```
75+
76+
### Step 3 — Start the update
77+
78+
```bash
79+
oc patch openstackversion openstack -n openstack \
80+
--type=merge -p '{"spec":{"targetVersion":"<new-version>"}}'
81+
```
82+
83+
The update begins immediately. The controller runs the `ovn-controlplane` stage and then
84+
pauses. The `MinorUpdateOVNControlplane` condition becomes `True` and the
85+
`MinorUpdateOVNDataplane` condition shows:
86+
87+
```
88+
Minor update progression stopped after stage: ovn-controlplane .
89+
Set annotation to next stage after ovn-controlplane to resume OpenStack update
90+
```
91+
92+
### Step 4 — Validate and advance stage by stage
93+
94+
After each pause, check the environment is healthy, then advance to the next stage.
95+
96+
#### Checking the current update status
97+
98+
```bash
99+
oc get openstackversion openstack -n openstack \
100+
-o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}' \
101+
| grep MinorUpdate
102+
```
103+
104+
Completed stages show `True`. The currently blocked stage shows `False` with a message
105+
telling you which stage just finished and what to set next.
106+
107+
#### Advancing to the next stage
108+
109+
Update the annotation value to the stage you want to run next. For example, after
110+
validating the OVN control-plane, advance to `ovn-dataplane`:
111+
112+
> **Before advancing to `ovn-dataplane`**, create the OVN dataplane deployment first —
113+
> see [Required manual deployments](#required-manual-deployments).
114+
115+
```bash
116+
oc annotate openstackversion openstack \
117+
core.openstack.org/minor-update-target-stage=ovn-dataplane \
118+
--overwrite -n openstack
119+
```
120+
121+
Continue advancing through the remaining stages as needed:
122+
123+
| To run through… | Set annotation to… |
124+
|--------------------|--------------------|
125+
| RabbitMQ | `rabbitmq` |
126+
| MariaDB | `mariadb` |
127+
| Memcached | `memcached` |
128+
| Keystone | `keystone` |
129+
| Full control-plane | `controlplane` |
130+
131+
### Step 5 — Complete the update
132+
133+
When you are ready to run the final data-plane update on compute nodes, first create the
134+
full dataplane deployment (see [Required manual deployments](#required-manual-deployments)),
135+
then remove the annotation to let the update finish:
136+
137+
```bash
138+
oc annotate openstackversion openstack \
139+
core.openstack.org/minor-update-target-stage- \
140+
-n openstack
141+
```
142+
143+
> The trailing `-` removes the annotation entirely.
144+
145+
The controller runs the remaining stages and, once complete, sets
146+
`status.deployedVersion` to the new version.
147+
148+
### Step 6 — Confirm completion
149+
150+
```bash
151+
oc get openstackversion openstack -n openstack \
152+
-o jsonpath='{.status.deployedVersion}'
153+
```
154+
155+
The output should show `<new-version>`.
156+
157+
---
158+
159+
## Required manual deployments
160+
161+
Two stages in the process do not self-start. You must create an
162+
`OpenStackDataPlaneDeployment` before (or at the same time as) advancing past each of them.
163+
164+
### OVN data-plane deployment
165+
166+
Required before the `ovn-dataplane` stage can complete. This deployment updates only the
167+
OVN-related services on compute nodes.
168+
169+
```yaml
170+
apiVersion: dataplane.openstack.org/v1beta1
171+
kind: OpenStackDataPlaneDeployment
172+
metadata:
173+
name: edpm-deployment-ovn-update
174+
namespace: openstack
175+
spec:
176+
nodeSets:
177+
- openstack-edpm-ipam
178+
servicesOverride:
179+
- ovn
180+
```
181+
182+
```bash
183+
oc apply -f edpm-deployment-ovn-update.yaml
184+
```
185+
186+
### Full data-plane update deployment
187+
188+
Required before the final completion step can finish. This deployment updates all remaining
189+
services on compute nodes.
190+
191+
```yaml
192+
apiVersion: dataplane.openstack.org/v1beta1
193+
kind: OpenStackDataPlaneDeployment
194+
metadata:
195+
name: edpm-deployment-update
196+
namespace: openstack
197+
spec:
198+
nodeSets:
199+
- openstack-edpm-ipam
200+
servicesOverride:
201+
- update
202+
```
203+
204+
```bash
205+
oc apply -f edpm-deployment-update.yaml
206+
```
207+
208+
---
209+
210+
## Pausing a running update
211+
212+
If you need to pause an update that is already in progress, add the annotation at any time.
213+
The controller completes whichever stage is currently running, then stops after the stage you
214+
named.
215+
216+
```bash
217+
oc annotate openstackversion openstack \
218+
core.openstack.org/minor-update-target-stage=<stage> \
219+
-n openstack
220+
```
221+
222+
Replace `<stage>` with the name of the last stage you want to run before pausing.
223+
224+
---
225+
226+
## Running the full update without pausing
227+
228+
If you do not need staged control, omit the annotation entirely and let the controller run
229+
all stages automatically. You still need to create both dataplane deployments at the right
230+
time:
231+
232+
1. Create the OVN dataplane deployment before or immediately after starting the update.
233+
2. Create the dataplane update deployment before the final completion step.
234+
235+
```bash
236+
oc patch openstackversion openstack -n openstack \
237+
--type=merge -p '{"spec":{"targetVersion":"<new-version>"}}'
238+
```
239+
240+
---
241+
242+
## Troubleshooting
243+
244+
### The update appears stuck
245+
246+
Check whether the blocked condition message contains `"stopped after stage"`. If it does,
247+
the update is intentionally paused — advance or remove the annotation to continue.
248+
249+
```bash
250+
oc get openstackversion openstack -n openstack -o json | \
251+
jq '[.status.conditions[] | select(.reason=="Requested" and .status=="False")]'
252+
```
253+
254+
### `MinorUpdateOVNDataplane` or `MinorUpdateDataplane` stays `False`
255+
256+
These stages wait for an `OpenStackDataPlaneDeployment` to complete. Check whether the
257+
required deployment exists and is running:
258+
259+
```bash
260+
oc get openstackdataplanedeployment -n openstack
261+
```
262+
263+
If the deployment is missing, create it as described in
264+
[Required manual deployments](#required-manual-deployments).
265+
266+
### Checking overall update progress
267+
268+
```bash
269+
watch -n 5 "oc get openstackversion openstack -n openstack \
270+
-o jsonpath='{range .status.conditions[*]}{.type}{\"\t\"}{.status}{\"\t\"}{.message}{\"\n\"}{end}' \
271+
| grep MinorUpdate"
272+
```

0 commit comments

Comments
 (0)