Track CustomContainerImages and prevent minor updates w/o images updates by dprince · Pull Request #1569 · openstack-k8s-operators/openstack-operator

dprince · 2025-08-18T16:22:31Z

Add tracking and validation to ensure CustomContainerImages are updated when changing targetVersion during minor updates:

Add TrackedCustomImages field to OpenStackVersionStatus to track CustomContainerImages used for each version
Implement validation webhook logic to prevent targetVersion updates when CustomContainerImages remain unchanged from previous version
Add helper functions for deep comparison of CustomContainerImages
Update controller to automatically track CustomContainerImages for each processed target version
Generate updated CRD manifests with new tracking field

This prevents inconsistent deployments where users update the target version but forget to update associated custom container images, ensuring proper version tracking and validation during minor updates.

Co-Authored-By: Claude noreply@anthropic.com

Jira: OSPRH-19183

dprince · 2025-08-18T16:26:20Z

letting CI run before I mark as draft. This is missing code to allow the user to override the restriction if they want to, essentially forcing the change regardless of the warning if they wish

softwarefactory-project-zuul · 2025-08-18T19:41:17Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3226a8e13a484ce28308f2ef50e3fec7

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 17m 27s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 09m 26s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 45m 52s
✔️ adoption-standalone-to-crc-ceph-provider SUCCESS in 3h 01m 51s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 24m 51s

dprince · 2025-09-16T22:46:41Z

@abays @stuggi I did have a question on this. Should we perhaps remove the new annotation if set once the minor update completes. This would force the administrator to manually re-apply the annotation for each minor update to override the behavior

softwarefactory-project-zuul · 2025-09-16T22:52:27Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/332b4e1f7f764474aa85e40e1132f0ba

❌ openstack-k8s-operators-content-provider POST_FAILURE in 3h 24m 09s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 16m 06s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 37m 20s
❌ adoption-standalone-to-crc-ceph-provider POST_FAILURE in 3h 09m 30s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 36m 31s

stuggi · 2025-09-17T05:48:33Z

@abays @stuggi I did have a question on this. Should we perhaps remove the new annotation if set once the minor update completes. This would force the administrator to manually re-apply the annotation for each minor update to override the behavior

for safety mechanism I think that makes sense. We'd have to document that behavior in minor updates doc.

…hanges Add tracking and validation to ensure CustomContainerImages are updated when changing targetVersion during minor updates: - Add TrackedCustomImages field to OpenStackVersionStatus to track CustomContainerImages used for each version - Implement validation webhook logic to prevent targetVersion updates when CustomContainerImages remain unchanged from previous version - Add helper functions for deep comparison of CustomContainerImages - Update controller to automatically track CustomContainerImages for each processed target version - Generate updated CRD manifests with new tracking field This prevents inconsistent deployments where users update the target version but forget to update associated custom container images, ensuring proper version tracking and validation during minor updates. Users can skip all validations related to this commit by setting the core.openstack.org/skip-custom-images-validation metadata annotations. Co-Authored-By: Claude <noreply@anthropic.com> Jira: OSPRH-19183

stuggi · 2025-09-19T10:51:11Z

 		instance.Status.DeployedVersion = &instance.Spec.TargetVersion
+
+		// Remove skip-custom-images-validation annotation after minor update completion
+		if instance.Annotations != nil {


Won't this remove the annotation on any reconcile when the ctlplane is ready, not just when a minor update finished? Like I do the minor update, it finishes, after that I want to re-add the condition. Maybe we should do it above after the dataplane got the minor update?

instance.Status.Conditions.MarkTrue( corev1beta1.OpenStackVersionMinorUpdateDataplane, corev1beta1.OpenStackVersionMinorUpdateReadyMessage)

it would only remove the annotation if targetVersion != availableVersion. My intention with the annotation is that you would set it only when enabling a minor update. If we move it above after the dataplane gets update it is possible someone could set the annotation at anytime and it would persist until the next minor update and then it would skip the check blindly.

right, thanks for clarification

stuggi

/lgtm

openshift-ci · 2025-10-01T11:41:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dprince, stuggi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dprince,stuggi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

stuggi · 2025-10-02T07:28:26Z

/retest

stuggi · 2025-10-03T07:05:14Z

/retest

stuggi · 2025-10-06T15:17:14Z

/retest

stuggi · 2025-10-07T07:16:48Z

/retest

…g minor update Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. During the update flow, `set_openstack_containers` updates the operator CSV env vars with the new delorean tag, but nothing updates the `OpenStackVersion` CR's `customContainerImages` -- they still carry the tag from the initial greenfield deployment. When `update_services` later patches `targetVersion`, the webhook rejects the request because the custom images haven't changed. Add a step after `set_openstack_containers` that reads the current `customContainerImages` from the `OpenStackVersion` CR, replaces every image tag with the new delorean md5 hash, and patches the CR. This ensures the custom images reflect the new version before `targetVersion` is changed, satisfying the webhook and guaranteeing the update uses the correct container images. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>

Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. In CI, the greenfield deployment and the simulated version bump share the same delorean hash, so `customContainerImages` are legitimately identical across versions -- there are no new container builds to reference. Retagging is a no-op. Add the operator-provided `skip-custom-images-validation` annotation before patching `targetVersion` in both update scripts. This annotation is the intended escape hatch for automation where the images are intentionally unchanged (see `ValidateUpdate` in `openstackversion_webhook.go`). The intermediate `openstackversionpatch.yaml` file is no longer generated; the patch is passed inline instead. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>

Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. In CI, the greenfield deployment and the simulated version bump share the same delorean hash, so `customContainerImages` are legitimately identical across versions. Clear `customContainerImages` (set to `null`) in the same merge patch that updates `targetVersion`. With no custom images present the webhook's `hasAnyCustomImage` check returns false and the validation is bypassed. The operator falls back to its CSV defaults, which `set_openstack_containers` has already updated to the correct tag. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>

…on` patch Starting with openstack-operator FR4 (openstack-k8s-operators/openstack-operator#1569, merged Oct 2025), the `OpenStackVersion` admission webhook rejects `targetVersion` changes when `spec.customContainerImages` are identical to those tracked for the previously deployed version. In CI the same delorean hash is used for both the greenfield deployment and the simulated version bump, so `customContainerImages` are legitimately identical. Simply clearing them is not safe because the operator falls back to CSV defaults which may point to an inaccessible upstream registry (`quay.io`), while the custom images point to the internal mirror (`images.paas.redhat.com`). Work around this by saving `customContainerImages`, nulling them in the same merge patch that sets `targetVersion` (so the webhook's `hasAnyCustomImage` check returns false and skips validation), then immediately restoring them before the operator reconciles. This preserves the correct internal-mirror image references while bypassing the webhook. Assisted-by: Cursor (claude-4.6-opus-high) Signed-off-by: Roberto Alfieri <ralfieri@redhat.com>

openshift-ci Bot requested review from abays and rebtoor August 18, 2025 16:22

openshift-ci Bot added the approved label Aug 18, 2025

dprince marked this pull request as draft August 19, 2025 00:13

openshift-ci Bot added the do-not-merge/work-in-progress label Aug 19, 2025

dprince force-pushed the tracked_custom_images branch from a760b91 to d5224a3 Compare August 19, 2025 12:23

dprince marked this pull request as ready for review August 19, 2025 12:23

openshift-ci Bot removed the do-not-merge/work-in-progress label Aug 19, 2025

openshift-ci Bot requested review from fultonj and stuggi August 19, 2025 12:24

dprince force-pushed the tracked_custom_images branch from d5224a3 to a4cbd4c Compare August 19, 2025 12:31

stuggi reviewed Aug 19, 2025

View reviewed changes

Comment thread apis/core/v1beta1/openstackversion_webhook.go Outdated

Comment thread apis/core/v1beta1/openstackversion_types.go

dprince force-pushed the tracked_custom_images branch 2 times, most recently from dc96ab3 to 7730fca Compare August 28, 2025 23:37

abays reviewed Sep 16, 2025

View reviewed changes

Comment thread apis/core/v1beta1/openstackversion_types.go Outdated

dprince force-pushed the tracked_custom_images branch from 7730fca to 2a416d7 Compare September 16, 2025 19:27

dprince force-pushed the tracked_custom_images branch from 2a416d7 to e97685d Compare September 17, 2025 15:50

stuggi reviewed Sep 19, 2025

View reviewed changes

stuggi approved these changes Oct 1, 2025

View reviewed changes

openshift-ci Bot assigned stuggi Oct 1, 2025

openshift-ci Bot added the lgtm label Oct 1, 2025

openshift-merge-bot Bot merged commit 70a918b into openstack-k8s-operators:main Oct 7, 2025
9 checks passed

rebtoor mentioned this pull request Mar 13, 2026

[update] Retag customContainerImages in OpenStackVersion CR during minor update openstack-k8s-operators/ci-framework#3764

Closed

rebtoor mentioned this pull request Mar 16, 2026

[update] Save and restore customContainerImages around targetVersion patch openstack-k8s-operators/install_yamls#1139

Closed

Conversation

dprince commented Aug 18, 2025 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dprince commented Aug 18, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Aug 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dprince commented Sep 16, 2025

Uh oh!

softwarefactory-project-zuul Bot commented Sep 16, 2025

Uh oh!

stuggi commented Sep 17, 2025

Uh oh!

stuggi Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

dprince Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

stuggi Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

stuggi left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Oct 1, 2025

Uh oh!

stuggi commented Oct 2, 2025

Uh oh!

stuggi commented Oct 3, 2025

Uh oh!

stuggi commented Oct 6, 2025

Uh oh!

stuggi commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dprince commented Aug 18, 2025 •

edited by openshift-ci Bot

Loading