🌱 Ensure COS phase immutability for referenced object approach (#2635)

ClusterObjectSet phases are immutable by design, but when objects are
stored in external Secrets via refs, the Secret content could be changed
by deleting and recreating the Secret. This enforces phase immutability
by:

- Verifying that referenced Secrets have `immutable: true` set
- Computing a per-phase SHA-256 content digest of pre-mutation resolved
  objects and recording it in `.status.observedPhases`
- Blocking reconciliation (`Progressing=False, Reason=Blocked`) if any
  referenced Secret is mutable or any phase's digest has changed
- Allowing blocked COS to recover when original content is restored

The digest is source-agnostic — it covers fully resolved phase content
regardless of whether objects are inline or from Secrets, making it
forward-compatible with future object sources.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: pedjak

api/v1/clusterobjectset_types.go

@@ -510,6 +510,39 @@ type ClusterObjectSetStatus struct {
 	// +listMapKey=type
 	// +optional
 	Conditions []metav1.Condition `json:"conditions,omitempty"`
+
+	// observedPhases records the content hashes of resolved phases
+	// at first successful reconciliation. This is used to detect if
+	// referenced object sources were deleted and recreated with
+	// different content. Each entry covers all fully-resolved object
+	// manifests within a phase, making it source-agnostic.
+	//
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf || oldSelf.size() == 0",message="observedPhases is immutable"
+	// +kubebuilder:validation:MaxItems=20
+	// +listType=map
+	// +listMapKey=name
+	// +optional
+	ObservedPhases []ObservedPhase `json:"observedPhases,omitempty"`
+}
+
+// ObservedPhase records the observed content digest of a resolved phase.
+type ObservedPhase struct {
+	// name is the phase name matching a phase in spec.phases.
+	//
+	// +required
+	// +kubebuilder:validation:MinLength=1
+	// +kubebuilder:validation:MaxLength=63
+	// +kubebuilder:validation:XValidation:rule=`!format.dns1123Label().validate(self).hasValue()`,message="the value must consist of only lowercase alphanumeric characters and hyphens, and must start with an alphabetic character and end with an alphanumeric character."
+	Name string `json:"name"`
+
+	// digest is the digest of the phase's resolved object content
+	// at first successful resolution, in the format "<algorithm>:<hex>".
+	//
+	// +required
+	// +kubebuilder:validation:MinLength=1
+	// +kubebuilder:validation:MaxLength=256
+	// +kubebuilder:validation:XValidation:rule=`self.matches('^[a-z0-9]+:[a-f0-9]+$')`,message="digest must be in the format '<algorithm>:<hex>'"
+	Digest string `json:"digest"`
 }
 
 // +genclient

api/v1/zz_generated.deepcopy.go

@@ -1,5 +1,5 @@
 processor:
-  ignoreTypes: [ClusterObjectSet, ClusterObjectSetList]
+  ignoreTypes: [ClusterObjectSet, ClusterObjectSetList, ObservedPhase]
   ignoreFields: []
 
 render:

applyconfigurations/api/v1/clusterobjectsetstatus.go

@@ -142,14 +142,18 @@ Recommended conventions:
 1. **Secret type**: Secrets should use the dedicated type
    `olm.operatorframework.io/object-data` to distinguish them from user-created
    Secrets and enable easy identification. The system always sets this type on
-   Secrets it creates. The reconciler does not enforce the type when resolving
-   refs — Secrets with any type are accepted — but producers should set it for
-   consistency.
-
-2. **Immutability**: Secrets should set `immutable: true`. Because COS phases
-   are immutable, the content backing a ref should not change after creation.
-   Mutable referenced Secrets are not rejected, but modifying them after the
-   COS is created leads to undefined behavior.
+   Secrets it creates. The reconciler does not enforce the Secret type when
+   resolving refs, but it does enforce that referenced Secrets have
+   `immutable: true` set and that their content has not changed since first
+   resolution.
+
+2. **Immutability**: Secrets must set `immutable: true`. The reconciler verifies
+   that all referenced Secrets have `immutable: true` set before proceeding.
+   Mutable referenced Secrets are rejected and reconciliation is blocked with
+   `Progressing=False, Reason=Blocked`. Additionally, the reconciler records
+   content hashes of the resolved phases on first successful reconciliation
+   and blocks reconciliation if the content changes (e.g., if a Secret is
+   deleted and recreated with the same name but different data).
 
 3. **Owner references**: Referenced Secrets should carry an ownerReference to
    the COS so that Kubernetes garbage collection removes them when the COS is
@@ -388,6 +392,14 @@ Key properties:
 
 ### COS reconciler behavior
 
+Before resolving individual object refs, the reconciler verifies that all
+referenced Secrets have `immutable: true` set. After successfully building
+the phases (resolving all refs), the reconciler computes a per-phase content
+digest and compares it against the digests recorded in `.status.observedPhases`
+(if present). If any phase's content has changed, reconciliation is blocked
+with `Progressing=False, Reason=Blocked`. On first successful build, phase
+content digests are persisted to status for future comparisons.
+
 When processing a COS phase:
 - For each object entry in the phase:
   - If `object` is set, use it directly (current behavior, unchanged).

applyconfigurations/api/v1/observedphase.go

@@ -621,6 +621,49 @@ spec:
                 x-kubernetes-list-map-keys:
                 - type
                 x-kubernetes-list-type: map
+              observedPhases:
+                description: |-
+                  observedPhases records the content hashes of resolved phases
+                  at first successful reconciliation. This is used to detect if
+                  referenced object sources were deleted and recreated with
+                  different content. Each entry covers all fully-resolved object
+                  manifests within a phase, making it source-agnostic.
+                items:
+                  description: ObservedPhase records the observed content digest of
+                    a resolved phase.
+                  properties:
+                    digest:
+                      description: |-
+                        digest is the digest of the phase's resolved object content
+                        at first successful resolution, in the format "<algorithm>:<hex>".
+                      maxLength: 256
+                      minLength: 1
+                      type: string
+                      x-kubernetes-validations:
+                      - message: digest must be in the format '<algorithm>:<hex>'
+                        rule: self.matches('^[a-z0-9]+:[a-f0-9]+$')
+                    name:
+                      description: name is the phase name matching a phase in spec.phases.
+                      maxLength: 63
+                      minLength: 1
+                      type: string
+                      x-kubernetes-validations:
+                      - message: the value must consist of only lowercase alphanumeric
+                          characters and hyphens, and must start with an alphabetic
+                          character and end with an alphanumeric character.
+                        rule: '!format.dns1123Label().validate(self).hasValue()'
+                  required:
+                  - digest
+                  - name
+                  type: object
+                maxItems: 20
+                type: array
+                x-kubernetes-list-map-keys:
+                - name
+                x-kubernetes-list-type: map
+                x-kubernetes-validations:
+                - message: observedPhases is immutable
+                  rule: self == oldSelf || oldSelf.size() == 0
             type: object
         type: object
     served: true