feat: migrate to Server-Side Apply (SSA) and enforce controller ownership

Fixes: #258

Migrate resource creation and updates from client-side POST/PUT to
Server-Side Apply (SSA) where appropriate, and strengthen ownership
semantics with controller references throughout.

This commit establishes ownership of various fields and CRD's, to prevent accidental overwrites or race conditions by other controllers
It doesn't migrate all patches to SSA, if we plan to migrate completely to SSA, there would be significant code rework, and honestly, its not necessary for now.

All resources created by the operator now use `controller: true` in
ownerReferences, enabling Kubernetes garbage collection:

- TEC owns: Deployments, Services, ConfigMaps, Jobs, ApprovedImages
- Machine owns: AttestationKey (transferred from TEC via merge patch once its approved automatically).
- AttestationKey owns: Secret (with controller: true + finalizer)

`generate_owner_reference` renamed to `generate_owner_controller_reference`
to reflect that all owner refs are now controller refs with
`blockOwnerDeletion: true`.

All controllers are owners, but not all owners have to be controllers. Kubernetes mandates only one controller.
Multiple owners are fine for Garbage Collection and delete cascades.

New `apply_resource!` macro replaces `create_or_info_if_exists!` for
idempotent resource creation via SSA (PATCH with Patch::Apply).

`update_reference_values`: uses full-data SSA patch on trustee-data
ConfigMap (includes all keys: kbs-config.toml, policy.rego,
reference-values.json) to prevent the field manager from implicitly
relinquishing keys not in a partial patch.

If a field manager (controller owning a field of a CRD)  makes a SSA patch, all fields of that patch are considered owned, and any previously owned fields not part of the patch are released from its control.

`update_attestation_keys`: uses full-object SSA patch on the Deployment
(sends entire deployment.spec) to avoid stripping required fields like
spec.selector. Metadata fields (managedFields, resourceVersion, uid)
are excluded from the SSA body to prevent 400 errors.

`do_mount_secret`: kept as GET+mutate+replace (PUT) since it follows a
read-modify-write pattern on the same field manager. Migrating this is significant code rework.

Owner transfer in `approve_ak` uses Merge patch (not SSA) because SSA
would upsert into the ownerReferences array, leaving stale references of both TEC. In the end TEC and Machine would co-exist as controllers, which we don't want.
Merge patch replaces the entire ownerReferences field atomically.

Single field manager `trusted-cluster-operator` used consistently via
`FIELD_MANAGER` constant. register-server uses its own `register-server`
field manager for Machine creation (POST with fieldManager query param,
not SSA).

`update_status!` macro now requires explicit type and field manager
parameters, enabling force-apply on the status subresource where
multiple writers exist (e.g. ApprovedImage status).

- Duplicate volume entries: `do_mount_secret` now checks for existing
  volumes before adding, preventing 422 errors on reconciler retry.
- Secret controller filtering: `update_attestation_keys` and
  `secret_reconcile` now require `controller: true` on AttestationKey
  owner refs, not just `kind == "AttestationKey"`.
- ConfigMap data loss: full-data SSA patch on trustee-data prevents
  kbs-config.toml and policy.rego from being garbage-collected when
  only reference-values.json is updated.

Added unit tests validating SSA body content, ownership, idempotency,
field manager usage, and filtering logic across trustee, reference
values, attestation key register, and register server modules.

Commit message co-authored-by: Cursor <cursoragent@cursor.com>

Author: SpaceFace02

attestation-key-register/src/main.rs

@@ -16,7 +16,8 @@ use uuid::Uuid;
 
 use trusted_cluster_operator_lib::endpoints::ATTESTATION_KEY_REGISTER_RESOURCE;
 use trusted_cluster_operator_lib::{
-    generate_owner_reference, get_trusted_execution_cluster, AttestationKey, AttestationKeySpec,
+    self, generate_owner_controller_reference, get_trusted_execution_cluster, AttestationKey,
+    AttestationKeySpec,
 };
 
 #[derive(Parser)]
@@ -64,15 +65,17 @@ async fn handle_registration(
 
     let api: Api<AttestationKey> = Api::default_namespaced(client.clone());
 
-    // Get the TrustedExecutionCluster to use as owner reference
+    // TEC is owner-controller of the AttestationKey until the operator approves the key and transfers control to Machine, which then becomes the controller and manages lifecycle of the AttestationKey.
     let cluster = match get_trusted_execution_cluster(client.clone()).await {
         Ok(c) => c,
         Err(e) => return internal_error(e.context("Failed to get TrustedExecutionCluster")),
     };
 
-    let owner_reference = match generate_owner_reference(&cluster) {
+    let owner_controller_reference = match generate_owner_controller_reference(&cluster) {
         Ok(o) => o,
-        Err(e) => return internal_error(e.context("Failed to generate owner reference")),
+        Err(e) => {
+            return internal_error(e.context("Failed to generate owner-controller reference"))
+        }
     };
 
     match api.list(&Default::default()).await {
@@ -104,7 +107,7 @@ async fn handle_registration(
     let attestation_key = AttestationKey {
         metadata: ObjectMeta {
             name: Some(name.clone()),
-            owner_references: Some(vec![owner_reference]),
+            owner_references: Some(vec![owner_controller_reference]),
             ..Default::default()
         },
         spec: AttestationKeySpec {
@@ -114,7 +117,17 @@ async fn handle_registration(
         status: None,
     };
 
-    match api.create(&Default::default(), &attestation_key).await {
+    // Client side apply, as this is a one time operation for each attestation key, and one owner. SSA is mainly used for patching. Setting field manager, so that we can identify the source of the creation later for reconciliation by operator.
+    match api
+        .create(
+            &kube::api::PostParams {
+                field_manager: Some(trusted_cluster_operator_lib::FIELD_MANAGER.to_string()),
+                ..Default::default()
+            },
+            &attestation_key,
+        )
+        .await
+    {
         Ok(created) => {
             let name = created.metadata.name.unwrap_or_default();
             info!("Successfully created AttestationKey: {name}",);

compute-pcrs/src/main.rs

@@ -12,6 +12,8 @@ use std::{fs::File, io::Read};
 
 use trusted_cluster_operator_lib::{conditions::INSTALLED_REASON, reference_values::*, *};
 
+const FIELD_MANAGER: &str = "compute-pcrs";
+
 #[derive(Parser)]
 #[command(version, about)]
 struct Args {
@@ -77,6 +79,16 @@ async fn main() -> Result<()> {
     let committed = committed_condition(INSTALLED_REASON, image.metadata.generation, &None);
     let conditions = Some(vec![committed]);
     let status = ApprovedImageStatus { conditions };
-    update_status!(approved_images, &args.resource_name, status)?;
+
+    // As operator also updates the status field, while requeuing every 10 seconds, we need to force this update to avoid race condition.
+    // TODO: Simplify ownership of this field.
+    update_status!(
+        approved_images,
+        &args.resource_name,
+        status,
+        ApprovedImage,
+        FIELD_MANAGER,
+        force
+    )?;
     Ok(())
 }

lib/src/lib.rs

@@ -30,13 +30,31 @@ use conditions::*;
 use k8s_openapi::apimachinery::pkg::apis::meta::v1::{Condition, OwnerReference, Time};
 use kube::Resource;
 
+pub const FIELD_MANAGER: &str = "trusted-cluster-operator";
+
+// 2 arms for updating status, one for normal updates and one for force updates.
 #[macro_export]
 macro_rules! update_status {
-    ($api:ident, $name:expr, $status:expr) => {{
-        let patch = kube::api::Patch::Merge(serde_json::json!({"status": $status}));
-        $api.patch_status($name, &Default::default(), &patch).await
+    ($api:ident, $name:expr, $status:expr, $type:ty, $field_manager:expr) => {{
+        let patch = kube::api::Patch::Apply(serde_json::json!({
+            "apiVersion": <$type as kube::Resource>::api_version(&()),
+            "kind": <$type as kube::Resource>::kind(&()),
+            "status": $status
+        }));
+        let params = kube::api::PatchParams::apply($field_manager);
+        $api.patch_status($name, &params, &patch).await
+            .map_err(Into::<anyhow::Error>::into)
+    }};
+    ($api:ident, $name:expr, $status:expr, $type:ty, $field_manager:expr, force) => {{
+        let patch = kube::api::Patch::Apply(serde_json::json!({
+            "apiVersion": <$type as kube::Resource>::api_version(&()),
+            "kind": <$type as kube::Resource>::kind(&()),
+            "status": $status
+        }));
+        let params = kube::api::PatchParams::apply($field_manager).force();
+        $api.patch_status($name, &params, &patch).await
             .map_err(Into::<anyhow::Error>::into)
-    }}
+    }};
 }
 
 pub fn condition_status(status: bool) -> String {
@@ -108,8 +126,8 @@ pub fn committed_condition(
     }
 }
 
-/// Generate an OwnerReference for any Kubernetes resource
-pub fn generate_owner_reference<T: Resource<DynamicType = ()>>(
+/// OwnerReference with `controller: true` for lifecycle and garbage collection.
+pub fn generate_owner_controller_reference<T: Resource<DynamicType = ()>>(
     object: &T,
 ) -> anyhow::Result<OwnerReference> {
     let name = object.meta().name.clone();

lib/src/reference_values.rs

@@ -29,6 +29,8 @@ macro_rules! update_image_pcrs {
         let map = (PCR_CONFIG_FILE.to_string(), image_pcrs_json.to_string());
         let data = std::collections::BTreeMap::from([map]);
         $map.data = Some(data);
+
+        // Operator and compute-pcr's both write to this configmap, hence client-side apply makes sense here.
         $api.replace(PCR_CONFIG_MAP, &Default::default(), &$map)
             .await?
     };