Support for expectations pattern

[csviri]

See references and description of expectations here:
https://ahmet.im/blog/controller-pitfalls/#expectations-pattern

This does not seems to be supported in controller runtime either:
kubernetes-sigs/controller-runtime#644

We should investigate how we could support this in different layers of the framework.

see sample expectations:
https://github.com/kubernetes/kubernetes/blob/8d9ab387273747fcccb9325269070987b8f1c66b/pkg/controller/controller_utils.go#L159
https://pkg.go.dev/github.com/aws/karpenter-core/pkg/test/expectations
https://pkg.go.dev/github.com/greths/brice/pkg/expectations

[csviri]

What I could imaging is to have an API that looks something like this:

UpdateControl reconcile(Primary primary, Context<Primary> context) {

// omit reconcile details

  return UpdateControl.expect((context,primary)-> {
     var resources = context.getSecondaryResources(MyResource.class)
     return checkIfResourcesInExpectedState(resources) 
   },Duration.ofSeconds(5))

}

In other words, that the reconcile could return an expectation with a timeout, and the reconciliation would not be triggered until that expectation predicate is true or it timeouts.

While this sounds nice, and might provide a benefit for the users, execution, we won't have too much space to optimize it, so in terms of execution, it would have to be done anyway in the executor service thread that we use now for reconciliation. So this can be directly transformed to something like this:

UpdateControl reconcile(Primary primary, Context<Primary> context) {
  var resources = context.getSecondaryResources(MyResource.class)
   if (!checkIfResourcesInexpectedState(resources)) {
      return UpdateControl.noUpdate()
   } 
}

Except the timeout for which user might have to track hat in a ConcurrentHashMap or such.

It would be a completely different story if this worked as a cross-cutting filter, where all the event source caches are available.

It would worth to make a prototype, how that would look like, but for that we would need a new abstraction, probably a subset of Context to access secondary resources from the cache and directly event sources in case.

[csviri]

cc @xstefank @metacosm @shawkins, maybe something to discuss in the next community meeting.

[csviri]

see also: https://kubernetes.slack.com/archives/C02MRBMN00Z/p1751979029540209

[csviri]

@metacosm
this is also where this would be needed:
#2893
So if the primary resource is deleted we cleanup the expectation map, without finalizer/cleaner interface that has to be done on delete event.

[metacosm]

To be honest, I'm not convinced at all by this feature and I'm not certain that this brings any value to the SDK, nor what use cases this addresses that cannot already be addressed or that we need to address.

[csviri]

@metacosm isn't is clear from thie blogpost: https://ahmet.im/blog/controller-pitfalls/#expectations-pattern linked to the issue.
we can explain it on community meeting, if that is not clear enough from there. In case pls let me know which part is not clear.

[csviri]

Added also link sample expectation from Kubernetes, that might help to understand