[Question] NoOp Event Source for Secondary Resource

[michaelkoepf]

Hi,

Assume I have a primary resource that reconciles a dependent resource via a standalone workflow. The workflow depends on information in the primary resource, thus, will be created on first reconciliation of the primary resource.

For the secondary resource, I do not want to be informed of changes of the dependent resource. However, when I do not override public List<EventSource<?, T>> prepareEventSources(final EventSourceContext<T> context) in the class implementing Reconciler<T> for the primary resource, or simply return an empty list, I get a NoEventSourceForClassException when I retrieve the secondary resource via context.getSecondaryResource().

Can I just use a GenericFilter to avoid change notifications of the secondary? E.g.,

InformerEventSourceConfiguration<MySecondary> configMapConfiguration =
        InformerEventSourceConfiguration.from(MySecondary.class, MyPrimary.class).withGenericFilter(resource -> false).build();

[metacosm]

I think using the filter would be a valid approach, indeed. I'd argue, though, that if you're not interested in changes in the resource, then the resource is NOT a secondary resource. Would you mind elaborating a little bit on your use case, please, so that we can understand better what you're trying to achieve?

[michaelkoepf]

@metacosm External state tracking.

While there are several ways to support this use case, we recommend storing such information in a dedicated Kubernetes resources (usually a ConfigMap or a Secret), so that it can be manipulated with common Kubernetes mechanisms.

The secondary config map stores some information that the primary receives during reconciliation and I do not want to bloat the status sub resource of the primary.

The reason I do not want to get informed of changes is that for my operator it's fine to say that the secondary is only modified by the operator itself. and if it's modified by any other means (which should not happen), the behavior is undefined.

[csviri]

I think that is a valid use case, although what I would consider is passing that information as environment variables to the Deployment of the operator

[csviri]

see, linked PR to support this, explicitly by having a noop SecondaryToPrimary mapper. Added a comment also there that this is not what you usually want, on the other hand it makes sense in some cases (I mean if you by definition have such use case).

[csviri]

just for the record ConfigMaps support immutability:
https://kubernetes.io/docs/concepts/configuration/configmap/#configmap-immutable

[metacosm]

I would assume that your operator would still like to know that the ConfigMap has been changed externally, maybe to report an error or to revert its state back to what it should be based on the primary resource, though, no?

[csviri]

Well if it is immutable it kinda the triggering is loosing it's purppose. Also usually reverting such changes where a resource is not managed by the controller, is not the responsibility of the controller. Usually it is FluxCD or other Gitops tools will do for you you anyways on a real life cluster.

Also note that the event itself is not accessed from the reconilier, so user don't know if the config map changed unless it compares it with some stored state, but in case of immutable secrets/config maps basically kubernetes is responsible making sure that those resources not change.

So in opinion this still makes sense to just cache resources with an informer and not trigger the reconciliation. Although I see this more like a corner case too.

[metacosm]

My comment was more meant for @michaelkoepf… Indeed, using the immutability feature of ConfigMap is probably the way to go and this should be correctly supported as a create-only dependent resource. If not, this is probably a bug.

However, if that's not an option, for some reason, I would think it would make sense for the operator to reset the ConfigMap value to whatever is required by the primary resource.

[michaelkoepf]

@metacosm i use the secondary to store some information about past reconciliation processes (e.g., some metrics) that are relevant for the next reconciliation. i know that it should be possible to recreate the secondary from the primary. but if a metric value is changed externally, there is no way to recalculate the observed, past metric value based on the primary. that's why it is easier to assume undefined behavior. or can i reset the secondary somehow to the previous version if it is changed externally?

[csviri]

My comment was more meant for @michaelkoepf… Indeed, using the immutability feature of ConfigMap is probably the way to go and this should be correctly supported as a create-only dependent resource. If not, this is probably a bug.

However, if that's not an option, for some reason, I would think it would make sense for the operator to reset the ConfigMap value to whatever is required by the primary resource.

Sorry @metacosm , I missunderstood some part, for some reason I though that this is a read only resource for the operator, not that it is created by it. Ehh hard day.

[csviri]

@metacosm i use the secondary to store some information about past reconciliation processes (e.g., some metrics) that are relevant for the next reconciliation. i know that it should be possible to recreate the secondary from the primary. but if a metric value is changed externally, there is no way to recalculate the observed, past metric value based on the primary. that's why it is easier to assume undefined behavior. or can i reset the secondary somehow to the previous version if it is changed externally?

Can't you just create an immutable config map for this purpose?

note that usually it is not the operators responsibility makind sure that resource is not changed, atlhough it is usually the case that it oevrrides some values if changed externally. But in this case it is not possible for you as I understand. So the best you could do probably is just to create an immutable config map. Note that it can be still deleted and recreated by someone, but you can prevent it with proper rbac in place for users.

Note that if its immutable if you receive an event or not does not play big role, the events does not influence the cache contents, they are produced when the resource is changed in the cache. So that will happen anyways if there is an informer - regardless if there is an event filter or not.

@michaelkoepf do I understand correctly that you want to use this config map in followup reconiliations? or how is it used further

[michaelkoepf]

@csviri yes. basically, i need a way to persist some data inside the cluster. i am using a config map, because there is no real advantage to creating a custom dependent resource.

the worfklow of the primary looks as follows:

• Get data (metrics) from secondary (config map)
• Observe metrics. Take action based on current metrics and data from past metrics (data from
  secondary)
• Write new metrics (and maybe additional data) back to secondary.

I know there might be no need for a secondary resource in this case. But it's easier to unit test and I do not want to blow up the status of the primary.

But I think the fundamental problem of external changes, however, stays the same even when putting it in the status? So if I would take an action in external changes, I would need a way to roll back to the previous version, which cannot be computed.

[csviri]

@michaelkoepf What do you exactly mean by external change?

Putting it into the status would be also legit in this case as far I can see.

[michaelkoepf]

@csviri external meaning not by the operator.

but even when putting it in the status, the problem remains the same. i cannot recompute the field if it is changed, e.g., manually.

to give an example from built-in kubernetes operators: for the horizontal pod autoscaler, the replicas of the past scaling actions are tracked for the stabilization window. this can also be not recomputed based on the definition HPA definition.

can i easily revert a secondary resource to a previous version? if not, i do not see a benefit of getting informed of changes in the secondary 🤔