11---
2- title : Welcome read-cache-after-write consistency!!!
2+ title : Welcome read-cache-after-write consistency!
33# todo issue with this?
44# date: 2026-03-25
55author : >-
@@ -13,6 +13,8 @@ and the framwork will guaratee that these updates will be instantly visible,
1313thus when accessing resources from caches;
1414and naturally also for subsequent reconciliations.
1515
16+ I briefly [ talked about this] ( https://www.youtube.com/watch?v=HrwHh5Yh6AM&t=1387s ) topic at KubeCon last year.
17+
1618``` java
1719
1820public UpdateControl<WebPage > reconcile(WebPage webPage, Context<WebPage > context) {
@@ -30,48 +32,114 @@ public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> contex
3032}
3133```
3234
33- In addition to that framework will automatically filter events for your updates, thus those
34- which are result of our own updates .
35+ In addition to that framework will automatically filter events for your own updates,
36+ so those are not triggering the reconciliation again .
3537
3638{{% alert color=success %}}
37- ** These should significantly simplify controller development, and will make reconciliation
39+ ** This should significantly simplify controller development, and will make reconciliation
3840much simpler to reason about!**
3941{{% /alert %}}
4042
4143This post will deep dive in this topic, explore the details and rationale behind it.
4244
43- I briefly [ talked about this] ( https://www.youtube.com/watch?v=HrwHh5Yh6AM&t=1387s ) topic at KubeCon last year.
44-
4545## Informers and eventual consistency
4646
47- First we have to understand a fundamental building block of Kubernetes operators, called Informers.
48- Since there is plentiful accessible information about this topic, just in a nutshell, we have to know,
49- that these components:
47+ First we have to understand a fundamental building block of Kubernetes operators: Informers.
48+ Since there is plentiful accessible information about this topic, just in a nutshell, informers:
5049
51501 . Watches Kubernetes resources - K8S API sends events if a resource changes to the client
52- though a websocket. An usually contains the whole resource. (There are some exceptions, see Bookmarks).
51+ though a websocket, An event usually contains the whole resource. (There are some exceptions, see Bookmarks).
5352 See details about watch as K8S API concept in the [ official docs] ( https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-watch ) .
54532 . Caches the actual latest state of the resource.
55543 . If an informer receives and event in which the ` metadata.resourceVersion ` is different from the version
56- in the cached resource it propagates and event further, in our case triggering the reconiliation .
55+ in the cached resource it call the event handled, thus in our case triggers the reconciliation .
5756
5857A controller is usually composed of multiple informers, one is tracking the primary resources, and
5958there are also informers registered for each (secondary) resource we manage.
6059Informers are great since we don't have to poll the Kubernetes API, it is push based; and they provide
6160a cache, so reconciliations are very fast since they work on top of cached resources.
6261
63- Now let's take a look on the flow when we do an update to a resources. Let's say we manage a Pod from our
64- controller:
62+ Now let's take a look on the flow when we do an update to a resources:
63+
6564
65+ ``` mermaid
66+ graph LR
67+ subgraph Controller
68+ Informer:::informer
69+ Cache[(Cache)]:::teal
70+ Reconciler:::reconciler
71+ Informer -->|stores| Cache
72+ Reconciler -->|reads| Cache
73+ end
74+ K8S[⎈ Kubernetes API Server]:::k8s
6675
76+ Informer -->|watches| K8S
77+ Reconciler -->|updates| K8S
78+
79+ classDef informer fill:#C0527A,stroke:#8C3057,color:#fff
80+ classDef reconciler fill:#E8873A,stroke:#B05E1F,color:#fff
81+ classDef teal fill:#3AAFA9,stroke:#2B807B,color:#fff
82+ classDef k8s fill:#326CE5,stroke:#1A4AAF,color:#fff
83+ ```
84+
85+ It is easy to see that, the cache of the informer is eventually consistent with the update we sent from the reconciler.
86+ It usually just takes a very short time (few milliseconds) to sync the caches until everything is ok. Well, sometimes
87+ it is not. Websocket can be disconnected (actually happens on purpose sometimes), the API Server is slow etc.
6788
6889
90+ ## The problem(s) we try to solve
6991
70- TODO
71- - we do not cover deletes
72- - thank to shaw
92+ Let's consider the following operator:
93+ - we have a custom resource ` PodPrefix ` where the spec contains only one field: ` podNamePrexix ` ,
94+ - goal of the operator is to create a pod with name that has the prefix and a random sequence
95+ - it should never run two pods at once, if the ` podNamePrefix ` changes it should delete
96+ the actual pod and after that create a new one
97+ - the status of the custom resource should contain the ` generatedPodName `
7398
99+ How the code would look like in 5.2.x:
100+
101+ ``` java
102+
103+ public UpdateControl<PodPrefix > reconcile(PodPrefix primary, Context<PodPrefix > context) {
104+
105+ Optional<Pod > currentPod = context. getSecondaryResource(Pod . class);
106+
107+ if (currentPod. isPresent()) {
108+ if (podNameHasPrefix(primary. getSpec(). getPodNamePrexix() ,currentPod. get())) {
109+ // all ok we can return
110+ return UpdateControl . noUpdate();
111+ } else {
112+ // deletes the current pod with different name pattern
113+ context. getClient(). resource(currentPod. get()). delete();
114+ // it returns pod delete event will trigger the reconciliation
115+ return UpdateControl . noUpdate();
116+ }
117+ } else {
118+ // creates new pod
119+ var newPod = context. getClient(). resource(createPodWithOwnerReference(primary)). serviceSideApply();
120+ return UpdateControl . patchStatus(setPodNameToStatus(primary,newPod));
121+ }
122+ }
123+
124+ @Override
125+ public List<EventSource<?, WebPage > > prepareEventSources(EventSourceContext<WebPage > context) {
126+
127+ // Code omitted for adding InformerEventsSource for the pod
128+
74129
130+ }
131+ ```
75132
133+ That is quite simple if there is a pod with different name prefix we delete it, otherwise we create the pod
134+ and update the status. The pod is created with an owner reference so any update on pod will trigger
135+ the reconciliation.
76136
137+ Now consider the following sequence of events:
77138
139+ 1 . We create a ` PodPrefix ` with ` podNamePrefix ` : "first-pod-prefix".
140+ 2 . Concurrently:
141+ - The reconciliation logic runs and creates a Pod with a name generated suffix: "first-pod-prefix-a3j3ka";
142+ also sets this to the status and updates the custom resource status.
143+ - While the reconciliation is running we update the custom resource to have the value
144+ "second-pod-prefix"
145+ 3 . The update of the custom resource triggers the reconciliation.
0 commit comments