Skip to content

Commit 4ee62d8

Browse files
committed
wip
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
1 parent 942ef64 commit 4ee62d8

File tree

1 file changed

+84
-16
lines changed

1 file changed

+84
-16
lines changed
Lines changed: 84 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Welcome read-cache-after-write consistency!!!
2+
title: Welcome read-cache-after-write consistency!
33
# todo issue with this?
44
#date: 2026-03-25
55
author: >-
@@ -13,6 +13,8 @@ and the framwork will guaratee that these updates will be instantly visible,
1313
thus when accessing resources from caches;
1414
and naturally also for subsequent reconciliations.
1515

16+
I briefly [talked about this](https://www.youtube.com/watch?v=HrwHh5Yh6AM&t=1387s) topic at KubeCon last year.
17+
1618
```java
1719

1820
public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {
@@ -30,48 +32,114 @@ public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> contex
3032
}
3133
```
3234

33-
In addition to that framework will automatically filter events for your updates, thus those
34-
which are result of our own updates.
35+
In addition to that framework will automatically filter events for your own updates,
36+
so those are not triggering the reconciliation again.
3537

3638
{{% alert color=success %}}
37-
**These should significantly simplify controller development, and will make reconciliation
39+
**This should significantly simplify controller development, and will make reconciliation
3840
much simpler to reason about!**
3941
{{% /alert %}}
4042

4143
This post will deep dive in this topic, explore the details and rationale behind it.
4244

43-
I briefly [talked about this](https://www.youtube.com/watch?v=HrwHh5Yh6AM&t=1387s) topic at KubeCon last year.
44-
4545
## Informers and eventual consistency
4646

47-
First we have to understand a fundamental building block of Kubernetes operators, called Informers.
48-
Since there is plentiful accessible information about this topic, just in a nutshell, we have to know,
49-
that these components:
47+
First we have to understand a fundamental building block of Kubernetes operators: Informers.
48+
Since there is plentiful accessible information about this topic, just in a nutshell, informers:
5049

5150
1. Watches Kubernetes resources - K8S API sends events if a resource changes to the client
52-
though a websocket. An usually contains the whole resource. (There are some exceptions, see Bookmarks).
51+
though a websocket, An event usually contains the whole resource. (There are some exceptions, see Bookmarks).
5352
See details about watch as K8S API concept in the [official docs](https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-watch).
5453
2. Caches the actual latest state of the resource.
5554
3. If an informer receives and event in which the `metadata.resourceVersion` is different from the version
56-
in the cached resource it propagates and event further, in our case triggering the reconiliation.
55+
in the cached resource it call the event handled, thus in our case triggers the reconciliation.
5756

5857
A controller is usually composed of multiple informers, one is tracking the primary resources, and
5958
there are also informers registered for each (secondary) resource we manage.
6059
Informers are great since we don't have to poll the Kubernetes API, it is push based; and they provide
6160
a cache, so reconciliations are very fast since they work on top of cached resources.
6261

63-
Now let's take a look on the flow when we do an update to a resources. Let's say we manage a Pod from our
64-
controller:
62+
Now let's take a look on the flow when we do an update to a resources:
63+
6564

65+
```mermaid
66+
graph LR
67+
subgraph Controller
68+
Informer:::informer
69+
Cache[(Cache)]:::teal
70+
Reconciler:::reconciler
71+
Informer -->|stores| Cache
72+
Reconciler -->|reads| Cache
73+
end
74+
K8S[⎈ Kubernetes API Server]:::k8s
6675
76+
Informer -->|watches| K8S
77+
Reconciler -->|updates| K8S
78+
79+
classDef informer fill:#C0527A,stroke:#8C3057,color:#fff
80+
classDef reconciler fill:#E8873A,stroke:#B05E1F,color:#fff
81+
classDef teal fill:#3AAFA9,stroke:#2B807B,color:#fff
82+
classDef k8s fill:#326CE5,stroke:#1A4AAF,color:#fff
83+
```
84+
85+
It is easy to see that, the cache of the informer is eventually consistent with the update we sent from the reconciler.
86+
It usually just takes a very short time (few milliseconds) to sync the caches until everything is ok. Well, sometimes
87+
it is not. Websocket can be disconnected (actually happens on purpose sometimes), the API Server is slow etc.
6788

6889

90+
## The problem(s) we try to solve
6991

70-
TODO
71-
- we do not cover deletes
72-
- thank to shaw
92+
Let's consider the following operator:
93+
- we have a custom resource `PodPrefix` where the spec contains only one field: `podNamePrexix`,
94+
- goal of the operator is to create a pod with name that has the prefix and a random sequence
95+
- it should never run two pods at once, if the `podNamePrefix` changes it should delete
96+
the actual pod and after that create a new one
97+
- the status of the custom resource should contain the `generatedPodName`
7398

99+
How the code would look like in 5.2.x:
100+
101+
```java
102+
103+
public UpdateControl<PodPrefix> reconcile(PodPrefix primary, Context<PodPrefix> context) {
104+
105+
Optional<Pod> currentPod = context.getSecondaryResource(Pod.class);
106+
107+
if (currentPod.isPresent()) {
108+
if (podNameHasPrefix(primary.getSpec().getPodNamePrexix() ,currentPod.get())) {
109+
// all ok we can return
110+
return UpdateControl.noUpdate();
111+
} else {
112+
// deletes the current pod with different name pattern
113+
context.getClient().resource(currentPod.get()).delete();
114+
// it returns pod delete event will trigger the reconciliation
115+
return UpdateControl.noUpdate();
116+
}
117+
} else {
118+
// creates new pod
119+
var newPod = context.getClient().resource(createPodWithOwnerReference(primary)).serviceSideApply();
120+
return UpdateControl.patchStatus(setPodNameToStatus(primary,newPod));
121+
}
122+
}
123+
124+
@Override
125+
public List<EventSource<?, WebPage>> prepareEventSources(EventSourceContext<WebPage> context) {
126+
127+
// Code omitted for adding InformerEventsSource for the pod
128+
74129

130+
}
131+
```
75132

133+
That is quite simple if there is a pod with different name prefix we delete it, otherwise we create the pod
134+
and update the status. The pod is created with an owner reference so any update on pod will trigger
135+
the reconciliation.
76136

137+
Now consider the following sequence of events:
77138

139+
1. We create a `PodPrefix` with `podNamePrefix`: "first-pod-prefix".
140+
2. Concurrently:
141+
- The reconciliation logic runs and creates a Pod with a name generated suffix: "first-pod-prefix-a3j3ka";
142+
also sets this to the status and updates the custom resource status.
143+
- While the reconciliation is running we update the custom resource to have the value
144+
"second-pod-prefix"
145+
3. The update of the custom resource triggers the reconciliation.

0 commit comments

Comments
 (0)