Skip to content

Commit 0664b6d

Browse files
committed
improve
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
1 parent 65541d6 commit 0664b6d

File tree

1 file changed

+123
-104
lines changed

1 file changed

+123
-104
lines changed
Lines changed: 123 additions & 104 deletions
Original file line numberDiff line numberDiff line change
@@ -1,75 +1,71 @@
11
---
2-
title: Working with EventSource caches
2+
title: Accessing resources in caches
33
weight: 48
44
---
55

66
As described in [Event sources and related topics](eventing.md) event sources are the backbone
7-
for caching resources and triggering reconciliation of primary resources related
8-
to these secondary resources.
7+
for caching resources and triggering the reconciliation for primary resources thar are related
8+
to cached resources.
99

10-
In Kubernetes parlance, `Informers` handle that responsibility. Without going into
11-
the details (there are plenty of good documents online regarding this topics), informers
12-
watch resources, cache them, and emit an event whenever watched resources change.
10+
In Kubernetes world, the component that does this is called Informer. Without going into
11+
the details (there are plenty of good documents online regarding informers), its responsibility
12+
is to watch resources, cache them, and emit an event if the resource changed.
1313

14-
`EventSource` generalizes this concept to also cover non-Kubernetes resources. Thus,
15-
allowing caching of external resources, and triggering reconciliation when those change.
14+
EventSource is a generalized concept of Informer to non-Kubernetes resources. Thus,
15+
to cache external resources, and trigger reconciliation if those change.
1616

1717
## The InformerEventSource
1818

19-
The underlying informer implementation comes from the Fabric8 client,
20-
called [DefaultSharedIndexInformer](https://github.com/fabric8io/kubernetes-client/blob/main/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/DefaultSharedIndexInformer.java).
21-
[InformerEventSource](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerEventSource.java)
22-
in Java Operator SDK wraps informers from Fabric8 client, thus presenting a unified front to deal with Kubernetes and
23-
non-Kubernetes resources with the `EventSource` architecture.
19+
The underlying informer implementation comes from the fabric8 client, called [DefaultSharedIndexInformer](https://github.com/fabric8io/kubernetes-client/blob/main/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/DefaultSharedIndexInformer.java).
20+
[InformerEventSource](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerEventSource.java)
21+
in Java Operator SDK wraps informers from fabric8 client.
22+
The purpose of such wrapping is to add additional capabilities required for controllers.
23+
(In general, Informers are not used only for implementing controllers).
2424

25-
However, `InformerEventSource` also provide additional capabilities such as:
26-
27-
- recording the relations between primary and secondary resources so that the event source knows which primary resource
28-
to trigger a reconciler with whenever one of the cached secondary resources cached by the informer changes,
29-
- setting up multiple informers for the same type if needed, for example to transparently watch multiple namespaces,
30-
without you having to worry about it,
31-
- dynamically adding/removing watched namespaces, if needed
32-
- and more, outside of the scope of this document.
25+
Such capabilities are:
26+
- maintaining and index to which primary are the secondary resources in informer cache are related to.
27+
- setting up multiple informers for the same type if needed. You need informer per namespace if the informer
28+
is not watching the whole cluster.
29+
- Dynamically adding/removing watched namespaces.
30+
- Some others, what is out of the scope of this document.
3331

3432
### Associating Secondary Resources to Primary Resource
3533

36-
Event sources need to trigger the appropriate reconciler, providing the correct primary resource, whenever one of their
37-
handled secondary resources changes. It is thus core to an event source's role to identify which primary resource (
38-
usually, your custom resource) is potentially impacted by that change.
39-
The framework uses [`SecondaryToPrimaryMapper`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/SecondaryToPrimaryMapper.java)
40-
for this purpose. For `InformerEventSources`, which target Kubernetes resources, this mapping is typically done using
41-
either the owner reference or an annotation on the secondary resource. For external resources, other mechanisms need to
42-
be used and there are also cases where the default mechanisms provided by the SDK do not work, even for Kubernetes
43-
resources.
44-
45-
However, once the event source has triggered a primary resource reconciliation, the associated reconciler needs to
46-
access the secondary resources which changes caused the reconciliation. Indeed, the information from the secondary
47-
resources might be needed during the reconciliation. For that purpose,
48-
`InformerEventSource` maintains a reverse
49-
index [PrimaryToSecondaryIndex](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/DefaultPrimaryToSecondaryIndex.java),
50-
based on the result of the `SecondaryToPrimaryMapper`result.
34+
The question is, how to trigger reconciliation of a primary resources (your custom resource),
35+
when Informer receives a new resource.
36+
For this purpose the framework uses [`SecondaryToPrimaryMapper`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/SecondaryToPrimaryMapper.java)
37+
that tells (usually) based on the resource which primary resource reconciliation to trigger.
38+
The mapping is usually done based on the owner reference or annotation on the secondary resource.
39+
(But not always, as we will see)
40+
41+
It is important to realize that if a resource triggers the reconciliation of a primary resource, that
42+
resource naturally will be used during reconciliation. So the reconciler will need to access them.
43+
Therefore, InformerEventSource maintains a revers index [PrimaryToSecondaryIndex](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/DefaultPrimaryToSecondaryIndex.java),
44+
based on the result of the `SecondaryToPrimaryMapper`result.
5145

5246
## Unified API for Related Resources
5347

54-
To access all related resources for a primary resource, the framework provides an API to access the related
55-
secondary resources using the `Set<R> getSecondaryResources(Class<R> expectedType)` method of the `Context` object
56-
provided as part of the `reconcile` method.
48+
To access all related resources for a primary resource, the framework provides an API to access the related
49+
secondary resources using:
50+
51+
```java
52+
Context.getSecondaryResources(Class<R> expectedType);
53+
```
5754

58-
For `InformerEventSource`, this will leverage the associated `PrimaryToSecondaryIndex`. Resources are then retrieved
59-
from the informer's cache. Note that since all those steps work
55+
That will list all the related resources of a certain type, based on the `InformerEventSource`'s `PrimaryToSecondaryIndex`.
56+
Based on that index, it reads the resources from the Informers cache. Note that since all those steps work
6057
on top of indexes, those operations are very fast, usually O(1).
6158

62-
While we've focused mostly on `InformerEventSource`, this concept can be extended to all `EventSources`, since
63-
[`EventSource`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/EventSource.java#L93)
64-
actually implements the `Set<R> getSecondaryResources(P primary)` method that can be called from the `Context`.
59+
We mostly talk about InformerEventSource, but this works in similar ways for generalized EventSources concept, since
60+
the [`EventSource`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/EventSource.java#L93)
61+
actually implements the `Set<R> getSecondaryResources(P primary);` method. That is just called from the context.
6562

66-
As there can be multiple event sources for the same resource types, things are a little more complex: the union of each
67-
event source results is returned.
63+
It is a bit more complex than that, since there can be multiple event sources for the same type, in that case
64+
the union of the results is returned.
6865

6966
## Getting Resources Directly from Event Sources
7067

71-
Note that nothing prevents you from directly accessing resources in the cache without going through
72-
`getSecondaryResources(...)`:
68+
Note that nothing stops you to directly access the resources in the cache (so not just through `getSecondaryResources(...)`):
7369

7470
```java
7571
public class WebPageReconciler implements Reconciler<WebPage> {
@@ -78,30 +74,35 @@ public class WebPageReconciler implements Reconciler<WebPage> {
7874

7975
@Override
8076
public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {
81-
// accessing resource directly from an event source
82-
var mySecondaryResource = configMapEventSource.get(new ResourceID("name", "namespace"));
83-
// details omitted
77+
// accessing resource directly from an event source
78+
var mySecondaryResource = configMapEventSource.get(new ResourceID("name","namespace"));
79+
// details omitted
8480
}
85-
81+
8682
@Override
8783
public List<EventSource<?, WebPage>> prepareEventSources(EventSourceContext<WebPage> context) {
88-
configMapEventSource = new InformerEventSource<>(
84+
configMapEventSource = new InformerEventSource<>(
8985
InformerEventSourceConfiguration.from(ConfigMap.class, WebPage.class)
9086
.withLabelSelector(SELECTOR)
9187
.build(),
9288
context);
93-
89+
9490
return List.of(configMapEventSource);
9591
}
9692
}
9793
```
9894

9995
## The Use Case for PrimaryToSecondaryMapper
10096

97+
TL;DR: `PrimaryToSecondaryMapper` is used to access secondary resources in `InformerEventSource` instead
98+
of the PrimaryToSecondaryIndex, thus `InfomerEventSource.getSecondaryResources(..)` will call this mapper
99+
to get the target secondary resources. This is usually required in cases when the `SecondaryToPrimaryMapper`
100+
is using the informer caches to list the target resources.
101+
101102
As we discussed, we provide a unified API to access related resources using `Context.getSecondaryResources(...)`.
102103
The name `Secondary` refers to resources that a reconciler needs to take into account to properly reconcile a primary
103104
resource. These resources cover more than only `child` resources as resources created by a reconciler are sometimes
104-
called and which usually have a owner reference pointing to the primary (and, typically, custom) resource. These also
105+
called and which usually have an owner reference pointing to the primary (and, typically, custom) resource. These also
105106
cover `related` resources (which might or might not be managed by Kubernetes) that serve as input for reconciliations.
106107

107108
There are cases where the SDK needs more information than what is readily available, in particular when some of these
@@ -118,81 +119,99 @@ resource.
118119
See full
119120
sample [here](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework/src/test/java/io/javaoperatorsdk/operator/baseapi/primarytosecondary).
120121

121-
Even writing a `SecondaryToPrimaryMapper` is not trivial in this case, if the cluster is updated, we want to trigger
122-
all `Job`s that are referencing it. So we have to efficiently get the list of jobs, and return their ResourceIDs in
123-
the mapper. So we need an index that maps `Cluster` to `Job`s. Here we can use indexing capabilities of the Informers:
124-
125122
```java
123+
InformerEventSourceConfiguration
124+
.from(Cluster.class, Job.class)
125+
.withSecondaryToPrimaryMapper(cluster -> context.getPrimaryCache()
126+
.list().filter(job -> job.getSpec().getClusterName().equals(cluster.getMetadata().getName()))
127+
.map(ResourceID::fromResource)
128+
.collect(Collectors.toSet()))
129+
```
130+
131+
This will trigger all the related `Jobs` if the related cluster changes. Also, the maintaining the `PrimaryToSecondaryIndex`.
132+
So we can use the `getSecondaryResources` in the `Job` reconciler to access the cluster.
133+
However, there is an issue, what if now there is a new `Job` created? The new job does not propagate
134+
automatically to `PrimaryToSecondaryIndex` in the `InformerEventSource` of the `Cluster`. That re-indexing
135+
happens where there is an event received for the `Cluster` and triggers all the `Jobs` again.
136+
Until that would happen again you could not use `getSecondaryResources` for the new `Job`, since the new
137+
job won't bre present in the reverse index.
138+
139+
You could access the Cluster directly from cache though in the reconciler:
140+
141+
```java
126142

127143
@Override
128-
public List<EventSource<?, Job>> prepareEventSources(EventSourceContext<Job> context) {
144+
public UpdateControl<Job> reconcile(Job resource, Context<Job> context) {
129145

130-
context.getPrimaryCache()
131-
.addIndexer(JOB_CLUSTER_INDEX,
132-
(job -> List.of(indexKey(job.getSpec().getClusterName(), job.getMetadata().getNamespace()))));
146+
clusterInformer.get(new ResourceID(job.getSpec().getClusterName(), job.getMetadata().getNamespace()));
133147

134148
// omitted details
135149
}
136150
```
137151

138-
where index key is a String that uniquely identifies a Cluster:
152+
But if you still want to use the unified API (thus `context.getSecondaryResources()`), we have to add
153+
`PrimaryToSecondaryMapper`:
139154

140155
```java
141-
private String indexKey(String clusterName, String namespace) {
142-
return clusterName + "#" + namespace;
143-
}
156+
clusterInformer.withPrimaryToSecondaryMapper( job ->
157+
Set.of(new ResourceID(job.getSpec().getClusterName(), job.getMetadata().getNamespace())));
144158
```
145159

146-
In the InformerEventSource for the cluster now we can get all the `Jobs` for the `Cluster` using this index:
160+
Using `PrimaryToSecondaryMapper` the InformerEventSource won't use the `PrimaryToSecondaryIndex`
161+
to get the resources, instead will call this mapper and will get the resources based on its result.
162+
In fact if this mapper is set the `PrimaryToSecondaryIndex` is not even initialized.
147163

148-
```java
164+
### Using Informer Indexes to Improve Performance
149165

150-
InformerEventSource<Job, Cluster> clusterInformer =
151-
new InformerEventSource(
152-
InformerEventSourceConfiguration.from(Cluster.class, Job.class)
153-
.withSecondaryToPrimaryMapper(
154-
cluster ->
155-
context.getPrimaryCache()
156-
.byIndex(
157-
JOB_CLUSTER_INDEX,
158-
indexKey(
159-
cluster.getMetadata().getName(),
160-
cluster.getMetadata().getNamespace()))
161-
.stream()
162-
.map(ResourceID::fromResource)
163-
.collect(Collectors.toSet()))
164-
.withNamespacesInheritedFromController().build(), context);
165-
```
166+
In the `SecondaryToPrimaryMapper` above we are looping through all the resources in the cache:
166167

167-
This will trigger all the related `Jobs` if a cluster changes. Also, the maintaining the `PrimaryToSecondaryIndex`.
168-
So we can use the `getSecondaryResources` in the `Job` reconciler to access the cluster.
169-
However, there is an issue, what if now there is a new `Job` created? The new job does not propagate
170-
automatically to `PrimaryToSecondaryIndex` in the `InformerEventSource` of the `Cluster`. That re-indexing
171-
happens where there is an event received for the `Cluster` and triggers all the `Jobs` again.
172-
Until that would happen again you could not use `getSecondaryResources` for the new `Job`.
168+
```java
169+
context.getPrimaryCache()
170+
.list().filter(job -> job.getSpec().getClusterName().equals(cluster.getMetadata().getName()))
171+
```
173172

174-
You could access the Cluster directly from cache though in the reconciler:
173+
This can be inefficient in case there is a large number of primary (Job) resources. To make it more efficient, we can
174+
create an index in the underlying Informer, that indexed the target jobs for a cluster:
175175

176-
```java
176+
```java
177177

178178
@Override
179-
public UpdateControl<Job> reconcile(Job resource, Context<Job> context) {
180-
181-
clusterInformer.get(new ResourceID(job.getSpec().getClusterName(), job.getMetadata().getNamespace()));
179+
public List<EventSource<?, Job>> prepareEventSources(EventSourceContext<Job> context) {
182180

181+
context.getPrimaryCache()
182+
.addIndexer(JOB_CLUSTER_INDEX,
183+
(job -> List.of(indexKey(job.getSpec().getClusterName(), job.getMetadata().getNamespace()))));
184+
183185
// omitted details
184186
}
185187
```
186188

187-
But if you still want to use the unified API (thus `context.getSecondaryResources()`), we can add
188-
`PrimaryToSecondaryMapper`:
189+
where `indexKey` is a String that uniquely identifies a Cluster:
189190

190191
```java
191-
clusterInformer.withPrimaryToSecondaryMapper( job ->
192-
Set.of(new ResourceID(job.getSpec().getClusterName(), job.getMetadata().getNamespace())));
192+
private String indexKey(String clusterName, String namespace) {
193+
return clusterName + "#" + namespace;
194+
}
193195
```
194196

195-
That will get the `Cluster` for the `Job` from the cache of `Cluster`'s `InformerEventSource`.
196-
So it won't use the `PrimaryToSecondaryIndex`, that might be outdated, but instead will use the
197-
`PrimaryToSecondaryMapper` to get
198-
the target `Cluster` ids.
197+
From this point, we can use the index to get the target resources very efficiently:
198+
199+
```java
200+
201+
InformerEventSource<Job,Cluster> clusterInformer =
202+
new InformerEventSource(
203+
InformerEventSourceConfiguration.from(Cluster.class, Job.class)
204+
.withSecondaryToPrimaryMapper(
205+
cluster ->
206+
context
207+
.getPrimaryCache()
208+
.byIndex(
209+
JOB_CLUSTER_INDEX,
210+
indexKey(
211+
cluster.getMetadata().getName(),
212+
cluster.getMetadata().getNamespace()))
213+
.stream()
214+
.map(ResourceID::fromResource)
215+
.collect(Collectors.toSet()))
216+
.withNamespacesInheritedFromController().build(), context);
217+
```

0 commit comments

Comments
 (0)