Skip to content

Commit a873794

Browse files
csviricsviri
committed
docs: add page to describe cache access model
Signed-off-by: Attila Mészáros <a_meszaros@apple.com>
1 parent 86603d9 commit a873794

File tree

3 files changed

+198
-0
lines changed

3 files changed

+198
-0
lines changed

‎docs/content/en/docs/documentation/_index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ This section contains detailed documentation for all Java Operator SDK features
1818
## Advanced Features
1919

2020
- **[Eventing](eventing/)** - Understanding the event-driven model
21+
- **[Accessing Resources in Caches](access-resources/) - How to access resources in caches
2122
- **[Observability](observability/)** - Monitoring and debugging your operators
2223
- **[Other Features](features/)** - Additional capabilities and integrations
2324

‎docs/content/en/docs/documentation/access-resources.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
---
2+
title: Accessing resources in caches
3+
weight: 48
4+
---
5+
6+
As described in [Event sources and related topics](eventing.md) event sources are the backbone
7+
for caching resources and triggering the reconciliation for primary resources thar are related
8+
to cached resources.
9+
10+
In Kubernetes world, the component that does this is called Informer. Without going into
11+
the details (there are plenty of good documents online regarding informers), its responsibility
12+
is to watch resources, cache them, and emit an event if the resource changed.
13+
14+
EventSource is a generalized concept of Informer to non-Kubernetes resources. Thus,
15+
to cache external resources, and trigger reconciliation if those change.
16+
17+
## The InformerEventSource
18+
19+
The underlying informer implementation comes from the fabric8 client, called [DefaultSharedIndexInformer](https://github.com/fabric8io/kubernetes-client/blob/main/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/DefaultSharedIndexInformer.java).
20+
[InformerEventSource](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerEventSource.java)
21+
in Java Operator SDK wraps informers from fabric8 client.
22+
The purpose of such wrapping is to add additional capabilities required for controllers.
23+
(In general, Informers are not used only for implementing controllers).
24+
25+
Such capabilities are:
26+
- maintaining and index to which primary are the secondary resources in informer cache are related to.
27+
- setting up multiple informers for the same type if needed. You need informer per namespace if the informer
28+
is not watching the whole cluster.
29+
- Dynamically adding/removing watched namespaces.
30+
- Some others, what is out of the scope of this document.
31+
32+
### Associating Secondary Resources to Primary Resource
33+
34+
The question is, how to trigger reconciliation of a primary resources (your custom resource),
35+
when Informer receives a new resource.
36+
For this purpose the framework uses [`SecondaryToPrimaryMapper`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/SecondaryToPrimaryMapper.java)
37+
that tells (mostly) based on the resource which primary resource reconciliation to trigger.
38+
The mapping is usually done based on the owner reference or annotation on the secondary resource.
39+
(But not always, as we will see)
40+
41+
It is important to realize that if a resource triggers the reconciliation of a primary resource, that
42+
resource naturally will be used during reconciliation. So the reconciler will need to access them.
43+
Therefore, InformerEventSource maintains a revers index [PrimaryToSecondaryIndex](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/DefaultPrimaryToSecondaryIndex.java),
44+
based on the result of the `SecondaryToPrimaryMapper`result.
45+
46+
## Unified API for Related Resources
47+
48+
To access all related resources for a primary resource, the framework provides an API to access the related
49+
secondary resources using:
50+
51+
```java
52+
Context.getSecondaryResources(Class<R> expectedType);
53+
```
54+
55+
That will list all the related resources of a certain type, based on the `InformerEventSource`'s `PrimaryToSecondaryIndex`.
56+
Based on that index, it reads the resources from the Informers cache. Note that since all those steps work
57+
on top of indexes, those operations are very fast, usually O(1).
58+
59+
We mostly talk about InformerEventSource, but this works in similar ways for generalized EventSources concept, since
60+
the [`EventSource`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/EventSource.java#L93)
61+
actually implements the `Set<R> getSecondaryResources(P primary);` method. That is just called from the context.
62+
63+
It is a bit more complex than that, since there can be multiple event sources for the same type, in that case
64+
the union of the results is returned.
65+
66+
## Getting Resources Directly from Event Sources
67+
68+
Note that nothing stops you to directly access the resources in the cache (so not just through `getSecondaryResources(...)`):
69+
70+
```java
71+
public class WebPageReconciler implements Reconciler<WebPage> {
72+
73+
InformerEventSource<ConfigMap, WebPage> configMapEventSource;
74+
75+
@Override
76+
public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {
77+
// accessing resource directly from an event source
78+
var mySecondaryResource = configMapEventSource.get(new ResourceID("name","namespace"));
79+
// details omitted
80+
}
81+
82+
@Override
83+
public List<EventSource<?, WebPage>> prepareEventSources(EventSourceContext<WebPage> context) {
84+
configMapEventSource = new InformerEventSource<>(
85+
InformerEventSourceConfiguration.from(ConfigMap.class, WebPage.class)
86+
.withLabelSelector(SELECTOR)
87+
.build(),
88+
context);
89+
90+
return List.of(configMapEventSource);
91+
}
92+
}
93+
```
94+
95+
## The Use Case for PrimaryToSecondaryMapper
96+
97+
As we discussed, we provide an unified API to access related resources using `Context.getSecondaryResources(...)`.
98+
This method was on purpose uses `Secondary` resource, since those are not only child resources - how
99+
resources that are created by the reconciler are sometimes called in Kubernetes world - and usually have owner references for the custom resources;
100+
neither related resources which are usually resources that serves as input for the primary (not managed). It is the union of both.
101+
102+
The issue is if we want to trigger reconciliation for a resource, that does not have an owner reference or other direct
103+
association with the primary resource.
104+
Typically, if you have ConfigMap where you have input parameters for a set of primary resources,
105+
and the primary is actually referencing the secondary resource. Like having the name of the ConfigMap in the spec part
106+
of the primary resource.
107+
108+
As an example we provide, have a primary resource a `Job` that references a `Cluster` resource.
109+
So multiple `Job` can reference the same `Cluster`, and we want to trigger `Job` reconciliation if cluster changes.
110+
See full sample [here](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework/src/test/java/io/javaoperatorsdk/operator/baseapi/primarytosecondary).
111+
But the `Cluster` (the secondary resource) does not reference the `Jobs`.
112+
113+
Even writing a `SecondaryToPrimaryMapper` is not trivial in this case, if the cluster is updated, we want to trigger
114+
all `Jobs` that are referencing it. So we have to efficiently get the list of jobs, and return their ResourceIDs in
115+
the mapper. So we need an index that maps `Cluster` to `Jobs`. Here we can use indexing capabilities of the Informers:
116+
117+
```java
118+
119+
@Override
120+
public List<EventSource<?, Job>> prepareEventSources(EventSourceContext<Job> context) {
121+
122+
context.getPrimaryCache()
123+
.addIndexer(JOB_CLUSTER_INDEX,
124+
(job -> List.of(indexKey(job.getSpec().getClusterName(), job.getMetadata().getNamespace()))));
125+
126+
// omitted details
127+
}
128+
```
129+
130+
where index key is a String that uniquely idetifies a Cluster:
131+
132+
```java
133+
private String indexKey(String clusterName, String namespace) {
134+
return clusterName + "#" + namespace;
135+
}
136+
```
137+
138+
In the InformerEventSource for the cluster now we can get all the `Jobs` for the `Cluster` using this index:
139+
140+
```java
141+
142+
InformerEventSource<Job,Cluster> clusterInformer =
143+
new InformerEventSource(
144+
InformerEventSourceConfiguration.from(Cluster.class, Job.class)
145+
.withSecondaryToPrimaryMapper(
146+
cluster ->
147+
context
148+
.getPrimaryCache()
149+
.byIndex(
150+
JOB_CLUSTER_INDEX,
151+
indexKey(
152+
cluster.getMetadata().getName(),
153+
cluster.getMetadata().getNamespace()))
154+
.stream()
155+
.map(ResourceID::fromResource)
156+
.collect(Collectors.toSet()))
157+
.withNamespacesInheritedFromController().build(), context);
158+
```
159+
160+
This will trigger all the related `Jobs` if a cluster changes. Also, the maintaining the `PrimaryToSecondaryIndex`.
161+
So we can use the `getSecondaryResources` in the `Job` reconciler to access the cluster.
162+
However, there is an issue, what if now there is a new `Job` created? The new job does not propagate
163+
automatically to `PrimaryToSecondaryIndex` in the `InformerEventSource` of the `Cluster`. That re-indexing
164+
happens where there is an event received for the `Cluster` and triggers all the `Jobs` again.
165+
Until that would happen again you could not use `getSecondaryResources` for the new `Job`.
166+
167+
You could access the Cluster directly from cache though in the reconciler:
168+
169+
```java
170+
@Override
171+
public UpdateControl<Job> reconcile(Job resource, Context<Job> context) {
172+
173+
clusterInformer.get(new ResourceID(job.getSpec().getClusterName(), job.getMetadata().getNamespace()));
174+
175+
// omitted details
176+
}
177+
```
178+
179+
But if still want to use the unified API (thus `context.getSecondaryResources()`), we can add
180+
`PrimaryToSecondaryMapper`:
181+
182+
```java
183+
clusterInformer.withPrimaryToSecondaryMapper( (PrimaryToSecondaryMapper<Job>)
184+
job -> Set.of(new ResourceID( job.getSpec().getClusterName(),job.getMetadata().getNamespace())));
185+
```
186+
187+
That will get the `Cluster` for the `Job` from the cache of `Cluster`'s `InformerEventSource`.
188+
So it won't use the `PrimaryToSecondaryIndex`, that might be outdated, but instead will use the `PrimaryToSecondaryMapper` to get
189+
the target `Cluster` ids.

‎operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/SecondaryToPrimaryMapper.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,15 @@
44

55
import io.javaoperatorsdk.operator.processing.event.ResourceID;
66

7+
/**
8+
* Maps secondary resource to primary resources.
9+
* @param <R>
10+
*/
711
@FunctionalInterface
812
public interface SecondaryToPrimaryMapper<R> {
13+
/**
14+
* @param resource - secondary
15+
* @return set of primary resource IDs
16+
*/
917
Set<ResourceID> toPrimaryResourceIDs(R resource);
1018
}

0 commit comments

Comments
 (0)