What would you like to be added:
In a discussion with @sttts and @embik, we concluded that this project might be able to replace the sharder's mutating webhook with a new controller in the sharder that watches (metadata-only) unassigned (unlabeled) objects. Instead of relying on the webhook for notifications about unassigned objects (without running an informer), we would again use a watch – but only for unlabeled objects.
The sharder would dynamically start/stop controllers/watches based on the ControllerRing spec (ref kubernetes-sigs/controller-runtime#3015 (comment)).
We need to compare the performance and scalability of this new approach to the existing design.
Why is this needed:
Eliminating the sharder's webhook would allow sharding controllers in scenarios where network connectivity from the API server to an external webhook server is not possible/desired. The sharder would only need to connect to the API server, but not the other way round.
This would also allow separating the "control cluster" (where ControllerRing and shard leases live) from the "workload cluster" (where the sharded API objects live).
Furthermore, we could run the sharder and the sharded controller for multiple clusters, i.e., in kubernetes-sigs/multicluster-runtime#58.
Originally, I switched from a watch-based approach to a webhook-based approach for the following reasons (also see https://github.com/timebertt/kubernetes-controller-sharding/blob/main/docs/design.md):
- Watching/caching all sharded objects in a (central) sharder causes overhead (CPU, memory, network traffic) that grows with object count and churn. This should be fine, though, when restricting the informer to unlabeled objects, as the objects disappear from the watch stream/cache after the label assignment.
- Centralizing the sharding implementation in a separate component allows reusing the mechanism with little implementation effort in the sharded controller. Also, it's independent of the controller framework and programming language. This aspect wouldn't change with the proposal in this issue.
What would you like to be added:
In a discussion with @sttts and @embik, we concluded that this project might be able to replace the sharder's mutating webhook with a new controller in the sharder that watches (metadata-only) unassigned (unlabeled) objects. Instead of relying on the webhook for notifications about unassigned objects (without running an informer), we would again use a watch – but only for unlabeled objects.
The sharder would dynamically start/stop controllers/watches based on the
ControllerRingspec (ref kubernetes-sigs/controller-runtime#3015 (comment)).We need to compare the performance and scalability of this new approach to the existing design.
Why is this needed:
Eliminating the sharder's webhook would allow sharding controllers in scenarios where network connectivity from the API server to an external webhook server is not possible/desired. The sharder would only need to connect to the API server, but not the other way round.
This would also allow separating the "control cluster" (where
ControllerRingand shard leases live) from the "workload cluster" (where the sharded API objects live).Furthermore, we could run the sharder and the sharded controller for multiple clusters, i.e., in kubernetes-sigs/multicluster-runtime#58.
Originally, I switched from a watch-based approach to a webhook-based approach for the following reasons (also see https://github.com/timebertt/kubernetes-controller-sharding/blob/main/docs/design.md):