feat(operator)!: support multiple controllers and selectors per entity (#1183)

## Summary

Add support for multiple controllers targeting the same entity type.

- Add `[LabelSelector]` and `[FieldSelector]` controller attributes.
- Add generator error `KOG001` for conflicting selector attributes.
- Add `WatchStrategy.PerController` and the opt-in `SharedPerEntity`
strategy.
- Add client-side selector matching through
`IClientSideEntitySelector<TEntity>`.
- Give each controller its own watcher, queue, and reconciliation
pipeline.
- Share parallelism limits and per-UID locking across controller
pipelines.
- Route `EntityQueue<TEntity>` requeues to the active controller
pipeline.
- Aggregate queue-depth metrics per entity type.
- Reject multiple controllers when using custom queue or leader-election
strategies.
- Update tests, migration guidance, and operator documentation.

Closes #909.

## Breaking changes

- Remove `IEntityQueueFactory` and `EntityQueueFactory`. Use
reconciliation results or the scoped `EntityQueue<TEntity>` delegate for
requeues.
- Add the required `OperatorSettings.WatchStrategy` property. Prefer
`OperatorSettingsBuilder`, which supplies the default `PerController`
strategy.
- Stop registering `IReconciler<TEntity>` and
`ITimedEntityQueue<TEntity>` in DI for the default in-memory pipeline.
- Resolving `EntityQueue<TEntity>` outside a reconciliation scope is
ambiguous and throws when multiple controller pipelines exist.
- Add a `cachePartition` parameter to custom `ResourceWatcher<TEntity>`
and `LeaderAwareResourceWatcher<TEntity>` constructors.
- Add `IEntityReconcileCoordinator<TEntity>` to custom
queue-background-service constructors.
- Add the concrete controller type to direct `Reconciler<TEntity>`
construction.
- Report controllers declaring both `[LabelSelector]` and
`[FieldSelector]` as generator error `KOG001`.

Author: kimpenhaus

docs/docs/operator/building-blocks/controllers.mdx

@@ -80,6 +80,8 @@ flowchart TB
 
 The `requeueAfter` path (dashed) is triggered when your controller returns a `ReconciliationResult` with a non-zero `RequeueAfter` value.
 
+KubeOps creates one such pipeline — watcher, queue, and background service — **per registered controller**. Multiple controllers for the same entity type (for example with different label selectors) therefore run fully independent pipelines that never suppress each other's events — see [Multiple Controllers per Entity](#multiple-controllers-per-entity). With the opt-in `WatchStrategy.SharedPerEntity`, a single shared watcher per entity type replaces the per-controller watchers and dispatches each event to all pipelines whose label selectors match — see [Watch Strategy](../configuration/watch-strategy) for the trade-offs.
+
 ## Reconciliation Loop
 
 The reconciliation loop is the core of your operator's functionality. It consists of two main methods:
@@ -452,6 +454,73 @@ When no field selector is needed, the built-in `DefaultEntityFieldSelector<TEnti
 | `AddControllerWithLabelSelector<TImpl, TEntity, TLabel>()` | Custom | Default (none) |
 | `AddControllerWithFieldSelector<TImpl, TEntity, TField>()` | Default (none) | Custom |
 
+### Declaring Selectors with Attributes
+
+Instead of calling the `AddControllerWith…Selector` methods manually, you can annotate the
+controller class. The source generator picks the attribute up and emits the matching registration
+into `RegisterControllers()` / `RegisterComponents()`:
+
+```csharp
+[LabelSelector(typeof(MyDemoEntityLabelSelector))]
+public class V1DemoEntityController : IEntityController<V1DemoEntity>
+{
+    // ...
+}
+
+// or
+[FieldSelector(typeof(MyDemoEntityFieldSelector))]
+public class V1DemoEntityController : IEntityController<V1DemoEntity>
+{
+    // ...
+}
+```
+
+A controller supports one selector kind at a time (label **or** field selector), matching the
+builder methods. Declaring both `[LabelSelector]` and `[FieldSelector]` on the same controller is a
+compile-time error (`KOG001`); split the work into two controllers instead.
+
+## Multiple Controllers per Entity
+
+The same entity type can be served by **multiple independent controllers**, each with its own label
+or field selector. Every registration creates a fully separate watch → queue → reconcile pipeline,
+so controllers with overlapping selectors both reconcile an entity that matches both:
+
+```csharp
+builder.Services.AddKubernetesOperator()
+    .AddControllerWithLabelSelector<ManagedController, V1DemoEntity, ManagedLabelSelector>()
+    .AddControllerWithLabelSelector<ConfigController, V1DemoEntity, ConfigLabelSelector>();
+```
+
+Notes:
+
+- Each pipeline maintains its own watch connection, queue, and deduplication cache, so the
+  controllers never suppress each other's events — even for the same object.
+- The same object is never reconciled by two of its controllers **concurrently**: a per-entity-type
+  coordinator serializes reconciliation per object UID (and enforces one shared
+  [parallelism budget](../configuration/parallel-reconciliation)). Finalizers are **entity-global** —
+  they are attached once and, under that same lock, are **not executed concurrently or duplicated
+  solely because multiple controllers match** the object. This is not general exactly-once execution
+  (e.g. a process crash between an external side effect and the finalizer removal can still re-run it),
+  so finalizers must remain idempotent.
+- Registering the exact same controller/selector combination twice throws at registration time
+  (it would only open a redundant watch stream).
+- The `EntityQueue<TEntity>` requeue delegate routes to the pipeline the current reconciliation
+  originated from. Inject it into the controller (or another scoped service); with multiple
+  controllers per entity it cannot be resolved from a singleton.
+- N controllers means N watch connections per entity type by default. If that becomes a concern
+  (many controllers, many objects, overlapping selectors), switch to the shared watch mode — see
+  [Watch Strategy](../configuration/watch-strategy).
+
+:::warning
+Multiple controllers per entity are supported only on the **default in-memory queue** with
+**non-custom leader election** (`LeaderElectionType.None` or `LeaderElectionType.Single`), where each
+pipeline owns its queue and reconciler. Under
+[`QueueStrategy.Custom`](../configuration/queue-strategy#custom) or
+[`LeaderElectionType.Custom`](../configuration/leader-election) all pipelines share a single
+user-owned queue and a single reconciler with no per-pipeline identity, so only **one controller per
+entity** is allowed — registering a second one throws an `InvalidOperationException`.
+:::
+
 ## Common Pitfalls
 
 1. **Infinite Loops**: Avoid creating reconciliation loops that trigger themselves

docs/docs/operator/configuration/caching.mdx

@@ -1,7 +1,7 @@
 ---
 title: Caching
 description: Caching - Memory and Distributed
-sidebar_position: 7
+sidebar_position: 8
 ---
 
 # Caching

docs/docs/operator/configuration/crd-installer.mdx

@@ -1,7 +1,7 @@
 ---
 title: CRD Installer
 description: Installing CRDs from the operator during development
-sidebar_position: 10
+sidebar_position: 11
 ---
 
 # CRD Installer

docs/docs/operator/configuration/external-triggers.mdx

@@ -1,7 +1,7 @@
 ---
 title: External Reconciliation Triggers
 description: Triggering reconciliation from outside the watch stream
-sidebar_position: 6
+sidebar_position: 7
 ---
 
 # External Reconciliation Triggers

docs/docs/operator/configuration/finalizer-lifecycle.mdx

@@ -1,7 +1,7 @@
 ---
 title: Finalizer Lifecycle
 description: Automatic attachment and detachment of finalizers
-sidebar_position: 5
+sidebar_position: 6
 ---
 
 # Finalizer Lifecycle

docs/docs/operator/configuration/generation-customization.mdx

@@ -1,7 +1,7 @@
 ---
 title: Customizing Resource Generation
 description: Replacing or extending the CRD, webhook, and event factories
-sidebar_position: 9
+sidebar_position: 10
 ---
 
 # Customizing Resource Generation

docs/docs/operator/configuration/index.mdx

@@ -26,6 +26,7 @@ Each `With*` method corresponds to one of the options described on the pages of
 | Page | What it controls |
 |---|---|
 | [Reconcile Strategy](./reconcile-strategy) | Which watch events trigger a reconciliation (`ByGeneration` vs. `ByResourceVersion`) |
+| [Watch Strategy](./watch-strategy) | How watch connections to the API server are created (`PerController` vs. `SharedPerEntity`) |
 | [Queue Strategy](./queue-strategy) | How reconciliation work is scheduled and dispatched (`InMemory` vs. `Custom`) |
 | [Parallel Reconciliation](./parallel-reconciliation) | Concurrency limits and conflict handling for simultaneous reconciliations |
 | [Leader Election](./leader-election) | How multiple operator instances coordinate (`None`, `Single`, `Custom`) |

docs/docs/operator/configuration/leader-election.mdx

@@ -1,7 +1,7 @@
 ---
 title: Leader Election
 description: Coordinating multiple operator instances
-sidebar_position: 4
+sidebar_position: 5
 ---
 
 # Leader Election

docs/docs/operator/configuration/parallel-reconciliation.mdx

@@ -1,7 +1,7 @@
 ---
 title: Parallel Reconciliation
 description: Concurrency limits and conflict handling
-sidebar_position: 3
+sidebar_position: 4
 ---
 
 # Parallel Reconciliation
@@ -14,18 +14,20 @@ the `WithParallelReconciliation` fluent method.
 
 ## Maximum Concurrency
 
-`MaxParallelReconciliations` sets the upper bound on concurrently running reconciliations across all
-entity types. The default is `Environment.ProcessorCount * 2`.
+`MaxParallelReconciliations` sets the upper bound on concurrently running reconciliations **per entity
+type**. The default is `Environment.ProcessorCount * 2`. When several controllers are registered for the
+same entity type, they **share** this single budget, so the limit is not multiplied by the controller
+count.
 
 ```csharp
 builder.Services
     .AddKubernetesOperator(settings => settings
         .WithParallelReconciliation(p => p.WithMaxParallelReconciliations(16)));
 ```
 
-A global `SemaphoreSlim` is acquired **before** an entry is dequeued, which means the queue acts as
-a natural buffer and back-pressure mechanism: producers (the watcher) can continue enqueuing while
-all reconciler slots are occupied.
+A `SemaphoreSlim` — one per entity type, shared by all of that entity's controllers — is acquired
+**before** an entry is dequeued, which means the queue acts as a natural buffer and back-pressure
+mechanism: producers (the watcher) can continue enqueuing while all reconciler slots are occupied.
 
 ## Conflict Strategy
 
@@ -48,9 +50,11 @@ builder.Services
 ```
 
 :::note
-The per-UID lock is independent of the global semaphore. Even with `MaxParallelReconciliations = 1`,
+The per-UID lock is independent of the parallelism semaphore. Even with `MaxParallelReconciliations = 1`,
 two events for *different* entities are never serialised by the UID lock; only events for the *same*
-UID are affected by `ConflictStrategy`.
+UID are affected by `ConflictStrategy`. Both the semaphore and the per-UID lock live in a coordinator
+shared by all controllers of an entity type, so the same object is never reconciled — or finalized —
+concurrently, even when two controllers match it.
 :::
 
 ## Best Practices

docs/docs/operator/configuration/queue-strategy.mdx

@@ -1,7 +1,7 @@
 ---
 title: Queue Strategy
 description: How reconciliation work is scheduled and dispatched
-sidebar_position: 2
+sidebar_position: 3
 ---
 
 # Queue Strategy
@@ -68,6 +68,14 @@ instead (or derive from `LeaderAwareEntityQueueBackgroundService<TEntity>`) so v
 consumer drives the leadership gate.
 :::
 
+:::warning[One controller per entity]
+`QueueStrategy.Custom` supports at most **one controller per entity type**. All pipelines share your
+single `ITimedEntityQueue<TEntity>`, whose entries carry no pipeline identity, and a single injected
+`IReconciler<TEntity>`, so a second controller for the same entity cannot be routed and is rejected at
+registration with an `InvalidOperationException`. Multiple controllers per entity require the default
+in-memory queue (see [Multiple Controllers per Entity](../building-blocks/controllers#multiple-controllers-per-entity)).
+:::
+
 Custom queues should follow the same de-duplication contract as the built-in implementation: keep at
 most one pending entry per namespace/name key and preserve the earliest scheduled execution time.