Reject storage version migrator in namespace scope

The StorageVersionMigrator watches cluster-scoped CustomResourceDefinitions
and re-stores custom resources across all namespaces, so it requires
cluster-scoped RBAC and a cluster-scoped manager cache. A namespace-scoped
operator gets only per-namespace RoleBindings and a namespace-restricted
cache, so the controller cannot sync its CRD informer and wedges the
manager — which is what broke the multi-tenancy chainsaw E2E once the
feature was defaulted on.

Add a Helm validation that fails the render when storageVersionMigrator is
true under operator.rbac.scope=namespace, with an actionable message. Fail
loudly rather than silently dropping the feature: a silent drop would let a
namespace-scoped admin believe storedVersions are being kept clean when they
are not, leading to the skip-a-version upgrade trap later.

- Add validateStorageVersionMigrator helper; invoke it from deployment.yaml
- Opt the namespace-scoped multi-tenancy chainsaw setup out of the migrator
- Pin the rejection and the supported namespace opt-out in helm unittests;
  disable the migrator in the namespace-scope suite's shared values
- Document the cluster-scope requirement and the standalone-migrator
  alternative for namespace-scoped clusters

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: ChrisJBurns

deploy/charts/operator/README.md

@@ -61,7 +61,7 @@ The command removes all the Kubernetes components associated with the chart and
 | operator.defaultRedis.existingSecretKey | string | `""` | existingSecretKey is the key within existingSecret that holds the password. Empty means use global.redis.existingSecretKey or fall back to "redis-password". |
 | operator.env | list | `[]` | Environment variables to set in the operator container. Supported toolhive-specific variables include: - TOOLHIVE_SKIP_UPDATE_CHECK: set to "true" to disable the operator's   periodic update check against the ToolHive update API. Also disables   the usage-metrics collection that is gated on the same check. |
 | operator.features.experimental | bool | `false` | Enable experimental features |
-| operator.features.storageVersionMigrator | bool | `true` | Enable the StorageVersionMigrator controller, which auto-cleans status.storedVersions on opted-in toolhive.stacklok.dev CRDs so a future release can drop deprecated versions (e.g. v1alpha1) without orphaning etcd objects in the cluster. Enabled by default; set to false to opt out and handle storage-version cleanup yourself. Sets TOOLHIVE_ENABLE_STORAGE_VERSION_MIGRATOR in the operator deployment. |
+| operator.features.storageVersionMigrator | bool | `true` | Enable the StorageVersionMigrator controller, which auto-cleans status.storedVersions on opted-in toolhive.stacklok.dev CRDs so a future release can drop deprecated versions (e.g. v1alpha1) without orphaning etcd objects in the cluster. Enabled by default; set to false to opt out and handle storage-version cleanup yourself. Sets TOOLHIVE_ENABLE_STORAGE_VERSION_MIGRATOR in the operator deployment. Requires `operator.rbac.scope=cluster` — the controller watches cluster-scoped CRDs and re-stores resources across all namespaces, so the chart rejects this being true when scope is namespace. |
 | operator.gc | object | `{"gogc":75,"gomemlimit":"110MiB"}` | Go memory limits and garbage collection percentage for the operator container |
 | operator.gc.gogc | int | `75` | Go garbage collection percentage for the operator container |
 | operator.gc.gomemlimit | string | `"110MiB"` | Go memory limits for the operator container |

deploy/charts/operator/templates/_helpers.tpl

@@ -68,4 +68,24 @@ Common labels for the toolhive resources
 {{- define "toolhive.labels" -}}
 app: toolhive
 app.kubernetes.io/name: toolhive
-{{- end }}
+{{- end }}
+
+{{/*
+Validate feature-flag / RBAC-scope combinations and fail the render early with
+an actionable message rather than deploying a wedged operator.
+
+The StorageVersionMigrator controller watches cluster-scoped
+CustomResourceDefinition objects and re-stores custom resources across every
+namespace, so it requires cluster-scoped RBAC and a cluster-scoped manager
+cache. A namespace-scoped operator (operator.rbac.scope=namespace) gets only
+per-namespace RoleBindings and a namespace-restricted cache, so the controller
+cannot sync its CRD informer and would prevent the manager from starting. We
+fail loudly here instead of silently dropping the feature, because a silent
+drop would let a namespace-scoped admin believe storedVersions are being kept
+clean when they are not.
+*/}}
+{{- define "toolhive-operator.validateStorageVersionMigrator" -}}
+{{- if and .Values.operator.features.storageVersionMigrator (ne .Values.operator.rbac.scope "cluster") -}}
+{{- fail "operator.features.storageVersionMigrator requires operator.rbac.scope=cluster: the StorageVersionMigrator controller watches cluster-scoped CustomResourceDefinitions and re-stores resources across all namespaces, which a namespace-scoped operator cannot do. Set operator.features.storageVersionMigrator=false for namespace-scoped installs." -}}
+{{- end -}}
+{{- end -}}

deploy/charts/operator/templates/deployment.yaml

@@ -1,3 +1,4 @@
+{{- include "toolhive-operator.validateStorageVersionMigrator" . }}
 apiVersion: apps/v1
 kind: Deployment
 metadata:

deploy/charts/operator/tests/feature_flags_test.yaml

@@ -3,6 +3,8 @@ suite: opt-in feature flags
 # vars. A dropped quote or a renamed var silently flips the feature, so pin both
 # the enabled and disabled states. storageVersionMigrator defaults to true, so
 # the disabled state is the one the baseline suite can't cover — pin it here.
+# The migrator also requires cluster-scoped RBAC, so the chart rejects it in
+# namespace scope; pin that validation too.
 templates:
   - deployment.yaml
 tests:
@@ -27,3 +29,30 @@ tests:
       - contains:
           path: 'spec.template.spec.containers[0].env'
           content: { name: TOOLHIVE_ENABLE_STORAGE_VERSION_MIGRATOR, value: "false" }
+
+  # The migrator watches cluster-scoped CRDs, so it must never be enabled in a
+  # namespace-scoped install. The chart fails the render rather than deploying a
+  # wedged operator. Pin both the rejection and the supported opt-out.
+  - it: rejects the storage version migrator in namespace scope
+    template: deployment.yaml
+    set:
+      operator.rbac.scope: namespace
+      operator.rbac.allowedNamespaces: [toolhive-system]
+      operator.features.storageVersionMigrator: true
+    asserts:
+      - failedTemplate:
+          errorPattern: "operator.features.storageVersionMigrator requires operator.rbac.scope=cluster"
+
+  - it: allows namespace scope when the storage version migrator is disabled
+    template: deployment.yaml
+    set:
+      operator.rbac.scope: namespace
+      operator.rbac.allowedNamespaces: [toolhive-system]
+      operator.features.storageVersionMigrator: false
+    asserts:
+      - contains:
+          path: 'spec.template.spec.containers[0].env'
+          content: { name: TOOLHIVE_ENABLE_STORAGE_VERSION_MIGRATOR, value: "false" }
+      - contains:
+          path: 'spec.template.spec.containers[0].env'
+          content: { name: WATCH_NAMESPACE, value: "toolhive-system" }

deploy/charts/operator/tests/namespace_scope_test.yaml

@@ -6,6 +6,9 @@ suite: namespace-scoped install
 set:
   operator.rbac.scope: namespace
   operator.rbac.allowedNamespaces: [team-a, team-b]
+  # storageVersionMigrator requires cluster scope; the chart rejects it under
+  # namespace scope (see feature_flags_test.yaml), so disable it here.
+  operator.features.storageVersionMigrator: false
 templates:
   - deployment.yaml
   - clusterrole/rolebinding.yaml

deploy/charts/operator/values.yaml

@@ -14,6 +14,9 @@ operator:
     # orphaning etcd objects in the cluster. Enabled by default; set to
     # false to opt out and handle storage-version cleanup yourself. Sets
     # TOOLHIVE_ENABLE_STORAGE_VERSION_MIGRATOR in the operator deployment.
+    # Requires `operator.rbac.scope=cluster` — the controller watches
+    # cluster-scoped CRDs and re-stores resources across all namespaces, so
+    # the chart rejects this being true when scope is namespace.
     storageVersionMigrator: true
   # -- Number of replicas for the operator deployment
   replicaCount: 1

docs/operator/storage-version-migration.md

@@ -83,6 +83,19 @@ operator:
 
 Once enabled, the controller is dormant on CRDs whose `storedVersions` already equals `[<currentStorageVersion>]` — most of the time, most CRDs. It only does meaningful work when a CRD's stored-versions list is dirty (typically right after a graduation release).
 
+### Requires cluster-scoped RBAC
+
+The migrator only works when the operator runs cluster-scoped (`operator.rbac.scope=cluster`, the default). It watches cluster-scoped `CustomResourceDefinition` objects and re-stores custom resources across **all** namespaces — neither is possible for a namespace-scoped operator, which gets only per-namespace `RoleBindings` and a namespace-restricted manager cache. To prevent a silently wedged operator, the chart **fails the render** if `storageVersionMigrator: true` is combined with `operator.rbac.scope=namespace`:
+
+```
+operator.features.storageVersionMigrator requires operator.rbac.scope=cluster: the
+StorageVersionMigrator controller watches cluster-scoped CustomResourceDefinitions and
+re-stores resources across all namespaces, which a namespace-scoped operator cannot do.
+Set operator.features.storageVersionMigrator=false for namespace-scoped installs.
+```
+
+Namespace-scoped installs must set `storageVersionMigrator: false` and handle storage-version cleanup by other means — e.g. running the standalone [`kube-storage-version-migrator`](https://github.com/kubernetes-sigs/kube-storage-version-migrator), which is a cluster-scoped component with its own identity, before any version-removal release.
+
 ## Per-CRD emergency escape hatch
 
 Removing the label on a live cluster excludes that single CRD from migration immediately:

test/e2e/chainsaw/operator/multi-tenancy/setup/chainsaw-test.yaml

@@ -42,6 +42,10 @@ spec:
         - operator.rbac.scope=namespace
         - --set
         - operator.rbac.allowedNamespaces={toolhive-system,test-namespace,toolhive-test-ns-1,toolhive-test-ns-2}
+        # The StorageVersionMigrator watches cluster-scoped CRDs, so it is
+        # unsupported (and chart-rejected) in namespace scope. Opt out here.
+        - --set
+        - operator.features.storageVersionMigrator=false
     - assert:
         file: assert-operator-ready.yaml
     - assert: