operator: user adoption and add credential sync for externally-managed secrets

[david-yu]

Closes #1354

Summary

The User controller has two related issues that prevent importing existing Redpanda users and block credential rotation via external secret management systems (like ESO + Azure Key Vault):

1. Existing users are never adopted — SyncResource only calls Create when !hasUser, so applying a User CR for a pre-existing Redpanda user leaves it permanently at status.managedUser=false
2. Managed users never get credential sync — even after the operator creates a user, subsequent reconciliation cycles (including the 5-minute drift correction) never re-read the password Secret or update Redpanda

This PR fixes both issues across three phases:

Phase 1: Fix user adoption

Changed: SyncResource branching logic in user_controller.go

The old code had a !hasUser && shouldManageUser gate that prevented adoption of existing users. The fix changes this to shouldManageUser && !hasManagedUser, which triggers the Create/upsert path regardless of whether the user already exists in Redpanda. This works because the underlying AlterUserSCRAMs with UpsertSCRAM is idempotent — it handles both create and update.

No opt-in needed — declaring spec.authentication is already the signal that the operator should manage the user.

Phase 2: Ongoing credential sync (opt-in)

New field: spec.authentication.syncCredentials (bool, default false)

When enabled, each reconciliation cycle re-reads the password from the referenced Secret and upserts credentials to Redpanda. This enables password rotation via external systems like ESO.

New method: users.Client.Update() — reads the current password from the Secret via Password.Fetch() (no generation or Secret creation) and upserts to Redpanda.

Phase 3: Immediate reconciliation on Secret changes

New index: Users are indexed by the password Secret they reference (spec.authentication.password.valueFrom.secretKeyRef.name)

New watch: A Watches(&corev1.Secret{}, ...) handler maps Secret changes to referencing User CRs and enqueues them. This means ESO-driven Secret updates trigger immediate reconciliation instead of waiting up to 5 minutes.

How to migrate existing users to operator management

Step-by-step: ESO + Azure Key Vault workflow

1. Ensure credentials exist in Azure Key Vault — the username/password pair must already be provisioned

2. Configure ESO to sync to a K8s Secret:

   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   metadata:
     name: my-user-password
   spec:
     refreshInterval: 1h
     secretStoreRef:
       name: azure-keyvault
       kind: ClusterSecretStore
     target:
       name: my-user-password
     data:
       - secretKey: password
         remoteRef:
           key: my-redpanda-user-password

3. Apply the User CR referencing the ESO-managed Secret:

   apiVersion: cluster.redpanda.com/v1alpha2
   kind: User
   metadata:
     name: my-user
   spec:
     cluster:
       clusterRef:
         name: my-cluster
     authentication:
       type: scram-sha-512
       password:
         valueFrom:
           secretKeyRef:
             name: my-user-password
             key: password
         noGenerate: true        # ESO manages the Secret, don't generate
       syncCredentials: true      # re-sync on each reconcile cycle
     authorization:
       acls:
         - type: allow
           resource:
             type: topic
             name: my-topic
           operations: [Read, Write, Describe]

4. Verify adoption:

   kubectl get users my-user
   # NAME      SYNCED   MANAGING USER   MANAGING ACLS
   # my-user   True     true            true

5. To rotate credentials: Update the secret in Azure Key Vault. ESO will sync the new value to the K8s Secret, the operator will detect the Secret change (via the new watch) and immediately reconcile, pushing the new password to Redpanda.

Step-by-step: Manual migration (no ESO)

1. Create a K8s Secret with the current password:

   kubectl create secret generic my-user-password \
     --from-literal=password='current-password'

2. Apply the User CR (same as step 3 above, but syncCredentials is optional since you're managing the Secret manually)

3. To rotate: Update the Secret, then either wait for the 5-minute periodic reconcile or trigger a manual reconcile by annotating the User CR

Key flags

Field	Purpose
password.noGenerate: true	Prevents operator from generating/overwriting the Secret — use when an external system (ESO) manages it
authentication.syncCredentials: true	Re-reads password from Secret on every reconcile and upserts to Redpanda — enables rotation

Files changed

• operator/api/redpanda/v1alpha2/user_types.go — add SyncCredentials field, ShouldSyncCredentials(), GetPasswordSecretName() helpers
• operator/internal/controller/redpanda/user_controller.go — fix adoption logic, add credential sync branch, add Secret index + watch
• operator/pkg/client/users/client.go — add Update() method
• operator/internal/controller/redpanda/user_controller_test.go — add TestUserAdoptExisting and TestUserCredentialSync tests

TODO

• Run make generate to regenerate apply configurations for the new SyncCredentials field
• Integration test with real ESO setup
• Consider adding a status condition or event when credential sync occurs

Test plan

• TestUserAdoptExisting — pre-creates a user via Kafka admin API, then applies a User CR and verifies managedUser=true and the new password works
• TestUserCredentialSync — creates a user with syncCredentials: true, rotates the password Secret, reconciles, and verifies the new password authenticates
• Existing TestUserReconcile table tests continue to pass (adoption case added to table)
• Manual verification with ESO + Azure Key Vault in a staging environment

🤖 Generated with Claude Code

[github-actions]

This PR is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days.