feat(keycloak): replace Keycloak realm setup with keycloak-config-cli

[viniciusdc]

Summary

Phase 1a of the declarative Keycloak configuration approach proposed in #154. Replaces the imperative shell script in realm-setup-job.yaml (chains of kcadm calls with || true for idempotency) with a declarative pipeline using keycloak-config-cli (kcc).

Structurally aligned with the design rewrite Chuck pushed to #154 on 2026-05-12: kcc input lives in an in-cluster Secret created by NIC at bootstrap from embedded defaults. The gitops repo never holds realm content — only the Job manifest.

Changes

• pkg/argocd/keycloak_defaults.yaml (new) — kcc realm input in native format. Parity with the previous shell setup: nebari realm settings, admin/user realm roles, admin user with both roles + argocd-admins membership, groups client scope + oidc-group-membership-mapper registered as defaultDefaultClientScopes (load-bearing for operator-managed app authorization via NebariApplication.allowedGroups), argocd OIDC client wired by ArgoCD SSO via Keycloak OIDC #234, argocd-admins/argocd-viewers groups.
• pkg/argocd/keycloak_defaults.go (new) — //go:embed of the realm YAML + RenderKeycloakDefaults(TemplateData) helper. {{ .Domain }} is substituted at deploy time (used in the argocd client's redirect URI).
• pkg/argocd/foundational.go — new createKeycloakImportSecret writes the rendered content to an in-cluster Secret named keycloak-config-import in the keycloak namespace. InstallFoundationalServices renders the defaults with cfg.Domain and calls the creator at bootstrap, immediately after the existing Keycloak secret creators.
• pkg/argocd/templates/manifests/keycloak/realm-setup-job.yaml (rewritten) — kcc Job pulling quay.io/adorsys/keycloak-config-cli:6.5.0-26.5.4. Volume mounts the new keycloak-config-import Secret (was the deleted gitops ConfigMap). Existing env-var passthrough for nebari-realm-admin-credentials and argocd-oidc-client-secret Secrets unchanged.
• pkg/argocd/templates/apps/keycloak.yaml — Keycloak chart image pinned to 26.5.4 to match the kcc build target.
• pkg/argocd/templates/manifests/keycloak/realm-config-cm.yaml (deleted) — the gitops ConfigMap is gone; realm content no longer leaks into git.

Scope (per #154 review)

Phase 1a only:

• No keycloak.yaml user-supplied file yet
• No NIC-side $(env:*) scanning yet
• No NIC-side merge engine (kcc owns reconciliation via remote-state)
• No backup/restore

User-supplied schema (Phase 1b) and the user-secrets pipeline (Phase 1c) follow in separate PRs.

Why an in-cluster Secret, not a gitops-repo ConfigMap

Original implementation used a ConfigMap rendered into the gitops repo with $(env:VAR) placeholders. The #154 design rewrite (commit 669458a on design/declarative-keycloak-config) explicitly rejected that pattern:

The gitops repo is treated as untrusted. Any pattern that requires "the file in the repo only contains placeholders, never literals" creates a footgun: one operator mistake leaks a secret into git history forever.

Refactored to the Secret model accordingly. Side benefit for Phase 1b: when a user supplies their own keycloak.yaml, the same Secret carrier accepts it without restructuring.

Operator coexistence

Three independent layers so operator-managed OAuth2 clients (created at runtime by NebariApplication CRDs) are never touched by kcc:

1. clients: in the kcc input only contains the argocd client. Operator-managed clients are not declared.
2. IMPORT_REMOTESTATE_ENABLED=true (default in kcc v6.x) — kcc tracks resources it created in a Keycloak realm attribute. Operator-created clients are invisible to kcc's delete logic.
3. IMPORT_MANAGED_CLIENT=no-delete — belt-and-braces. Even if the input ever changes shape, kcc still won't delete unlisted clients.

My earlier #154 review said "omit clients: entirely" — that was too strict. NIC owns the argocd client at bootstrap and has to declare it. The three-layer protection above is the practical shape.

Migration from the shell-based setup

On existing deployments where the previous realm-setup-job shell script ran successfully, kcc detects the existing realm/users/groups/scope/clients by their natural keys and updates them in place, adding them to remote state. First run is effectively a no-op for already-correct values. The pre-generated argocd OIDC client secret is the same value across both K8s Secrets and Keycloak's stored value (verified during testing), so the OIDC trust survives.

Testing

Verified end-to-end on a fresh kind cluster (kind-nebari-local):

• keycloak-config-import Secret created at bootstrap from the embedded defaults
• Gitops repo manifests/keycloak/ directory contains only realm-setup-job.yaml (no realm content)
• Job's volume references secretName: keycloak-config-import (not the deleted ConfigMap)
• kcc Job runs in ~4s after Keycloak is reachable, auto-cleaned by HookSucceeded
• Realm state: nebari realm with all 7 defaultDefaultClientScopes (basic, profile, email, roles, web-origins, acr, groups); argocd client default scopes same set; argocd-admins and argocd-viewers groups created; admin user member of argocd-admins
• Pre-generated OIDC secret byte-identical across all 3 sites (keycloak-namespace Secret, argocd-secret, Keycloak client config)

pkg/argocd/keycloak_defaults_test.go::TestRenderKeycloakDefaults (new) asserts each load-bearing field of the rendered kcc input + that no unresolved Go-template markers leak into the output. Catches regressions at unit-test time before they hit the cluster.

Findings the testing surfaced (and is now baked into source)

Five non-obvious-from-the-design things came up during real deploys:

• kcc image tags are tighter than the CI matrix advertises. kcc v6.5.0's CI lists Keycloak 26.5.5 but the actual built image only exists for 6.5.0-26.5.4. Both Keycloak chart image and kcc image pinned to 26.5.4.
• HookSucceeded-only delete policy traps stuck Jobs. A kcc Job that fails (bad image, network blip, parsing error) never succeeds, so HookSucceeded never fires its cleanup, and ArgoCD's hook-finalizer keeps the Job around indefinitely. Subsequent reconciles can't update the Job's pod template either (Jobs are immutable post-create). Added BeforeHookCreation to the policy so every fresh sync first deletes any existing Job.
• kcc applies list fields as full-replace, not merge. The previous shell script added groups to defaultDefaultClientScopes via a PUT on the specific scope path (additive). kcc's declarative defaultDefaultClientScopes: [groups] replaces the entire list. All built-in default scopes have to be listed alongside groups, both at realm level and on the argocd client.
• kcc runs string substitution over the whole file before parsing as YAML. That includes comments. A comment block that literally contained the $(env:VAR) placeholder shape tripped the substitutor with Cannot resolve variable 'env:VAR'. Comments now describe the syntax without writing the literal pattern, plus a warning so the next editor doesn't reintroduce it.
• ArgoCD's hook-finalizer can leave the application in a stuck operation state if a Job is force-deleted (via stripped finalizer). Surfaces only when iterating on the same cluster (force-deleting a stuck Job to push a corrected manifest). Not a deploy-flow bug — normal first deploys don't hit this. Unstick pattern is kubectl patch app <name> --type=merge -p '{"operation":null,"status":{"operationState":null}}' followed by a fresh sync.

Refs

• Design: Declarative Keycloak configuration #154 (design doc, supersedes the merge-engine + operator-prefix proposal there)
• ArgoCD SSO via Keycloak OIDC #234 (the ArgoCD OIDC SSO PR whose realm-setup state is preserved through this migration)

[viniciusdc]

Two small in-manifest follow-ups surfaced while talking through user/group lifecycle, both worth doing but holding off this PR to keep the diff focused on the shell-to-kcc swap. Filing here so they don't get lost.

1. userLabel: initial on the admin user's password

Currently:

credentials:
  - type: password
    value: $(env:REALM_ADMIN_PASSWORD)

kcc can't read existing password hashes from Keycloak, so it re-applies the password on every reconcile. If an operator manually rotates the admin password via the Keycloak UI, the next nic deploy reverts it.

Standard kcc mitigation is userLabel: initial, which tells kcc to set the password only at user creation, not on subsequent updates:

credentials:
  - type: password
    value: $(env:REALM_ADMIN_PASSWORD)
    userLabel: initial

After this change, the env-var value is the bootstrap password; any later rotation done in Keycloak survives nic deploy. Doc reference: docs/FEATURES.md "preserveExistingCredentials" pattern in adorsys/keycloak-config-cli.

2. IMPORT_MANAGED_GROUP=no-delete on the kcc Job

Same belt-and-braces shape as the existing IMPORT_MANAGED_CLIENT=no-delete. With IMPORT_REMOTESTATE_ENABLED=true (already set), kcc only deletes groups it itself created — so manually-created groups are already protected. But layering no-delete adds defense against future regressions in remote-state handling, with zero cost. Mirrors the rationale that's already in the IMPORT_MANAGED_CLIENT comment.

- name: IMPORT_MANAGED_GROUP
  value: "no-delete"

Out of scope for this comment but worth noting

The admin user's groups: [/argocd-admins] field in the realm config is also full-replace on kcc re-runs. If an operator manually adds the admin user to additional Keycloak groups via the UI, those memberships get reverted. That's by-design for a managed bootstrap user, but worth documenting in the eventual user-facing docs (#288 follow-up).

User accounts NOT in our users: list (IdP-synced, manually-created, self-registered) are safe — kcc has no IMPORT_MANAGED_USER policy and explicitly leaves the broader user set alone.

[viniciusdc]

One more note on kcc's capabilities relevant to future scope: identity providers are first-class in kcc's input schema. Setting up SSO (Google, GitHub, GitLab, generic OIDC/SAML, etc.) is the same shape as everything else — a top-level field on the realm with secret values via $(env:VAR):

realms:
  - realm: nebari
    # ...
    identityProviders:
      - alias: github
        providerId: github
        enabled: true
        trustEmail: true
        config:
          clientId: "$(env:GITHUB_CLIENT_ID)"
          clientSecret: "$(env:GITHUB_CLIENT_SECRET)"
          defaultScope: "user:email"

    identityProviderMappers:
      - identityProviderAlias: github
        name: email-mapper
        identityProviderMapper: github-user-attribute-mapper
        config:
          jsonField: email
          userAttribute: email

Not in scope for this PR (Phase 1a is defaults-only), but the shape is what users get in Phase 1b/1c when they put an idps: (or similar) block in their config.yaml keycloak: section. That maps directly through to the realm's identityProviders array in the rendered kcc YAML.

Two reasons this is worth surfacing now:

1. It validates the broader design — kcc is one tool covering realm config + IdPs + mappers + groups + scopes + clients. We don't need a separate code path for "SSO setup" vs "realm setup". The April 9 discussion about declarative IdPs as an enabler for Google-groups → Keycloak-groups → admin-role mapping lands naturally on top of this PR.

2. By default kcc applies IMPORT_MANAGED_IDENTITYPROVIDER=full, meaning IdPs not in the input get deleted on re-run. Combined with IMPORT_REMOTESTATE_ENABLED=true, this only affects IdPs kcc itself created — so manually-added IdPs in the Keycloak admin UI are safe today. But once users start declaring IdPs through config.yaml, removing one from config.yaml and re-running nic deploy will delete that IdP from Keycloak. That's by-design for a declarative tool, but worth being explicit about in the user-facing docs (Follow-up to #234: user-facing docs for ArgoCD SSO #288 follow-up).