Change kubeconfigs from kcp-admin credentials to service specific credentials

[aaronschweig]

Current State
Currently, the kcp-admin kubeconfigs is used in all platform mesh services. We strive to have separate kubeconfigs for the service that give contain service specific kubeconfigs only with the rbacs needed, rather than full admin access

Target

We need to come to a state where using the admin-kubeconfig is not the default but rather an explicit opt-in to properly scope down the permissions our operators have across the kcp instance. For that the platform-mesh operator needs to be aware what permissions are needed by which operator.

We already have that information - for any future provider - as part of the APIExport that these provider bring with them. The platform-mesh operator needs to be able to generate a ServiceAccount token based kubeconfig for a given provider, with RBACs attached to it, that give:

• Full Access to the resources listed in the latestResourcheSchemas section
• the access specified in the permissionClaims with the corresponding scopes, verbs etc.

Todos

• Update to generate service-specific kubeconfigs
• Update to generate specific RBACs for a generated service account based on the above criteria
• Provide the capability to explicitly opt-in for an admin kubeconfig if an operator really needs it
• Test the generated configurations work and do not cause logic to break

[philtk79]

SCOPED_KUBECONFIG_OPERATOR_STATUS.md
So I added the findings of my testsing with scoped kubeconfig. In order to make operators work, they need more permissions than could be derived from APIExport. The document contains a list.

My proposal, add flags to the providerconnection that enable those, at least to make the http bin provider run. For the internal operators that are required for platform mesh, one could think of having a fixed ruleset implemented for them. In this way we could make sure future external providers could not gain permissions that are only intended for internal use

[simontesar]

We are still in the middle of making this work with all our services by going through them one by one.

A common issue that we have is that most our services still build KCP clients "manually" by modifying the URL to given KCP kubeconfig. This was mostly caused by the apiexport-provider not supporting getting cluster clients by cluster name but only by ID. This has been resolved by the pathaware-provider wrapping the apiexport provider.

A security-operator-specific issue that we are encountering is that the generated kubeconfigs based on an APIExport make no sense for the initializer and terminator of the security-operator. Both run in their own pods, with their own manager and kubeconfigs using their own providers that don't "connect" to an APIExport but the virtual initializing/terminating workspace of the security WorkspaceType. We need to verify that what we're doing in the initializer and terminator is possible without an admin-config and if so, we need to decide based on what we want to generate scoped kubeconfigs for them. This is a feature/extension of the scoped-kubeconfigs-feature that needs to be implemented in the platform-mesh-operator that generates these kubeconfigs primarily.

These are some related items:

• account-operator: Retrieve clients via manager: feat: retrieve orgs client via manager account-operator#206
• security-operator: Retrieve clients via manager: Prepare for scoped kubeconfigs security-operator#481 feat: prepare operator, system and generator for scoped kubeconfigs security-operator#494
• security-operator terminator/initializer: Scoped kubeconfigs for initializer and terminator platform-mesh#44

We also depend on upstream KCP fixes/enhancements for this to work properly: kcp-dev/kcp#4038

[philtk79]

The findings of @OlegErshov are documented in this kcp bug report kcp-dev/kcp#4038
Future enhancements of kcp's rbac is also described here
https://github.com/kcp-dev/enhancements/blob/main/keps/core/0003-workspace-lifecycle-rbac.md

[OlegErshov]

Also there was some workaround to make iam service work with scoped kubeconfigs: platform-mesh/iam-service#450

In account operator there also was a problem that workspace types in :root:orgs extend workspace types from :root workspace. With scoped kubeconfigs we have permissions for working in root:orgs but not in root. Therefore we've decided to move some WorkspaceTypes into :root:orgs workspace. Here's a draft pr which does it #576.
But even with this account operator depends on the upstream changes from kcp which Simon has mentioned

As I understand the current state is the following (feel free to change this list, if something has changed):

1. security-operator - everything what Simon has described
2. account-operator - some works was done, but next progress depends on the upstream kcp issue
3. graphql-gateway - needs more clarification as gateway needs a lot of permissions and not clear if it will work with scoped kubeconfig which is created based on APIExport
4. rebac-authz-webhook - OK
5. iam-service - OK