diff --git a/content/en/docs/operating/best-practices/_index.md b/content/en/docs/operating/best-practices/_index.md index bdc89a1..537442d 100644 --- a/content/en/docs/operating/best-practices/_index.md +++ b/content/en/docs/operating/best-practices/_index.md @@ -3,3 +3,34 @@ title: Best Practices weight: 2 description: Best Practices when running Capsule in production --- + + +This is general advice you should consider before making Kubernetes Distribution consideration. They are partly relevant for Multi-Tenancy with Capsule. + +### Authentication + +User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider). +The rationale is that other central platform components — such as ArgoCD, Grafana, Headlamp, or Harbor — should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions. + +[Capsule relies on native Kubernetes RBAC](/docs/operating/authentication/), so it's important to consider how the Kubernetes API handles user authentication. + +### OCI Pull-Cache + +By default, Kubernetes clusters pull images directly from upstream registries like `docker.io`, `quay.io`, `ghcr.io`, or `gcr.io`. In production environments, this can lead to issues — especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled). + +To ensure availability, performance, and control over container images, it's essential to provide an on-premise OCI mirror. +This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., `docker.io`). +This way, all nodes automatically benefit from caching without requiring developers to change image URLs. + +### Secrets Management + +In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical. +Instead, a centralized secrets management system should be established — such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project [OpenBao](https://openbao.org/) (formerly the Vault community fork). + +To integrate these external secret stores with Kubernetes, the [External Secrets Operator (ESO)](https://external-secrets.io/latest/) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing. + +If no external secret store is available, there should at least be a secure way to store sensitive data in Git. +In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case. + +[👉 Demonstration](https://killercoda.com/peakscale/course/playgrounds/sops-secrets) + diff --git a/content/en/docs/operating/best-practices/general-advice.md b/content/en/docs/operating/best-practices/general-advice.md deleted file mode 100644 index d50d9b8..0000000 --- a/content/en/docs/operating/best-practices/general-advice.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: General Advice -weight: 2 -description: This is general advice you should consider before making Kubernetes Distribution consideration ---- - -This is general advice you should consider before making Kubernetes Distribution consideration. They are partly relevant for Multi-Tenancy with Capsule. - -### Authentication - -User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider). -The rationale is that other central platform components — such as ArgoCD, Grafana, Headlamp, or Harbor — should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions. - -[Capsule relies on native Kubernetes RBAC](/docs/operating/authentication/), so it's important to consider how the Kubernetes API handles user authentication. - -### OCI Pull-Cache - -By default, Kubernetes clusters pull images directly from upstream registries like `docker.io`, `quay.io`, `ghcr.io`, or `gcr.io`. In production environments, this can lead to issues — especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled). - -To ensure availability, performance, and control over container images, it's essential to provide an on-premise OCI mirror. -This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., `docker.io`). -This way, all nodes automatically benefit from caching without requiring developers to change image URLs. - -### Secrets Management - -In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical. -Instead, a centralized secrets management system should be established — such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project [OpenBao](https://openbao.org/) (formerly the Vault community fork). - -To integrate these external secret stores with Kubernetes, the [External Secrets Operator (ESO)](https://external-secrets.io/latest/) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing. - -If no external secret store is available, there should at least be a secure way to store sensitive data in Git. -In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case. - -[👉 Demonstration](https://killercoda.com/peakscale/course/playgrounds/sops-secrets) diff --git a/content/en/docs/operating/setup/admission-policies.md b/content/en/docs/operating/setup/admission-policies.md new file mode 100644 index 0000000..6252cef --- /dev/null +++ b/content/en/docs/operating/setup/admission-policies.md @@ -0,0 +1,497 @@ +--- +title: Admission Policies +weight: 2 +description: Recommended Admission Policies to enforce best practices in multi-tenant environments. +--- + +As Capsule we try to provide a secure multi-tenant environment out of the box, there are however some additional Admission Policies you should consider to enforce best practices in your cluster. Since Capsule only covers the core multi-tenancy features, such as Namespaces, Resource Quotas, Network Policies, and Container Registries, Classes, you should consider using an additional Admission Controller to enforce best practices on workloads and other resources. + +## Mutate User Namespace + +You should enforce the usage of [User Namespaces](/docs/operating/best-practices/workloads/#user-namespaces). Most Helm-Charts currently don't support this out of the box. With Kyverno you can enforce this on Pod level. + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="Kyverno" >}} +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: tenants-user-namespace +spec: + rules: + - name: enforce-no-host-users + match: + any: + - resources: + kinds: + - Pod + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + # selector: + # matchExpressions: + # - key: company.com/allow-host-users + # operator: NotIn + # values: + # - "true" + preconditions: + all: + - key: "{{request.operation || 'BACKGROUND'}}" + operator: AnyIn + value: + - CREATE + - UPDATE + skipBackgroundRequests: true + mutate: + patchStrategicMerge: + spec: + hostUsers: false{{< /tab >}} +{{% /tabpane %}} + +Note that users still can override this setting by adding the label `company.com/allow-host-users=true` to their namespace. You can change the label to your needs. This is because NFS does not support user namespaces and you might want to allow this for specific tenants. + +## Disallow Daemonsets + +Tenant's should not be allowed to create Daemonsets, unless they have dedicated nodes. + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="VAP" >}} +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: deny-daemonset-create +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: ["apps"] + apiVersions: ["v1"] + resources: ["daemonsets"] + operations: ["CREATE"] + scope: "Namespaced" + validations: + - expression: "false" + message: "Creating DaemonSets is not allowed in this cluster." +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: deny-daemonset-create-binding +spec: + policyName: deny-daemonset-create + validationActions: ["Deny"] + matchResources: + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists{{< /tab >}} + {{< tab header="Kyverno" >}} +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: tenant-workload-restrictions +spec: + validationFailureAction: Enforce + rules: + - name: block-daemonset-create + match: + any: + - resources: + kinds: + - DaemonSet + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + preconditions: + all: + - key: "{{ request.operation || 'BACKGROUND' }}" + operator: Equals + value: CREATE + validate: + message: "Creating DaemonSets is not allowed in this cluster." + deny: + conditions: + any: + - key: "true" + operator: Equals + value: "true"{{< /tab >}} +{{% /tabpane %}} + +## Disallow Scheduling on Control Planes + +If a Pods are not scoped to specific nodes, they could be scheduled on control plane nodes. You should disallow this by enforcing that Pods do not use tolerations for control plane nodes. + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="Capsule" >}} +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + owners: + - name: alice + kind: User + nodeSelector: + customer: public-services{{< /tab >}} + {{< tab header="VAP" >}} +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: disallow-controlplane-scheduling +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: [""] + apiVersions: ["v1"] + resources: ["pods"] + operations: ["CREATE","UPDATE"] + scope: "Namespaced" + validations: + - expression: > + // deny if any toleration targets control-plane taints + !has(object.spec.tolerations) || + !exists(object.spec.tolerations, t, + t.key in ['node-role.kubernetes.io/master','node-role.kubernetes.io/control-plane'] + ) + message: "Pods may not use tolerations which schedule on control-plane nodes." +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: disallow-controlplane-scheduling +spec: + policyName: disallow-controlplane-scheduling + validationActions: ["Deny"] + matchResources: + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists{{< /tab >}} + {{< tab header="Kyverno" >}} +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: disallow-controlplane-scheduling +spec: + validationFailureAction: Enforce + rules: + - name: restrict-controlplane-scheduling-master + match: + resources: + kinds: + - Pod + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + validate: + message: Pods may not use tolerations which schedule on control plane nodes. + pattern: + spec: + =(tolerations): + - key: "!node-role.kubernetes.io/master" + + - name: restrict-controlplane-scheduling-control-plane + match: + resources: + kinds: + - Pod + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + validate: + message: Pods may not use tolerations which schedule on control plane nodes. + pattern: + spec: + =(tolerations): + - key: "!node-role.kubernetes.io/control-plane"{{< /tab >}} +{{% /tabpane %}} + +## Enforce EmptDir Requests/Limits + +By Defaults `emptyDir` Volumes do not have any limits. This could lead to a situation, where a tenant fills up the node disk. To avoid this, you can enforce limits on `emptyDir` volumes. You may also consider restricting the usage of `emptyDir` with the `medium: Memory` option, as this could lead to memory exhaustion on the node. + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="Kyverno" >}} +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: tenant-workload-restrictions +spec: + rules: + - name: default-emptydir-sizelimit + match: + any: + - resources: + kinds: + - Pod + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + mutate: + foreach: + - list: "request.object.spec.volumes[]" + preconditions: + all: + - key: "{{element.keys(@)}}" + operator: AnyIn + value: emptyDir + - key: "{{element.emptyDir.sizeLimit || ''}}" + operator: Equals + value: '' + patchesJson6902: |- + - path: "/spec/volumes/{{elementIndex}}/emptyDir/sizeLimit" + op: add + value: 250Mi{{< /tab >}} +{{% /tabpane %}} + +## Block Ephemeral Containers + +Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the `kubectl debug` functionality and attach a temporary container to an existing Pod. This may potentially be used to gain access to unauthorized information executing inside one or more containers in that Pod. This policy blocks the use of ephemeral containers. + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="VAP" >}} +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: block-ephemeral-containers +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + # 1) Regular Pods (ensure spec doesn't carry ephemeralContainers) + - apiGroups: [""] + apiVersions: ["v1"] + resources: ["pods"] + operations: ["CREATE","UPDATE"] + scope: "Namespaced" + # 2) Subresource used by `kubectl debug` to inject ephemeral containers + - apiGroups: [""] + apiVersions: ["v1"] + resources: ["pods/ephemeralcontainers"] + operations: ["UPDATE","CREATE"] # UPDATE is typical, CREATE included for future-proofing + scope: "Namespaced" + validations: + # Deny any request that targets the pods/ephemeralcontainers subresource + - expression: request.subResource != "ephemeralcontainers" + message: "Ephemeral (debug) containers are not permitted (subresource)." + # For direct Pod create/update, allow only if the field is absent or empty + - expression: > + !has(object.spec.ephemeralContainers) || + size(object.spec.ephemeralContainers) == 0 + message: "Ephemeral (debug) containers are not permitted in Pod specs." +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: block-ephemeral-containers-binding +spec: + policyName: block-ephemeral-containers + validationActions: ["Deny"] + matchResources: + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists{{< /tab >}} + {{< tab header="Kyverno" >}} +# Source: https://kyverno.io/policies/other/block-ephemeral-containers/block-ephemeral-containers/ +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: block-ephemeral-containers + annotations: + policies.kyverno.io/title: Block Ephemeral Containers + policies.kyverno.io/category: Other + policies.kyverno.io/severity: medium + kyverno.io/kyverno-version: 1.6.0 + policies.kyverno.io/minversion: 1.6.0 + kyverno.io/kubernetes-version: "1.23" + policies.kyverno.io/subject: Pod + policies.kyverno.io/description: >- + Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the + `kubectl debug` functionality and attach a temporary container to an existing Pod. + This may potentially be used to gain access to unauthorized information executing inside + one or more containers in that Pod. This policy blocks the use of ephemeral containers. +spec: + validationFailureAction: Enforce + background: true + rules: + - name: block-ephemeral-containers + match: + any: + - resources: + kinds: + - Pod + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + validate: + message: "Ephemeral (debug) containers are not permitted." + pattern: + spec: + X(ephemeralContainers): "null"{{< /tab >}} +{{% /tabpane %}} + +## Image Registry + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="Capsule" >}} +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + containerRegistries: + allowed: + - "docker.io" + - "public.ecr.aws" + - "quay.io" + - "mcr.microsoft.com"{{< /tab >}} + {{< tab header="Kyverno" >}} +# Or with a Kyverno Policy. Here the default registry is `docker.io`, when no registry prefix is specified: +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: restrict-image-registries + annotations: + policies.kyverno.io/title: Restrict Image Registries + policies.kyverno.io/category: Best Practices, EKS Best Practices + policies.kyverno.io/severity: medium +spec: + validationFailureAction: Audit + background: true + rules: + - name: validate-registries + match: + any: + - resources: + kinds: + - Pod + validate: + message: "Using unknown image registry." + foreach: + - list: "request.object.spec.initContainers" + deny: + conditions: + - key: '{{images.initContainers."{{element.name}}".registry }}' + operator: NotIn + value: + - "docker.io" + - "public.ecr.aws" + - "quay.io" + - "mcr.microsoft.com" + + - list: "request.object.spec.ephemeralContainers" + deny: + conditions: + - key: '{{images.ephemeralContainers."{{element.name}}".registry }}' + operator: NotIn + value: + - "docker.io" + - "public.ecr.aws" + - "quay.io" + - "mcr.microsoft.com" + + - list: "request.object.spec.containers" + deny: + conditions: + - key: '{{images.containers."{{element.name}}".registry }}' + operator: NotIn + value: + - "docker.io" + - "public.ecr.aws" + - "quay.io" + - "mcr.microsoft.com"{{< /tab >}} +{{% /tabpane %}} + +## Image PullPolicy + +[Read More](/docs/operating/best-practices/images/). + + +{{% tabpane lang="yaml" %}} + {{% tab header="**Engines**:" disabled=true /%}} + {{< tab header="Capsule" >}} +--- +apiVersion: capsule.clastix.io/v1beta2 +kind: Tenant +metadata: + name: solar +spec: + owners: + - name: alice + kind: User + imagePullPolicies: + - Always{{< /tab >}} + {{< tab header="Kyverno" >}} +--- +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: always-pull-images + annotations: + policies.kyverno.io/title: Always Pull Images + policies.kyverno.io/category: Sample + policies.kyverno.io/severity: medium + policies.kyverno.io/subject: Pod + policies.kyverno.io/minversion: 1.6.0 + policies.kyverno.io/description: >- + By default, images that have already been pulled can be accessed by other + Pods without re-pulling them if the name and tag are known. In multi-tenant scenarios, + this may be undesirable. This policy mutates all incoming Pods to set their + imagePullPolicy to Always. An alternative to the Kubernetes admission controller + AlwaysPullImages. +spec: + rules: + - name: always-pull-images + match: + any: + - resources: + kinds: + - Pod + namespaceSelector: + matchExpressions: + - key: capsule.clastix.io/tenant + operator: Exists + mutate: + patchStrategicMerge: + spec: + initContainers: + - (name): "?*" + imagePullPolicy: Always + containers: + - (name): "?*" + imagePullPolicy: Always + ephemeralContainers: + - (name): "?*" + imagePullPolicy: Always{{< /tab >}} +{{% /tabpane %}} + +## QOS Classes + +You may consider the upstream policies, depending on your needs: + +* [QoS Burstable](https://kyverno.io/policies/other/require-qos-burstable/require-qos-burstable/) +* [QoS Guaranteed](https://kyverno.io/policies/other/require-qos-guaranteed/require-qos-guaranteed/) diff --git a/content/en/docs/operating/setup/installation.md b/content/en/docs/operating/setup/installation.md index 125674c..3166b24 100644 --- a/content/en/docs/operating/setup/installation.md +++ b/content/en/docs/operating/setup/installation.md @@ -60,9 +60,7 @@ Here are some key considerations to keep in mind when installing Capsule. Also c ### Admission Policies -While Capsule provides a robust framework for managing multi-tenancy in Kubernetes, it does not include built-in admission policies for enforcing specific security or operational standards for all possible aspects of a Kubernetes cluster. Therefore, it is recommended to use additional tools like [Kyverno](https://kyverno.io/) to enforce admission policies that align with your organization's requirements. - -[We provide policy recommendations for Kyverno here](/ecosystem/integrations/kyverno/#recommended-policies). +While Capsule provides a robust framework for managing multi-tenancy in Kubernetes, it does not include built-in admission policies for enforcing specific security or operational standards for all possible aspects of a Kubernetes cluster. [We provide additional policy recommendations here](/docs/operating/setup/admission-policies/). ### Certificate Management @@ -88,16 +86,16 @@ Generally we recommend to use [matchconditions](https://kubernetes.io/docs/refer #### Nodes -There is a webhook which catches interactions with the Node resource. This Webhook is mainly interesting, when you make use of [Node Metadata](/docs/tenants/enforcement/#nodes). In any other case it will just case you problems. By default the webhook is enabled, but you can disable it by setting the following value: +There is a webhook which catches interactions with the Node resource. This Webhook is mainly interesting, when you make use of [Node Metadata](/docs/tenants/enforcement/#nodes). In any other case it will just case you problems. By default the webhook is **disabled**, but you can enabled it by setting the following value: ```yaml webhooks: hooks: nodes: - enabled: false + enabled: true ``` -Or you could at least consider to set the failure policy to `Ignore`: +Or you could at least consider to set the failure policy to `Ignore`, if you don't want to disrupt critical nodes: ```yaml webhooks: @@ -132,10 +130,6 @@ webhooks: expression: '!("system:serviceaccounts:kube-system" in request.userInfo.groups)' ``` -## Compatibility - -The Kubernetes compatibility is announced for each [Release](https://github.com/projectcapsule/capsule/releases). Generally we are up to date with the latest upstream Kubernetes Version. Note that the Capsule project offers support only for the latest minor version of Kubernetes. Backwards compatibility with older versions of Kubernetes and OpenShift is offered by [vendors](/support/). - ## GitOps There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource. @@ -186,14 +180,13 @@ spec: capsuleUserGroups: - oidc:kubernetes-users - system:serviceaccounts:capsule-argo-addon - webhooks: - hooks: - nodes: - failurePolicy: Ignore - serviceMonitor: - enabled: true - annotations: - argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true + monitoring: + dashboards: + enabled: true + serviceMonitor: + enabled: true + annotations: + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true proxy: enabled: true webhooks: @@ -263,7 +256,7 @@ spec: chart: spec: chart: capsule - version: "0.10.6" + version: "0.11.0" sourceRef: kind: HelmRepository name: capsule @@ -291,12 +284,11 @@ spec: capsuleUserGroups: - oidc:kubernetes-users - system:serviceaccounts:capsule-argo-addon - webhooks: - hooks: - nodes: - failurePolicy: Ignore - serviceMonitor: - enabled: true + monitoring: + dashboards: + enabled: true + serviceMonitor: + enabled: true proxy: enabled: true webhooks: @@ -387,3 +379,7 @@ To inspect the SBOM of the docker image, run the following command. Replace `` with an [available release tag](https://github.com/projectcapsule/capsule/pkgs/container/charts%2Fcapsule): COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign download sbom ghcr.io/projectcapsule/charts/capsule: + +## Compatibility + +The Kubernetes compatibility is announced for each [Release](https://github.com/projectcapsule/capsule/releases). Generally we are up to date with the latest upstream Kubernetes Version. Note that the Capsule project offers support only for the latest minor version of Kubernetes. Backwards compatibility with older versions of Kubernetes and OpenShift is offered by [vendors](/support/). diff --git a/content/en/docs/tenants/permissions.md b/content/en/docs/tenants/permissions.md index 09fa76e..92dcfcc 100644 --- a/content/en/docs/tenants/permissions.md +++ b/content/en/docs/tenants/permissions.md @@ -216,6 +216,10 @@ spec: - capsule-namespace-deleter kind: User name: alice + labels: + projectcapsule.dev/sample: "true" + annotations: + projectcapsule.dev/sample: "true" resourceQuotas: scope: Tenant status: @@ -248,6 +252,9 @@ items: labels: capsule.clastix.io/role-binding: 8fb969aaa7a67b71 capsule.clastix.io/tenant: solar + projectcapsule.dev/sample: "true" + annotations: + projectcapsule.dev/sample: "true" name: capsule-solar-0-admin namespace: solar-production ownerReferences: @@ -274,6 +281,9 @@ items: labels: capsule.clastix.io/role-binding: b8822dde20953fb1 capsule.clastix.io/tenant: solar + projectcapsule.dev/sample: "true" + annotations: + projectcapsule.dev/sample: "true" name: capsule-solar-1-capsule-namespace-deleter namespace: solar-production ownerReferences: @@ -457,6 +467,10 @@ spec: - apiGroup: rbac.authorization.k8s.io kind: User name: joe + labels: + projectcapsule.dev/sample: "true" + annotations: + projectcapsule.dev/sample: "true" EOF ``` @@ -536,8 +550,3 @@ roleRef: With the above example, Capsule is leaving the tenant owner to create namespaced custom resources. > Take Note: a tenant owner having the admin scope on its namespaces only, does not have the permission to create Custom Resources Definitions (CRDs) because this requires a cluster admin permission level. Only Bill, the cluster admin, can create CRDs. This is a known limitation of any multi-tenancy environment based on a single shared control plane. - - - - - diff --git a/content/en/docs/whats-new.md b/content/en/docs/whats-new.md index 76b68e2..b0d0db4 100644 --- a/content/en/docs/whats-new.md +++ b/content/en/docs/whats-new.md @@ -19,7 +19,7 @@ We have added new documentation for a better experience. See the following Topic * **[Best Practices](/docs/operating/best-practices/)** * **[Installation](/docs/operating/setup/installation/)** -* **[Kyverno Policy Recommendations](/ecosystem/integrations/kyverno/#recommended-policies)** +* **[Admission Policy Recommendations](/docs/operating/setup/admission-policies/)** ## Ecosystem diff --git a/content/en/ecosystem/integrations/kyverno.md b/content/en/ecosystem/integrations/kyverno.md index 5bd1730..2020b90 100644 --- a/content/en/ecosystem/integrations/kyverno.md +++ b/content/en/ecosystem/integrations/kyverno.md @@ -13,345 +13,7 @@ integration: true Not all relevant settings are covered by Capsule. We recommend to use Kyverno to enforce additional policies, as their policy implementation is of a very high standard. Here are some policies you might want to consider in multi-tenant environments: -### Workloads (Pods) - -Admission Rules for Pods are a good way to enforce security best practices. - -#### Mutate User Namespace - -You should enforce the usage of [User Namespaces](/docs/operating/best-practices/workloads/#user-namespaces). Most Helm-Charts currently don't support this out of the box. With Kyverno you can enforce this on Pod level: - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: tenant-workload-restrictions -spec: - rules: - - name: enforce-no-host-users - match: - any: - - resources: - kinds: - - Pod - namespaceSelector: - matchExpressions: - - key: capsule.clastix.io/tenant - operator: Exists - # selector: - # matchExpressions: - # - key: company.com/allow-host-users - # operator: NotIn - # values: - # - "true" - preconditions: - all: - - key: "{{request.operation || 'BACKGROUND'}}" - operator: AnyIn - value: - - CREATE - - UPDATE - skipBackgroundRequests: true - mutate: - patchStrategicMerge: - spec: - hostUsers: false -``` - -Note that users still can override this setting by adding the label `company.com/allow-host-users=true` to their namespace. You can change the label to your needs. This is because NFS does not support user namespaces and you might want to allow this for specific tenants. - -#### Disallow Daemonsets - -Tenant's should not be allowed to create Daemonsets, unless they have dedicated nodes: - -```yaml - -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: tenant-workload-restrictions -spec: - validationFailureAction: Enforce - rules: - - name: block-daemonset-create - match: - any: - - resources: - kinds: - - DaemonSet - namespaceSelector: - matchExpressions: - - key: capsule.clastix.io/tenant - operator: Exists - preconditions: - all: - - key: "{{ request.operation || 'BACKGROUND' }}" - operator: Equals - value: CREATE - validate: - message: "Creating DaemonSets is not allowed in this cluster." - deny: - conditions: - any: - - key: "true" - operator: Equals - value: "true" -``` - -#### Disallow Scheduling on Control Planes - -If a Pods are not scoped to specific nodes, they could be scheduled on control plane nodes. You should disallow this by enforcing that Pods do not use tolerations for control plane nodes: - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: tenant-workload-restrictions -spec: - validationFailureAction: Enforce - rules: - - name: restrict-controlplane-scheduling-master - match: - resources: - kinds: - - Pod - namespaceSelector: - matchExpressions: - - key: capsule.clastix.io/tenant - operator: Exists - validate: - message: Pods may not use tolerations which schedule on control plane nodes. - pattern: - spec: - =(tolerations): - - key: "!node-role.kubernetes.io/master" - - - name: restrict-controlplane-scheduling-control-plane - match: - resources: - kinds: - - Pod - namespaceSelector: - matchExpressions: - - key: capsule.clastix.io/tenant - operator: Exists - validate: - message: Pods may not use tolerations which schedule on control plane nodes. - pattern: - spec: - =(tolerations): - - key: "!node-role.kubernetes.io/control-plane" -``` - -#### Enforce EmptDir Requests/Limits - -By Defaults `emptyDir` Volumes do not have any limits. This could lead to a situation, where a tenant fills up the node disk. To avoid this, you can enforce limits on `emptyDir` volumes. You may also consider restricting the usage of `emptyDir` with the `medium: Memory` option, as this could lead to memory exhaustion on the node. - -```yaml - -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: tenant-workload-restrictions -spec: - rules: - - name: default-emptydir-sizelimit - match: - any: - - resources: - kinds: - - Pod - namespaceSelector: - matchExpressions: - - key: capsule.clastix.io/tenant - operator: Exists - mutate: - foreach: - - list: "request.object.spec.volumes[]" - preconditions: - all: - - key: "{{element.keys(@)}}" - operator: AnyIn - value: emptyDir - - key: "{{element.emptyDir.sizeLimit || ''}}" - operator: Equals - value: '' - patchesJson6902: |- - - path: "/spec/volumes/{{elementIndex}}/emptyDir/sizeLimit" - op: add - value: 250Mi -``` - -### Block Ephemeral Containers - -Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the `kubectl debug` functionality and attach a temporary container to an existing Pod. This may potentially be used to gain access to unauthorized information executing inside one or more containers in that Pod. This policy blocks the use of ephemeral containers. - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: block-ephemeral-containers - annotations: - policies.kyverno.io/title: Block Ephemeral Containers - policies.kyverno.io/category: Other - policies.kyverno.io/severity: medium - kyverno.io/kyverno-version: 1.6.0 - policies.kyverno.io/minversion: 1.6.0 - kyverno.io/kubernetes-version: "1.23" - policies.kyverno.io/subject: Pod - policies.kyverno.io/description: >- - Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the - `kubectl debug` functionality and attach a temporary container to an existing Pod. - This may potentially be used to gain access to unauthorized information executing inside - one or more containers in that Pod. This policy blocks the use of ephemeral containers. -spec: - validationFailureAction: Enforce - background: true - rules: - - name: block-ephemeral-containers - match: - any: - - resources: - kinds: - - Pod - validate: - message: "Ephemeral (debug) containers are not permitted." - pattern: - spec: - X(ephemeralContainers): "null" -``` - -[Source](https://kyverno.io/policies/other/block-ephemeral-containers/block-ephemeral-containers/) - -### Image Registry - -This can alos be achieved using Capsule's [Container Registries](/docs/tenants/enforcement/#images-registries) feature. Here is an example of allowing specific registries for a tenant: - -```yaml -apiVersion: capsule.clastix.io/v1beta2 -kind: Tenant -metadata: - name: solar -spec: - containerRegistries: - allowed: - - "docker.io" - - "public.ecr.aws" - - "quay.io" - - "mcr.microsoft.com" -``` - -Or with a Kyverno Policy. Here the default registry is `docker.io`, when no registry prefix is specified: - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: restrict-image-registries - annotations: - policies.kyverno.io/title: Restrict Image Registries - policies.kyverno.io/category: Best Practices, EKS Best Practices - policies.kyverno.io/severity: medium -spec: - validationFailureAction: Audit - background: true - rules: - - name: validate-registries - match: - any: - - resources: - kinds: - - Pod - validate: - message: "Using unknown image registry." - foreach: - - list: "request.object.spec.initContainers" - deny: - conditions: - - key: '{{images.initContainers."{{element.name}}".registry }}' - operator: NotIn - value: - - "docker.io" - - "public.ecr.aws" - - "quay.io" - - "mcr.microsoft.com" - - - list: "request.object.spec.ephemeralContainers" - deny: - conditions: - - key: '{{images.ephemeralContainers."{{element.name}}".registry }}' - operator: NotIn - value: - - "docker.io" - - "public.ecr.aws" - - "quay.io" - - "mcr.microsoft.com" - - - list: "request.object.spec.containers" - deny: - conditions: - - key: '{{images.containers."{{element.name}}".registry }}' - operator: NotIn - value: - - "docker.io" - - "public.ecr.aws" - - "quay.io" - - "mcr.microsoft.com" -``` - -### Image PullPolicy - -As stated [here](/docs/operating/best-practices/images/) on shared nodes you must use the `Always` pull policy. You can enforce this with the following policy: - -```yaml -apiVersion: kyverno.io/v1 -kind: ClusterPolicy -metadata: - name: always-pull-images - annotations: - policies.kyverno.io/title: Always Pull Images - policies.kyverno.io/category: Sample - policies.kyverno.io/severity: medium - policies.kyverno.io/subject: Pod - policies.kyverno.io/minversion: 1.6.0 - policies.kyverno.io/description: >- - By default, images that have already been pulled can be accessed by other - Pods without re-pulling them if the name and tag are known. In multi-tenant scenarios, - this may be undesirable. This policy mutates all incoming Pods to set their - imagePullPolicy to Always. An alternative to the Kubernetes admission controller - AlwaysPullImages. -spec: - rules: - - name: always-pull-images - match: - any: - - resources: - kinds: - - Pod - namespaceSelector: - matchExpressions: - - key: capsule.clastix.io/tenant - operator: Exists - mutate: - patchStrategicMerge: - spec: - initContainers: - - (name): "?*" - imagePullPolicy: Always - containers: - - (name): "?*" - imagePullPolicy: Always - ephemeralContainers: - - (name): "?*" - imagePullPolicy: Always -``` - -### QOS Classes - -You may consider the upstream policies, depending on your needs: - -* [QoS Burstable](https://kyverno.io/policies/other/require-qos-burstable/require-qos-burstable/) -* [QoS Guaranteed](https://kyverno.io/policies/other/require-qos-guaranteed/require-qos-guaranteed/) +[Moved to new page](/docs/operating/setup/admission-policies/) ## References