feat: add best-practices

Author: oliverbaehler

content/en/_index.md

@@ -13,13 +13,14 @@ title: Project Capsule
   Demo <i class="fas fa-arrow-alt-circle-right ms-2"></i>
 </a>
 </div>
+
 {{< blocks/link-down color="info" >}}
 {{< /blocks/cover >}}
 
-
-<a href="/adopters">
 {{< blocks/section color="white" type="row" >}}
 
+## Developed in 🇮🇹 / 🇨🇭 / 🇧🇬 { class="text-center mb-4" }
+
 {{< adopters-slider >}}
 
 {{< /blocks/section >}}

content/en/docs/operating/best-practices/_index.md

@@ -0,0 +1,5 @@
+---
+title: Best Practices
+weight: 2
+description: Best Practices when running Capsule in production 
+---

content/en/docs/overview/architecture.md …operating/best-practices/architecture.mdcontent/en/docs/overview/architecture.md renamed to content/en/docs/operating/best-practices/architecture.md content/en/docs/overview/architecture.md renamed to content/en/docs/operating/best-practices/architecture.md

@@ -1,6 +1,6 @@
 ---
 title: Architecture
-weight: 10
+weight: 1
 description: Architecture references and considerations
 ---
 
@@ -14,6 +14,43 @@ In Capsule, we introduce a new persona called the `Tenant Owner`. The goal is to
 
 Capsule provides robust tools to strictly enforce tenant boundaries, ensuring that each tenant operates within its defined limits. This separation of duties promotes both security and efficient resource management.
 
+### Key Decisions
+
+Introducing a new separation of duties can lead to a significant paradigm shift. This has technical implications and may also impact your organizational structure. Therefore, when designing a multi-tenant platform pattern, carefully consider the following aspects. As **Cluster Administrator**, ask yourself:
+
+  * 🔑 **How much ownership can be delegated to Tenant Owners (Platform Users)?**
+
+The answer to this question may be influenced by the following aspects:
+
+* **Are the Cluster Adminsitrators willing to grant permissions to Tenant Owners**? 
+  * _You might have a problem with know-how and probably your organisation is not yet pushing Kubernetes itself enough as a key strategic plattform. The key here is enabling Plattform Users through good UX and know-how transfers_
+
+* **Who is responsible for the deployed workloads within the Tenants?**? 
+  * _If Platform Administrators are still handling this, a true “shift left” has not yet been achieved._
+
+* **Who gets paged during a production outage within a Tenant’s application?**?
+  * _You’ll need robust monitoring that enables Tenant Owners to clearly understand and manage what’s happening inside their own tenant._
+
+* **Are your customers technically capable of working directly with the Kubernetes API?**? 
+  * _If not, you may need to build a more user-friendly platform with better UX — for example, a multi-tenant ArgoCD setup, or UI layers like Headlamp._
+
+
+## Layouts
+
+Let's dicuss different Tenant Layouts which could be used . These are just approaches we have seen, however you might also find a combination of these which fits your use-case.
+
+### Tenant As A Service
+
+With this approach you essentially just provide your Customers with the Tenant on your cluster. The rest is their responsability. This concludes to a shared responsibility model. This can be achieved when also the Tenant Owners are responsible for everything they are provisiong within their Tenant's namespaces.
+
+![Resourcepool Dashboard](/images/content/architecture/layout-taas.drawio.png)
+
+
+| Platform Administrators | Platform Users (Tenant Users) |
+| :---------------------- | :----------------------------------------- |
+|||
+
+
 ## Scheduling
 
 Workload distribution across your compute infrastructure can be approached in various ways, depending on your specific priorities. Regardless of the use case, it's essential to preserve maximum flexibility for your platform administrators. This means ensuring that:
@@ -32,7 +69,7 @@ Strong tenant isolation, ensuring that any noisy neighbor effects remain confine
 
 ### Shared
 
-With this approach you share the nodes amongst all Tenants, therefor giving you more potential for optimizing resources on a node level. It's a common pattern to separate the controllers needed to power your distro (operators) form the actual workload. This ensures smooth operations for the clust
+With this approach you share the nodes amongst all Tenants, therefor giving you more potential for optimizing resources on a node level. It's a common pattern to separate the controllers needed to power your Distribution (operators) form the actual workload. This ensures smooth operations for the cluster
 
 **Overview**:
 
@@ -43,7 +80,8 @@ With this approach you share the nodes amongst all Tenants, therefor giving you
 
 ![Shared Nodepool](/images/content/scheduling-shared.drawio.png)
 
-There's some further aspects you must think about with shared approaches:
+
+We provide the concept of [ResourcePools](/docs/resourcepools/) to manage resources cross namespaces. There's some further aspects you must think about with shared approaches:
 
   * [PriorityClasses](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)
   * [ResourceQuotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/)

…n/docs/operating/control-pod-security.md …g/best-practices/control-pod-security.mdcontent/en/docs/operating/control-pod-security.md renamed to content/en/docs/operating/best-practices/control-pod-security.md content/en/docs/operating/control-pod-security.md renamed to content/en/docs/operating/best-practices/control-pod-security.md

@@ -1,6 +1,6 @@
 ---
-title: Pod Security
-weight: 10
+title: Pod Security Standards
+weight: 3
 description: Control the security of the pods running in the tenant namespaces
 ---
 
@@ -62,9 +62,10 @@ metadata:
 ```
 
 ### Capsule
+
 According to the regular Kubernetes segregation model, the cluster admin has to operate either at cluster level or at namespace level. Since Capsule introduces a further segregation level (the _Tenant_ abstraction), the cluster admin can implement Pod Security Standards at tenant level by simply forcing specific labels on all the namespaces created in the tenant.
 
-As cluster admin, create a tenant with additional labels:
+You can distribute these profiles via namespace. Here's how this could look like:
 
 ```yaml
 apiVersion: capsule.clastix.io/v1beta2
@@ -73,14 +74,26 @@ metadata:
   name: solar
 spec:
   namespaceOptions:
-    additionalMetadata:
+    additionalMetadataList:
+    - namespaceSelector:
+        matchExpressions:
+          - key: projectcapsule.dev/low_security_profile
+            operator: NotIn
+            values: ["system"]
       labels:
-        pod-security.kubernetes.io/enforce: baseline
-        pod-security.kubernetes.io/audit: restricted
+        pod-security.kubernetes.io/enforce: restricted
         pod-security.kubernetes.io/warn: restricted
-  owners:
-  - kind: User
-    name: alice
+        pod-security.kubernetes.io/audit: restricted
+    - namespaceSelector:
+        matchExpressions:
+          - key: company.com/env
+            operator: In
+            values: ["system"]
+      labels:
+        pod-security.kubernetes.io/enforce: privileged
+        pod-security.kubernetes.io/warn: privileged
+        pod-security.kubernetes.io/audit: privileged
+
 ```
 
 All namespaces created by the tenant owner, will inherit the Pod Security labels: 
@@ -152,7 +165,78 @@ kubectl --kubeconfig alice-solar.kubeconfig label ns solar-production \
 Error from server (Label pod-security.kubernetes.io/audit is forbidden for namespaces in the current Tenant ...
 ```
 
+## User Namespaces
+
+{{% alert title="Info" color="info" %}}
+The FeatureGate `UserNamespacesSupport` is active by default since [Kubernetes 1.33](https://kubernetes.io/blog/2025/04/25/userns-enabled-by-default/). However every pod must still [opt-in](#admission)
+
+When you are also enabling the FeatureGate `UserNamespacesPodSecurityStandards` you may relax the Pod Security Standards for your workloads. [Read More](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#integration-with-pod-security-admission-checks)
+{{% /alert %}}
+
+A process running as root in a container can run as a different (non-root) user in the host; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace. [Read More](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/)
+
+### Kubelet
+
+On your Kubelet you must use the [FeatureGates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/):
+
+* `UserNamespacesSupport`
+* `UserNamespacesPodSecurityStandards` (Optional)
+
+### Sysctls
+
+```yaml
+user.max_user_namespaces: "11255"
+```
+
+### Admission (Kyverno)
+
+To make sure all the workloads are forced to use dedicated User Namespaces, we recommend to mutate pods at admission. See the following examples.
+
+#### Kyverno
+
+```yaml
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: add-hostusers-spec
+  annotations:
+    policies.kyverno.io/title: Add HostUsers
+    policies.kyverno.io/category: Security
+    policies.kyverno.io/subject: Pod,User Namespace
+    kyverno.io/kubernetes-version: "1.31"
+    policies.kyverno.io/description: >-
+      Do not use the host's user namespace. A new userns is created for the pod. 
+      Setting false is useful for mitigating container breakout vulnerabilities even allowing users to run their containers as root
+      without actually having root privileges on the host. This field is
+      alpha-level and is only honored by servers that enable the
+      UserNamespacesSupport feature.
+spec:
+  rules:
+  - name: add-host-users
+    match:
+      any:
+      - resources:
+          kinds:
+          - Pod
+          namespaceSelector:
+            matchExpressions:
+            - key: capsule.clastix.io/tenant
+              operator: Exists
+    preconditions:
+      all:
+      - key: "{{request.operation || 'BACKGROUND'}}"
+        operator: AnyIn
+        value:
+          - CREATE
+          - UPDATE
+    mutate:
+      patchStrategicMerge:
+        spec:
+          hostUsers: false
+```
+
 ## Pod Security Policies
+
 As stated in the documentation, *"PodSecurityPolicies enable fine-grained authorization of pod creation and updates. A Pod Security Policy is a cluster-level resource that controls security sensitive aspects of the pod specification. The `PodSecurityPolicy` objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields."*
 
 Using the [Pod Security Policies](https://kubernetes.io/docs/concepts/security/pod-security-policy), the cluster admin can impose limits on pod creation, for example the types of volume that can be consumed, the linux user that the process runs as in order to avoid running things as root, and more. From multi-tenancy point of view, the cluster admin has to control how users run pods in their tenants with a different level of permission on tenant basis.

content/en/docs/operating/best-practices/general-advice.md

@@ -0,0 +1,34 @@
+---
+title: General Advice
+weight: 2
+description: This is general advice you should consider before making Kubernetes Distribution consideration
+---
+
+This is general advice you should consider before making Kubernetes Distribution consideration. They are partly relevant for Multi-Tenancy with Capsule.
+
+### Authentication
+
+User authentication for the platform should be handled via a central OIDC-compatible identity provider system (e.g., Keycloak, Azure AD, Okta, or any other OIDC-compliant provider).
+The rationale is that other central platform components — such as ArgoCD, Grafana, Headlamp, or Harbor — should also integrate with the same authentication mechanism. This enables a unified login experience and reduces administrative complexity in managing users and permissions.
+
+[Capsule relies on native Kubernetes RBAC](/docs/operating/authentication/), so it's important to consider how the Kubernetes API handles user authentication.
+
+### OCI Pull-Cache
+
+By default, Kubernetes clusters pull images directly from upstream registries like `docker.io`, `quay.io`, `ghcr.io`, or `gcr.io`. In production environments, this can lead to issues — especially because Docker Hub enforces rate limits that may cause image pull failures with just a few nodes or frequent deployments (e.g., when pods are rescheduled).
+
+To ensure availability, performance, and control over container images, it's essential to provide an on-premise OCI mirror.
+This mirror should be configured via the CRI (Container Runtime Interface) by defining it as a mirror endpoint in registries.conf for default registries (e.g., `docker.io`).
+This way, all nodes automatically benefit from caching without requiring developers to change image URLs.
+
+### Secrets Management
+
+In more complex environments with multiple clusters and applications, managing secrets manually via YAML or Helm is no longer practical.
+Instead, a centralized secrets management system should be established — such as Vault, AWS Secrets Manager, Azure Key Vault, or the CNCF project [OpenBao](https://openbao.org/) (formerly the Vault community fork).
+
+To integrate these external secret stores with Kubernetes, the [External Secrets Operator (ESO)](https://external-secrets.io/latest/) is a recommended solution. It automatically syncs defined secrets from external sources as Kubernetes secrets, and supports dynamic rotation, access control, and auditing.
+
+If no external secret store is available, there should at least be a secure way to store sensitive data in Git.
+In our ecosystem, we provide a solution based on SOPS (Secrets OPerationS) for this use case.
+
+[👉 Demonstration](https://killercoda.com/peakscale/course/playgrounds/sops-secrets)

content/en/docs/operating/best-practices/images.md

@@ -0,0 +1,9 @@
+---
+title: Container Images
+weight: 5
+description: Multi-Tenant Container Images considerations
+---
+
+> [Until this issue is resolved (might be in Kubernetes 1.34)](https://github.com/kubernetes/enhancements/issues/2535)
+
+it's recommended to use the [ImagePullPolicy](https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy) `Always` for private registries on shared nodes. This ensures that no images can be used which are already pulled to the node.