Merge pull request #5080 from MyMirelHub/affinity-docs

[1.18] Docs on affinity spread for placement and scheduler

Author: marcduiker

daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md

@@ -122,6 +122,30 @@ To scale scheduler and placement to three instances independently of the `global
     --wait
    ```
 
+### Spreading Placement and Scheduler replicas
+
+With HA enabled, the default Helm values add pod anti-affinity so Scheduler and Placement service replicas are attempted to be scheduled across different failure domains if cluster resourcing allows. This is configured using the following Helm chart [options](https://github.com/dapr/dapr/blob/master/charts/dapr/README.md)
+
+- `global.ha.topologyKey`: Kubernetes label used meaning “different domain”. Set by default to `topology.kubernetes.io/zone`.
+- `global.ha.podAntiAffinityPolicy`: Enables soft spreading with `preferredDuringSchedulingIgnoredDuringExecution` (the default) or hard spreading with `requiredDuringSchedulingIgnoredDuringExecution`. 
+
+When running in production, consider the following recommendations.
+
+1. **Availability zones**: The default topology key spreads pods across zones. This limits the chance that a single zone outage removes multiple Scheduler or Placement pods at the same time.
+2. **Single zone, many nodes**:  Zone labels are often identical across the cluster; set `topologyKey` to `kubernetes.io/hostname` to enable spreading by node.
+3. **Soft spreading**: The default behaviour is recommended here allowing pod scheduling to succeed when the cluster is short on zones or nodes.
+4. **Hard spreading**: Only use when you must never place two replicas on the same topology value. This requires as many distinct values for `topologyKey` as replicas, or pods will stay `Pending`.
+
+For example, too configure hard spreading by hostname in a single-zone cluster, use the following Helm values:
+
+```yaml
+global:
+  ha:
+    enabled: true
+    topologyKey: kubernetes.io/hostname
+    podAntiAffinityPolicy: requiredDuringSchedulingIgnoredDuringExecution
+```
+
 ## Setting cluster critical priority class name for control plane services
 
 In some scenarios, nodes may have memory and/or cpu pressure and the Dapr control plane pods might get selected