Stabilize sandbox Elasticsearch node placement (#1032)

## Summary
- Pin sandbox Elasticsearch pods to the sandbox-large managed node
group.
- Mark sandbox Elasticsearch pods as not safe to evict by
cluster-autoscaler.
- Require both sandbox Elasticsearch pods to remain available for
voluntary disruptions.
- Keep sandbox-large node group minimum capacity at 2 nodes.

## Validation
- Applied the Kubernetes manifest changes to `credreg-sandbox`.
- Updated live EKS node group `ce-registry-eks-ng-sandbox-large` to
`min=2`, `desired=2`, `max=4`.
- Verified Elasticsearch StatefulSet rollout completed.
- Verified Elasticsearch cluster health is `green` with 2 nodes and 0
unassigned, initializing, or relocating shards.

---------

Co-authored-by: Ariel Rolfo <arielr-lt+username@users.noreply.github.com>

Author: arielr-lt

terraform/environments/eks/k8s-manifests-sandbox/elasticsearch-pdb.yaml

@@ -4,7 +4,7 @@ metadata:
   name: elasticsearch-pdb
   namespace: credreg-sandbox
 spec:
-  minAvailable: 1
+  minAvailable: 2
   selector:
     matchLabels:
       app: elasticsearch

terraform/environments/eks/k8s-manifests-sandbox/elasticsearch-statefulset.yaml

@@ -16,11 +16,14 @@ spec:
     metadata:
       labels:
         app: elasticsearch
+      annotations:
+        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
     spec:
       subdomain: elasticsearch-discovery
       priorityClassName: prod-high
       nodeSelector:
         env: sandbox
+        eks.amazonaws.com/nodegroup: ce-registry-eks-ng-sandbox-large
       tolerations:
         - key: "env"
           operator: "Equal"

terraform/environments/eks/terraform.tfvars

@@ -34,7 +34,7 @@ ng_staging_max_size           = 6
 ng_sandbox_min_size           = 1
 ng_sandbox_desired_size       = 1
 ng_sandbox_max_size           = 5
-ng_sandbox_large_min_size     = 1
+ng_sandbox_large_min_size     = 2
 ng_sandbox_large_desired_size = 2
 ng_sandbox_large_max_size     = 4
 ng_prod_min_size              = 2