infra: switch nomad server update policy to OPPORTUNISTIC (#2356)

* infra: switch nomad server update policy to OPPORTUNISTIC

Prevents automatic instance replacements on template changes,
requiring manual intervention to roll out updates to the control
plane. This reduces risk of unintended disruptions to the Nomad
server cluster.

* fix(infra): use dev conditional and fix redistribution for server update policy

- Use var.environment == "dev" ? "PROACTIVE" : "OPPORTUNISTIC" to match
  the pattern used by all other nodepools (api, clickhouse, loki, worker)
- Set instance_redistribution_type to NONE for non-dev to prevent GCP
  zone rebalancing from propagating new templates as a side effect
- Add reference to hashicorp/nomad#9390 (missed heartbeats during rotation)
- Update stale comment to reflect OPPORTUNISTIC semantics

* fix(infra): keep PROACTIVE redistribution for server pool

Server cluster needs even zone distribution for Raft quorum resilience.
Redistribution only triggers on zone imbalance, not on template changes,
so it doesn't undermine the OPPORTUNISTIC update policy.

* docs(infra): note redistribution interaction with OPPORTUNISTIC updates

Author: ValentaTomas

iac/provider-gcp/nomad-cluster/nodepool-control-server.tf

@@ -49,13 +49,17 @@ resource "google_compute_region_instance_group_manager" "server_pool" {
     port = var.nomad_port
   }
 
-  # Server is a stateful cluster, so the update strategy used to roll out a new GCE Instance Template must be
-  # a rolling update.
+  # Server is a stateful cluster. In non-dev environments, use OPPORTUNISTIC updates so instance template
+  # changes are only applied when instances are recreated for other reasons (e.g., auto-healing).
+  # Proactive rolling replacements of servers can cause missed client heartbeats and secret revocations:
+  # https://github.com/hashicorp/nomad/issues/9390
   update_policy {
-    type           = "PROACTIVE"
+    type           = var.environment == "dev" ? "PROACTIVE" : "OPPORTUNISTIC"
     minimal_action = "REPLACE"
 
-    // We want to keep the instance distribution even
+    // Keep PROACTIVE redistribution to maintain even server distribution across zones for Raft quorum resilience.
+    // Note: redistributed instances will pick up the current instance template, which may apply pending template
+    // changes as a side effect of zone rebalancing. This is an acceptable trade-off for server quorum safety.
     instance_redistribution_type = "PROACTIVE"
     max_unavailable_fixed        = 0