You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-`ready` flag starts `False` and is set to `True` after the first successful poll cycle. This drives the `/healthz` readiness probe endpoint — Kubernetes won't route traffic until the service has completed its first poll.
206
+
-`k8s_api_reachable` starts `True` (fail-open). The background health check thread flips it to `False` after 3 consecutive failures. See [Fencing Safety Model](#fencing-safety-model) for the startup trade-off rationale.
205
207
-`shutdown_event` is set by the SIGTERM handler to signal the main loop and all `wait()` calls to exit gracefully.
206
208
-`kdump_lock` protects all kdump-related state since the UDP listener thread writes concurrently with the main poll loop.
207
209
-`heartbeat_lock` similarly protects heartbeat state from concurrent UDP listener writes.
@@ -674,6 +676,93 @@ The `disabled_reason` field tracks evacuation state:
674
676
675
677
---
676
678
679
+
## Fencing Safety Model
680
+
681
+
InstanceHA uses a two-tier safety model to prevent false-positive fencing. Each gate answers a distinct question — there is no redundancy between them.
682
+
683
+
### Gate Chain
684
+
685
+
When `CHECK_HEARTBEAT=true` (recommended for production):
686
+
687
+
```
688
+
Poll cycle starts
689
+
│
690
+
├─ Gate 1: K8s API reachable?
691
+
│ Question: "Can we trust our own observations?"
692
+
│ If no (3 consecutive failures) → skip entire cycle
693
+
│ Rationale: if InstanceHA's worker node is network-isolated,
694
+
│ Nova's service list may be stale — fencing would hit healthy hosts
695
+
│
696
+
├─ Gate 2: Per-host heartbeat
697
+
│ Question: "Is this specific host actually down?"
698
+
│ If host sending heartbeats within HEARTBEAT_TIMEOUT → skip that host
699
+
│ Rationale: Nova reports host as down, but heartbeat proves the OS is alive —
700
+
│ only nova-compute crashed, not the host
701
+
│
702
+
├─ Gate 3: Heartbeat cliff detection
703
+
│ Question: "Did we lose visibility, or did hosts actually fail?"
704
+
│ If >HEARTBEAT_CLIFF_THRESHOLD% of heartbeats vanish at once → skip ALL fencing
705
+
│ Rationale: simultaneous mass heartbeat loss is more likely a listener-side
706
+
│ network partition than N simultaneous host failures
│ Question: "Is this host crashing and dumping memory?"
710
+
│ If kdump message received → fence immediately, skip power-on during recovery
711
+
│
712
+
├─ Gate 5: Global threshold
713
+
│ Question: "Are too many hosts down to safely evacuate?"
714
+
│ If failed percentage > THRESHOLD → skip all fencing
715
+
│
716
+
└─ Gate 6: Per-aggregate threshold
717
+
Question: "Is this aggregate losing too many hosts?"
718
+
If failed count > instanceha:max_failures metadata → skip hosts in that aggregate
719
+
```
720
+
721
+
When `CHECK_HEARTBEAT=false`, gates 2 and 3 are not evaluated. A warning is logged at startup to alert operators that split-brain protection is limited to the K8s API check.
722
+
723
+
### Fail-Open vs Fail-Closed Semantics
724
+
725
+
| Gate | On error / no data | Rationale |
726
+
|------|-------------------|-----------|
727
+
| K8s API (startup) | Fail-open (allow fencing) | No data yet — blocking fencing on startup would silently prevent all evacuations until the health check thread runs |
728
+
| K8s API (after 3 failures) | Fail-closed (block fencing) | Confirmed partition — fencing would hit healthy hosts |
729
+
| Heartbeat (grace period) | Fail-open (allow fencing) | No heartbeat history in first `HEARTBEAT_TIMEOUT` seconds — cannot distinguish "no heartbeat" from "never received one" |
730
+
| Heartbeat (steady state) | Fail-open (allow fencing) | No heartbeat = host is down = proceed with fencing |
731
+
| Cliff detection | Fail-closed (block fencing) | Mass heartbeat loss is treated as observer failure |
732
+
733
+
### Startup Race Window
734
+
735
+
`k8s_api_reachable` initializes to `True` before the background health check thread confirms K8s API connectivity. If the K8s API is genuinely unreachable at startup, fencing could proceed for up to ~45 seconds (3 failures × 15s `K8S_API_CHECK_INTERVAL`) before the check blocks it.
736
+
737
+
In practice this window is smaller: the health check thread starts during `_initialize_service()` and runs its first probe almost immediately, while the main loop still needs to complete `_establish_nova_connection()` and `_reconcile_orphaned_hosts()` before its first poll. The health check typically completes its first probe before the main loop is ready to fence.
738
+
739
+
This is a deliberate trade-off. The alternative (`k8s_api_reachable = False` at startup) caused a deadlock: if the health check thread failed to start or was delayed, fencing was silently blocked forever with no log output indicating the problem.
740
+
741
+
### Fencing Decision Log
742
+
743
+
Every poll cycle that reaches the fencing decision point emits a single summary line:
744
+
745
+
```
746
+
INFO Fencing decision: 3 stale, 1 heartbeat-alive, 2 to fence, cliff=False
747
+
```
748
+
749
+
This line is emitted in all cases — including when all hosts are filtered out by heartbeat or cliff detection — so operators can diagnose fencing decisions from a single log grep.
750
+
751
+
### Relationship to MedIK8s
752
+
753
+
InstanceHA and MedIK8s (NHC/FAR/SNR) operate at different layers with no overlap:
| InstanceHA | EDPM data plane | Compute host fencing, VM evacuation |
759
+
760
+
MedIK8s watches Kubernetes Node objects. EDPM compute nodes are **not** Kubernetes nodes — they are external bare-metal hosts managed outside the cluster. MedIK8s cannot detect or remediate EDPM compute node failures; InstanceHA handles those independently.
761
+
762
+
MedIK8s is relevant for control plane recovery (e.g., if the node running RabbitMQ or the InstanceHA pod itself fails). InstanceHA is relevant for data plane recovery (compute host failures requiring VM evacuation).
![openshift-merge-bot[bot]](https://avatars.githubusercontent.com/in/412865?v=4&size=40){kind=avatar}
0 commit comments