fix(cluster): spec-5.15 Hardening v1.2 — cold-bootstrap proof on durable slot, not live CSSD

A founding survivor could mis-classify itself as a rejoiner and refuse its own
writes (53R61) after an UNRELATED node fail-stop, blocking 3-node reconfig (the
join-remaster barrier never completes — surfaced by spec-5.16 t/326).

Root cause: the v1.1 cold-bootstrap proof (cluster_reconfig_bootstrap_quorum_at_initial)
counted live CSSD-alive peers.  Transient IC / heartbeat churn could drop the
quorum below threshold, leaving a genuine co-booting member UNDECIDED (its
joiner gate never latched BOOTSTRAP).  An unrelated peer's later fail-stop then
advanced the cluster epoch, and joiner_self_tick reclassified the still-UNDECIDED
member as a rejoiner: it closed its own write gate and timed out to 53R61.
INV-J14's latch only covered the BOOTSTRAP branch, not the UNDECIDED window.

Fix: anchor the co-boot quorum on the durable voting-disk slot instead of live
CSSD.  Count a declared peer toward the quorum only on a valid observed slot
(cluster_reconfig_get_observed_slot true, generation > 0) at epoch INITIAL —
never on a default-0 placeholder, never on live CSSD state.  Stable across CSSD
churn, so a founding member latches BOOTSTRAP reliably during formation, before
any unrelated epoch advance.  The quorum threshold (declared/2+1) and the
rejoiner predicate (any peer epoch > INITIAL) are unchanged; only the "alive at
INITIAL" predicate moves to the durable slot.

Unit: test_cluster_reconfig U20 (valid-slot-vs-CSSD differentiator + default-0
fail-closed, TDD red->green) and U19 updated to the durable-slot semantics.
No on-disk / wire / catalog / GUC change; no catversion bump.

Spec: spec-5.15-online-declared-node-join-membership.md (Hardening v1.2)

Author: SqlRush

src/backend/cluster/cluster_reconfig.c

@@ -1200,39 +1200,69 @@ cluster_reconfig_cluster_already_running(void)
 }
 
 /*
- * spec-5.15 Hardening v1.1 (HF-2 / INV-J14) — positive cold-bootstrap proof.
+ * spec-5.15 Hardening v1.1 (HF-2 / INV-J14) — positive cold-bootstrap proof,
+ * REVISED by Hardening v1.2 (INV-J14 self-join-gate race) to rest on the durable
+ * voting-disk slot rather than live CSSD.
+ *
  * A node may keep its write gate open WITHOUT online-join admission only when a
- * majority of declared nodes are CSSD-alive AND no declared peer is observed
- * past CLUSTER_EPOCH_INITIAL (everyone co-booting at epoch 0).  This is an EPOCH
- * proof, not a timing grace: a slow qvotec leaves the decision UNDECIDED (gate
- * stays closed, fail-closed) instead of mis-deciding bootstrap and permanently
- * fail-opening (P1-2).  A rejoiner can never satisfy it — by the time it sees
- * its peers they are already at epoch > INITIAL.
+ * majority of declared nodes are observed CO-BOOTING at CLUSTER_EPOCH_INITIAL on
+ * a VALID durable slot (qvotec saw a real voting-disk slot, generation > 0, at
+ * epoch INITIAL) AND no declared peer is observed past CLUSTER_EPOCH_INITIAL.
+ * This is an EPOCH proof, not a timing grace: a slow qvotec leaves the decision
+ * UNDECIDED (gate stays closed, fail-closed) instead of mis-deciding bootstrap
+ * and permanently fail-opening (P1-2).  A rejoiner can never satisfy it — by the
+ * time it sees its peers they are already at epoch > INITIAL.
+ *
+ * v1.2 RATIONALE: the v1.1 proof counted live CSSD-alive peers, which a
+ * transient IC / heartbeat churn could momentarily drop below quorum — leaving a
+ * GENUINE founding member UNDECIDED (never latched).  An UNRELATED peer's later
+ * fail-stop then advanced the epoch, and joiner_self_tick reclassified that
+ * still-UNDECIDED member as a rejoiner: it closed its own write gate and timed
+ * out to 53R61 (refused its own writes), never participating again.  Anchoring
+ * the quorum on the DURABLE slot (stable across CSSD churn) lets a founding
+ * member latch reliably during formation, closing the UNDECIDED window before
+ * any unrelated epoch advance.  A default-0 placeholder (generation 0) is NOT
+ * proof and must never count (else a node with no real evidence fail-opens) —
+ * the v1.2 user constraint: latch only on a valid co-boot slot, never on 0.
+ * Quorum (not all-declared) is retained so a degraded co-boot (e.g. 2 of 3) can
+ * still form, and because requiring every peer would only WIDEN the UNDECIDED
+ * window the race exploits.
  */
 bool
 cluster_reconfig_bootstrap_quorum_at_initial(void)
 {
 	uint32 declared = 0;
-	uint32 alive_at_initial = 0;
+	uint32 proven_at_initial = 0;
 	int i;
 
 	for (i = 0; i < CLUSTER_MAX_NODES; i++) {
+		uint64 inc = 0;
+		uint64 gen = 0;
+		uint64 ep;
+
 		if (cluster_conf_lookup_node(i) == NULL)
 			continue;
 		declared++;
 		if (i == cluster_node_id) {
-			alive_at_initial++; /* self is up, at INITIAL (not yet admitted) */
+			proven_at_initial++; /* self is up, at INITIAL (not yet admitted) */
 			continue;
 		}
+		ep = cluster_reconfig_get_observed_epoch(i);
 		/* any declared peer past INITIAL => a running cluster, NOT a bootstrap */
-		if (cluster_reconfig_get_observed_epoch(i) > CLUSTER_EPOCH_INITIAL)
+		if (ep > CLUSTER_EPOCH_INITIAL)
 			return false;
-		if (cluster_cssd_get_peer_state(i) != CLUSTER_CSSD_PEER_DEAD)
-			alive_at_initial++;
+		/*
+		 * Count a peer only on a VALID durable co-boot slot: a real observed
+		 * voting-disk slot (generation > 0) at epoch INITIAL.  Never count a
+		 * default-0 placeholder (generation 0) nor rely on live CSSD state.
+		 */
+		if (cluster_reconfig_get_observed_slot(i, &inc, &gen) && gen > 0
+			&& ep == CLUSTER_EPOCH_INITIAL)
+			proven_at_initial++;
 	}
 	if (declared == 0)
 		return false;
-	return alive_at_initial >= ((declared / 2u) + 1u);
+	return proven_at_initial >= ((declared / 2u) + 1u);
 }
 
 /*

src/test/cluster_unit/test_cluster_reconfig.c

@@ -1413,20 +1413,70 @@ UT_TEST(test_reconfig_bootstrap_quorum_epoch_proof)
 	ut_join_setup();		   /* self = node 0 */
 	ut_declared_set[1] = true; /* 3 declared nodes */
 	ut_declared_set[2] = true;
-	ut_peer_state[1] = CLUSTER_CSSD_PEER_ALIVE;
-	ut_peer_state[2] = CLUSTER_CSSD_PEER_ALIVE;
 
-	/* all CSSD-alive at epoch 0 -> cold bootstrap proven. */
+	/* quorum of declared on VALID co-boot slots at INITIAL -> bootstrap proven
+	 * (v1.2: anchored on the durable voting-disk slot, not live CSSD). */
+	cluster_reconfig_record_observed_slot(1, 1, 1, 0);
+	cluster_reconfig_record_observed_slot(2, 1, 1, 0);
 	UT_ASSERT(cluster_reconfig_bootstrap_quorum_at_initial());
 
 	/* a peer past INITIAL (running cluster) -> NOT a bootstrap (fail-closed). */
 	cluster_reconfig_record_observed_slot(1, 1, 1, 4);
 	UT_ASSERT(!cluster_reconfig_bootstrap_quorum_at_initial());
 
-	/* back at epoch 0 but only self alive (peers DEAD) -> below quorum -> false. */
-	cluster_reconfig_record_observed_slot(1, 1, 1, 0);
-	ut_peer_state[1] = CLUSTER_CSSD_PEER_DEAD;
+	/* no valid co-boot slot on either peer (generation 0) -> only self proven ->
+	 * below quorum -> false (never latch on a default-0 placeholder). */
+	cluster_reconfig_record_observed_slot(1, 0, 0, 0);
+	cluster_reconfig_record_observed_slot(2, 0, 0, 0);
+	UT_ASSERT(!cluster_reconfig_bootstrap_quorum_at_initial());
+}
+
+
+/* ======================================================================
+ * U20 (spec-5.15 Hardening v1.2 / INV-J14 self-join-gate race) -- the
+ * cold-bootstrap proof must rest on a VALID durable co-boot slot
+ * (cluster_reconfig_get_observed_slot true, generation > 0, observed_epoch
+ * == INITIAL), NOT on live CSSD state and NOT on a default-0 placeholder.
+ *
+ * Root cause it guards: a founding survivor that has durable proof of co-
+ * booting at INITIAL but whose peers' live CSSD is momentarily DOWN (IC /
+ * heartbeat churn) was denied bootstrap by the v1.1 CSSD-quorum proof, so it
+ * stayed UNDECIDED; a later UNRELATED node fail-stop then advanced the epoch
+ * and reclassified this genuine member as a rejoiner -> 53R61 (refused its own
+ * writes).  Anchoring the proof on the durable voting-disk slot lets the member
+ * latch reliably during formation (immune to CSSD churn), closing the window.
+ * ====================================================================== */
+UT_TEST(test_reconfig_bootstrap_proof_valid_slot_not_cssd)
+{
+	/* --- A. valid co-boot slots at INITIAL but peers CSSD-DEAD -> still a
+	 *        proven bootstrap (durable slot, not live CSSD).  v1.1 returned
+	 *        false here (the race window); v1.2 returns true. --- */
+	ut_join_setup();		   /* self = node 0 */
+	ut_declared_set[1] = true; /* 3 declared nodes */
+	ut_declared_set[2] = true;
+	cluster_reconfig_record_observed_slot(1, 7, 1, 0); /* valid slot, INITIAL */
+	cluster_reconfig_record_observed_slot(2, 7, 1, 0); /* valid slot, INITIAL */
+	ut_peer_state[1] = CLUSTER_CSSD_PEER_DEAD;		   /* live CSSD churned down */
 	ut_peer_state[2] = CLUSTER_CSSD_PEER_DEAD;
+	UT_ASSERT(cluster_reconfig_bootstrap_quorum_at_initial());
+
+	/* --- B. CSSD-alive but NO valid slot (generation 0 placeholder) must
+	 *        NOT prove bootstrap — never latch on a default-0 epoch. --- */
+	ut_join_setup();
+	ut_declared_set[1] = true;
+	ut_declared_set[2] = true;
+	ut_peer_state[1] = CLUSTER_CSSD_PEER_ALIVE;
+	ut_peer_state[2] = CLUSTER_CSSD_PEER_ALIVE;
+	/* no record_observed_slot -> generation 0 -> not a valid co-boot proof */
+	UT_ASSERT(!cluster_reconfig_bootstrap_quorum_at_initial());
+
+	/* --- C. a peer observed past INITIAL is a running cluster, never a
+	 *        bootstrap (rejoiner fail-closed) — unchanged from v1.1. --- */
+	ut_join_setup();
+	ut_declared_set[1] = true;
+	ut_declared_set[2] = true;
+	cluster_reconfig_record_observed_slot(1, 7, 1, 0); /* valid, INITIAL */
+	cluster_reconfig_record_observed_slot(2, 7, 1, 5); /* valid, past INITIAL */
 	UT_ASSERT(!cluster_reconfig_bootstrap_quorum_at_initial());
 }
 
@@ -1508,6 +1558,7 @@ main(void)
 	UT_RUN(test_reconfig_join_publish_proven_member_quorum);
 	UT_RUN(test_reconfig_join_publish_proven_no_member_failclosed);
 	UT_RUN(test_reconfig_bootstrap_quorum_epoch_proof);
+	UT_RUN(test_reconfig_bootstrap_proof_valid_slot_not_cssd);
 
 	UT_DONE();
 	return ut_failed_count == 0 ? 0 : 1;