fix(cluster): spec-5.15 online-join Hardening v1.1 — half-publish proof, bootstrap epoch-proof, marker identity-grouping

Spec: spec-5.15-online-declared-node-join-membership.md (Hardening v1.1)

Three post-ship correctness findings on the online declared-node join path,
all fail-closed (behind cluster.online_join, default off):

- HF-1 / INV-J9 (half-publish window): the v1.0 joiner opened its write gate
  on the durable COMMITTED join marker alone (note_self_admitted), so a
  coordinator that crashed/stalled after the marker was majority-durable but
  before the publish (epoch advance + survivor state=MEMBER) left the joiner
  writable while every other node still saw it JOINING. New pure fn
  cluster_reconfig_join_publish_proven(admitted_epoch): qvotec opens the gate
  only after a majority of MEMBER survivors have advanced their durable
  observed epoch to >= admitted_epoch (the publish actually propagated). A
  marker-durable-but-unpublished state keeps the gate CLOSED; the joiner then
  times out to 53R61 and restarts with a fresh incarnation.

- HF-2 / INV-J14 (bootstrap fail-open): the v1.0 joiner_self_tick used a
  timing-grace heuristic — if no running peer was observed within a window it
  declared cold-bootstrap and left the gate open forever, so a slow qvotec
  could mis-see a rejoiner as a bootstrap and permanently fail-open. Replaced
  with a positive epoch proof cluster_reconfig_bootstrap_quorum_at_initial()
  (quorum of declared CSSD-alive AND no declared peer past EPOCH_INITIAL);
  undecided keeps the gate CLOSED (fail-closed). shmem init defaults the gate
  CLOSED when online_join is on (open when off — no behavior change off).

- HF-3 / INV-J13 (false majority): self-admit and the startup seed counted
  "any COMMITTED marker" toward majority, so two minority writes from
  different commit attempts (different coordinator/epoch) could aggregate.
  ClusterJoinCommitMarker gains a per-attempt commit_nonce (CLUSTER_JCMK_VERSION
  1->2; v1 markers fail-closed rejected); new cluster_join_marker_same_commit()
  groups by full identity, so only a single commit present on a disk majority
  is honored.

Test-infra fix the L8 e2e surfaced: the three reconfig-join injection fire
sites were never in the cluster_inject registry, so cluster.injection_points
rejected them as unknown and the half-publish injection never armed. Register
cluster-reconfig-join-commit-marker-durable (the one L8 needs) — total registry
139 -> 140; ripple the count baselines in t/015/017/018/020/021/022/023/024/030
and the cluster-reconfig-% count in cluster_regress reconfig_smoke (5 -> 6).

Tests: cluster_unit 131/131 (U16-U19 new: same_commit identity group, version
fail-closed, publish-proven member quorum, bootstrap epoch proof); t/315 18/18
incl. new L8 real crash-in-window e2e (coordinator paused post-marker-durable,
joiner stays 53R60 then 53R61, never half-published); cluster_regress 11/11;
PG 219/219; clang-format v18 clean. No catversion bump (marker is voting-disk
on-disk, self-versioned, not catalog).

Author: SqlRush

src/backend/cluster/cluster_inject.c

@@ -229,6 +229,13 @@ static ClusterInjectPoint cluster_injection_points[] = {
 	{ .name = "cluster-reconfig-decide-coordinator" },
 	{ .name = "cluster-reconfig-epoch-bump-pre" },
 	{ .name = "cluster-reconfig-broadcast-procsig-pre" },
+	/*
+	 * spec-5.15 Hardening v1.1 (HF-1): pause the join coordinator inside
+	 * commit_member after the COMMITTED marker is majority-durable but before
+	 * the publish, so t/315 L8 can prove the half-publish window keeps the
+	 * joiner's write gate CLOSED (publish-proof, not the marker alone, opens it).
+	 */
+	{ .name = "cluster-reconfig-join-commit-marker-durable" },
 	{ .name = "cluster-cssd-mark-peer-dead" },
 	/* Stage 1.15 (spec-1.15 D11 inject) — 4 SCN encoding-layer injects. */
 	{ .name = "cluster-scn-advance-pre" },

src/backend/cluster/cluster_membership.c

@@ -180,7 +180,7 @@ cluster_membership_is_member(int32 node_id)
  * ============================================================
  */
 
-/* Set m->crc32c = CRC32C over [magic .. supersedes_leave_epoch]. */
+/* Set m->crc32c = CRC32C over [magic .. commit_nonce]. */
 void
 cluster_join_marker_compute_crc(ClusterJoinCommitMarker *m)
 {
@@ -227,6 +227,23 @@ cluster_join_marker_is_committed_basis(const ClusterJoinCommitMarker *m, int32 e
 		   && m->phase == CLUSTER_JCMK_PHASE_COMMITTED;
 }
 
+/*
+ * INV-J13 (Hardening v1.1): same commit attempt?  Compares the whole identity
+ * range [0 .. offsetof(crc32c)) -- magic/version/node_id/phase/_pad/generation/
+ * admitted_incarnation/admitted_epoch/supersedes_leave_epoch/commit_nonce.  The
+ * markers are memset-0 before fill so _pad never differs.  Used by the self-
+ * admit and the startup-seed majority judgements so that two minority writes
+ * from DIFFERENT commit attempts (different coordinator / epoch / nonce) cannot
+ * be counted together as a false majority (P1-3).
+ */
+bool
+cluster_join_marker_same_commit(const ClusterJoinCommitMarker *a, const ClusterJoinCommitMarker *b)
+{
+	if (a == NULL || b == NULL)
+		return false;
+	return memcmp(a, b, offsetof(ClusterJoinCommitMarker, crc32c)) == 0;
+}
+
 /*
  * Apply one durable marker to the admitted floor (INV-J7).  Only a committed
  * basis raises the floor; record_admitted is monotonic so re-applying / lower

src/backend/cluster/cluster_qvotec.c

@@ -821,15 +821,26 @@ qvotec_poll_once(void)
 	/*
 	 * spec-5.15 D5: detect THIS node's own admission — a §2.6 COMMITTED join
 	 * marker in region-3 slot self, with admitted_incarnation == our incarnation,
-	 * on a quorum-majority of disks.  note_self_admitted then adopts the admitted
-	 * epoch + sets self MEMBER + opens the joiner write gate (one extra slot read
-	 * per disk; qvotec already has the fds open).
+	 * on a quorum-majority of disks (one extra slot read per disk; qvotec already
+	 * has the fds open).
+	 *
+	 * Hardening v1.1:
+	 *   HF-3 (INV-J13): require a majority of the SAME commit (identical nonce),
+	 *     not "any COMMITTED marker" — two minority writes from different commit
+	 *     attempts (different coordinator / epoch) must not aggregate (P1-3).
+	 *   HF-1 (INV-J9): open the gate only after the publish-proof also holds (a
+	 *     member quorum reached admitted_epoch).  marker-durable-but-coordinator-
+	 *     crashed-before-publish keeps the gate CLOSED -> the joiner times out ->
+	 *     53R61 -> restarts (P1-1 half-publish window).  Until then the lmon
+	 *     epoch catch-up keeps transport alive and the next poll re-checks.
 	 */
 	if (cluster_node_id >= 0 && cluster_node_id < CLUSTER_MAX_NODES) {
-		uint32 self_agree = 0;
-		uint64 self_admitted_epoch = 0;
+		ClusterJoinCommitMarker self_markers[CLUSTER_MAX_VOTING_DISKS];
+		int n_self = 0;
 		uint32 majority = ((uint32)qvotec_n_disks / 2u) + 1u;
-		int d;
+		int d, a, b;
+		int win = -1;
+		uint32 win_agree = 0;
 
 		for (d = 0; d < qvotec_n_disks; d++) {
 			union {
@@ -847,12 +858,27 @@ qvotec_poll_once(void)
 				continue;
 			if (m.admitted_incarnation != qvotec_self_incarnation)
 				continue; /* a stale prior-incarnation admission — not us */
-			self_agree++;
-			if (m.admitted_epoch > self_admitted_epoch)
-				self_admitted_epoch = m.admitted_epoch;
+			self_markers[n_self++] = m;
+		}
+
+		/* HF-3: find a single commit (nonce) present on >= majority disks. */
+		for (a = 0; a < n_self; a++) {
+			uint32 same = 0;
+
+			for (b = 0; b < n_self; b++)
+				if (cluster_join_marker_same_commit(&self_markers[a], &self_markers[b]))
+					same++;
+			if (same >= majority) {
+				win = a;
+				win_agree = same;
+				break;
+			}
 		}
-		if (self_agree >= majority)
-			cluster_reconfig_note_self_admitted(self_admitted_epoch);
+
+		/* HF-1: gate-open requires the publish-proof too, not the marker alone. */
+		if (win >= 0 && win_agree >= majority
+			&& cluster_reconfig_join_publish_proven(self_markers[win].admitted_epoch))
+			cluster_reconfig_note_self_admitted(self_markers[win].admitted_epoch);
 	}
 
 	/*