fix(cluster): spec-5.18 Hardening v1.1 — survivor-local removal apply + self-demote durability

Post-ship review of permanent node removal found 4 P1 + 1 P2, all in the
non-coordinator survivor path and the removed-node self path (the coordinator
main path was already covered).  Root cause: a runtime removal was applied only
on the coordinator and on the removed node itself; non-coordinator survivors
never applied it locally.

- HF-1/HF-3: a non-coordinator survivor now APPLIES the removal locally on the
  NODE_REMOVE_ANNOUNCE — seeds removed_bitmap + membership_state[N]=REMOVED with
  the coordinator's incarnation floor, permanently remasters N-mastered shards,
  clears N's GES/PCM, and proves zero leftover before it ACKs (was: only dropped
  refs + ACKed, leaving the permanent-removal fact absent from its memory until a
  restart, and ACKing without a leftover proof).  Retried each survivor lmon tick
  until verify converges.  Announce payload gains removed_incarnation (IC wire
  version 1->2).  New INV-LF11 (survivor-local-apply).
- HF-2: a removed node that is still running / restarts keeps self REMOVED — the
  lmon self-state maintenance and joiner_self_tick no longer flip it to
  JOINING/MEMBER, and the 53R64 self-demote gate now consults the durable
  removed_bitmap (lock-free) as the authoritative floor, not just membership_state.
- HF-4: the survivor adopts the announce's identity when its recorded one is
  absent/terminal/stale and ACKs with the payload identity; the coordinator
  validates the full (target, epoch, event_id) tuple — consecutive removals no
  longer wedge the cleanup barrier on a stale event_id.
- HF-5: pg_cluster_remove_node returns removal_in_progress (not a stale accepted)
  when a concurrent request reserved first.

Tests: cluster_unit U-H (payload v2 + removed_incarnation in CRC); t/325 +HF-2
(restarted removed node fail-closes xid assignment via 53R64); new 3-node t/326
(non-coordinator survivor records the removal + retains it after the coordinator
leaves).  Local: cluster_unit + cluster_regress(12) + PG219 + t/325/t/326/t/313
all green; clang-format v18 clean.  No catversion / shmem region / wait event /
errcode change.  HF-4 4-node consecutive-removal e2e + HF-5 concurrency forward
to spec-5.19.

Spec: spec-5.18-online-node-leave-fence-cleanup.md (## Hardening v1.1)

Author: SqlRush

.github/workflows/nightly.yml

@@ -149,7 +149,7 @@ jobs:
           - { name: stage3-recent,     ranges: "228-237", unit: false, regress: false }
           - { name: stage4-wal,        ranges: "242-248 273 275", unit: false, regress: false }
           - { name: stage4-hardgate,   ranges: "274-274", unit: false, regress: false }
-          - { name: stage5-ges-locking, ranges: "276-325", unit: false, regress: false }
+          - { name: stage5-ges-locking, ranges: "276-326", unit: false, regress: false }
     steps:
       - name: Checkout
         uses: actions/checkout@v4

src/backend/cluster/cluster_node_remove.c

@@ -270,14 +270,21 @@ cluster_node_remove_self_is_removed(void)
 	if (!cluster_enabled || cluster_node_id < 0)
 		return false;
 	/*
-	 * Lock-free hot-path check (called at every writable-xid assignment): the
-	 * membership_state[self] byte is the naturally-atomic SSOT.  REMOVED is published
-	 * into this node's own table by the startup durable-marker rebuild
+	 * Lock-free hot-path check (called at every writable-xid assignment).  REMOVED
+	 * is published into this node's own table by the startup durable-marker rebuild
 	 * (rebuild_from_disks) and, for a still-running removed node, by the
-	 * NODE_REMOVE_ANNOUNCE handler (nr_announce_handler self-demote).  No reconfig
-	 * lock is taken here (mirrors the clean-leave refuse-writes gate).
+	 * NODE_REMOVE_ANNOUNCE handler (nr_announce_handler self-demote).
+	 *
+	 * HF-2: consult the durable removed_bitmap (lock-free, monotonic) IN ADDITION to
+	 * the membership_state[self] byte.  membership_state[self] is rewritten every
+	 * LMON tick by the joiner / self-state maintenance paths; although those now
+	 * carry a REMOVED terminal guard, the durable bitmap is the authoritative floor
+	 * and closes any residual window where a stale self-state write could transiently
+	 * un-REMOVE this node and open the 53R64 write gate.  Either signal = removed.
+	 * No reconfig lock is taken (mirrors the clean-leave refuse-writes gate).
 	 */
-	return cluster_membership_get_state(cluster_node_id) == CLUSTER_MEMBER_REMOVED;
+	return cluster_membership_get_state(cluster_node_id) == CLUSTER_MEMBER_REMOVED
+		   || cluster_reconfig_is_removed_unlocked(cluster_node_id);
 }
 
 
@@ -441,6 +448,16 @@ cluster_node_remove_request(int32 node_id)
 		pg_atomic_write_u32(&nr_state->phase, CLUSTER_REMOVE_REQUESTED);
 		pg_atomic_fetch_add_u64(&nr_state->removal_request_count, 1);
 	}
+	/*
+	 * HF-5: ACCEPTED at the lock-free precheck but the phase advanced out of the
+	 * reservable set between the precheck and this exclusive section — a concurrent
+	 * request reserved first.  Do NOT return a stale ACCEPTED without actually
+	 * reserving (the caller would believe its target is being removed when it is
+	 * not); downgrade to removal_in_progress so it retries.
+	 */
+	else if (verdict == CLUSTER_REMOVE_REQ_ACCEPTED) {
+		verdict = CLUSTER_REMOVE_REQ_IN_PROGRESS;
+	}
 	/* RESUME: keep the existing target/epoch; just re-arm the drive from SHRUNK by
 	 * moving CLEANUP_BLOCKED back to CLEANUP (the lmon_tick retries the cleanup). */
 	else if (verdict == CLUSTER_REMOVE_REQ_RESUME && cur_phase == CLUSTER_REMOVE_CLEANUP_BLOCKED) {
@@ -656,16 +673,50 @@ cluster_node_remove_drive(void)
 void
 cluster_node_remove_survivor_ack(int32 target_node_id, uint64 remove_epoch)
 {
-	/* a non-coordinator survivor drops its local refs to the removed node and
-	 * records its ACK so the coordinator's barrier can complete. */
+	uint64 removal_event_id;
+	uint64 removed_incarnation;
+	int32 coordinator;
+
+	/*
+	 * HF-1/HF-3 (INV-LF11): a non-coordinator survivor APPLIES the removal
+	 * locally — not just drops its N-refs.  It seeds the durable removed set +
+	 * membership_state[N]=REMOVED (so its own removed_bitmap carries N for any
+	 * fence baseline it later publishes, INV-LF10), permanently remasters
+	 * N-mastered shards onto a MEMBER survivor, clears N's GES/PCM, and PROVES
+	 * zero leftover — and only ACKs once verify passes.  The coordinator's final
+	 * REMOVED marker (the trust source) is built on "local verify + all-survivor
+	 * ACK", so an ACK must mean THIS survivor is genuinely clean, not merely
+	 * "dropped some refs".  Idempotent: the survivor lmon_tick re-runs it every
+	 * tick until it converges (the announce is one-shot, so a transient leftover
+	 * must be retried locally, not re-announced).
+	 */
 	if (nr_state == NULL || target_node_id < 0 || target_node_id >= CLUSTER_MAX_NODES)
 		return;
 
-	(void)cluster_grd_cleanup_on_node_dead(target_node_id);
-	(void)cluster_pcm_lock_clear_pending_x_for_node(target_node_id);
+	/* identity for the seed + ACK = THIS attempt (recorded by the announce
+	 * handler / lmon_tick adopt path, HF-4). */
+	LWLockAcquire(&nr_state->lock, LW_SHARED);
+	removal_event_id = nr_state->removal_event_id;
+	removed_incarnation = nr_state->target_last_incarnation;
+	coordinator = nr_state->coordinator_node_id;
+	LWLockRelease(&nr_state->lock);
+
+	/* seed the durable removed set + membership REMOVED with the coordinator's
+	 * pinned incarnation floor (carried in the announce, HF-1). */
+	cluster_reconfig_seed_removed_membership(target_node_id, remove_epoch, removed_incarnation,
+											 /*raise_epoch_floor*/ true);
+
+	/* full cluster-wide cleanup on THIS survivor + zero-leftover proof (HF-3).
+	 * run_cleanup bumps leftover_detected_count + returns false when not clean. */
+	if (!cluster_node_remove_run_cleanup(target_node_id, remove_epoch)) {
+		pg_atomic_write_u32(&nr_state->survivor_acked, 0);
+		return; /* leftover -> retry next survivor tick, do NOT ACK */
+	}
 
-	cluster_node_remove_ic_send_ack(nr_state->coordinator_node_id, target_node_id, remove_epoch,
-									nr_state->removal_event_id);
+	/* clean: ACK with THIS attempt's identity so the coordinator's barrier keys on
+	 * this removal, not a stale prior one. */
+	cluster_node_remove_ic_send_ack(coordinator, target_node_id, remove_epoch, removal_event_id);
+	pg_atomic_write_u32(&nr_state->survivor_acked, 1);
 }
 
 void
@@ -674,6 +725,7 @@ cluster_node_remove_lmon_tick(void)
 	ClusterRemovePhase phase;
 	int32 node_id;
 	int32 coordinator;
+	uint64 remove_epoch;
 
 	if (nr_state == NULL || !cluster_enabled || !cluster_online_node_removal)
 		return;
@@ -684,6 +736,7 @@ cluster_node_remove_lmon_tick(void)
 	phase = (ClusterRemovePhase)pg_atomic_read_u32(&nr_state->phase);
 	node_id = nr_state->target_node_id;
 	coordinator = nr_state->coordinator_node_id;
+	remove_epoch = nr_state->remove_epoch;
 	LWLockRelease(&nr_state->lock);
 
 	if (node_id < 0)
@@ -695,10 +748,20 @@ cluster_node_remove_lmon_tick(void)
 		if (phase >= CLUSTER_REMOVE_CLEANUP && phase <= CLUSTER_REMOVE_CLEANUP_BLOCKED
 			&& pg_atomic_read_u32(&nr_state->announce_sent) == 0) {
 			cluster_node_remove_ic_broadcast_announce(node_id, nr_state->remove_epoch,
-													  nr_state->removal_event_id);
+													  nr_state->removal_event_id,
+													  nr_state->target_last_incarnation);
 			pg_atomic_write_u32(&nr_state->announce_sent, 1);
 		}
 		cluster_node_remove_drive();
+	} else {
+		/*
+		 * HF-1/HF-3 (INV-LF11): survivor side — (re)apply the recorded removal
+		 * locally + ACK; retry each tick until verify converges.  The announce is
+		 * one-shot, so a transient leftover (or an announce that arrived before the
+		 * GRD/PCM state was settleable) must be retried here, not re-announced.
+		 */
+		if (pg_atomic_read_u32(&nr_state->survivor_acked) == 0)
+			cluster_node_remove_survivor_ack(node_id, remove_epoch);
 	}
 }
 
@@ -707,7 +770,7 @@ cluster_node_remove_lmon_tick(void)
  * IC wire (D10): NODE_REMOVE_ANNOUNCE (broadcast) + REMOVE_CLEANUP_ACK (p2p).
  * ============================================================ */
 
-/* survivor side: a coordinator announced a removal — drop refs + ACK. */
+/* survivor side: a coordinator announced a removal — apply it locally + ACK. */
 static void
 nr_announce_handler(const ClusterICEnvelope *env, const void *payload)
 {
@@ -721,24 +784,38 @@ nr_announce_handler(const ClusterICEnvelope *env, const void *payload)
 		 * into our own membership table (lock-free SSOT for the self-demote write
 		 * gate) + the durable removed set so a still-running removed node fail-closes
 		 * new writable transactions (53R64) instead of serving as a phantom member.
-		 * Do NOT send a cleanup ACK — a removed node is not a survivor.
+		 * HF-1: pin the coordinator's incarnation floor from the announce so a future
+		 * re-admit must present a strictly newer incarnation.  Do NOT send a cleanup
+		 * ACK — a removed node is not a survivor.
 		 */
 		cluster_reconfig_seed_removed_membership(cluster_node_id, p->remove_epoch,
-												 0 /* incarnation floor set by coordinator */,
+												 p->removed_incarnation,
 												 /*raise_epoch_floor*/ false);
 		return;
 	}
 
-	/* record the attempt so our ACK echoes the right identity. */
+	/*
+	 * HF-4: adopt THIS removal attempt's identity when our recorded one is absent,
+	 * terminal, or a different attempt — a survivor never runs the driver's
+	 * abort/commit resets, so a prior attempt's identity would otherwise linger and
+	 * get this attempt's ACK rejected by the coordinator (event_id mismatch),
+	 * wedging the next removal's cleanup barrier.  Same event_id = an idempotent
+	 * re-announce: keep progress (do not reset survivor_acked).
+	 */
 	LWLockAcquire(&nr_state->lock, LW_EXCLUSIVE);
-	if (nr_state->target_node_id < 0) {
+	if (nr_state->removal_event_id != p->removal_event_id
+		|| nr_state->target_node_id != p->target_node_id) {
 		nr_state->target_node_id = p->target_node_id;
 		nr_state->coordinator_node_id = p->coordinator_node_id;
 		nr_state->remove_epoch = p->remove_epoch;
 		nr_state->removal_event_id = p->removal_event_id;
+		nr_state->target_last_incarnation = p->removed_incarnation;
+		pg_atomic_write_u32(&nr_state->survivor_acked, 0); /* re-apply for the new attempt */
 	}
 	LWLockRelease(&nr_state->lock);
 
+	/* INV-LF11: apply the removal locally + ACK when clean (the survivor lmon_tick
+	 * retries until verify converges). */
 	cluster_node_remove_survivor_ack(p->target_node_id, p->remove_epoch);
 }
 
@@ -753,11 +830,18 @@ nr_ack_handler(const ClusterICEnvelope *env, const void *payload)
 		return;
 	if (p->survivor_node_id < 0 || p->survivor_node_id >= CLUSTER_NODE_REMOVE_ACK_BITMAP_BYTES * 8)
 		return;
-	if (p->removal_event_id != nr_state->removal_event_id)
-		return; /* stale ACK from a prior attempt */
 
+	/*
+	 * HF-4: an ACK counts toward the barrier only for THIS exact removal attempt —
+	 * validate the full identity tuple (target, epoch, event_id) under the lock, not
+	 * just event_id, so a stale ACK from a prior attempt can never satisfy the
+	 * current barrier (and the snapshot is consistent with the bitmap write).
+	 */
 	LWLockAcquire(&nr_state->lock, LW_EXCLUSIVE);
-	nr_state->ack_bitmap[p->survivor_node_id / 8] |= (uint8)(1u << (p->survivor_node_id % 8));
+	if (p->removal_event_id == nr_state->removal_event_id
+		&& p->target_node_id == nr_state->target_node_id
+		&& p->remove_epoch == nr_state->remove_epoch)
+		nr_state->ack_bitmap[p->survivor_node_id / 8] |= (uint8)(1u << (p->survivor_node_id % 8));
 	LWLockRelease(&nr_state->lock);
 }
 
@@ -785,7 +869,7 @@ cluster_node_remove_register_ic_msg_types(void)
 
 void
 cluster_node_remove_ic_broadcast_announce(int32 target_node_id, uint64 remove_epoch,
-										  uint64 removal_event_id)
+										  uint64 removal_event_id, uint64 removed_incarnation)
 {
 	ClusterNodeRemoveAnnouncePayload p;
 	ClusterICFanoutResult per_peer[CLUSTER_MAX_NODES];
@@ -797,6 +881,7 @@ cluster_node_remove_ic_broadcast_announce(int32 target_node_id, uint64 remove_ep
 	p.target_node_id = target_node_id;
 	p.remove_epoch = remove_epoch;
 	p.removal_event_id = removal_event_id;
+	p.removed_incarnation = removed_incarnation; /* HF-1: incarnation floor for survivor seed */
 	cluster_node_remove_announce_compute_crc(&p);
 
 	cluster_ic_send_envelope_fanout(PGRAC_IC_MSG_NODE_REMOVE_ANNOUNCE, &p, (uint32)sizeof(p),

src/backend/cluster/cluster_reconfig.c

@@ -844,6 +844,25 @@ cluster_reconfig_is_removed(int32 node_id)
 	return removed;
 }
 
+/*
+ * spec-5.18 HF-2: lock-free removed test for the 53R64 self-demote write gate
+ * (cluster_node_remove_self_is_removed), called at every writable-xid assignment.
+ * Reads a single removed_bitmap bit without the reconfig lock — safe because the
+ * removed set is monotonic at runtime (a removal is terminal, INV-LF1; only an
+ * operator un-fence, not implemented, could clear it), so a one-byte read cannot
+ * tear and the bit never spuriously clears.  The durable bitmap is the
+ * authoritative floor: unlike membership_state[self] (which the joiner / lmon
+ * self-state paths rewrite each tick), it cannot be flipped REMOVED -> not-removed
+ * by membership churn, so the write gate stays fail-closed for a removed node.
+ */
+bool
+cluster_reconfig_is_removed_unlocked(int32 node_id)
+{
+	if (ReconfigShmem == NULL || node_id < 0 || node_id >= CLUSTER_RECONFIG_DEAD_BITMAP_BYTES * 8)
+		return false;
+	return (ReconfigShmem->removed_bitmap[node_id / 8] & (uint8)(1u << (node_id % 8))) != 0;
+}
+
 uint64
 cluster_reconfig_get_removed_epoch(int32 node_id)
 {
@@ -1246,6 +1265,17 @@ cluster_reconfig_joiner_self_tick(void)
 	if (cluster_node_id < 0 || cluster_node_id >= CLUSTER_MAX_NODES)
 		return;
 
+	/*
+	 * spec-5.18 INV-LF9 (HF-2): a durably-removed node (this node was permanently
+	 * removed; startup rebuild seeded removed_bitmap[self]) must NOT run the joiner
+	 * gate — doing so would flip its own membership_state REMOVED -> JOINING/MEMBER
+	 * (the REJOINER/bootstrap branches below) and defeat the 53R64 self-demote write
+	 * gate.  A removed node can only return via operator un-fence + a fresh-
+	 * incarnation join (external plane, §1.3) — never by re-running this gate.
+	 */
+	if (cluster_reconfig_is_removed_unlocked(cluster_node_id))
+		return;
+
 	now_us = (uint64)GetCurrentTimestamp();
 
 	/*
@@ -1444,9 +1474,21 @@ cluster_reconfig_lmon_tick(void)
 
 		LWLockAcquire(&ReconfigShmem->lock, LW_EXCLUSIVE);
 
+		/*
+		 * spec-5.18 INV-LF9 (HF-2): REMOVED is TERMINAL for self too.  A removed
+		 * node still running / restarted (rebuild seeded removed_bitmap[self] +
+		 * membership_state[self]=REMOVED) must keep self REMOVED — the joiner gate
+		 * below must NOT flip it back to MEMBER/JOINING, which would defeat the
+		 * 53R64 self-demote write gate and let a removed node serve writes.  The
+		 * durable removed_bitmap is the authoritative floor (mirrors the peer
+		 * REMOVED terminal guard further down + the joiner_self_tick guard).
+		 */
+		if (clean_departed_test_bit_locked(ReconfigShmem->removed_bitmap, self_id)
+			|| cluster_membership_get_state(self_id) == CLUSTER_MEMBER_REMOVED)
+			cluster_membership_set_state(self_id, CLUSTER_MEMBER_REMOVED);
 		/* self-state follows the joiner gate (D5): MEMBER when admitted / steady,
 		 * JOINING while this node is itself a joiner whose gate is closed. */
-		if (ReconfigShmem->self_join_admitted && !ReconfigShmem->self_join_failed)
+		else if (ReconfigShmem->self_join_admitted && !ReconfigShmem->self_join_failed)
 			cluster_membership_set_state(self_id, CLUSTER_MEMBER_MEMBER);
 		else
 			cluster_membership_set_state(self_id, CLUSTER_MEMBER_JOINING);

src/include/cluster/cluster_node_remove.h

@@ -251,7 +251,13 @@ StaticAssertDecl(sizeof(ClusterNodeRemoveState) <= 512, "ClusterNodeRemoveState
  *	message being acted on as a removal command (8.A defense-in-depth).
  * ============================================================ */
 #define CLUSTER_NODE_REMOVE_IC_MAGIC 0x50524943 /* "PRIC" — pgrac remove IC */
-#define CLUSTER_NODE_REMOVE_IC_VERSION 1
+/*
+ * Wire version.  v2 (spec-5.18 Hardening v1.1, HF-1): the announce payload
+ * carries removed_incarnation so a non-target survivor can seed its own
+ * membership_state[N]=REMOVED with the correct incarnation floor when it
+ * applies the removal locally (INV-LF11), not just drop its N-refs.
+ */
+#define CLUSTER_NODE_REMOVE_IC_VERSION 2
 
 /*
  * NODE_REMOVE_ANNOUNCE payload — coordinator -> all survivors (broadcast).
@@ -263,10 +269,12 @@ typedef struct ClusterNodeRemoveAnnouncePayload {
 	uint16 version;
 	uint16 _pad0;
 	int32 coordinator_node_id;
-	int32 target_node_id;	 /* who is removed */
-	uint64 remove_epoch;	 /* the removal reconfig new_epoch */
-	uint64 removal_event_id; /* identity bound to this removal attempt */
-	uint32 crc;				 /* CRC32C over [magic..removal_event_id] */
+	int32 target_node_id;		/* who is removed */
+	uint64 remove_epoch;		/* the removal reconfig new_epoch */
+	uint64 removal_event_id;	/* identity bound to this removal attempt */
+	uint64 removed_incarnation; /* HF-1: target's pinned incarnation floor (for
+								 * survivor-local membership REMOVED seed, INV-LF11) */
+	uint32 crc;					/* CRC32C over [magic..removed_incarnation] */
 } ClusterNodeRemoveAnnouncePayload;
 
 /*
@@ -435,7 +443,8 @@ extern bool cluster_node_remove_verify_no_leftover(int32 node_id);
  * msg-type registration block, postmaster phase 1). */
 extern void cluster_node_remove_register_ic_msg_types(void);
 extern void cluster_node_remove_ic_broadcast_announce(int32 target_node_id, uint64 remove_epoch,
-													  uint64 removal_event_id);
+													  uint64 removal_event_id,
+													  uint64 removed_incarnation);
 extern void cluster_node_remove_ic_send_ack(int32 dest_node_id, int32 target_node_id,
 											uint64 remove_epoch, uint64 removal_event_id);
 

src/include/cluster/cluster_reconfig.h

@@ -659,6 +659,8 @@ extern uint64 cluster_reconfig_get_clean_departed_count(void);
 extern void cluster_reconfig_record_removed(int32 node_id, uint64 remove_epoch,
 											bool raise_epoch_floor);
 extern bool cluster_reconfig_is_removed(int32 node_id);
+/* HF-2: lock-free durable removed test for the 53R64 self-demote write-gate hot path. */
+extern bool cluster_reconfig_is_removed_unlocked(int32 node_id);
 extern uint64 cluster_reconfig_get_removed_epoch(int32 node_id);
 extern uint64 cluster_reconfig_get_removed_count(void);
 extern void

src/test/cluster_tap/t/325_cluster_5_18_node_remove.pl

@@ -181,6 +181,34 @@ sub poll_until
 	'L8 node1 write fails with a cluster fail-closed error (never a silent success, INV-LF8)');
 
 
+# ----------
+# L12 (Hardening v1.1, HF-2): a removed node that RESTARTS must STAY removed.  On
+# bring-up node1 reads its own durable removal marker (removed_bitmap[self] +
+# membership_state[self]=REMOVED); its lmon self-state maintenance must NOT flip
+# that back to JOINING/MEMBER (before the fix the joiner / self-state path rewrote
+# self each tick, defeating the 53R64 self-demote write gate and letting a removed
+# node serve writes).  node0 stays up so node1 is held removed + fenced.
+# ----------
+$pair->node1->restart;
+usleep(6_000_000);    # several lmon ticks (1s heartbeat / 500ms qvotec poll)
+# node1's own view of itself stays removed across the tick churn (the HF-2 signal:
+# self membership_state is NOT flipped REMOVED -> JOINING/MEMBER).
+ok(poll_until($pair->node1,
+		q{SELECT state = 'removed' FROM pg_cluster_membership WHERE node_id = 1},
+		't', 30, 'L12 node1 self stays removed after restart'),
+	'L12 node1 self-state stays removed across lmon ticks after restart (HF-2)');
+# Forcing an xid assignment (txid_current) hits the AssignTransactionId
+# self-demote gate directly: a removed node must fail closed there (53R64),
+# proving cluster_node_remove_self_is_removed held across the tick churn (HF-2) and
+# was not defeated by a stale self-state write.
+my ($hf2rc, $hf2out, $hf2err) =
+  $pair->node1->psql('postgres', 'SELECT txid_current()');
+ok($hf2rc != 0, 'L12 restarted removed node fail-closes xid assignment (HF-2)');
+like(($hf2err // ''),
+	qr/53R64|removed/i,
+	'L12 restarted removed node xid assignment blocked by 53R64 self-demote gate (HF-2)');
+
+
 # ----------
 # L10 crash-recovery-from-marker (INV-LF7): restart node0; the durable §2.5
 # SHRUNK/REMOVED marker must rebuild removed_bitmap + membership REMOVED so node1