Skip to content

Prevent unnecessary RabbitMQ rolling restarts during scaling and cert rotation - #643

Open
lmiccini wants to merge 1 commit into
openstack-k8s-operators:mainfrom
lmiccini:fix-rabbitmq-tls-hash-rolling-restart
Open

Prevent unnecessary RabbitMQ rolling restarts during scaling and cert rotation#643
lmiccini wants to merge 1 commit into
openstack-k8s-operators:mainfrom
lmiccini:fix-rabbitmq-tls-hash-rolling-restart

Conversation

@lmiccini

Copy link
Copy Markdown
Contributor

Three changes to align with the upstream rabbitmq-cluster-operator and prevent quorum queue corruption caused by unnecessary rolling restarts:

  1. Remove TLS secret data and replica count from config hash

    Stop hashing TLS secret content (tlsSecretData, caSecretData) and the replica count into the pod-template annotation. The secret names are still hashed so that switching to a different cert triggers a restart, but content changes (cert-manager re-issuance, CA chain updates, rotation) no longer cause rolling restarts. TLS certificates are mounted via projected volumes that kubelet updates in-place, and RabbitMQ 4.x reloads them from disk automatically.

    During initial deployment the playbook scales RabbitMQ from 1 to 3 replicas, which adds new DNS SANs to the Certificate CR and triggers cert-manager to re-issue. When secret data was hashed, this caused a rolling restart before quorum queue replicas had synced, leading to Raft log divergence and a permanent crash loop (ra_server assertion "PLIdx < LastApplied").

  2. Make PreStop hook replica-count-independent

    The PreStop command previously varied based on spec.replicas (skipping quorum checks for single-replica clusters). This meant scaling from 1 to 3 changed the pod template, triggering a restart of existing pods. Now the quorum check is conditional at runtime: it queries rabbit_nodes:list_running() and only runs await_online_quorum_plus_one when more than one node is active.

    Timeouts are reduced from 600s to 10s for quorum checks and 30s for drain, fitting within the 60s terminationGracePeriodSeconds. If quorum is already satisfied (normal rolling restart), the check passes instantly; in degraded scenarios it times out quickly rather than blocking.

  3. Set explicit UpdateStrategy on the StatefulSet

    Always set RollingUpdate with Partition=0, matching the upstream cluster-operator. This prevents stale partition values (from migration or manual debugging) from silently blocking pod updates.

@openshift-ci
openshift-ci Bot requested review from abays and dciabrin July 28, 2026 08:42
@openshift-ci

openshift-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lmiccini

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

… rotation

Three changes to align with the upstream rabbitmq-cluster-operator and
prevent quorum queue corruption caused by unnecessary rolling restarts:

1. Remove TLS secret data and replica count from config hash

   Stop hashing TLS secret content (tlsSecretData, caSecretData) and
   the replica count into the pod-template annotation. The secret names
   are still hashed so that switching to a different cert triggers a
   restart, but content changes (cert-manager re-issuance, CA chain
   updates, rotation) no longer cause rolling restarts. TLS certificates
   are mounted via projected volumes that kubelet updates in-place, and
   RabbitMQ 4.x reloads them from disk automatically.

   During initial deployment the playbook scales RabbitMQ from 1 to 3
   replicas, which adds new DNS SANs to the Certificate CR and triggers
   cert-manager to re-issue. When secret data was hashed, this caused a
   rolling restart before quorum queue replicas had synced, leading to
   Raft log divergence and a permanent crash loop (ra_server assertion
   "PLIdx < LastApplied").

2. Make PreStop hook replica-count-independent

   The PreStop command previously varied based on spec.replicas (skipping
   quorum checks for single-replica clusters). This meant scaling from
   1 to 3 changed the pod template, triggering a restart of existing
   pods. Now the quorum check is conditional at runtime: it queries
   rabbit_nodes:list_running() and only runs await_online_quorum_plus_one
   when more than one node is active.

   Per-command timeouts are derived from TerminationGracePeriodSeconds
   using a proportional split (T/6 for quorum and mirror checks, T/2
   for drain) so that drain always gets a chance to run within the grace
   period. With the default 60s grace period this yields 10s/10s/30s.

3. Set explicit UpdateStrategy on the StatefulSet

   Always set RollingUpdate with Partition=0, matching the upstream
   cluster-operator. This prevents stale partition values (from migration
   or manual debugging) from silently blocking pod updates.

4. Add unit tests for PreStop hook and UpdateStrategy

   Cover default and custom TerminationGracePeriodSeconds values, and
   validate RollingUpdate with Partition=0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@lmiccini
lmiccini force-pushed the fix-rabbitmq-tls-hash-rolling-restart branch from ace25a4 to 0d008d5 Compare July 28, 2026 09:20
@lmiccini

Copy link
Copy Markdown
Contributor Author

/test infra-operator-build-deploy-kuttl

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/53660dab84234301ad76bad70b0fa2a1

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 15m 17s
podified-multinode-edpm-deployment-crc RETRY_LIMIT in 3m 47s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 46m 10s

@lmiccini

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/4a28530fcd25490abe6d48211d3d9be5

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 55m 23s
podified-multinode-edpm-deployment-crc RETRY_LIMIT in 3m 40s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 38m 38s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant