Prevent unnecessary RabbitMQ rolling restarts during scaling and cert rotation - #643
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: lmiccini The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
… rotation Three changes to align with the upstream rabbitmq-cluster-operator and prevent quorum queue corruption caused by unnecessary rolling restarts: 1. Remove TLS secret data and replica count from config hash Stop hashing TLS secret content (tlsSecretData, caSecretData) and the replica count into the pod-template annotation. The secret names are still hashed so that switching to a different cert triggers a restart, but content changes (cert-manager re-issuance, CA chain updates, rotation) no longer cause rolling restarts. TLS certificates are mounted via projected volumes that kubelet updates in-place, and RabbitMQ 4.x reloads them from disk automatically. During initial deployment the playbook scales RabbitMQ from 1 to 3 replicas, which adds new DNS SANs to the Certificate CR and triggers cert-manager to re-issue. When secret data was hashed, this caused a rolling restart before quorum queue replicas had synced, leading to Raft log divergence and a permanent crash loop (ra_server assertion "PLIdx < LastApplied"). 2. Make PreStop hook replica-count-independent The PreStop command previously varied based on spec.replicas (skipping quorum checks for single-replica clusters). This meant scaling from 1 to 3 changed the pod template, triggering a restart of existing pods. Now the quorum check is conditional at runtime: it queries rabbit_nodes:list_running() and only runs await_online_quorum_plus_one when more than one node is active. Per-command timeouts are derived from TerminationGracePeriodSeconds using a proportional split (T/6 for quorum and mirror checks, T/2 for drain) so that drain always gets a chance to run within the grace period. With the default 60s grace period this yields 10s/10s/30s. 3. Set explicit UpdateStrategy on the StatefulSet Always set RollingUpdate with Partition=0, matching the upstream cluster-operator. This prevents stale partition values (from migration or manual debugging) from silently blocking pod updates. 4. Add unit tests for PreStop hook and UpdateStrategy Cover default and custom TerminationGracePeriodSeconds values, and validate RollingUpdate with Partition=0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ace25a4 to
0d008d5
Compare
|
/test infra-operator-build-deploy-kuttl |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 15m 17s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 55m 23s |
Three changes to align with the upstream rabbitmq-cluster-operator and prevent quorum queue corruption caused by unnecessary rolling restarts:
Remove TLS secret data and replica count from config hash
Stop hashing TLS secret content (tlsSecretData, caSecretData) and the replica count into the pod-template annotation. The secret names are still hashed so that switching to a different cert triggers a restart, but content changes (cert-manager re-issuance, CA chain updates, rotation) no longer cause rolling restarts. TLS certificates are mounted via projected volumes that kubelet updates in-place, and RabbitMQ 4.x reloads them from disk automatically.
During initial deployment the playbook scales RabbitMQ from 1 to 3 replicas, which adds new DNS SANs to the Certificate CR and triggers cert-manager to re-issue. When secret data was hashed, this caused a rolling restart before quorum queue replicas had synced, leading to Raft log divergence and a permanent crash loop (ra_server assertion "PLIdx < LastApplied").
Make PreStop hook replica-count-independent
The PreStop command previously varied based on spec.replicas (skipping quorum checks for single-replica clusters). This meant scaling from 1 to 3 changed the pod template, triggering a restart of existing pods. Now the quorum check is conditional at runtime: it queries rabbit_nodes:list_running() and only runs await_online_quorum_plus_one when more than one node is active.
Timeouts are reduced from 600s to 10s for quorum checks and 30s for drain, fitting within the 60s terminationGracePeriodSeconds. If quorum is already satisfied (normal rolling restart), the check passes instantly; in degraded scenarios it times out quickly rather than blocking.
Set explicit UpdateStrategy on the StatefulSet
Always set RollingUpdate with Partition=0, matching the upstream cluster-operator. This prevents stale partition values (from migration or manual debugging) from silently blocking pod updates.