LOGC-63: Enable rdkafka cgrp debug logs on workbench crr-queue-processor

Add a workflow step that appends RDKAFKA_DEBUG_LOGS=cgrp to the
backbeat container's /conf/env and SIGKILLs the queue-processor so
supervisord respawns it with the new env. With this set, librdkafka
surfaces consumer-group state transitions (JoinGroup/SyncGroup/
Heartbeat/rebalance triggers) as rdkafka.log events in the process
stdout, which are dumped by the existing "Dump backbeat logs on
failure" step. Needed to identify the broker-side cause of the 1s
rebalance loop the next time the CRR e2e flake reproduces.

The env mutation has to live in CI rather than supervisord.conf
because env/default/config/backbeat/supervisord.conf is generated
by workbench's docker-compose action on each run. Scope kept to
cgrp to keep log volume manageable. Workbench-only; production
deployments use Federation and are unaffected.

Author: dvasilas

.github/scripts/enable-qproc-rdkafka-debug.sh

@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+# Enable librdkafka cgrp debug logs on the workbench backbeat
+# crr-queue-processor so consumer-group state transitions
+# (JoinGroup/SyncGroup/Heartbeat/rebalance triggers) surface as
+# rdkafka.log events in the process stdout. Needed to diagnose an
+# intermittent 1s rebalance loop that leaves CRR e2e tests stuck at
+# ReplicationStatus=PENDING.
+#
+# We can't bake this into the supervisord.conf because that file is
+# generated by workbench's docker-compose action. Instead, append the
+# export to /conf/env inside the container and SIGKILL the QProc;
+# supervisord's autorestart=true respawns it, the new instance sources
+# /conf/env and starts with RDKAFKA_DEBUG_LOGS=cgrp set.
+#
+# Workbench-only; production deployments use Federation and are
+# unaffected.
+set -e
+
+echo "Enabling librdkafka cgrp debug on crr-queue-processor..."
+
+docker exec workbench-backbeat sh -c \
+  'grep -q "RDKAFKA_DEBUG_LOGS" /conf/env || echo "export RDKAFKA_DEBUG_LOGS=cgrp" >> /conf/env'
+
+proc=$(docker exec workbench-backbeat supervisord ctl -c /conf/supervisord.conf status \
+  | sed 's/\x1b\[[0-9;]*m//g' \
+  | awk '/^crr-queue-processor_[0-9]+[[:space:]]/ {print $1; exit}')
+if [ -z "$proc" ]; then
+  echo "ERROR: crr-queue-processor not found in supervisord status"
+  docker exec workbench-backbeat supervisord ctl -c /conf/supervisord.conf status || true
+  exit 1
+fi
+
+# Count current "ready to consume" markers; after restart we expect
+# this count to grow by one.
+prev_ready=$(docker exec workbench-backbeat sh -c \
+  "cat /logs/crr-queue-processor_*.log 2>/dev/null | grep -c 'queue processor is ready to consume replication entries' || true")
+
+docker exec workbench-backbeat supervisord ctl -c /conf/supervisord.conf signal KILL "$proc" || true
+
+# Wait for the respawned instance to log a fresh ready marker.
+for i in {1..60}; do
+  sleep 1
+  new_ready=$(docker exec workbench-backbeat sh -c \
+    "cat /logs/crr-queue-processor_*.log 2>/dev/null | grep -c 'queue processor is ready to consume replication entries' || true")
+  if [ "${new_ready:-0}" -gt "${prev_ready:-0}" ]; then
+    echo "✓ crr-queue-processor restarted with RDKAFKA_DEBUG_LOGS=cgrp"
+    exit 0
+  fi
+done
+
+echo "ERROR: crr-queue-processor did not re-emit 'ready to consume' within 60s after restart"
+docker exec workbench-backbeat supervisord ctl -c /conf/supervisord.conf status || true
+exit 1

.github/workflows/e2e-tests.yaml

@@ -41,6 +41,9 @@ jobs:
       - name: Start Workbench
         uses: scality/workbench@v0.16.0
 
+      - name: Enable rdkafka cgrp debug on crr-queue-processor
+        run: ./.github/scripts/enable-qproc-rdkafka-debug.sh
+
       - name: Wait for all services
         run: |
           set -e