Add ControllerEventQueueSizeGauge to detect a wedged ("zombie leader" controller) by LZD-PratyushBhatt · Pull Request #201 · linkedin/helix

LZD-PratyushBhatt · 2026-06-18T08:50:40Z

Issues

My PR addresses the following Helix issues and references them in the PR description:

Fixes #200

Description

Here are some details about my PR:

What: Add a new per-cluster controller JMX gauge, ControllerEventQueueSizeGauge, on
ClusterStatusMonitor, exposing the depth of the DEFAULT cluster-event pipeline queue
(GenericHelixController._eventQueue).

Why: The controller pipeline is drained by a single ClusterEventProcessor thread while
the ZK session is kept alive by a separate ZK-client thread. As a result a wedged controller
can keep leadership and a live ZK session yet stop processing events entirely. None of the
existing controller signals (ZK session expiries, controllership-takeover failures, ZK op
latency, WAGED internal failure) catch this, because the process stays up and remains leader.
This "zombie leader" failure mode is currently invisible to monitoring.

How: Expose the DEFAULT cluster-event pipeline backlog as a per-cluster JMX gauge on
ClusterStatusMonitor, updated from GenericHelixController on both the enqueue side
(ZK-callback / periodic-rebalance threads, in enqueueEvent when the target is _eventQueue)
and the dequeue side (the pipeline thread, in ClusterEventProcessor.run() after take() for
the DEFAULT queue). Updates are guarded by _isMonitoring and a null check on the monitor.

A healthy controller keeps this near 0: the queue dedups by event type
(DedupEventBlockingQueue), so depth saturates at the few distinct pending event types and the
pipeline drains them quickly. A wedged controller lets it climb and stay elevated. EKG / alerts
should therefore gate on a sustained average above ~0 (not MAX), with a small threshold.

Sensor / attribute exposed: ObjectName ClusterStatus:cluster={cluster}, attribute
ControllerEventQueueSizeGauge.

Files changed:

helix-core/.../monitoring/mbeans/ClusterStatusMonitor.java - new AtomicLong field, setter
setControllerEventQueueSizeGauge(long), getter getControllerEventQueueSizeGauge().
helix-core/.../monitoring/mbeans/ClusterStatusMonitorMBean.java - getter declared on the MBean
interface (auto-exports the JMX attribute).
helix-core/.../controller/GenericHelixController.java - updateControllerEventQueueSizeGauge()
helper, invoked at enqueue and dequeue of the DEFAULT _eventQueue.

Tests

The following tests are written for this issue:
TestClusterStatusMonitor#testControllerEventQueueSizeGaugeStartsAtZero
TestClusterStatusMonitor#testControllerEventQueueSizeGaugeReflectsLatestSetter
The following is the result of the "mvn test" command on the appropriate module
(mvn test -pl helix-core -Dtest=TestClusterStatusMonitor):

[INFO] Running org.apache.helix.monitoring.mbeans.TestClusterStatusMonitor
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.226 s - in org.apache.helix.monitoring.mbeans.TestClusterStatusMonitor
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

Changes that Break Backward Compatibility (Optional)

My PR contains changes that break backward compatibility or previous assumptions for certain methods or API.

No backward-incompatible behavior changes. The only API surface change is additive: a new
read-only getter getControllerEventQueueSizeGauge() on the ClusterStatusMonitorMBean JMX
interface. No existing method signatures are changed or removed, and the sole implementor
(ClusterStatusMonitor) is updated in this PR. The new value defaults to 0 until the controller
reports it, so existing consumers and dashboards are unaffected.

Documentation (Optional)

In case of new functionality, my PR adds documentation in the following wiki page:

Not applicable. The change adds a single self-describing JMX gauge (documented via Javadoc on the
MBean interface); no public wiki change is required.

Commits

My commits all reference appropriate Apache Helix GitHub issues in their subject lines.
In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Note: the current commit subject is
Add ControllerEventQueueSizeGauge to detect a wedged ("zombie leader") controller.
It uses the imperative mood, has a blank line before the body, does not end with a period, and
the body explains what/why. It does not yet reference #200 and exceeds 50 characters. Amend
to e.g. Fix #200: add ControllerEventQueueSizeGauge for wedged controllers before merge if
strict compliance is required.

Code Quality

My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)

…) controller The controller pipeline is drained by a single ClusterEventProcessor thread while the ZK session is kept alive by a separate ZK-client thread, so a wedged controller can keep leadership and a live ZK session yet stop processing events entirely. None of the existing controller signals (ZK session expiries, controllership-takeover failures, ZK op latency, WAGED internal failure) catch this, since the process stays up and leader. Expose the DEFAULT cluster-event pipeline backlog as a per-cluster JMX gauge on ClusterStatusMonitor, updated from GenericHelixController on both the enqueue side (ZK-callback / periodic-rebalance threads) and the dequeue side (pipeline thread). A healthy controller keeps this near 0 (the queue dedups by event type, so depth saturates at the few distinct pending event types); a wedged controller lets it climb. EKG/alerts should gate on a sustained average above ~0. Add unit tests for the gauge default and reversible setter.

kabragaurav · 2026-06-23T05:56:24Z

The CI is failing but Copilot analysis tells that this failure is not because of this PR's changes.

kabragaurav · 2026-06-23T06:10:31Z

      _maxInstanceMsgQueueSize.set(0L);
      _totalPastDueMsgSize.set(0L);
      _totalMsgQueueSize.set(0L);
      _rebalanceFailureCount.set(0L);


reset() runs on leadership change/monitor reset and zeros every counter. So stale numbers from a prior leadership period aren't attributed to the new one._controllerEventQueueSizeGauge appears only at its declaration, the setter, and the getter - it is not in reset().

Add this line:

_controllerEventQueueSizeGauge.set(0L);

Good point, done

kabragaurav · 2026-06-23T06:17:57Z

+   * Backlog of the DEFAULT controller cluster-event pipeline (events enqueued but not yet
+   * processed). Stays near 0 on a healthy controller because the pipeline drains quickly and the
+   * queue dedups by event type; climbs when the controller holds leadership but has stopped
+   * processing events ("zombie leader"). Alert on a sustained average above ~0.


We should combine this gauge with the progress signal Helix already has, ClusterEventStatus...TotalProcessed.EventCounter

The disambiguator is to combine this gauge with the progress signal Helix already has, ClusterEventStatus...TotalProcessed.EventCounter:

Idle + healthy: queue depth 0, counter flat.

Busy + healthy: queue depth occasionally > 0, counter advancing.

Wedged (zombie leader): queue depth sustained > 0, counter NOT advancing (and InQueue wait-time climbing).

So the robust alert is the conjunction: ControllerEventQueueSizeGauge sustained > 0 AND TotalProcessed.EventCounter rate ~= 0.

Yeah, that will be done on alerting end. I've updated the java doc

kabragaurav · 2026-06-23T06:25:30Z

    Assert.assertEquals(monitor.getWagedFallbackInUseGauge(), 0L);
  }

+  @Test


We should add tests that reflect this PR's behaviour like:

that enqueueing to the DEFAULT _eventQueue makes the gauge climb, and dequeueing brings it down

that only the DEFAULT queue updates it and the TASK queue (_taskEventQueue) does not (the queue == _eventQueue / _eventBlockingQueue == _eventQueue guards — the easiest thing to get wrong here, and untestable by the current tests)

that the value resets to 0 on leadership change (comment 2 — and a test here would have caught that the reset is missing)

nqueue several distinct event types without draining, assert the gauge reflects the backlog depth.

I've added for some which looked related. The rest (gauge climbs on enqueue / drops on dequeue, DEFAULT-vs-TASK queue discrimination, backlog depth across distinct event types) touch the GenericHelixController enqueue/dequeue wiring, so I'll cover those as a follow-up integration test.

Zero _controllerEventQueueSizeGauge in ClusterStatusMonitor.reset() so a backlog from a prior leadership period is not re-reported by the reused monitor instance after re-election, matching the existing rebalance/WAGED counter reset semantics. Add a regression test that sets the gauge, calls reset(), and asserts it returns to 0. Add a brief MBean Javadoc note that depth alone is ambiguous (pair with ClusterEventStatus...TotalProcessed.EventCounter) and that the alert threshold/windowing belongs in the alerting layer, keeping the operational detail in the PR discussion rather than in source.

…line Complements ControllerEventQueueSizeGauge. Raw queue depth alone cannot tell a wedged controller from a busy-but-progressing one: the event queue dedups by event type (depth saturates at the few distinct pending types), and a producer faster than the single processor thread can keep it non-empty while the controller is still making progress. This adds a reversible 0/1 gauge that is 1 only when the DEFAULT event queue is non-empty AND no pipeline run has completed within a stall threshold (5s for now), i.e. the controller holds events but is not draining them. It is computed lazily in the JMX getter from the queue-size gauge plus a last-pipeline- completion timestamp reported by GenericHelixController, so it stays correct even if the pipeline thread is dead and no setter runs. Producer-rate- independent; gate EKG/alerts on == 1. Add unit tests for the idle, busy-but-progressing, no-baseline, wedged, and leadership-reset cases.

Replace the hardcoded 5s stall threshold with a per-cluster ClusterConfig value (CONTROLLER_PIPELINE_STALL_THRESHOLD_MS, default 5000ms) so ControllerPipelineStalledGauge can be tuned without a redeploy. GenericHelixController reads the value from ClusterConfig on each DEFAULT pipeline completion and pushes it to ClusterStatusMonitor, which now holds the threshold in a field (defaulting until first reported) and uses it in the lazily-computed stalled gauge. Non-positive values are ignored so a misconfiguration cannot silently disable the gauge. Add ClusterConfig getter/setter + default/round-trip test and a monitor test asserting the configured threshold changes the gauge result.

ControllerPipelineStalledGauge returns 0 while its last-completion timestamp is 0 (no baseline), so a controller that wedges before completing its very first pipeline run after acquiring leadership (e.g. a freshly-migrated Customer cluster whose first run hangs) would never be flagged. Seed the baseline to the current time when monitoring is enabled (leadership acquired) so "no pipeline completed since becoming active" is measurable and such a wedge is caught after the stall threshold. Cold-start slowness that could briefly read as stalled is covered by the EKG warm-up window and the configurable threshold.

LZD-PratyushBhatt requested review from PranaviAncha, arkmish, kabragaurav, laxman-ch, ngngwr, proud-parselmouth and thestreak101 as code owners June 18, 2026 08:50

kabragaurav reviewed Jun 23, 2026

View reviewed changes

LZD-PratyushBhatt requested a review from kabragaurav June 23, 2026 16:58

LZD-PratyushBhatt requested a review from sjainit as a code owner July 3, 2026 15:03

LZD-PratyushBhatt added 2 commits July 3, 2026 21:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add ControllerEventQueueSizeGauge to detect a wedged ("zombie leader" controller)#201

Add ControllerEventQueueSizeGauge to detect a wedged ("zombie leader" controller)#201
LZD-PratyushBhatt wants to merge 5 commits into
devfrom
lzd/controllerEventQueueMetric

LZD-PratyushBhatt commented Jun 18, 2026 •

edited

Loading

Uh oh!

kabragaurav commented Jun 23, 2026

Uh oh!

kabragaurav Jun 23, 2026

Uh oh!

LZD-PratyushBhatt Jun 23, 2026

Uh oh!

kabragaurav Jun 23, 2026

Uh oh!

LZD-PratyushBhatt Jun 23, 2026

Uh oh!

kabragaurav Jun 23, 2026

Uh oh!

LZD-PratyushBhatt Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

LZD-PratyushBhatt commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issues

Description

Tests

Changes that Break Backward Compatibility (Optional)

Documentation (Optional)

Commits

Code Quality

Uh oh!

kabragaurav commented Jun 23, 2026

Uh oh!

kabragaurav Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

LZD-PratyushBhatt Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

kabragaurav Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

LZD-PratyushBhatt Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

kabragaurav Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

LZD-PratyushBhatt Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LZD-PratyushBhatt commented Jun 18, 2026 •

edited

Loading