Fix Prometheus delayed_messages_inprogress gauge never returning to zero (#672)

[apoorvdarshan]

Closes #672

Root cause

The Prometheus middleware tracks dramatiq_delayed_messages_inprogress by incrementing the gauge in before_delay_message and decrementing it in before_process_message, keyed on (queue_name, actor_name) labels.

A delayed message lives on the ".DQ" queue while it waits for its eta, so when before_delay_message fires the message's queue_name is e.g. default.DQ, and the gauge is incremented for the label ("default.DQ", actor).

When the eta passes, Worker.handle_delayed_messages re-enqueues the message onto the canonical queue using q_name() (which strips the .DQ suffix). The re-enqueued message keeps its message_id, so before_process_message matches it and decrements the gauge — but now the message's queue_name is default, so the decrement lands on the label ("default", actor).

Because the increment and decrement use different queue_name labels, the ".DQ" series is only ever incremented and never settles back to zero (the "metric never goes down" behavior reported in the issue), while the canonical series drifts negative.

Reproduction

Driving a real Worker with a StubBroker and the Prometheus middleware, sending one delayed message and letting it run to completion, then dumping every child of the gauge:

do_work.send_with_options(delay=100)
broker.join(do_work.queue_name, timeout=5000)
worker.join()

for labels, child in prometheus.inprogress_delayed_messages._metrics.items():
    print(labels, "=", child._value.get())

Observed on master (before the fix) — the message is delayed on default.DQ but processed on default, so the two events hit different labels:

before_delay   queue=default.DQ  actor=do_work
before_process queue=default     actor=do_work

('default.DQ', 'do_work') = 1.0     <-- only ever incremented, never returns to zero
('default', 'do_work')    = -1.0

After the fix (both events recorded against the canonical queue name):

before_delay   queue=default.DQ  actor=do_work
before_process queue=default     actor=do_work

('default', 'do_work') = 0.0        <-- gauge settles back to zero

Fix

In before_delay_message, normalize the queue name with the existing q_name() helper before building the label tuple — the same helper the worker already uses when re-enqueueing the delayed message. This makes the increment and decrement land on the same label set. The change is a one-line label normalization plus the import; it does not alter the id-tracking or any other metric.

Tests

Added test_prometheus_middleware_delayed_messages_inprogress_returns_to_zero in tests/middleware/test_prometheus.py. It runs a delayed message through a real worker and asserts every inprogress_delayed_messages child returns to 0 (and that the in-memory delayed_messages set does not leak). The test fails on master (the ('default.DQ', 'do_work') child is stuck at 1.0) and passes with this change.

The middleware test module and a broad stub/worker/message/common smoke subset pass locally, and flake8, black, isort and mypy are clean on the modified files.

Disclosure: prepared with AI assistance; reviewed and verified locally.

[LincolnPuzey]

You claim

reviewed and verified locally.

Yet you opened 67 Pull requests with similar claims, on the same day as this one, in a 4-hour period.

Please don't include us in your LLM-spam list.