docs(redis): warn about sep change on a live Redis breaking binding lookups#2533
docs(redis): warn about sep change on a live Redis breaking binding lookups#2533notromdavid wants to merge 4 commits into
Conversation
…ookups
Changing `sep` in `broker_transport_options` after a Redis broker already
has `_kombu.binding.*` entries written with the previous separator causes
`get_table()` (redis.py) to produce 1-element tuples when it splits the
stored entries on the new separator. `DirectExchange.lookup()` at
`kombu/transport/virtual/exchange.py:67` then raises:
ValueError: not enough values to unpack (expected 3, got 1)
because it unpacks each entry as `(routing_key, _, queue)`.
Expand the bare `sep` bullet in the module-level Transport Options
docstring to document the failure mode and the two safe migration paths:
delete the stale `_kombu.binding.*` keys before the deploy, or cut over
to a fresh Redis logical database. Also document the default value and
note that it rarely needs to change.
No code changes — docs only.
There was a problem hiding this comment.
Pull request overview
Updates the Redis transport documentation to explain what the sep transport option does, its default value, and to warn operators about a migration hazard when changing sep on a live Redis broker (which can break binding lookups and crash workers).
Changes:
- Expands the
septransport option docstring entry with a detailed description and default. - Adds a
.. warning::admonition describing the failure mode whensepis changed after bindings already exist. - Documents a suggested mitigation (deleting existing binding keys or using a fresh Redis logical DB).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| from ``DirectExchange.lookup()`` | ||
| (``kombu/transport/virtual/exchange.py``, line 67) when those | ||
| stale entries are read. ``get_table()`` splits each stored entry | ||
| on the *new* separator; if the old separator is not present the | ||
| split returns a 1-element tuple rather than the required | ||
| ``(routing_key, pattern, queue)`` 3-tuple. |
| either delete the existing binding keys: | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| redis-cli --scan --pattern '_kombu.binding.*' | xargs redis-cli DEL |
There was a problem hiding this comment.
@notromdavid can you please cross check this and other suggestions?
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2533 +/- ##
=======================================
Coverage 82.53% 82.53%
=======================================
Files 79 79
Lines 10213 10213
Branches 1172 1172
=======================================
Hits 8429 8429
Misses 1582 1582
Partials 202 202 ☔ View full report in Codecov by Sentry. |
|
|
||
| .. code-block:: bash | ||
|
|
||
| redis-cli --scan --pattern '_kombu.binding.*' | xargs -r -n 100 redis-cli DEL |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Title (under 72 chars)
docs(redis): warn about sep change on a live Redis breaking binding lookups
Body
What
Expands the bare
sepbullet inkombu/transport/redis.py's module-levelTransport Options docstring to document the field separator, its default
value, and a
.. warning::admonition explaining the migration hazard whensepis changed on a live Redis deployment.Why
Changing
sepinbroker_transport_optionsafter a Redis broker alreadyhas
_kombu.binding.*entries written with the previous separator causesget_table()to split each stored entry on the new separator. If the oldseparator is not present in the stored string, the split returns a
1-element tuple.
DirectExchange.lookup()atkombu/transport/virtual/exchange.py:67then unpacks each table row as(routing_key, _, queue)and raises:The kombu code is correct — this is a deploy-state migration trap, not a
kombu bug. There is currently no documentation warning operators about
this failure mode. The symptom appears at runtime (workers crash on
celery inspect ping/ pidbox operations) with no indication that asepmismatch is the cause. A real-world hit is recorded at:https://github.com/notromdavid/HomeCooking/blob/main/ai-worker/app/celery_app.py#L60-L72
How tested
python3 -c "import ast; ast.parse(open('kombu/transport/redis.py').read())"— no errors.sphinx-build -b html docs/ _build/) completeswith zero warnings introduced by this change. Pre-existing warnings in
the repo (QoS.restore_visible title underline, SentinelChannel title
underline, ambiguous Channel cross-references) were present before this
edit and are unchanged.
repro.py) demonstratesthe exact
ValueErrorwhenDirectExchange.lookup()is called with atable built by splitting a
\x06\x16-separated binding string on:(1-tuple instead of 3-tuple). The same repro confirms that a fresh
binding set written with
sep=":"does not raise.CLA
Pending — maintainer (notromdavid) to sign before pushing the PR.
See: https://github.com/celery/celery/blob/main/CONTRIBUTING.rst for CLA
information (celery org shares a CLA process across repos; kombu is part
of the celery org).
Related
documentation gap, not a code bug)
https://github.com/notromdavid/HomeCooking/blob/main/ai-worker/app/celery_app.py#L60-L72