Skip to content

docs(redis): warn about sep change on a live Redis breaking binding lookups#2533

Open
notromdavid wants to merge 4 commits into
celery:mainfrom
notromdavid:docs/redis-priority-sep-migration-warning
Open

docs(redis): warn about sep change on a live Redis breaking binding lookups#2533
notromdavid wants to merge 4 commits into
celery:mainfrom
notromdavid:docs/redis-priority-sep-migration-warning

Conversation

@notromdavid
Copy link
Copy Markdown

Title (under 72 chars)

docs(redis): warn about sep change on a live Redis breaking binding lookups

Body

What

Expands the bare sep bullet in kombu/transport/redis.py's module-level
Transport Options docstring to document the field separator, its default
value, and a .. warning:: admonition explaining the migration hazard when
sep is changed on a live Redis deployment.

Why

Changing sep in broker_transport_options after a Redis broker already
has _kombu.binding.* entries written with the previous separator causes
get_table() to split each stored entry on the new separator. If the old
separator is not present in the stored string, the split returns a
1-element tuple. DirectExchange.lookup() at
kombu/transport/virtual/exchange.py:67 then unpacks each table row as
(routing_key, _, queue) and raises:

ValueError: not enough values to unpack (expected 3, got 1)

The kombu code is correct — this is a deploy-state migration trap, not a
kombu bug. There is currently no documentation warning operators about
this failure mode. The symptom appears at runtime (workers crash on
celery inspect ping / pidbox operations) with no indication that a
sep mismatch is the cause. A real-world hit is recorded at:
https://github.com/notromdavid/HomeCooking/blob/main/ai-worker/app/celery_app.py#L60-L72

How tested

  • Python AST parse clean: python3 -c "import ast; ast.parse(open('kombu/transport/redis.py').read())" — no errors.
  • Sphinx HTML build (sphinx-build -b html docs/ _build/) completes
    with zero warnings introduced by this change. Pre-existing warnings in
    the repo (QoS.restore_visible title underline, SentinelChannel title
    underline, ambiguous Channel cross-references) were present before this
    edit and are unchanged.
  • No code changes — docs only (module-level docstring only).
  • Empirical confirmation: a minimal repro (repro.py) demonstrates
    the exact ValueError when DirectExchange.lookup() is called with a
    table built by splitting a \x06\x16-separated binding string on :
    (1-tuple instead of 3-tuple). The same repro confirms that a fresh
    binding set written with sep=":" does not raise.

CLA

Pending — maintainer (notromdavid) to sign before pushing the PR.
See: https://github.com/celery/celery/blob/main/CONTRIBUTING.rst for CLA
information (celery org shares a CLA process across repos; kombu is part
of the celery org).

Related

…ookups

Changing `sep` in `broker_transport_options` after a Redis broker already
has `_kombu.binding.*` entries written with the previous separator causes
`get_table()` (redis.py) to produce 1-element tuples when it splits the
stored entries on the new separator.  `DirectExchange.lookup()` at
`kombu/transport/virtual/exchange.py:67` then raises:

    ValueError: not enough values to unpack (expected 3, got 1)

because it unpacks each entry as `(routing_key, _, queue)`.

Expand the bare `sep` bullet in the module-level Transport Options
docstring to document the failure mode and the two safe migration paths:
delete the stale `_kombu.binding.*` keys before the deploy, or cut over
to a fresh Redis logical database.  Also document the default value and
note that it rarely needs to change.

No code changes — docs only.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Redis transport documentation to explain what the sep transport option does, its default value, and to warn operators about a migration hazard when changing sep on a live Redis broker (which can break binding lookups and crash workers).

Changes:

  • Expands the sep transport option docstring entry with a detailed description and default.
  • Adds a .. warning:: admonition describing the failure mode when sep is changed after bindings already exist.
  • Documents a suggested mitigation (deleting existing binding keys or using a fresh Redis logical DB).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread kombu/transport/redis.py Outdated
Comment on lines +44 to +49
from ``DirectExchange.lookup()``
(``kombu/transport/virtual/exchange.py``, line 67) when those
stale entries are read. ``get_table()`` splits each stored entry
on the *new* separator; if the old separator is not present the
split returns a 1-element tuple rather than the required
``(routing_key, pattern, queue)`` 3-tuple.
Comment thread kombu/transport/redis.py Outdated
Comment on lines +52 to +56
either delete the existing binding keys:

.. code-block:: bash

redis-cli --scan --pattern '_kombu.binding.*' | xargs redis-cli DEL
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@notromdavid can you please cross check this and other suggestions?

Comment thread kombu/transport/redis.py Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 29, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.53%. Comparing base (e66e4c6) to head (f7f7c1a).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2533   +/-   ##
=======================================
  Coverage   82.53%   82.53%           
=======================================
  Files          79       79           
  Lines       10213    10213           
  Branches     1172     1172           
=======================================
  Hits         8429     8429           
  Misses       1582     1582           
  Partials      202      202           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@auvipy auvipy added this to the 5.7.0 milestone May 7, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Comment thread kombu/transport/redis.py Outdated
Comment thread kombu/transport/redis.py

.. code-block:: bash

redis-cli --scan --pattern '_kombu.binding.*' | xargs -r -n 100 redis-cli DEL
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants