Skip to content

[13.x] Add first-class Redis Cluster support for Queue and ConcurrencyLimiter#59533

Merged
taylorotwell merged 7 commits into
laravel:13.xfrom
timmylindh:feature/redis-cluster
Apr 9, 2026
Merged

[13.x] Add first-class Redis Cluster support for Queue and ConcurrencyLimiter#59533
taylorotwell merged 7 commits into
laravel:13.xfrom
timmylindh:feature/redis-cluster

Conversation

@timmylindh
Copy link
Copy Markdown
Contributor

@timmylindh timmylindh commented Apr 5, 2026

When using AWS ElastiCache Serverless (Valkey) or any Redis Cluster deployment, Laravel's Redis Queue and
ConcurrencyLimiter fail with CROSSSLOT errors because their Lua scripts operate on keys that hash to different cluster slots. Even the Laravel Redis tests explicitly set REDIS_QUEUE: {default} to work-around this issue.

For example, the queue pop() Lua script atomically reads from queues:default, writes to queues:default:reserved, and pops from queues:default:notify — three keys that land on three different slots.

This PR makes these subsystems cluster-safe by automatically wrapping queue names in Redis hash tags when the connection is a cluster, ensuring all related keys hash to the same slot:

queues:{default} → slot for "default"
queues:{default}:delayed → slot for "default"
queues:{default}:reserved → slot for "default"
queues:{default}:notify → slot for "default"

Different queues ({emails}, {notifications}, {exports}) still distribute across the cluster naturally. This is the
same pattern used by Sidekiq, BullMQ, and other cluster-aware queue systems.

Changes

Redis Queue (RedisQueue)

  • Added getRedisKey() method that wraps queue names in {...} hash tags when on a cluster connection
  • Kept getQueue() unchanged — payload hooks and events still see the original format, which avoids breaking Horizon or other packages that consume the queue name
  • Replaced getQueue() with getRedisKey() in all Redis operation methods (pushRaw, laterRaw, size, pop, clear, etc.)
  • Added isClusterConnection() with cached result for the lifetime of the queue instance

ConcurrencyLimiter

  • Slot keys are now wrapped in hash tags on cluster connections ({limiter}1, {limiter}2 instead of limiter1,
    limiter2) so the mget Lua script doesn't cross slots
  • The ARGV prefix passed to the Lua script matches the key format so release() uses the correct key

Connection infrastructure

  • Added Connection::isCluster() method (returns false by default, true in PhpRedisClusterConnection and
    PredisClusterConnection)
  • Added Connection::hasHashTag() static helper for detecting existing {...} hash tags in key names

PhpRedis cluster connector

  • Added ACL auth support (['username', 'password'] array format) for cluster connections via formatClusterPassword(), gated to phpredis ≥ 5.3.2
  • Added max_retries, backoff_algorithm, backoff_base, backoff_cap options to cluster instances (parity with standalone connections) — important for ElastiCache Serverless which returns transient errors during scaling

Backward Compatibility

  • Non-cluster users: Zero change. getRedisKey() returns the same keys as getQueue() when not on a cluster.
  • Cluster users with REDIS_QUEUE={default}: The existing workaround continues to work — hash tag detection prevents double-wrapping.
  • Horizon: getQueue() is unchanged, so Horizon's payload hooks and event listeners see the same queue name format.
    Horizon's readyNow() method would need a one-line update in a follow-up Horizon PR to call the new getRedisKey().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants