Add Redis scheduler backend with distributed leadership support#6
Merged
Conversation
Introduces a Redis-powered distributed scheduler backend (RedisSchedulerBackend) with leader election and state synchronization for scheduled tasks. Updates the beat loop to support distributed backends, adds schedule descriptors for serialization, and extends cron/interval schedule types for persistence. Includes a new example (redis_beat.rs) and updates documentation for usage.
Added leader and follower sleep hint logic to the RedisSchedulerBackend, ensuring more responsive failover and efficient scheduling. Updated documentation in README and beat module to clarify backend selection and failover testing. Also added a new TickDecision::execute_with_hint method for more flexible scheduling.
Added [project] section to pyproject.toml with project name, version, Python requirement, and dependencies on celery and redis. This sets up the initial project configuration for dependency management.
There was a problem hiding this comment.
Pull Request Overview
This PR adds Redis-backed distributed scheduler functionality with leadership coordination for Celery beat instances. The implementation enables multiple beat instances to coordinate task scheduling through Redis, with automatic leader election and failover capabilities.
Key changes:
- Added
RedisSchedulerBackendwith distributed leadership support via Redis locks - Extended the scheduler architecture with
DistributedSchedulertrait and optional hooks - Introduced schedule descriptors for serializing/deserializing schedule state in Redis
Reviewed Changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/error.rs | Added RedisError variant to handle Redis-specific errors |
| src/beat/schedule/mod.rs | Extended Schedule trait with serialization support and added Any bound |
| src/beat/schedule/descriptor.rs | New module for serializable schedule descriptors with roundtrip conversion |
| src/beat/schedule/cron/*.rs | Added serialization support methods to cron time units and schedule |
| src/beat/mod.rs | Updated beat loop to support distributed scheduling decisions |
| src/beat/backend/redis.rs | Complete Redis backend implementation with leadership coordination |
| src/beat/backend.rs | Added distributed scheduler traits and decision structures |
| examples/redis_beat.rs | Example demonstrating Redis-backed beat usage |
| pyproject.toml | Added Python dependencies for examples |
| README.md | Updated documentation with Redis backend information |
| Cargo.toml | Added Redis script feature and redis_beat example |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting
Introduced a Redis-based scheduler backend with configurable lock management and task state persistence. Added documentation and examples for multi-instance beat deployments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Added a Redis-backed distributed scheduler backend with leadership coordination and Redis state persistence, including config builder, lock handling, and task synchronization logic (src/beat/backend.rs, src/beat/backend/redis.rs, src/beat/schedule/**).
Extended SchedulerBackend with optional distributed hooks; updated beat loop to honor distributed decisions while keeping legacy backends unchanged.
Introduced schedule descriptors, examples (examples/redis_beat.rs), documentation updates, and helper tests/config for multi-instance scenarios (README.md, pyproject.toml, src/beat/mod.rs).
Testing
cargo fmt
cargo check
AMQP_ADDR=amqp://localhost:5672 cargo test --lib
AMQP_ADDR=amqp://localhost:5672 cargo test --test codegen