Skip to content

Commit 5c50ffc

Browse files
Morry98Morry98
authored
Improve task management with Redis Streams, leader election and management API (#2)
## Summary Major overhaul of fastapi-task-manager introducing a more robust and scalable architecture: - **Redis Streams mode**: Replaced the legacy polling mode with Redis Streams + consumer groups for distributed task execution, ensuring reliable message delivery and at-least-once processing - **Leader election**: Added distributed leader election via Redis so only one instance schedules tasks across multiple app replicas - **Reconciler & heartbeat**: New reconciler component detects and recovers stale/failed tasks; tasks now emit heartbeats to track liveness - **Exponential backoff retry**: Failed tasks are automatically retried with configurable exponential backoff - **Dynamic task CRUD API**: New endpoints for creating, updating, and deleting tasks at runtime with Redis persistence - **Revamped management API**: Async bulk operations for task groups and tasks (enable/disable, trigger, health checks) - **Statistics via Redis Streams**: Migrated statistics storage from Redis Lists to Streams for better performance and queryability - **Worker identity**: Each worker instance now has a unique identity tracked via `WorkerIdentity` schema - **Centralized Redis keys**: All Redis key patterns consolidated in `redis_keys.py` - **Simplified config**: Removed redundant TTL config fields; renamed `app_name` → `redis_key_prefix` and `log_level` config - **Comprehensive test suite**: Added 12 test modules covering all new components (~3,500+ lines of tests) - **Full documentation site**: MkDocs Material docs with getting started guide, architecture overview, API reference, deployment guide, and dynamic tasks tutorial - **CI workflow**: Added GitHub Actions CI pipeline with Python 3.14 support - **Pre-commit improvements**: Migrated from mypy+vermin to Astral's `ty`, added bandit security linter
1 parent d04f85d commit 5c50ffc

77 files changed

Lines changed: 11867 additions & 561 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
pre-commit:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- uses: tox-dev/action-pre-commit-uv@v1
15+
16+
test:
17+
runs-on: ubuntu-latest
18+
strategy:
19+
matrix:
20+
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
21+
22+
steps:
23+
- uses: actions/checkout@v4
24+
25+
- name: Install uv
26+
uses: astral-sh/setup-uv@v6
27+
28+
- name: Set up Python ${{ matrix.python-version }}
29+
run: uv python install ${{ matrix.python-version }}
30+
31+
- name: Install dependencies
32+
run: uv sync
33+
34+
- name: Run tests with coverage
35+
run: uv run coverage run -m pytest
36+
37+
- name: Generate lcov report
38+
run: uv run coverage lcov -o coverage.lcov
39+
40+
- name: Upload coverage to Coveralls
41+
uses: coverallsapp/github-action@v2
42+
with:
43+
github-token: ${{ secrets.GITHUB_TOKEN }}
44+
file: coverage.lcov
45+
flag-name: python-${{ matrix.python-version }}
46+
parallel: true
47+
48+
coveralls-finished:
49+
needs: test
50+
if: always()
51+
runs-on: ubuntu-latest
52+
steps:
53+
- name: Coveralls finished
54+
uses: coverallsapp/github-action@v2
55+
with:
56+
github-token: ${{ secrets.GITHUB_TOKEN }}
57+
parallel-finished: true

‎.github/workflows/publish.yml‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
name: Publish
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
9+
jobs:
10+
check-version:
11+
runs-on: ubuntu-latest
12+
outputs:
13+
should_publish: ${{ steps.check.outputs.should_publish }}
14+
version: ${{ steps.check.outputs.version }}
15+
is_official: ${{ steps.check.outputs.is_official }}
16+
steps:
17+
- uses: actions/checkout@v4
18+
with:
19+
fetch-depth: 0
20+
21+
- name: Install uv
22+
uses: astral-sh/setup-uv@v6
23+
24+
# Extract version from pyproject.toml, check if tag exists, and determine version type
25+
- name: Check version
26+
id: check
27+
run: |
28+
VERSION=$(uv run python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
29+
TAG="v${VERSION}"
30+
echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
31+
32+
# Check if version matches x.y.z (official) or has pre-release suffix
33+
if echo "$VERSION" | grep -qP '^\d+\.\d+\.\d+$'; then
34+
echo "is_official=true" >> "$GITHUB_OUTPUT"
35+
echo "Version $VERSION is an official release"
36+
else
37+
echo "is_official=false" >> "$GITHUB_OUTPUT"
38+
echo "Version $VERSION is a pre-release"
39+
fi
40+
41+
if git tag -l "$TAG" | grep -q "$TAG"; then
42+
echo "Tag $TAG already exists, skipping publish"
43+
echo "should_publish=false" >> "$GITHUB_OUTPUT"
44+
else
45+
echo "Tag $TAG does not exist, will publish"
46+
echo "should_publish=true" >> "$GITHUB_OUTPUT"
47+
fi
48+
49+
# Pre-release versions (e.g. 1.0.0-rc.1, 1.0.0-beta.1) are published from PRs
50+
publish-prerelease:
51+
needs: check-version
52+
if: >-
53+
needs.check-version.outputs.should_publish == 'true'
54+
&& needs.check-version.outputs.is_official == 'false'
55+
&& github.event_name == 'pull_request'
56+
runs-on: ubuntu-latest
57+
environment:
58+
name: pypi
59+
permissions:
60+
id-token: write
61+
contents: write
62+
steps:
63+
- uses: actions/checkout@v4
64+
65+
- name: Install uv
66+
uses: astral-sh/setup-uv@v6
67+
68+
- name: Build
69+
run: uv build
70+
71+
- name: Publish to PyPI
72+
run: uv publish
73+
74+
- name: Create git tag and GitHub Release
75+
env:
76+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
77+
run: |
78+
TAG="v${{ needs.check-version.outputs.version }}"
79+
git tag "$TAG"
80+
git push origin "$TAG"
81+
gh release create "$TAG" dist/* --generate-notes --prerelease
82+
83+
# Official versions (x.y.z) are published only from pushes to main
84+
publish:
85+
needs: check-version
86+
if: >-
87+
needs.check-version.outputs.should_publish == 'true'
88+
&& needs.check-version.outputs.is_official == 'true'
89+
&& github.event_name == 'push'
90+
runs-on: ubuntu-latest
91+
environment:
92+
name: pypi
93+
permissions:
94+
id-token: write
95+
contents: write
96+
steps:
97+
- uses: actions/checkout@v4
98+
99+
- name: Install uv
100+
uses: astral-sh/setup-uv@v6
101+
102+
- name: Build
103+
run: uv build
104+
105+
- name: Publish to PyPI
106+
run: uv publish
107+
108+
- name: Create git tag and GitHub Release
109+
env:
110+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
111+
run: |
112+
TAG="v${{ needs.check-version.outputs.version }}"
113+
git tag "$TAG"
114+
git push origin "$TAG"
115+
gh release create "$TAG" dist/* --generate-notes

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,9 @@ cover/
5454
# Environments
5555
.env
5656
.venv
57+
58+
# Docs
59+
docs/site
60+
61+
# AI
62+
.claude

‎.pre-commit-config.yaml‎

Lines changed: 28 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,7 @@
11
default_install_hook_types: [ pre-commit ]
22
default_stages: [ pre-commit ]
33
default_language_version:
4-
python: "3.13"
5-
exclude: |
6-
(?x)^(
7-
.*test.*\.py|
8-
.*/test/.*|
9-
.*.md
10-
)$
4+
python: "3.14"
115

126
repos:
137
- repo: https://github.com/pre-commit/pre-commit-hooks
@@ -22,39 +16,36 @@ repos:
2216
- --unsafe
2317
- id: mixed-line-ending
2418
- id: trailing-whitespace
19+
exclude: \.md$
2520

2621
- repo: https://github.com/Lucas-C/pre-commit-hooks
27-
rev: v1.5.5
22+
rev: v1.5.6
2823
hooks:
2924
- id: remove-tabs
3025

31-
- repo: https://github.com/pre-commit/mirrors-mypy
32-
rev: v1.18.2
33-
hooks:
34-
- id: mypy
35-
additional_dependencies: [ "types-redis", "fastapi", "pydantic", "cronsim"]
36-
args: ["--scripts-are-modules"]
37-
3826
- repo: https://github.com/astral-sh/uv-pre-commit
3927
# uv version.
40-
rev: 0.8.22
28+
rev: 0.10.9
4129
hooks:
4230
- id: uv-lock
4331

4432
- repo: https://github.com/astral-sh/ruff-pre-commit
45-
rev: v0.13.2
33+
rev: v0.15.5
4634
hooks:
47-
- id: ruff-check
35+
- id: ruff-check
4836
args:
49-
- --fix
50-
- --unsafe-fixes
51-
- id: ruff-format
37+
- --fix
38+
- id: ruff-format
5239

53-
- repo: https://github.com/netromdk/vermin
54-
rev: v1.6.0
40+
- repo: https://github.com/PyCQA/bandit
41+
rev: 1.9.4
5542
hooks:
56-
- id: vermin
57-
args: ['-t=3.13-', '--violations']
43+
- id: bandit
44+
exclude: |
45+
(?x)^(
46+
.*test.*\.py|
47+
.*/test/.*
48+
)$
5849
5950
- repo: local
6051
hooks:
@@ -63,3 +54,15 @@ repos:
6354
language: system
6455
entry: git fetch origin --prune --prune-tags --tags
6556
pass_filenames: false
57+
stages: [ manual ]
58+
59+
- id: ty
60+
name: ty check
61+
entry: uv run ty check .
62+
language: system
63+
pass_filenames: false
64+
exclude: |
65+
(?x)^(
66+
.*test.*\.py|
67+
.*/test/.*
68+
)$

‎.python-version‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎.vscode/settings.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"files.exclude": {
3+
"**/.git": true,
4+
"**/.svn": true,
5+
"**/.hg": true,
6+
"**/.DS_Store": true,
7+
"**/Thumbs.db": true,
8+
".idea": true,
9+
".mypy_cache": true,
10+
".ruff_cache": true,
11+
".vscode": true,
12+
".venv": true
13+
},
14+
"explorerExclude.backup": {}
15+
}

‎CHANGELOG.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Release Notes
2+
3+
## 1.0.0 - 2026-03-09
4+
5+
### Features
6+
7+
* 🚀 **Redis Streams execution engine**: Replace polling-based task execution with Redis Streams and leader election as the sole execution strategy. Includes `LeaderElector` for distributed leader election via Redis `SET NX`, `Coordinator` for evaluating cron expressions and publishing tasks to streams, and `StreamConsumer` for consuming and executing tasks from dual-priority streams.
8+
* 🚀 **Exponential backoff retry**: Add configurable retry with exponential backoff for failed tasks. On success the backoff state resets automatically. Includes per-task override support and a `/reset-retry` API endpoint.
9+
* 🚀️ **Reconciler and task heartbeat**: Leader-only reconciliation loop that detects missed or stuck tasks and republishes them to the Redis stream. Tasks maintain a heartbeat via a running key to distinguish between actively-running and stalled executions.
10+
* 🚀 **Statistics storage with Redis Streams**: Migrate statistics from Redis Lists to a single Redis Stream per task. Each entry contains both `ts` and `dur` fields, ensuring data correlation. Uses `XADD` with approximate `MAXLEN` for bounded storage.
11+
* 🚀 **Dynamic task CRUD API**: Runtime task management allowing tasks to be created and deleted via the management API. Functions are registered in a `TaskGroup` registry and definitions are persisted in a Redis Hash to survive restarts.
12+
* 🚀 **Worker identity**: Introduce `WorkerIdentity` for traceable worker identification with hostname, PID, and short UUID. Add `RedisKeyBuilder` to centralize all Redis key construction.
13+
14+
### Reworks
15+
16+
* ♻️ **Revamp management API**: Rewrite task router services to use the shared async Redis client. Convert
17+
single-task actions to bulk operations.
18+
Add new endpoints: `GET /health`, `GET /config`, `POST /tasks/trigger`, `DELETE /tasks/statistics`.
19+
* ♻️ Switch all modules to per-component loggers.
20+
* ♻️ Rename config `app_name` to `redis_key_prefix` to better reflect its purpose.
21+
22+
### Docs
23+
24+
* 📝 Add first implementation of docs available at [https://fastapi-task-manager.morando.uk](https://fastapi-task-manager.morando.uk)
25+
26+
### Internals
27+
28+
* 🔒 Add bandit pre-commit hook for security linting.
29+
* ✅ Add `pytest-asyncio` with async test coverage for dynamic tasks, statistics, config, schemas, and more.
30+
* ♻️ Extract `interruptible_sleep` into a shared `async_utils` module.
31+
* ♻️ Fix race condition in distributed lock by using `SET NX` instead of `EXISTS + SET` pattern.
32+
* ♻️ Move pre-commit checks from mypy+vermin to [ty from astral](https://docs.astral.sh/ty/)
33+
* ⬆️ Pre-commit bump uv-pre-commit from 0.9.7 to 0.9.18.
34+
* ⬆️ Pre-commit bump ruff-pre-commit from 0.14.3 to 0.14.10.
35+
* ⬆️ Bump dependencies in uv.lock file, for dev purposes:
36+
* - annotated-doc added 0.0.4
37+
- anyio from 4.11.0 to 4.12.1
38+
- backrefs from 6.1 to 6.2
39+
- certifi from 2026.1.4 to 2026.2.25
40+
- charset-normalizer from 3.4.4 to 3.4.5
41+
- coverage from 7.11.1 to 7.13.0
42+
- exceptiongroup from 1.3.0 to 1.3.1
43+
- fastapi from 0.121.0 to 0.135.1
44+
- idna from 3.10 to 3.11
45+
- markdown-include-variants from 0.0.5 to 0.0.8
46+
- mkdocs-macros-plugin from 1.4.1 to 1.5.0
47+
- mkdocs-material from 9.6.23 to 9.7.4
48+
- platformdirs from 4.5.0 to 4.9.4
49+
- pydantic from 2.12.4 to 2.12.5
50+
- pydantic-core from 2.33.2 to 2.41.5
51+
- pydantic-settings from 2.12.0 to 2.13.1
52+
- pymdown-extensions from 10.16.1 to 10.21
53+
- pytest from 9.0.1 to 9.0.2
54+
- python-dotenv from 1.2.1 to 1.2.2
55+
- redis from 7.0.1 to 7.3.0
56+
- ruff from 0.15.0 to 0.15.5
57+
- selectolax from 0.4.3 to 0.4.7
58+
- sniffio removed
59+
- starlette from 0.49.3 to 0.52.1
60+
- ty from 0.0.16 to 0.0.21
61+
- typing-inspection from 0.4.1 to 0.4.2
62+
- urllib3 from 2.5.0 to 2.6.2
63+
64+
## 0.8.0 - 2026-02-12
65+
66+
### Internals
67+
68+
* ♻️ Change pypi package 'cronexpr' with 'cronsim' due to maintenance issues.
69+
No side effects in package usage.
70+
71+
## 0.7.0 - 2025-xx-xx
72+
73+
TBC

0 commit comments

Comments
 (0)