Skip to content

Commit 34604d2

Browse files
authored
docs: clarify Redis maintenance for CeleryExecutor (#67393)
1 parent b5a60ce commit 34604d2

1 file changed

Lines changed: 21 additions & 0 deletions

File tree

providers/celery/docs/celery_executor.rst

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,27 @@ Some caveats:
9595
- Tasks can consume resources. Make sure your worker has enough resources to run ``worker_concurrency`` tasks
9696
- Queue names are limited to 256 characters, but each broker backend might have its own restrictions
9797

98+
Redis maintenance
99+
-----------------
100+
101+
When Redis is used as the Celery broker, Airflow uses it for transient Celery messages: queued task
102+
commands, task acknowledgement data, and other broker-side state. Airflow task history, Dag runs,
103+
connections, Variables, and XCom records are stored in the Airflow metadata database and should be
104+
maintained with database tooling such as ``airflow db clean``. Cleaning the metadata database does not
105+
clean Redis, and cleaning Redis does not clean Airflow's metadata database.
106+
107+
Do not flush or delete keys from the Redis database used by Airflow while schedulers or Celery workers
108+
are running. Removing broker keys while tasks are queued, running, or being acknowledged can lose
109+
pending work or make task state temporarily inconsistent. If you need to discard stale broker data
110+
after an outage, first stop Airflow schedulers and Celery workers, make sure there are no tasks that
111+
you intend to preserve in ``queued`` or ``running`` state, back up the Redis database if it is
112+
persistent, and then remove only the Redis database or keyspace configured in ``[celery] broker_url``.
113+
114+
If you configured Redis as the Celery ``result_backend``, the same maintenance window should also
115+
account for result backend keys. Deleting those keys can remove Celery task result information that the
116+
scheduler may still query. For production deployments, Airflow recommends a database-backed result
117+
backend so broker cleanup and task-result storage are handled separately.
118+
98119
See :doc:`apache-airflow:administration-and-deployment/modules_management` for details on how Python and Airflow manage modules.
99120

100121
Architecture

0 commit comments

Comments
 (0)