Django 6 Tasks Framework support

[jrobichaud]

Django 6 includes a new tasks framework.

https://docs.djangoproject.com/en/dev/topics/tasks/

It would be nice if django-structlog could work with this new system.

To be discussed.

[jrobichaud]

There are undocumented signals (task_enqueued, task_started and task_finished):

https://github.com/django/django/blob/e49e14fd9032feb7a8cf254658ac4e74a4ffb712/django/tasks/signals.py#L13-L15

I made this diagram to show how it needs to work.

(It is similar to django_structlog celery's implementation.)

sequenceDiagram
    autonumber
    participant app as django
    participant broker as Task Broker
    participant django_structlog_app as django_structlog<br>(in App's Process)
    participant shared_storage as Shared storage
    participant django_structlog_worker as django_structlog<br>(in Worker's Process)
    participant worker as Task Worker
    participant task as Task

    %% Broker
    app ->>+ broker: Enqueue Task
    broker ->> django_structlog_app: Send "task_enqueued" signal
    django_structlog_app ->> shared_storage: Store context
    broker ->> app: Return task id

    %% Worker
    worker ->>+ django_structlog_worker: Send "task_started" signal
    shared_storage ->> django_structlog_worker: Restore context
    django_structlog_worker ->> django_structlog_worker: Log "task_started"
    worker ->>+ task: Start task
    task ->>- worker: Return result
    worker ->> django_structlog_worker: Send "task_finished" signal
    django_structlog_worker ->> django_structlog_worker: Log "task_finished"

Loading

Django-structlog needs to have shared storage/cache between the app's process (See Store Context (3)) and the worker's process (See Restore context (6)).

In django_structlog's Celery's implementation the data is stored directly in celery's message.

Django's implementation uses frozen dataclass which prevents the code to change the task.

We could use django's cache api.

[jrobichaud]

Anyone knows a third party minimalist django task backend?

I need to test my implementation of the django task support.

[LucidDan]

Anyone knows a third party minimalist django task backend?

I need to test my implementation of the django task support.

https://github.com/RealOrangeOne/django-tasks-db perhaps?

[jrobichaud]

I tested django-tasks-db and i faced this issue: RealOrangeOne/django-tasks-db#27 where their signals are not propagated to the django task signals.

That will be a problem with any worker based on django-tasks.