multiple workers?

[zetaab]

I have tried to debug how this codecov/worker works and it looks like its launching celery beat per worker instance. And if multiple beats are executed the tasks are duplicated as well. Which basically means that there is no way to run multiple worker instances?

So my question is: is there way to run multiple instances? Is codecov really running one worker in production?

[thomasrockhu-codecov]

@jason-ford-codecov let me know if you can help

[zetaab]

and the part which starts the beat worker is here https://github.com/codecov/worker/blob/main/main.py#L108

from https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html#introduction

You have to ensure only a single scheduler is running for a schedule at a time, otherwise you’d end up with duplicate tasks. Using a centralized approach means the schedule doesn’t have to be synchronized, and the service can operate without using locks.

One way of making this scalable is that there could be one beat worker and multiple normal workers (which should execute normal tasks).

[jason-ford-codecov]

Thank you for the feedback. We consider our worker tasks idempotent and we run multiple workers and have duplicate tasks. We are going to investigate moving to a single beat worker as an optimization.