[Bug]: encoder06 seems to be the only instance working

[don-vip]

Contact Details

No response

What happened?

Hi @Amdrel ,

I wonder if I have broken something with the latest updates :(

It seems that only encoder06 is now picking up tasks from redis, I don't understand why. Could you please have a look?

What browser(s) are you experiencing the problem on?

No response

Relevant error messages

[Amdrel]

Sorry for missing this. I wonder if tweaking some of the routing settings and upgrading the base image would help.

We currently send restarted tasks and manually uploaded tasks to encoder05 and encoder06 since we don't have a good way to estimate how resource intensive the conversions will be until the worker receives them. Upgrading the base image to a newer Python version with ffmpeg installed alongside it will allow us to probe file metadata and not have to route as much to those instances.

[Amdrel]

I shelled into some of the instances with low CPU activity and noticed hung ffmpeg processes that were multiple days old and doing nothing. I've restarted celery on those instances and they seem to be working again for now.

I'm not sure how that happened. At the very least we should probably monitor the subprocesses and end them if they get hung. We can check if the progress percentage changed within a specific timeout, and if not, end the process and fail the task. That would at least prevent us having to ssh into the workers to fix them until we find the root cause.