Pipeline count_all_scheduled_jobs for significantly faster execution

[top-sigrid]

Problem

count_all_scheduled_jobs issues one sequential LLEN Redis call per delayed timestamp. With many delayed jobs, this becomes a significant bottleneck, especially when called from metric collectors like yabeda-resque during Prometheus scrapes.

On our production system (20k timestamps, 36k delayed jobs) this single method took 5.6 seconds, causing Prometheus scrape timeouts. For the same number of jobs and timestamps, the pipelined implementation took only ~300ms.

	Jobs	Timestamps	Duration
Sequential (before)	35,948	20,309	5,641ms
Pipelined (after)	35,969	20,309	304ms

18.6x faster.

Solution

Pipeline all LLEN calls in batches of 10,000 to avoid blocking the Redis thread for too long while still being significantly faster than sequential calls.

Production benchmarks

We have run this implementation in production by inclduing it in https://github.com/jbockler/yabeda-resque first See PR, so I am confident this works in practice. We have seen a reduction of query times by an order of magnitide confirming our simple measerement above.