sched: disperse threads faster in the load balancer - #1464
Open
gburd wants to merge 1 commit into
Open
Conversation
The per-CPU load balancer was the only mechanism that spreads runnable threads across CPUs: thread::start() places a new thread on its creator's CPU and thread::wake() re-wakes a blocked thread on the CPU it last ran on, so neither disperses work on its own. Running load_balance() once every 100ms and migrating a single thread per wakeup is far too slow for a workload that fans many threads out from one parent -- for example a server that forks or spawns a worker per connection. The workers pile onto the few CPUs they were started or woken on while the rest of the machine sits idle, and a one-thread-per-100ms drip cannot catch up under a high request rate. Measured on a 16-vCPU guest running a thread-per-connection server driven over the network: as client concurrency rose past the point where workers should have spread out, CPU utilization stayed low (the machine was ~90% idle) yet throughput regressed, because requests queued behind workers stuck on a handful of CPUs. A sampling profile confirmed the CPUs were idle, not contended, and a thread dump showed dozens of workers concentrated on a few CPUs with the majority idle. Run the balancer more often (10ms) and, on each pass, keep migrating the most-migratable queued thread to the least-loaded CPU until this CPU is no longer meaningfully more loaded -- draining the whole imbalance in one pass instead of one thread of it. The number of migrations per pass is bounded by the CPU count so a transient spike cannot become an unbounded migration storm. The migration mechanics are unchanged. On the workload above this flattened the throughput regression: at high concurrency throughput improved by 50-70% and no longer collapsed, at a small (single-digit percent) cost at low concurrency from the extra migration traffic. Only the thread placement path is affected; the runqueue and runtime-accounting logic are untouched. Signed-off-by: Greg Burd <greg@burd.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
cpu::load_balance()is the only mechanism in the scheduler that spreads threads across CPUs:thread::start()places a new thread on its creator's CPU, andthread::wake()re-wakes a blocked thread on the CPU it last ran on. Neither disperses work on its own. The balancer runs once every 100ms and migrates a single thread per pass.That is far too slow for a workload that fans many threads out from one parent (a server that forks or spawns a worker per connection). The workers pile onto the few CPUs they were started or woken on while the rest of the machine sits idle, and a one-thread-per-100ms drip cannot catch up under a high request rate.
This changes the balancer to run every 10ms and, on each pass, keep migrating the most-migratable queued thread to the least-loaded CPU until this CPU is no longer meaningfully more loaded, draining the whole imbalance in one pass. Work per pass is bounded by the CPU count so a transient spike cannot become a migration storm. It reuses the existing, proven migration path unchanged.
Why (measured)
A many-connection server benchmark on a 16-vCPU guest, driven over the network, was collapsing past 8 connections. Profiling showed the guest was ~93% idle at 32 connections while throughput fell: the sampler put 93.5% of CPU time in
cpu::idle(), thread dumps showed ~37 worker threads crammed onto 4 of 16 vCPUs with 12 idle, and the host saw the VM using about 6 of 16 cores. The bottleneck was thread placement, not lock contention, CPU saturation, or I/O.Before / after (same guest, fresh boot each, connections = c)
It trades ~6% at low concurrency (extra migration traffic) for the tail no longer collapsing. Measured effect on placement: idle time 93.5% -> 89.7%, worker spread 4 -> 6 CPUs.
Scope / honest note
General scheduler change, no application specifics. It does not fully saturate the machine: even after the fix the guest is still substantially idle at high concurrency, because the remaining limit is per-request round-trip latency through the single network receive path and per-packet wakeup IPIs, which grows with concurrency. That is a separate, larger change (batching wakeups) and is not attempted here; this PR removes the placement collapse, which is the dominant effect.