virtio-net: implement multiqueue (VIRTIO_NET_F_MQ) - #1463
Conversation
The virtio-net driver hardcoded a single Rx/Tx queue pair drained by one
receive poll thread. Every inbound packet for every connection funnelled
through that one queue and one thread on one CPU, so receive-side network
throughput could not scale past what a single core can drain, regardless of
how many vCPUs the guest had. A concurrency sweep of a network-bound server
workload showed throughput saturating at a low connection count and then
regressing as contention grew, with most vCPUs idle: the load was serialized
on the single queue, not CPU-bound.
Negotiate VIRTIO_NET_F_MQ (and VIRTIO_NET_F_CTRL_VQ, required to activate it)
and drive N Rx/Tx queue pairs:
- Read max_virtqueue_pairs from config space and pick the pair count as
min(advertised pairs, number of vCPUs, MSI-X vector budget).
- Replace the single rxq/txq with vectors of queue pairs. Each Rx queue has
its own poll thread, pinned to a distinct CPU so receivers run in parallel;
each Tx queue has its own xmitter. Transmit selects a Tx queue by CPU id.
- Register one MSI-X vector per data virtqueue (entry = virtqueue index, per
the 1:1 mapping setup_queue() programs) so a completion on queue i wakes
that queue's thread.
- Send VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET on the control virtqueue to tell the
device how many pairs the guest will use.
When the device does not offer VIRTIO_NET_F_MQ (or offers a single pair, as
with the MMIO transport's shared interrupt) the driver uses exactly one queue
pair and behaves as before.
Also add a process-wide MSI-X vector budget shared by all virtio devices. The
x86-64 IDT has only 224 usable vectors (32..255), shared by every device. A
high-vCPU guest with several multiqueue virtio devices (for example a handful
of multiqueue virtio-blk disks plus multiqueue virtio-net) could request more
one-vector-per-queue bindings than there are vectors and abort during driver
load, so the guest could not boot at high vCPU counts at all. Now:
- register_interrupt_handler() returns 0 (an impossible MSI vector, since
0..31 are CPU exceptions) instead of aborting when the IDT is full, and
msix_vector construction / request_vectors() treat that as a soft failure;
- virtio_driver::reserve_msix_vectors() hands out vectors from a shared pool
so each multiqueue device caps its interrupt-bearing queue count and still
gets at least one vector.
virtio-blk and virtio-net both consult the budget, so a guest with many vCPUs
and several multiqueue virtio devices boots instead of crashing, and each
device runs as many queues as the budget allows.
Signed-off-by: Greg Burd <greg@burd.me>
Update: benchmark results and a regression to fix before mergeI ran a direct A/B (multiqueue nqp>=8 vs single-queue nqp=1, same image, same NIC, an external client) with PostgreSQL/pgbench across concurrency levels, and the results change what I can claim for this PR. Converting to draft. Two honest findings: 1. The multiqueue feature does not improve concurrent throughput on this workload. Single-queue (nqp=1) equals or beats nqp>=8 at every concurrency level (e.g. read-only tps at 16 connections: nqp=1 gave ~66k vs nqp=16 ~25k, reproduced). The throughput ceiling this workload hits is a per-connection CPU/scheduler cost and a storage write path, not RX-queue depth. So multiqueue is not the scaling win I expected; it is correct and it activates (N queues, N Rx threads, device ACKs CTRL_MQ_VQ_PAIRS_SET), but it does not move the numbers here. 2. The MSI-X vector budget introduces a virtio-blk regression under load. When virtio-net claims 2*nqp vectors at high vCPU count, it starves the several virtio-blk devices; the soft-fail path (register_interrupt_handler returning 0 instead of aborting) then leaves a blk queue with no interrupt vector, so completions are never delivered and reads fail with EIO under sustained write. The pre-MQ image was clean. This needs fixing: a queue that cannot get its own vector must fall back to a shared/polled vector rather than run interrupt-less, and net must not be allowed to starve blk. What stands: the vector-budget change does fix a real pre-existing boot crash (many-vCPU guests with several multiqueue virtio-blk devices previously aborted in register_interrupt_handler on IDT vector exhaustion; they now boot). That part is worth keeping. Plan: I will either (a) fix the vector arbitration so blk always keeps a working (shared/polled) vector and re-take the A/B, or (b) split the boot-crash MSI-X fix into its own PR and hold the multiqueue feature until it demonstrates a real gain. Marking draft until then. |
|
I am not surprised by your performance findings. I am seeing similar ones with the virtio-blk multi-queue multi-consumer -threads changes. Single queue, single consumer thread seems to be beating all other setups. |
|
That lines up with what I found, and it points at a shared root cause rather than anything queue-specific. I profiled the concurrency collapse separately (a many-connection server workload over the network) and the guest was ~90% idle while throughput fell: threads were piling onto a few CPUs while the rest sat idle. The scheduler places a new thread on its creator's CPU and re-wakes a blocked thread on its last CPU, and the load balancer only migrated one thread per 100ms, far too slow to disperse a fan-out of many worker threads. That is why more queues (net or blk) did not help: adding consumers does not change where the woken threads land, so they still serialize on the same few CPUs. I opened #1464 for the scheduler side (run the balancer more often and drain the whole imbalance per pass); it recovered +53% to +73% at 32 to 64 connections in an A/B. Your virtio-blk observation is consistent with the same effect: single queue/single consumer wins because it avoids paying migration/placement overhead for consumers that the balancer cannot spread anyway. Given all that, I am holding this virtio-net MQ PR as draft. The multiqueue feature does not earn its keep on its own, and the vector-budget change in it also needs the virtio-blk starvation fix I noted above. The genuinely useful part is the MSI-X vector-budget boot fix (many-vCPU guests with several multiqueue virtio-blk devices previously aborted at boot); if useful I can split that out into its own small PR and drop the multiqueue machinery. |
2 participants
What
OSv's virtio-net driver hardcodes a single Rx/Tx virtqueue pair drained by one
receiver()poll thread. Every guest's inbound network I/O funnels through that one queue on one CPU.VIRTIO_NET_F_MQexists as a feature constant but was never negotiated.This implements multiqueue:
VIRTIO_NET_F_MQ+VIRTIO_NET_F_CTRL_VQ; readmax_virtqueue_pairsfrom device config; usenqp = min(advertised, num_vCPUs, MSI-X budget)._rxq/_txqsingle structs become N-element vectors; N Rx poll threads, one per pair, pinned across distinct CPUs.VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SETissued on the control queue.nqp == 1path (device without MQ, e.g. QEMU SLIRP) is unchanged.Prerequisite fix: MSI-X vector budget (also fixes a boot crash)
Enabling multiqueue on a many-vCPU guest exposed a pre-existing crash: OSv
abort()s ininterrupt_descriptor_table::register_interrupt_handlerwhen the 224 usable IDT vectors (32..255) are exhausted. With several multiqueue virtio-blk devices at high-smpthis already crashed at boot before any net change. This PR adds a process-wide MSI-X vector budget:register_interrupt_handlerreturns 0 (no vector) instead of aborting when the IDT is full, and virtio-blk and virtio-net cap their interrupt-bearing queues against the shared pool.Validation
CTRL_MQ_VQ_PAIRS_SET(8)reaching DRIVER_OK. A single-queue device falls back to nqp=1.-smp 4,-smp 16, and-smp 48.-smp 48): stock master aborts 3/3 inregister_interrupt_handler; patched boots 5/5.Note on throughput
The motivation is concurrent-workload scaling (single-queue net serializes many concurrent connections' RX through one thread). I verified the MQ mechanism (8 active queues/threads) and boot stability here; the end-to-end concurrent-throughput curve is best shown with a many-connection server workload and I have not included those numbers in this PR. Bulk single-stream TCP is not a faithful measure (vhost GRO coalesces it so one Rx thread suffices). The change is a strict superset of the current single-queue behavior (nqp=1 unchanged), so it is safe independent of the throughput delta.