From 75a9dcd132b0a5aab69bf234dd621c9035208174 Mon Sep 17 00:00:00 2001 From: Michael Still Date: Mon, 29 Jun 2026 10:11:47 +1000 Subject: [PATCH] Tune libvirtd connection limits for OpenStack The libvirt daemon ships with max_workers=20 and max_client_requests=5. Those defaults suit a multi-tenant host serving many mutually distrustful clients, where the per-client request cap protects the daemon from any single client monopolising it. Under Kolla-Ansible, nova-compute is effectively the sole privileged client of libvirtd, so the cap of 5 concurrent requests per client becomes the binding constraint rather than a safety net. When several instances build at once -- as they do during the tempest run of the upgrade jobs -- nova-compute's concurrent libvirt RPCs (instance defineXML calls plus the resource-tracker periodic) exceed five. Further requests queue, the connection's keepalive responses are starved, and after keepalive_interval (5s) x keepalive_count (5) = 25s libvirtd closes the connection. nova then sees "Cannot recv data: Input/output error" mid-defineXML, the build is rescheduled, and once all hosts are exhausted the instance is left in ERROR. This presents as flaky upgrade-job failures. Example failure, on the voting kolla-ansible-debian-bookworm-upgrade job (stable/2025.2): https://zuul.opendev.org/t/openstack/build/73ff5a97567d4e05b1774170e0f4aada with primary/logs/kolla/libvirt/libvirtd.txt showing libvirt's own recommendation to tune this parameter: warning : Client hit max requests limit 5. This may result in keep-alive timeouts. Consider tuning the max_client_requests server parameter ... error : connection closed due to keepalive timeout Raise the per-client cap to 20 -- comfortably above the handful of concurrent builds plus periodics seen in practice -- and the worker pool to 50 so that lifting the per-client limit does not starve other connections such as live migration, health checks and periodic tasks. Both values are exposed as variables (libvirt_max_client_requests and libvirt_max_workers) so operators can tune them further. This is not a Kolla-specific workaround -- the stock libvirtd.conf comment warns that setting max_client_requests too low "may cause keepalive timeouts", and since 2022 libvirtd emits the runtime warning quoted above. As Daniel Berrange explained when he added that warning, the keepalive packets "are subject to the 'max_client_requests' limit just like any API calls", so a client that saturates the limit can have its connection dropped: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/5NIJPA7SJVY55VSK7RTEDEHKWOZYJKT6/ Red Hat documents the same tuning for libvirt hosts under load in its knowledge base, notably "Too many active clients are dropping the connection" at https://access.redhat.com/solutions/28146. Assisted-By: Claude Opus 4.8 (1M context, medium effort) Partial-Bug: 2158515 Change-Id: I131c3d4d57dc73ca4dd421cc8f28465248c9830e (cherry picked from commit 5acd2f91d22213ef04bd6988a2a0261982b0cf05) Signed-off-by: Michael Still --- ansible/roles/nova-cell/defaults/main.yml | 11 +++++++++ .../nova-cell/templates/libvirtd.conf.j2 | 10 ++++++++ .../reference/compute/libvirt-guide.rst | 23 +++++++++++++++++++ ...td-connection-tuning-7c4f1e9a2b8d6f30.yaml | 18 +++++++++++++++ 4 files changed, 62 insertions(+) create mode 100644 releasenotes/notes/libvirtd-connection-tuning-7c4f1e9a2b8d6f30.yaml diff --git a/ansible/roles/nova-cell/defaults/main.yml b/ansible/roles/nova-cell/defaults/main.yml index 790190fb08..918de4463d 100644 --- a/ansible/roles/nova-cell/defaults/main.yml +++ b/ansible/roles/nova-cell/defaults/main.yml @@ -639,6 +639,17 @@ libvirt_sasl_authname: "nova" libvirt_sasl_mech_list: - "{{ 'SCRAM-SHA-256' if libvirt_tls | bool else 'DIGEST-MD5' }}" +# Connection processing limits for libvirtd. nova-compute is effectively the +# only privileged client of libvirtd under Kolla, so libvirt's stock per-client +# request cap (max_client_requests=5) is the binding constraint rather than a +# safety net: concurrent instance builds plus the resource-tracker periodic +# routinely exceed five in-flight RPCs, which starves keepalive responses until +# libvirtd drops the connection. Raise the per-client cap, and the worker pool +# along with it so that lifting the per-client limit does not starve other +# connections (live migration, health checks, periodics). +libvirt_max_client_requests: 20 +libvirt_max_workers: 50 + #################### # Kolla #################### diff --git a/ansible/roles/nova-cell/templates/libvirtd.conf.j2 b/ansible/roles/nova-cell/templates/libvirtd.conf.j2 index aa01f743d8..d156fb4be7 100644 --- a/ansible/roles/nova-cell/templates/libvirtd.conf.j2 +++ b/ansible/roles/nova-cell/templates/libvirtd.conf.j2 @@ -25,3 +25,13 @@ listen_addr = "{{ migration_interface_address }}" # Enable read-only access to libvirt socket auth_unix_ro = "none" {% endif %} + +# Connection processing limits. libvirt's defaults (max_workers=20, +# max_client_requests=5) assume a multi-tenant host with many mutually +# distrustful clients. Under Kolla, nova-compute is effectively the sole +# privileged client, so the per-client cap of 5 becomes the binding +# constraint: concurrent builds plus the resource-tracker periodic exceed +# five in-flight RPCs and starve keepalive responses, causing libvirtd to +# drop the connection after keepalive_interval (5s) x keepalive_count (5). +max_workers = {{ libvirt_max_workers }} +max_client_requests = {{ libvirt_max_client_requests }} diff --git a/doc/source/reference/compute/libvirt-guide.rst b/doc/source/reference/compute/libvirt-guide.rst index 24a55e1fbe..d6212b9086 100644 --- a/doc/source/reference/compute/libvirt-guide.rst +++ b/doc/source/reference/compute/libvirt-guide.rst @@ -198,3 +198,26 @@ Since the Yoga release, the ``kolla-ansible certificates`` command generates certificates for libvirt TLS. A single key and certificate is used for all hosts, with a Subject Alternative Name (SAN) entry for each compute host hostname. + +Libvirt Connection Limits +========================= + +By default ``libvirt`` is configured by upstream packages with reasonable +defaults for an environment where many users are starting and stopping virtual +machines. This isn't true for Kolla Hypervisor nodes, where a single +``nova-compute`` container manages all of the virtual machine instances on the +machine. + +Kolla-Ansible therefore tunes the following values in ``libvirtd.conf``: + +* libvirt_max_client_requests: raised to 20 compared to the usual default of + 5. +* libvirt_max_workers: raised to 50 compared to the usual default of 20. + +Both of these values may be overridden in ``globals.yml``. + +This means that ``nova-compute`` can therefore perform 20 operations via +libvirt simultaneously instead of the usual default of 5. These values should +be safe, but if you see libvirt error messages saying ``Client hit max requests +limit``, you may want to consider an additional increase -- especially if you +have particularly large hypervisors. diff --git a/releasenotes/notes/libvirtd-connection-tuning-7c4f1e9a2b8d6f30.yaml b/releasenotes/notes/libvirtd-connection-tuning-7c4f1e9a2b8d6f30.yaml new file mode 100644 index 0000000000..fcc8a65d41 --- /dev/null +++ b/releasenotes/notes/libvirtd-connection-tuning-7c4f1e9a2b8d6f30.yaml @@ -0,0 +1,18 @@ +--- +features: + - | + The ``nova_libvirt`` container's ``libvirtd.conf`` now raises libvirt's + connection processing limits above their stock defaults, which are tuned + for a multi-tenant host rather than for Kolla's single-privileged-client + pattern. ``max_client_requests`` is raised from 5 to 20 and ``max_workers`` + from 20 to 50. Both are exposed as the ``libvirt_max_client_requests`` and + ``libvirt_max_workers`` variables so they can be tuned further. +fixes: + - | + Fixed intermittent instance build failures (``Cannot recv data: + Input/output error`` during ``defineXML``, leading to instances in + ``ERROR`` state) that occurred when several instances were built + concurrently. nova-compute is effectively the sole client of ``libvirtd``, + and libvirt's default per-client request cap of 5 was being exceeded by + concurrent builds plus periodic tasks, starving keepalive responses until + ``libvirtd`` dropped the connection.