Synchronise 2025.1 with upstream#896
Merged
Merged
Conversation
The libvirt daemon ships with max_workers=20 and max_client_requests=5. Those defaults suit a multi-tenant host serving many mutually distrustful clients, where the per-client request cap protects the daemon from any single client monopolising it. Under Kolla-Ansible, nova-compute is effectively the sole privileged client of libvirtd, so the cap of 5 concurrent requests per client becomes the binding constraint rather than a safety net. When several instances build at once -- as they do during the tempest run of the upgrade jobs -- nova-compute's concurrent libvirt RPCs (instance defineXML calls plus the resource-tracker periodic) exceed five. Further requests queue, the connection's keepalive responses are starved, and after keepalive_interval (5s) x keepalive_count (5) = 25s libvirtd closes the connection. nova then sees "Cannot recv data: Input/output error" mid-defineXML, the build is rescheduled, and once all hosts are exhausted the instance is left in ERROR. This presents as flaky upgrade-job failures. Example failure, on the voting kolla-ansible-debian-bookworm-upgrade job (stable/2025.2): https://zuul.opendev.org/t/openstack/build/73ff5a97567d4e05b1774170e0f4aada with primary/logs/kolla/libvirt/libvirtd.txt showing libvirt's own recommendation to tune this parameter: warning : Client hit max requests limit 5. This may result in keep-alive timeouts. Consider tuning the max_client_requests server parameter ... error : connection closed due to keepalive timeout Raise the per-client cap to 20 -- comfortably above the handful of concurrent builds plus periodics seen in practice -- and the worker pool to 50 so that lifting the per-client limit does not starve other connections such as live migration, health checks and periodic tasks. Both values are exposed as variables (libvirt_max_client_requests and libvirt_max_workers) so operators can tune them further. This is not a Kolla-specific workaround -- the stock libvirtd.conf comment warns that setting max_client_requests too low "may cause keepalive timeouts", and since 2022 libvirtd emits the runtime warning quoted above. As Daniel Berrange explained when he added that warning, the keepalive packets "are subject to the 'max_client_requests' limit just like any API calls", so a client that saturates the limit can have its connection dropped: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/5NIJPA7SJVY55VSK7RTEDEHKWOZYJKT6/ Red Hat documents the same tuning for libvirt hosts under load in its knowledge base, notably "Too many active clients are dropping the connection" at https://access.redhat.com/solutions/28146. Assisted-By: Claude Opus 4.8 (1M context, medium effort) Partial-Bug: 2158515 Change-Id: I131c3d4d57dc73ca4dd421cc8f28465248c9830e (cherry picked from commit 5acd2f9) Signed-off-by: Michael Still <mikal@stillhq.com>
Set the Ironic tempdir to be within the Ironic Docker container volume. Modified to work without directories support in config.json Closes-Bug: #2156758 Depends-On: https://review.opendev.org/c/openstack/kolla/+/995011 Change-Id: Ib24a04657f297ebb323686aa1d80f9bf019f44f2 Signed-off-by: Jed Preist <jed@stackhpc.com> (cherry picked from commit ba5d421)
Due to implementation choices in qemu and libvirt, standard log rotate does not work on these console files, and instead the usual approach is to have libvirt's virtlogd rotate the contents of the console log file when it gets a bit big. OpenStack Compute knows how to read data from more than one rotated console log file if required to service a REST API request. OpenStack Kolla and Kolla-Ansible deploy OpenStack components in docker or podman containers. An error in the startup configuration for the nova_libvirt container results in the libvirt virtlogd daemon not being run, which results in this console log rotation not occurring. This opens up the risk of a disk usage denial of service attack on a hypervisor. The console log is stored on the same filesystem as ephemeral disks, and OpenStack Nova will take a hypervisor out of service if disk space becomes constrained. There is some protection from the instance console not being able to write data particularly quickly, but it is definitely still possible to fill the hypervisor file system -- especially if multiple instances are used. This misconfiguration appears to exist in all versions of Kolla / Kolla-Ansible as best as I can tell. Flip qemu.conf's stdio_handler from "file" to "logd" so libvirt routes instance console chardevs through virtlogd, which rotates them per /etc/libvirt/virtlogd.conf. Without this, virtlogd runs but is never consulted and console.log grows unbounded. The nova_libvirt command now points at the /usr/local/bin/kolla_nova_libvirt_start helper shipped in the image. Deploying this Kolla-Ansible change against an older image that predates that helper would leave the container unable to start. Add a deploy-time check that inspects the nova_libvirt image for the helper and fails with an explanatory message when it is absent, so an incomplete upgrade (new code, stale image) is caught before the container is reconfigured. Partial-Bug: 2155214 Change-Id: Ia543939b17cd06714695dff8ef8bd29bc3f8175e Signed-off-by: Michael Still <mikal@stillhq.com> (cherry picked from commit 4244e6a) (cherry picked from commit 6303668) Signed-off-by: Michael Still <mikal@stillhq.com>
github-actions
Bot
added
automated
Alex-Welsh
approved these changes
Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains a snapshot of 2025.1 from upstream stable/2025.1.