openstack/kolla-ansible.git/doc/source/reference/compute/libvirt-guide.rst, branch stable/2026.1

Tune libvirtd connection limits for OpenStack

2026-06-29T00:11:47+00:00

The libvirt daemon ships with max_workers=20 and max_client_requests=5. Those defaults suit a multi-tenant host serving many mutually distrustful clients, where the per-client request cap protects the daemon from any single client monopolising it. Under Kolla-Ansible, nova-compute is effectively the sole privileged client of libvirtd, so the cap of 5 concurrent requests per client becomes the binding constraint rather than a safety net. When several instances build at once -- as they do during the tempest run of the upgrade jobs -- nova-compute's concurrent libvirt RPCs (instance defineXML calls plus the resource-tracker periodic) exceed five. Further requests queue, the connection's keepalive responses are starved, and after keepalive_interval (5s) x keepalive_count (5) = 25s libvirtd closes the connection. nova then sees "Cannot recv data: Input/output error" mid-defineXML, the build is rescheduled, and once all hosts are exhausted the instance is left in ERROR. This presents as flaky upgrade-job failures. Example failure, on the voting kolla-ansible-debian-bookworm-upgrade job (stable/2025.2): https://zuul.opendev.org/t/openstack/build/73ff5a97567d4e05b1774170e0f4aada with primary/logs/kolla/libvirt/libvirtd.txt showing libvirt's own recommendation to tune this parameter: warning : Client hit max requests limit 5. This may result in keep-alive timeouts. Consider tuning the max_client_requests server parameter ... error : connection closed due to keepalive timeout Raise the per-client cap to 20 -- comfortably above the handful of concurrent builds plus periodics seen in practice -- and the worker pool to 50 so that lifting the per-client limit does not starve other connections such as live migration, health checks and periodic tasks. Both values are exposed as variables (libvirt_max_client_requests and libvirt_max_workers) so operators can tune them further. This is not a Kolla-specific workaround -- the stock libvirtd.conf comment warns that setting max_client_requests too low "may cause keepalive timeouts", and since 2022 libvirtd emits the runtime warning quoted above. As Daniel Berrange explained when he added that warning, the keepalive packets "are subject to the 'max_client_requests' limit just like any API calls", so a client that saturates the limit can have its connection dropped: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/5NIJPA7SJVY55VSK7RTEDEHKWOZYJKT6/ Red Hat documents the same tuning for libvirt hosts under load in its knowledge base, notably "Too many active clients are dropping the connection" at https://access.redhat.com/solutions/28146. Assisted-By: Claude Opus 4.8 (1M context, medium effort) Partial-Bug: 2158515 Change-Id: I131c3d4d57dc73ca4dd421cc8f28465248c9830e (cherry picked from commit 5acd2f91d22213ef04bd6988a2a0261982b0cf05) Signed-off-by: Michael Still

Fix docs rendering errors.

2025-06-03T15:54:01+00:00

I was reading https://docs.openstack.org's version of the kolla-ansible advanced configuration guide and noticed some RST rendering errors. This patch corrects those errors so that the page renders correctly. Change-Id: I330678d8971d286e67f115eace1cb606d30433b6

docs: fix information about libvirt SASL auth

2023-02-03T09:55:29+00:00

Change-Id: I0ff303a2fad2edbcedbe88486b272d2efa765d8d

docs: Fix reference to Libvirt TLS

2022-12-06T10:39:23+00:00

Change-Id: I2df1eff4e9eda42127db5b0bfed4f84b6955c763

Adds a wrapper script to run ovs-vsctl commands in the container

2022-11-15T10:16:02+00:00

Libvirt needs to be able to plug ports into openvswitch bridges. It does this using the ovs-vsctl command, which it searches for in $PATH[1, 2]. This change will optionally install a wrapper script that executes the ovs-vsctl commands in the context of the openvswitchd container. This is useful when running libvirt on the host whilst still running openvswitch in a container. The advantage of this method over install the packages on the host is that it ensures client compatability with the daemon. The default is set to false as the wrapper could overwrite ovs-vsctl installed on the host. [1] https://github.com/libvirt/libvirt/blob/ee51ab86c2e52b6ff1b17a4c7ad11439fd610c9e/src/util/virnetdevopenvswitch.c#L59 [2] https://github.com/libvirt/libvirt/blob/a89b17c2a75cfbaeb9e430f88e0f8a7475eb4f54/docs/kbase/internals/command.rst#id3 Closes-Bug: #1995409 Change-Id: Iaa6bfb012ae847f5f6aa0a1fc1c27970ac265f93

Merge "docs: add information about development libvirt TLS certs"

2022-05-30T13:36:36+00:00

libvirt: add nova-libvirt-cleanup command

2022-03-21T11:54:54+00:00

Change Ia1239069ccee39416b20959cbabad962c56693cf added support for running a libvirt daemon on the host, rather than using the nova_libvirt container. It did not cover migration of existing hosts from using a container to using a host daemon. This change adds a kolla-ansible nova-libvirt-cleanup command which may be used to clean up the nova_libvirt container, volumes and related items on hosts, once it has been disabled. The playbook assumes that compute hosts have been emptied of VMs before it runs. A future extension could support migration of existing VMs, but this is currently out of scope. Change-Id: I46854ed7eaf1d5b5e3ccd8531c963427848bdc99

libvirt: make it possible to run libvirt on the host

2022-03-21T11:54:31+00:00

In some cases it may be desirable to run the libvirt daemon on the host. For example, when mixing host and container OS distributions or versions. This change makes it possible to disable the nova_libvirt container, by setting enable_nova_libvirt_container to false. The default values of some Docker mounts and other paths have been updated to point to default host directories rather than Docker volumes when using a host libvirt daemon. This change does not handle migration of existing systems from using a nova_libvirt container to libvirt on the host. Depends-On: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/830504 Change-Id: Ia1239069ccee39416b20959cbabad962c56693cf

libvirt: support SASL authentication

2022-03-10T16:57:16+00:00

In Kolla Ansible OpenStack deployments, by default, libvirt is configured to allow read-write access via an unauthenticated, unencrypted TCP connection, using the internal API network. This is to facilitate migration between hosts. By default, Kolla Ansible does not use encryption for services on the internal network (and did not support it until Ussuri). However, most other services on the internal network are at least authenticated (usually via passwords), ensuring that they cannot be used by anyone with access to the network, unless they have credentials. The main issue here is the lack of authentication. Any client with access to the internal network is able to connect to the libvirt TCP port and make arbitrary changes to the hypervisor. This could include starting a VM, modifying an existing VM, etc. Given the flexibility of the domain options, it could be seen as equivalent to having root access to the hypervisor. Kolla Ansible supports libvirt TLS [1] since the Train release, using client and server certificates for mutual authentication and encryption. However, this feature is not enabled by default, and requires certificates to be generated for each compute host. This change adds support for libvirt SASL authentication, and enables it by default. This provides base level of security. Deployments requiring further security should use libvirt TLS. [1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021 Closes-Bug: #1964013 Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e

docs: add information about development libvirt TLS certs

2022-02-11T09:57:20+00:00

Adds docs for I1bde9fa018f66037aec82dc74c61ad1f477a7c12. Change-Id: I88a07bb3bfeb0c98bea9dbe8674033208ec3fb9f