| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
The libvirt daemon ships with max_workers=20 and max_client_requests=5.
Those defaults suit a multi-tenant host serving many mutually
distrustful clients, where the per-client request cap protects the
daemon from any single client monopolising it. Under Kolla-Ansible,
nova-compute is effectively the sole privileged client of libvirtd, so
the cap of 5 concurrent requests per client becomes the binding
constraint rather than a safety net.
When several instances build at once -- as they do during the tempest
run of the upgrade jobs -- nova-compute's concurrent libvirt RPCs
(instance defineXML calls plus the resource-tracker periodic) exceed
five. Further requests queue, the connection's keepalive responses are
starved, and after keepalive_interval (5s) x keepalive_count (5) = 25s
libvirtd closes the connection. nova then sees "Cannot recv data:
Input/output error" mid-defineXML, the build is rescheduled, and once
all hosts are exhausted the instance is left in ERROR. This presents as
flaky upgrade-job failures.
Example failure, on the voting kolla-ansible-debian-bookworm-upgrade
job (stable/2025.2):
https://zuul.opendev.org/t/openstack/build/73ff5a97567d4e05b1774170e0f4aada
with primary/logs/kolla/libvirt/libvirtd.txt showing libvirt's own
recommendation to tune this parameter:
warning : Client hit max requests limit 5. This may result in
keep-alive timeouts. Consider tuning the max_client_requests server
parameter
...
error : connection closed due to keepalive timeout
Raise the per-client cap to 20 -- comfortably above the handful of
concurrent builds plus periodics seen in practice -- and the worker
pool to 50 so that lifting the per-client limit does not starve other
connections such as live migration, health checks and periodic tasks.
Both values are exposed as variables (libvirt_max_client_requests and
libvirt_max_workers) so operators can tune them further.
This is not a Kolla-specific workaround -- the stock libvirtd.conf
comment warns that setting max_client_requests too low "may cause
keepalive timeouts", and since 2022 libvirtd emits the runtime warning
quoted above. As Daniel Berrange explained when he added that warning,
the keepalive packets "are subject to the 'max_client_requests' limit
just like any API calls", so a client that saturates the limit can have
its connection dropped:
https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/5NIJPA7SJVY55VSK7RTEDEHKWOZYJKT6/
Red Hat documents the same tuning for libvirt hosts under load in its
knowledge base, notably "Too many active clients are dropping the
connection" at https://access.redhat.com/solutions/28146.
Assisted-By: Claude Opus 4.8 (1M context, medium effort)
Partial-Bug: 2158515
Change-Id: I131c3d4d57dc73ca4dd421cc8f28465248c9830e
(cherry picked from commit 5acd2f91d22213ef04bd6988a2a0261982b0cf05)
Signed-off-by: Michael Still <mikal@stillhq.com>
|
|
I was reading https://docs.openstack.org's version of the kolla-ansible
advanced configuration guide and noticed some RST rendering errors.
This patch corrects those errors so that the page renders correctly.
Change-Id: I330678d8971d286e67f115eace1cb606d30433b6
|
|
Change-Id: I0ff303a2fad2edbcedbe88486b272d2efa765d8d
|
|
Change-Id: I2df1eff4e9eda42127db5b0bfed4f84b6955c763
|
|
Libvirt needs to be able to plug ports into openvswitch bridges.
It does this using the ovs-vsctl command, which it searches for
in $PATH[1, 2]. This change will optionally install a wrapper
script that executes the ovs-vsctl commands in the context of the
openvswitchd container. This is useful when running libvirt on the
host whilst still running openvswitch in a container. The advantage
of this method over install the packages on the host is that it
ensures client compatability with the daemon. The default is set
to false as the wrapper could overwrite ovs-vsctl installed on the
host.
[1] https://github.com/libvirt/libvirt/blob/ee51ab86c2e52b6ff1b17a4c7ad11439fd610c9e/src/util/virnetdevopenvswitch.c#L59
[2] https://github.com/libvirt/libvirt/blob/a89b17c2a75cfbaeb9e430f88e0f8a7475eb4f54/docs/kbase/internals/command.rst#id3
Closes-Bug: #1995409
Change-Id: Iaa6bfb012ae847f5f6aa0a1fc1c27970ac265f93
|
|
|
|
Change Ia1239069ccee39416b20959cbabad962c56693cf added support for
running a libvirt daemon on the host, rather than using the nova_libvirt
container. It did not cover migration of existing hosts from using a
container to using a host daemon.
This change adds a kolla-ansible nova-libvirt-cleanup command which may
be used to clean up the nova_libvirt container, volumes and related
items on hosts, once it has been disabled.
The playbook assumes that compute hosts have been emptied of VMs before
it runs. A future extension could support migration of existing VMs, but
this is currently out of scope.
Change-Id: I46854ed7eaf1d5b5e3ccd8531c963427848bdc99
|
|
In some cases it may be desirable to run the libvirt daemon on the host.
For example, when mixing host and container OS distributions or
versions.
This change makes it possible to disable the nova_libvirt container, by
setting enable_nova_libvirt_container to false. The default values of
some Docker mounts and other paths have been updated to point to default
host directories rather than Docker volumes when using a host libvirt
daemon.
This change does not handle migration of existing systems from using
a nova_libvirt container to libvirt on the host.
Depends-On: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/830504
Change-Id: Ia1239069ccee39416b20959cbabad962c56693cf
|
|
In Kolla Ansible OpenStack deployments, by default, libvirt is
configured to allow read-write access via an unauthenticated,
unencrypted TCP connection, using the internal API network. This is to
facilitate migration between hosts.
By default, Kolla Ansible does not use encryption for services on the
internal network (and did not support it until Ussuri). However, most
other services on the internal network are at least authenticated
(usually via passwords), ensuring that they cannot be used by anyone
with access to the network, unless they have credentials.
The main issue here is the lack of authentication. Any client with
access to the internal network is able to connect to the libvirt TCP
port and make arbitrary changes to the hypervisor. This could include
starting a VM, modifying an existing VM, etc. Given the flexibility of
the domain options, it could be seen as equivalent to having root access
to the hypervisor.
Kolla Ansible supports libvirt TLS [1] since the Train release, using
client and server certificates for mutual authentication and encryption.
However, this feature is not enabled by default, and requires
certificates to be generated for each compute host.
This change adds support for libvirt SASL authentication, and enables it
by default. This provides base level of security. Deployments requiring
further security should use libvirt TLS.
[1] https://docs.openstack.org/kolla-ansible/latest/reference/compute/libvirt-guide.html#libvirt-tls
Depends-On: https://review.opendev.org/c/openstack/kolla/+/833021
Closes-Bug: #1964013
Change-Id: Ia91ceeb609e4cdb144433122b443028c0278b71e
|
|
Adds docs for I1bde9fa018f66037aec82dc74c61ad1f477a7c12.
Change-Id: I88a07bb3bfeb0c98bea9dbe8674033208ec3fb9f
|
|
Adds a top-level guide for Nova, with links off to the various virt
driver guides.
Generalises the libvirt TLS guide into a libvirt guide, and adds info on
hardware virtualisation and qemu vs. kvm.
Adds information on configuring consoles.
Change-Id: I36beaaee313bdbc4bcf8cc15c41dda245a5a81ba
|
