Unnamed repository; edit this file 'description' to name the repository. https://git.platform-os.org/openstack/kolla-ansible.git/atom?h=stable%2F2025.1 2026-06-29T00:48:45+00:00 2026-06-29T00:48:45+00:00 Michael Still mikal@stillhq.com 2026-06-29T00:48:45+00:00 urn:sha1:f740c5beb25f9eedee8fa1916faace8188892664 The libvirt daemon ships with max_workers=20 and max_client_requests=5. Those defaults suit a multi-tenant host serving many mutually distrustful clients, where the per-client request cap protects the daemon from any single client monopolising it. Under Kolla-Ansible, nova-compute is effectively the sole privileged client of libvirtd, so the cap of 5 concurrent requests per client becomes the binding constraint rather than a safety net. When several instances build at once -- as they do during the tempest run of the upgrade jobs -- nova-compute's concurrent libvirt RPCs (instance defineXML calls plus the resource-tracker periodic) exceed five. Further requests queue, the connection's keepalive responses are starved, and after keepalive_interval (5s) x keepalive_count (5) = 25s libvirtd closes the connection. nova then sees "Cannot recv data: Input/output error" mid-defineXML, the build is rescheduled, and once all hosts are exhausted the instance is left in ERROR. This presents as flaky upgrade-job failures. Example failure, on the voting kolla-ansible-debian-bookworm-upgrade job (stable/2025.2): https://zuul.opendev.org/t/openstack/build/73ff5a97567d4e05b1774170e0f4aada with primary/logs/kolla/libvirt/libvirtd.txt showing libvirt's own recommendation to tune this parameter: warning : Client hit max requests limit 5. This may result in keep-alive timeouts. Consider tuning the max_client_requests server parameter ... error : connection closed due to keepalive timeout Raise the per-client cap to 20 -- comfortably above the handful of concurrent builds plus periodics seen in practice -- and the worker pool to 50 so that lifting the per-client limit does not starve other connections such as live migration, health checks and periodic tasks. Both values are exposed as variables (libvirt_max_client_requests and libvirt_max_workers) so operators can tune them further. This is not a Kolla-specific workaround -- the stock libvirtd.conf comment warns that setting max_client_requests too low "may cause keepalive timeouts", and since 2022 libvirtd emits the runtime warning quoted above. As Daniel Berrange explained when he added that warning, the keepalive packets "are subject to the 'max_client_requests' limit just like any API calls", so a client that saturates the limit can have its connection dropped: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/5NIJPA7SJVY55VSK7RTEDEHKWOZYJKT6/ Red Hat documents the same tuning for libvirt hosts under load in its knowledge base, notably "Too many active clients are dropping the connection" at https://access.redhat.com/solutions/28146. Assisted-By: Claude Opus 4.8 (1M context, medium effort) Partial-Bug: 2158515 Change-Id: I131c3d4d57dc73ca4dd421cc8f28465248c9830e (cherry picked from commit 5acd2f91d22213ef04bd6988a2a0261982b0cf05) Signed-off-by: Michael Still <mikal@stillhq.com> 2024-04-25T12:48:36+00:00 Zuul zuul@review.opendev.org 2024-04-25T12:48:36+00:00 urn:sha1:d524c25ba8a4e8eb28379d6d5f66b01c76df7ea7 2024-04-19T07:43:59+00:00 Mark Goddard mark@stackhpc.com 2024-04-19T07:32:45+00:00 urn:sha1:c3b566218128c3819b127c3b3b66b57cd1bdde83 This reverts commit 9675347fecd2907e7f60e84e405abcedf426b229. Reason for revert: Masakari CI is working Change-Id: Ifb44a437b96c0c029210b1491ee6c6403ee9e6fb 2024-01-22T12:25:51+00:00 hongbin hongbin034@gmail.com 2023-12-21T07:41:18+00:00 urn:sha1:1eeb3a5395529b40994d7e25f89a0a7cf9799132 This reverts commit f8d4db54db8f2ab2c7f99f0fcbc456177d33ddad. Reason for revert: Zun was deprecated due to the hard dependency on old version of docker and etcd. Right now, the problem was fixed. Change-Id: I9fe760e2b9b774b5ea623e6b305de004fc0c6dd2 2024-01-22T12:24:37+00:00 hongbin hongbin034@gmail.com 2023-12-21T05:54:06+00:00 urn:sha1:9c77220f6eeef2698a1f185438ead393b9253c21 This reverts commit 94a74f58c7891e41ca637c4e2a6450ca88cbfb56. Reason for revert: Zun/Kuryr-libnetwork has switched to "local" scope which doesn't require docker 20 anymore. It should work for latest docker version. Related patches: https://review.opendev.org/c/openstack/zun/+/903884 Change-Id: Ieb545ae5a5917322f599728587c3f04ea8356126 2023-11-16T17:45:15+00:00 Dr. Jens Harbott harbott@osism.tech 2023-11-16T17:44:06+00:00 urn:sha1:9675347fecd2907e7f60e84e405abcedf426b229 Since the CI is failing and there are no contributors working on fixing this, we are deprecating the service. Also mention other deprecations in the README. Change-Id: Ifff11645f6b4682810c042a0421d1eb569ab0ede 2023-09-26T20:48:12+00:00 Jan Gutter github@jangutter.com 2023-09-26T20:37:32+00:00 urn:sha1:f8d4db54db8f2ab2c7f99f0fcbc456177d33ddad * Zun currently has a dependency on a combination of Docker and etcd that is no longer installable (or supported) in newer distros. * These components are core dependencies of other systems and vendoring them will be very painful. * Work to update Zun past this is scheduled for the 'C' cycle. * If Zun regains compatibility, backports to this series will be considered. Change-Id: I6a2a4ffdcaf1c4fab1097e94d5f255ffb9a2947d 2023-09-13T15:56:33+00:00 Jan Gutter github@jangutter.com 2023-09-11T19:11:51+00:00 urn:sha1:994ab9bb9b318a1cddca9dec81b866ad963d1553 * When the Debian job got upgraded from Bullseye to Bookworm, the non-voting Zun job started failing. * This is because Debian Bookworm doesn't support the required Docker 20 pinning. * Until Zun gains support for newer dependencies this job will keep failing. Change-Id: I1e1d9ea7c76d1b2f77b09b7948f138b88ef483d2 2023-03-21T08:34:31+00:00 Michal Nasiadka mnasiadka@gmail.com 2023-02-15T09:50:49+00:00 urn:sha1:94a74f58c7891e41ca637c4e2a6450ca88cbfb56 Related-Bug: #2007142 Change-Id: I9ce2a9ce5413e77625201f3986967e31a679ad9c 2023-02-03T09:55:29+00:00 Pierre Riteau pierre@stackhpc.com 2023-02-03T09:55:29+00:00 urn:sha1:cbf6ce640a10414492a10584be1b537c6a510257 Change-Id: I0ff303a2fad2edbcedbe88486b272d2efa765d8d