diff options
| author | Michael Still <mikal@stillhq.com> | 2026-08-14 06:22:22 +1000 |
|---|---|---|
| committer | Michal Nasiadka <mnasiadka@gmail.com> | 2026-08-19 14:40:16 +0000 |
| commit | 762181ae4918e3e261df06ed983a1ef1cb61b10c (patch) | |
| tree | 33c3f9be23ba766fb124cee90ea10984d31fe1fc /etc/kolla | |
| parent | fe4f6adaf01e93af39cd28d3ad57d45b1db11884 (diff) | |
| download | kolla-ansible-762181ae4918e3e261df06ed983a1ef1cb61b10c.tar.gz kolla-ansible-762181ae4918e3e261df06ed983a1ef1cb61b10c.zip | |
Extend the nova_scheduler healthcheck timeout.
Nova changed in 2026.1 how long their services take to gracefully
shutdown.
The default value of [DEFAULT]/graceful_shutdown_timeout has been
changed from 60 to 180 seconds for all Nova services. [...]
Operators using external system (e.g. k8s, systemd) to manage the
Nova services should ensure that their service stop timeouts are
set to at least graceful_shutdown_timeout to avoid forcefully
killing service before Nova finish its graceful shutdown.
Kolla-Ansible failed to update the period of time that ansible waits
for Nova services to shut down. This causes both CI failures and
failures for deployers when trying to manage some Nova containers.
Prompt: Michael asked for assistance debugging why CI jobs were
failing with nova-scheduler timeouts.
Assisted-By: Claude Opus 5 (1M context, high effort) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context, high effort) <noreply@anthropic.com>
Closes-Bug: #2162860
Change-Id: If7c5a6d885b19e39cfc305440424f10de3a5ea05
Signed-off-by: Michael Still <mikal@stillhq.com>
(cherry picked from commit 4ce86ef1db631c15e6e5e142d2908f73348377d4)
Diffstat (limited to 'etc/kolla')
| -rw-r--r-- | etc/kolla/globals.yml | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/etc/kolla/globals.yml b/etc/kolla/globals.yml index 2d7cdcda4..f5f8cdad2 100644 --- a/etc/kolla/globals.yml +++ b/etc/kolla/globals.yml @@ -219,6 +219,21 @@ workaround_ansible_issue_8743: true #default_container_healthcheck_timeout: 30 #default_container_healthcheck_retries: 3 #default_container_healthcheck_start_period: 5 +# Individual services accept per-service overrides of the options above, +# named <service>_healthcheck_<option>. nova_scheduler ships with a wider +# one: its healthcheck asserts an established AMQP connection, but every +# deploy and every reconfigure signals the scheduler to restart its +# workers, and they take [DEFAULT]/manager_shutdown_timeout in nova.conf +# (160 seconds by default) to come back. The default 30 second interval +# with 3 retries only covers 90 seconds of that. +#nova_scheduler_healthcheck_retries: 8 +# The conductors wait for the same period, but are only ever stopped by +# the container engine, which kills them after docker_graceful_timeout +# and so stays inside what the default healthcheck covers. Raise these +# to match if docker_graceful_timeout is raised towards +# manager_shutdown_timeout. +#nova_conductor_healthcheck_retries: 3 +#nova_super_conductor_healthcheck_retries: 3 ################## # Firewall options |