| Age | Commit message (Collapse) | Author | Files | Lines |
|
This reverts commit c62d5e509b74fd39b09c272fb731b73088c3a55b.
Reason for revert: Fixing broken tests
Change-Id: Iea1d9ad71c64642fadb7d02d50f9421cbe64acb7
Signed-off-by: Dawud <dawud@stackhpc.com>
(cherry picked from commit 1839ef9a957432b08c81756780e20d7374e5bb89)
|
|
Closes-Bug: #2112280
Change-Id: I52499bdf2b7fc70f91a4c2920749bfef83a79850
Signed-off-by: Rafal Lewandowski <rafal@stackhpc.com>
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
(cherry picked from commit e302bab31f30296f7d6f41c3d46e19cf5668b97a)
|
|
The migration to uWSGI has been successfully completed.
All API services now run exclusively under uWSGI. The
*_wsgi_provider variables have been removed from all affected
roles along with leftover Apache wsgi config templates and
fluentd Apache log handling.
Rework Apache fluentd input configuration so it doesn't glob out
whole /var/log/kolla/*/* and just include required services when
enabled.
Change-Id: I1475e629d033163a50a5c681c8fe2557a7b10bae
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
Signed-off-by: Bartosz Bezak <bartosz@stackhpc.com>
|
|
Configure dedicated health check endpoints for Load Balancers
Standardize health check paths in OpenStack and Grafana
Ansible defaults to improve Load Balancer (HAProxy) reliability.
The 'httpchk' configuration for the modified roles now uses
dedicated health check endpoints (e.g., /healthcheck or
/api/health) and enforces an HTTP 200 OK expectation.
This ensures faster and more robust failure detection.
Additionally, by moving health checks away from generic
endpoints (such as OPTIONS /), application logs are now
much cleaner. The log files will no longer be flooded
every second with HAProxy probe entries like:
"OPTIONS / HTTP/1.0" 200 94 0.000954
NOTE: The following services are exceptions to this
standardization and still use the root endpoint (`/`)
for health checks due to the lack of a dedicated health
check middleware in their default API paste configuration:
**cyborg_api**, **masakari**, **mistral_api**, **nova_api**,
**octavia_api**, **placement**, **tacker**, **venus**, and **watcher**.
Updated roles:
* aodh
* barbican
* cinder
* cloudkitty
* cyborg
* designate
* etcd
* glance
* gnocchi
* grafana
* heat
* horizon
* influxdb
* ironic
* keystone
* loadbalancer
* magnum
* manila
* masakari
* mistral
* neutron
* nova
* nova-cell
* octavia
* opensearch
* placement
* rabbitmq
* skyline
* tacker
* trove
* venus
* watcher
* zun
Change-Id: Ibbb9fda45a98565ab12e784bbef80be787bd6638
Signed-off-by: Piotr Milewski <vurmil@gmail.com>
|
|
|
|
This reverts commit 7aab7e997e4e99e4d227a997ffc90d376bb49ab1.
Reason for revert: This broke cloudkitty
Change-Id: I8735eb5ef07b9a3c7d15ac730a7fccec9ced64e2
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
|
|
|
|
Change-Id: I37fab87536e882541ef675449265bef9599d79b1
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
Change-Id: I3f001ce2e44b092851779c881cc1eab2f7cec639
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
|
|
When Prometheus and Openstack Exporter are enabled, set ``scope_key`` to
``tenant_id``, to ensure CloudKitty can fetch valid metrics from the
OpenStack Exporter.
Change-Id: I70197aa9b8accbb937371ce3da8832b8759abc5a
Signed-off-by: Leonie Chamberlin-Medd <leonie@stackhpc.com>
|
|
Kolla delivers InfluxDB v1, which is EOL (unless you have paid support).
There are no plans in upgrading InfluxDB to v2 - so let's drop it.
Telegraf was deprecated in June 2025.
Removing both in one patch for convenience.
Change-Id: Ieee2be3a06a5ac9504a822cc0a727e662146b260
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
Adds haproxy_cloudkitty_api_client_timeout and
haproxy_cloudkitty_api_server_timeout for the cloudkitty_api and
cloudkitty_api_external HAProxy frontend/backend entries. Defaults
are set to haproxy_server_timeout and haproxy_client_timeout.
Change-Id: I0294cae7379b017cb470fe43fe2594d66a627499
Signed-off-by: Dawud <dawud@stackhpc.com>
|
|
Since ipwrap filter is getting moved another time
inside ansible code - let's switch to j2lint that is
actively maintained and it's rules will help in
readability of jinja2 templates
Change-Id: I6762b691b927de31fa1abd83c4c14e87226fe5e2
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
Remove /etc/timezone bind mount since Debian does not provide
that file anymore.
Depends-On: https://review.opendev.org/c/openstack/kolla/+/964793
Change-Id: If1a289068093843184f0d7b495815db006f2d563
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
Change-Id: I25a950ce5de121726abad7ccb660a205e88f6086
Signed-off-by: Bertrand Lanson <bertrand.lanson@protonmail.com>
|
|
Change-Id: I7c693403b1768c6c97eea90c3dfe9326764c3230
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
Change-Id: I215731b5cccf8354e69e6552472a9781f29a78d9
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
Bumped collections in requirements-core.yml to Ansible 12 major versions,
see [1].
[1]: https://github.com/ansible-community/ansible-build-data/blob/main/12/ansible-12.1.0.yaml
Depends-On: https://review.opendev.org/c/openstack/kolla/+/960307
Co-authored-by: Doug Szumski doug@stackhpc.com
Change-Id: If8a29052d8a43ffc0fef11514adabb5e4ab9d977
Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com>
|
|
Since Kolla-Ansible now supports TLS for MariaDB,
this patch adds support for services that can use
this feature.
This feature can be used only when ProxySQL is used
as LB.
Signed-off-by: Michal Arbet <michal.arbet@ultimum.io>
Change-Id: I052c763073c5da29cb8e78004a01a22217dde5da
|
|
|
|
Some queries (e.g. SET AUTOCOMMIT, ROLLBACK) may be executed before
schema is selected, causing them to be routed to hostgroup 0, which
does not exist if user using different `mariadb_default_shard_id`.
This change sets default_hostgroup (shard_id) for users to ensure
correct routing even when schema is not specified.
Also adjusts nova-cell user generation logic to include shard assignment,
so queries from nova_<cell> users are properly routed, even if schema is
not used directly.
This avoids silent misrouting and improves reliability in multi-shard
deployments.
Note: These errors are not visible in CI because
mariadb_default_shard_group_id = 0 and ProxySQL routes to hostgroup 0
by default when no routing rule matches.
Depends-On: https://review.opendev.org/c/openstack/kolla/+/951741
Closes-Bug: #2112339
Change-Id: I4f3380a83fa992c3413e82a4692c26a07b25d57f
|
|
Stop repeating the same code (actually 158 times).
Change-Id: I2fd04e6ed72f3702eb41fe5cf172f47a33e93bbd
|
|
We will be upgrading RabbitMQ to version 4.0 in Epoxy. This will not
work without quorum queues being enabled.
Change-Id: Ic6ad64bf8c62bbff175e15029eb121814032c40e
|
|
We're going to upgrade RabbitMQ to 4.0, so this option will not longer
be supported.
Change-Id: Ide75a8c9086798bf4bdf5bc02d4a1be17017884f
|
|
The global configuration of rabbit_qos_prefetch_count = 1
in oslo.messaging is a deliberate choice that balances
compatibility, fairness, and reliability across all
OpenStack services.
While some services (particularly those using quorum queues)
could theoretically benefit from a higher or even unlimited
prefetch count for performance reasons, this is not universally
safe. Specifically, RabbitMQ stream queues - which are used in
certain OpenStack components do not support a prefetch_count of
0. Setting this value to 0 would result in runtime errors
or stalls, as the stream protocol requires a positive credit
window to function properly. Oslo.messaging enforces this by
raising an error if prefetch_count = 0 is set while using stream
queues.
A value of 1 is the lowest legal and universally compatible
setting. It works safely with both stream and quorum queues.
It also improves fairness in message distribution across worker
processes. In distributed services like Nova, Neutron, or
Cinder, this helps avoid uneven load where one worker might
prefetch many messages while others remain idle.
A low prefetch count ensures that messages are pulled only
when the worker is ready, promoting better load balancing and
more predictable performance.
Although a higher prefetch value could increase throughput in
certain scenarios, it comes at the cost of memory usage, risk
of overloading specific nodes, and potential starvation of
others. Until oslo.messaging supports queue-type-specific or
per-service tuning, the value of 1 remains the safest and
most predictable option that works well in mixed OpenStack
environments.
In summary, rabbit_qos_prefetch_count = 1 is not about
optimizing raw throughput for any one service, but about
ensuring stable, fair, and reliable behavior across all
services that rely on oslo.messaging, regardless of the queue
type or broker backend used.
Change-Id: I541f704bfa2e98068096331afbdb591659cbc40b
|
|
This helps to improve the reliability of openstack services when
a rabbitmq node has issues.
See also: https://bugs.launchpad.net/oslo.messaging/+bug/2031497
Adjust the upgrade tests similar to what was done
during the introduction of quorum queues in I6c033d460a5c9b93c346e9e47e93b159d3c27830
Closes-Bug: #2078339
Partial-Bug: #2077448
Depends-On: https://review.opendev.org/c/openstack/oslo.messaging/+/888479
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/924623
Signed-off-by: Sven Kieske <kieske@osism.tech>
Change-Id: Idb8a8d2e560206f7697c0771c9ae3913268fa6dd
|
|
Adds the variable ``om_enable_queue_manager`` to configure enabling
oslo.messaging Queue Manager for all services which use RabbitMQ. This
is enabled by default.
This is required before we can move away from transient queues
(``rabbit_transient_quorum_queue``) as Queue Manager is needed to avoid
consuming all erlang atoms after some time. It is also a useful feature
for debugging, as queues are now named with hostnames and services
included.
Also setting ``lock_path`` and mounting ``dev/shm`` is dependent on
``om_enable_queue_manager``, these are now enabled too. This will allow
us to backport these features to Caracal and Dalamation without
enforcing the changes right away.
Why is -t added for exec commands?
When using Podman, behavior differs from Docker regarding exec
commands. If -t (which allocates a pseudo-terminal) is not used, the
process runs as a child of podman exec without a PTY. This causes
Python inside the container to assign PGID=0 because the pseudo-terminal
is missing, leading to issues with non-existent paths like /proc/0/....
Adding -t ensures proper PGID assignment, preventing these issues and
ensuring consistent behavior across Docker and Podman.
Side-notes:
The quorum queue precheck needed to be updated as queue manager puts
"_fanout" on the end of the name instead.
We need to override the ``[oslo_messaging_rabbit] processname`` for
services running udner wsgi, as they will otherwise all use the same
processname ``mod_wsgi``. This will cause Permission Errors trying to
access the same file in shared memory as services run with different
users.
Change-Id: Iae5f268e778fbbd2b744dc71a84253ec9e758a99
|
|
In the past, we agreed to use lowercase letters
for boolean representation. This patch implements
exactly that, just changing True -> true, False -> false.
Change-Id: Ia601d84283dd8bc45e56e1029106de8836139173
|
|
Move actions responsible for info about containers
from kolla_container module to kolla_container_facts.
Also fixes a bug with inconsistencies between docker
and podman in kolla_container_facts.
Closes-bug: #2084878
Change-Id: I1db88e28a828ebf073f018b2bae1d9556ec22807
Signed-off-by: Ivan Halomi <ivan.halomi@tietoevry.com>
Signed-off-by: Martin Hiner <martin.hiner@tietoevry.com>
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
|
|
|
|
This allows operators quickly diagnose all containers across
all hosts by running kolla-ansible check. It returns a list
of containers that are missing, not running or in unhealthy
state for each OpenStack service.
Change-Id: I36119ccdeb264aa3de928ec2254d6ff4cc955bfb
Implements: blueprint check-containers
Co-Authored-By: Roman Krček <roman.krcek@tietoevry.com>
|
|
The Oslo.messaging project implemented a nice feature called Queue
Manager. This means that instead of using random queue names in
RabbitMQ, it uses queue names composed of hostname, process-name and a
counter. For proper functioning, the code stores some informations in
/dev/shm. This is needed to avoid creating queues with the same name.
We'd otherwise hit this where multiple services run under mod_wsgi, or
with services such as Magnum just within a single container as it needs
to create multiple "reply" queues.
This patch mounts /dev/shm in containers where oslo_messaging is used.
This is enabled when ``om_enable_queue_manager`` is set to ``True``.
Queue Manager will be supported in a follow-up patch.
Change-Id: Ib85ce252374fae917d329e1824800a288c6bc9f1
|
|
The `www_authenticate_uri` parameter is used to indicate to clients
where they should get a token from in order to authenticate against a
service. Most clients are not expected to be able to talk to the
internal identity endpoint, so this parameter should refer to the public
endpoint instead, see also [0].
[0] https://opendev.org/openstack/keystonemiddleware/src/branch/master/keystonemiddleware/auth_token/_opts.py#L31-L50
Change-Id: Ic99804967b5a62b5a9e39486749474520734ba48
|
|
Address occasional issues where TCP connections appear healthy,
yet the web servers within containers fail to respond,
resulting in requests being sent to unhealthy servers.
Implemented for services I currently use.
- Aodh (OPTIONS to /, expects 2XX or 3XX)
- Barbican (OPTIONS to /, expects 2XX or 3XX)
- Blazar (OPTIONS to /, expects 401)
- Cinder API (OPTIONS to /, expects 2XX or 3XX)
- CloudKitty (OPTIONS to /, expects 2XX or 3XX)
- Designate (OPTIONS to /, expects 2XX or 3XX)
- Glance (OPTIONS to /, expects 2XX or 3XX)
- Gnocchi (OPTIONS to /, expects 2XX or 3XX)
- Grafana (OPTIONS to /, expects 2XX or 3XX)
- Heat (OPTIONS to /, expects 2XX or 3XX)
- Horizon (OPTIONS to /, expects 2XX or 3XX)
- Ironic (OPTIONS to /, expects 2XX or 3XX)
- Keystone (OPTIONS to /, expects 2XX or 3XX)
- Magnum (OPTIONS to /, expects 2XX or 3XX)
- Manila (OPTIONS to /, expects 2XX or 3XX)
- Masakari (OPTIONS to /, expects 2XX or 3XX)
- Mistral (OPTIONS to /, expects 2XX or 3XX)
- Nova API (OPTIONS to /, expects 2XX or 3XX)
- Nova Metadata (OPTIONS to /, expects 2XX or 3XX)
- Neutron (OPTIONS to /, expects 2XX or 3XX)
- Opensearch (OPTIONS to /, expects 2XX or 3XX)
- Opensearch Dashboards (OPTIONS to /, expects 401)
- Placement (GET to /, expects 2XX or 3XX)
- Prometheus (OPTIONS to /, expects 2XX or 3XX)
- Prometheus AlertManager (OPTIONS to /, expects 2XX or 3XX)
- Prometheus Openstack Exporter (OPTIONS to /, expects 2XX or 3XX)
- Prometheus Server (OPTIONS to /, expects 2XX or 3XX)
- Skyline API (OPTIONS to /docs, expects 2XX or 3XX)
- Skyline Console (GET to /, expects 2XX or 3XX)
- Swift (OPTIONS to /info, expects 2XX or 3XX)
- Trove (OPTIONS to /, expects 2XX or 3XX)
- Venus (OPTIONS to /, expects 2XX or 3XX)
- Watcher (GET to /, expects 2XX or 3XX)
- Zun (OPTIONS to /, expects 2XX or 3XX)
Change-Id: I839f7f1051182fe797394e5436571d64d5c5b5a4
|
|
This might fix some hidden bugs where the check tasks forgot to
include params important for the service.
We also get a nice optimisation by using a filtered loop instead
of task skipping per service with 'when'. As proven in
https://review.opendev.org/c/openstack/kolla-ansible/+/914997
This refactoring allows for further optimisation and
fixing work to proceed with much less hassle. Including getting
rid of many notify statements as the restarts are now safely handled
by check-containers. Some notifies had to stay, because of special
edge cases eg. in rolling upgrades and loadbalancer config.
One downside is we remove the little optimisation for Zun that
ignored config change for copying loopback but this is an
acceptable tradeoff considering the benefits above.
Co-Authored-By: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: I855dfef33aa0f3fd1301295bb8ede3e587e7162a
Partially-Implements: blueprint performance-improvements
|
|
This patch builds upon genconfig optimisation and it takes it
further by not having genconfig ever touch the handlers!
Calling the handlers and skipping them created an unnecessary slow down
if only config was ran. It also depends on the config checking fix.
This gets us closer to the single responsibility principle -
config only generates the config, container checks only validate
whether container restart is needed.
And this also means that we will have single place were containers
are restarted, were we can fix the ansible quirk of it restarting
the whole group even when one container changed in the following patches.
The only exception is the loadbalance role. As the loadbalancer services
have their config altered by other roles registering their services
using loadbalancer-config. This is in contrast to typical roles,
which do config in one step and can then run check-containers in
the next step.
Fixes some handlers that were missing the necessary guard,
making genconfig actually able to restart some containers.
Future work:
- optimise config by doing local generation and mass rsync
- support for reloads
- unconditional restart/reload (separate action)
- make 'reconfigure' act like 'genconfig' + 'deploy-containers'
- this would avoid calling bootstrapping each time but might
be tricky as it would break current compatibility
- could call this 'reconfigure-containers' and deprecate
'reconfigure'
- fix the ansible quirk that notifies more handlers then intended
Change-Id: I0ce24043ae5486b2b55489ba40abe2b96b0991a6
Partially-Implements: blueprint performance-improvements
Co-Authored-By: Roman Krček <roman.krcek@tietoevry.com>
|
|
This is a prerequisite for patchset #745164
This fixes unwanted restarts during copying of certificates.
By removing conditional statements from role handlers in #745164,
copying certificates caused containers to restart, this is unwanted
during the genconfig process. However, if we would remove handler
notifiers from copying certificates, the container would never
restart, since from #745164, containers will restart only if any
of the files specified in config.json change. Certificates are now
copied to intermediary location inside of the container, from which
the script kolla_copy_cacerts will install them in the trust store.
Depends-on: https://review.opendev.org/c/openstack/kolla/+/926882
Change-Id: Ib89048c7e0f250182c4bf57d8c8a1b5478e9b4ab
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
|
|
|
|
|
|
Build upon changes in kolla which change strategy of installing projects
in containers when in dev mode. This fixes problems where when package
file manifest changes, the changes were not reflected in to
devmode-enabled container.
It changes the strategy of installing projects in dev mode in containers.
Instead of bind mounting the project's git repository to the venv
of the container, the repository is bind mounted to
/dev-mode/<project_name> from which the it is installed using pip
on every startup of the container using kolla_install_projects script.
Also updates docs to reflect the changes.
Depends-On: https://review.opendev.org/c/openstack/kolla/+/925712
Closes-Bug: #1814515
Singed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: If191cd0e3fcf362ee058549a1b6c244d109b6d9a
|
|
|
|
For possible config options see docs
https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#memcache-protection
Closes-bug: #1850733
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
Change-Id: I169e27899f7350f5eb8adb1f81a062c51e6cbdfc
|
|
Refactor that prepares kolla_container_facts
module for introducing more actions that will be moved
from kolla_container module and kolla_container_volume_facts.
This change is based on a discussion about adding a new action
to kolla_container module that retrieves all names of the running
containers. It was agreed that kolla-ansible should follow Ansible's
direction of splitting modules between action modules and facts
modules. Because of this, kolla_container_facts needs to be able
to handle different requests for data about containers or volumes.
Change-Id: Ieaec8f64922e4e5a2199db2d6983518b124cb4aa
Signed-off-by: Ivan Halomi <ivan.halomi@tietoevry.com>
|
|
|
|
The Kolla project supports building images with
user-defined prefixes. However, Kolla-ansible is unable
to use those images for installation.
This patch fixes that issue.
Closes-Bug: #2073541
Change-Id: Ia8140b289aa76fcd584e0e72686e3786215c5a99
|
|
Most roles are not leveraging the jinja filters available.
According to [1] filtering the list of services makes the execution
faster than skipping the tasks.
This patchset also includes some cosmetic changes to genconfig.
Individual services are now also using a jinja filter. This has
no impact on performance, just makes the tasks look cleaner.
Naming of some vars in genconfig was changed to "service" to make
the tasks more uniform as some were previously using
the service name and some were using "service".
Three metrics from the deployment were taken and those were
- overall deployment time [s]
- time spent on the specific role [s]
- CPU usage (measured with perf) [-]
Overall genconfig time went down on avg. from 209s to 195s
Time spent on the loadbalancer role went down on avg. from 27s to 23s
Time spent on the neutron role went down on avg from 102s to 95s
Time spent on the nova-cell role went down on avg. from 54s to 52s
Also the average CPUs utilized reported by perf went down
from 3.31 to 3.15.
For details of how this was measured see the comments in gerrit.
[1] - https://github.com/stackhpc/ansible-scaling/blob/master/doc/skip.md
Change-Id: Ib0f00aadb6c7022de6e8b455ac4b9b8cd6be5b1b
Signed-off-by: Roman Krček <roman.krcek@tietoevry.com>
|
|
This change automates the prometheus blackbox monitoring configuration
for common endpoints. Custom endpoints can be added to
prometheus_blackbox_exporter_endpoints_custom.
Change-Id: Id6f51a2bebee3ab63b84ca7032aad17c2933838c
|
|
Depends-On: https://review.opendev.org/c/openstack/cloudkitty/+/880739
Change-Id: Ib8d7182cc4b8a0c7d320ba2c51b2157782030317
|