diff options
| -rw-r--r-- | contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh | 24 | ||||
| -rw-r--r-- | doc/source/reference/message-queues/rabbitmq.rst | 19 |
2 files changed, 43 insertions, 0 deletions
diff --git a/contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh b/contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh new file mode 100644 index 000000000..ffe2c7c20 --- /dev/null +++ b/contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh @@ -0,0 +1,24 @@ +#!/bin/bash + +running_nodes=$(docker exec rabbitmq rabbitmqctl cluster_status \ + | awk '/Running Nodes/{f=1;next}/Versions/{f=0}f{print $1}') + +streams=$(docker exec rabbitmq rabbitmqctl list_queues name type --silent | awk '/stream/ {print $1}') + +for stream in $streams; do + stream_nodes=$(docker exec rabbitmq rabbitmq-streams stream_status -s $stream \ + | awk 'NR>3 {print $4}') + + missing=0 + for node in $running_nodes; do + if ! grep -qx "$node" <<< "$stream_nodes"; then + echo "MISSING replica of $stream on $node" + missing=1 + docker exec rabbitmq rabbitmq-streams add_replica $stream $node + fi + done + + if [ $missing -eq 0 ]; then + echo "OK: All running nodes have a replica of $stream" + fi +done diff --git a/doc/source/reference/message-queues/rabbitmq.rst b/doc/source/reference/message-queues/rabbitmq.rst index 375d58fe4..3bd364564 100644 --- a/doc/source/reference/message-queues/rabbitmq.rst +++ b/doc/source/reference/message-queues/rabbitmq.rst @@ -159,3 +159,22 @@ different type, the follow procedure will be needed. .. code-block:: console kolla-ansible deploy --tags <service-tags> + +Handling Stream Replicas +~~~~~~~~~~~~~~~~~~~~~~~~ + +RabbitMQ streams are expected to be replicated across the nodes in the +cluster. However, RabbitMQ itself will only create replicas of a stream when +the stream is initially declared. This means that any streams declared when a +RabbitMQ node is out of service must be explicitly managed by an operator. +RabbitMQ documents how to manage stream replicas here: +https://www.rabbitmq.com/docs/streams#member-management + +An example script to create any missing stream replicas can be found under +`kolla-ansible/contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh +<https://opendev.org/openstack/kolla-ansible/src/branch/master/contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh>`__. +This should be executed from a host running the RabbitMQ container. +Currently, membership changes for streams `is not entirely safe +<https://github.com/rabbitmq/rabbitmq-server/discussions/14246>`__, so this +script should only be used when the RabbitMQ cluster is in a known healthy +state. |