mariadb: add support for custom Galera notification script

This patch adds the ability for operators to provide a custom
wsrep-notify.sh script to handle Galera cluster events (e.g., node
status changes, membership updates).

If a script is placed in the custom configuration directory on the
control host (/etc/kolla/config/mariadb/wsrep-notify.sh), Kolla
Ansible will copy it to the nodes, mount it into the MariaDB
container at /usr/local/bin/ and enable 'wsrep_notify_cmd'.

This enables integration with external systems like Alertmanager
or local logging without hardcoding scripts into the roles.

Change-Id: I7887889cb2e4038baeb566884cb847a71b5f99a6
Signed-off-by: Piotr Milewski <vurmil@gmail.com>

1 files changed, 105 insertions, 0 deletions

diff --git a/doc/source/reference/databases/mariadb-guide.rst b/doc/source/reference/databases/mariadb-guide.rst
index c4a40740e..b09ab11b6 100644
--- a/doc/source/reference/databases/mariadb-guide.rst
+++ b/doc/source/reference/databases/mariadb-guide.rst
@@ -69,3 +69,108 @@ Backup and restore is working only for default shard as kolla currently
 using HAProxy solution for MariaDB loadbalancer which is simple TCP and
 has configured only default shard hosts as backends, therefore backup
 script will reach only default shard on ``kolla_internal_vip_address``.
+
+.. _mariadb-notifications:
+
+Cluster Event Notifications
+---------------------------
+
+Kolla Ansible supports native MariaDB Galera cluster notifications. This allows
+operators to execute custom logic whenever a node status or cluster membership
+changes.
+
+To enable this feature, you must provide a custom script named
+``wsrep-notify.sh`` in the following directory on the control host:
+``/etc/kolla/config/mariadb/wsrep-notify.sh``.
+
+Kolla Ansible will automatically detect the presence of this file, copy it to
+the MariaDB nodes, and configure the ``wsrep_notify_cmd`` directive in the
+MariaDB configuration.
+
+Example 1: Integration with Prometheus Alertmanager
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can use this feature to send alerts directly to Prometheus Alertmanager.
+Save the following content as ``/etc/kolla/config/mariadb/wsrep-notify.sh``:
+
+.. code-block:: bash
+
+    #!/bin/bash
+
+    # List of Alertmanager instances (direct IP addresses)
+    # Using kolla_address to bypass VIP for higher reliability
+    ALERTMANAGERS=(
+    {% for host in groups['prometheus-alertmanager'] %}
+      "{{ internal_protocol }}://{{ 'api' | kolla_address(host) |
+      put_address_in_context('url') }}:{{ hostvars[host]
+      ['prometheus_alertmanager_port'] }}/api/v2/alerts"
+    {% endfor %}
+    )
+
+    # Authentication credentials from Kolla configuration
+    AUTH="{{ prometheus_alertmanager_user }}:{{
+    prometheus_alertmanager_password }}"
+    HOSTNAME=$(hostname)
+
+    # Prepare the JSON payload for the Alertmanager API
+    PAYLOAD=$(cat <<EOF
+    [
+      {
+        "labels": {
+          "alertname": "GaleraEvent",
+          "severity": "warning",
+          "instance": "$HOSTNAME"
+        },
+        "annotations": {
+          "description": "Galera cluster event detected: $*",
+          "summary": "Galera Status Change on $HOSTNAME"
+        }
+      }
+    ]
+    EOF
+    )
+
+    # Iterate through instances until the first successful POST
+    for URL in "${ALERTMANAGERS[@]}"; do
+        echo "Attempting to send alert to: $URL"
+        if curl -X POST "$URL" \
+             -u "$AUTH" \
+             -H "Content-Type: application/json" \
+             -d "$PAYLOAD" \
+             --fail --silent --show-error; then
+            echo "Alert sent successfully."
+            exit 0
+        fi
+        echo "Failed to reach $URL, trying next instance..."
+    done
+
+    echo "Error: Could not send alert to any Alertmanager instance."
+    exit 1
+
+.. note::
+
+   When providing this script, ensure it has the correct permissions. Kolla
+   Ansible will attempt to set execution permissions (``0755``) automatically
+   during the deployment.
+
+Example 2: Logging Cluster Changes to a File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Alternatively, you can log all cluster events to a local file for later audit
+or simple monitoring. Save the following content as
+``/etc/kolla/config/mariadb/wsrep-notify.sh``:
+
+.. code-block:: bash
+
+   #!/bin/bash
+
+   LOG_FILE="/var/log/kolla/mariadb/galera_cluster_events.log"
+   TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
+
+   echo "[$TIMESTAMP] Host: $HOSTNAME | Event: $*" >> "$LOG_FILE"
+
+.. note::
+
+   When providing this script, ensure it has the correct permissions. Kolla
+   Ansible will attempt to set execution permissions (``0755``) automatically
+   during the deployment.