Wait for slaves to sync before the recovery master restart.

During mariadb_recovery, recover_cluster.yml starts the bootstrap
node, starts the slave containers, and then recreates the master
container to clear pc.bootstrap. The "Wait for slave MariaDB" task
between those steps only waits for the service port to open, but a
freshly started slave is still receiving its state transfer (SST
then IST) at that point. The master restart is a Galera cluster
membership change, and a membership change while a joiner is
mid-IST makes the joiner abort with "IST didn't contain all write
sets, expected last: N last received: N-1 ... node restart
required". The joiner then drops to non-primary, self-leaves, and
sits permanently wsrep-disconnected; nothing restarts it, and the
Galera readiness check added by commit aed9d890 fails the recovery.

Over the last 30 days this hit 16 of 954 mariadb scenario CI runs
(1.7%), was terminal in every case, and accounts for about 30% of
all failures in those jobs, across Debian, Ubuntu and Rocky. The
Ubuntu failures are on MariaDB 11.4.12, which already contains the
upstream fixes for MDEV-36621 and MDEV-39643, so this window has to
be closed at the orchestration layer.

Add a wait for every slave to report wsrep_local_state_comment
Synced before the master restart, using the same defensive until
pattern as the bootstrap WSREP sync wait. The retry budget is
larger than the port waits because a slave may legitimately still
be receiving an SST at this point.

Prompt: Michael asked for a kerbside-patches patch fixing the
mariadb_recovery IST race we diagnosed from the kolla-ansible
989612 gate failures, where the master restart during recovery
races slaves that are still receiving state transfer.

Assisted-By: Claude Fable 5 (200K context, medium effort) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 (200K context, medium effort) <noreply@anthropic.com>

Closes-Bug: #2161773
Change-Id: Ifd0260fa4875051b1c38ee52de4f0de1806f6cc9
Signed-off-by: Michael Still <mikal@stillhq.com>
(cherry picked from commit cb259a3914180234cef888ddfe34b54f60f240a0)

1 files changed, 24 insertions, 0 deletions

diff --git a/ansible/roles/mariadb/tasks/recover_cluster.yml b/ansible/roles/mariadb/tasks/recover_cluster.yml
index 16f0d05fc..0592cb390 100644
--- a/ansible/roles/mariadb/tasks/recover_cluster.yml
+++ b/ansible/roles/mariadb/tasks/recover_cluster.yml
@@ -215,6 +215,30 @@
     - bootstrap_host is defined
     - bootstrap_host != inventory_hostname
 
+# The master restart below is a Galera cluster membership change. If it
+# happens while a slave is still receiving its IST, the joiner aborts with
+# "IST didn't contain all write sets ... node restart required" and stays
+# permanently disconnected, so wait for every slave to finish syncing first.
+- name: Wait for slave MariaDB to sync WSREP
+  become: true
+  kolla_toolbox:
+    container_engine: "{{ kolla_container_engine }}"
+    module_name: mysql_query
+    module_args:
+      login_db: "mysql"
+      login_host: "{{ api_interface_address }}"
+      login_port: "{{ mariadb_port }}"
+      login_user: "{{ database_user }}"
+      login_password: "{{ database_password }}"
+      query: 'SHOW STATUS LIKE "wsrep_local_state_comment"'
+  register: result
+  until: result.query_result is defined and result.query_result[0][0]['Value'] == "Synced"
+  retries: 30
+  delay: 10
+  when:
+    - bootstrap_host is defined
+    - bootstrap_host != inventory_hostname
+
 - name: Unset pc.bootstrap for primary MariaDB galera.cnf for next restart
   vars:
     service_name: "mariadb"