systemd: handle running container without systemd unit

MariaDB bootstrap has a phase where the first MariaDB container
is running with Galera bootstrap - after a check that WSREP
is synced is successful - we restart the container.

The bootstrap container is named mariadb and running with
docker_restart_policy: "no" - the restarted container should be running
in systemd.

Before this patch the code created a systemd unit but it was initially
stopped - so stopping was always a success - and the container would be
killed with SIGKILL on removal (which obviously breaks MariaDB).

This patch also improves docker/systemd stops by waiting for real
unit/container stop and adds failing CI for containers that are
killed with signal 9.

Closes-Bug: #2029613

Change-Id: I0a03e509ce228a50e081fcab44d2b4831251190c

1 files changed, 8 insertions, 0 deletions

diff --git a/tests/check-logs.sh b/tests/check-logs.sh
index 42a8a4541..2da6b5596 100755
--- a/tests/check-logs.sh
+++ b/tests/check-logs.sh
@@ -22,6 +22,10 @@ function check_fluentd_log_file_for_level {
     sudo egrep "\[$2\]:" $1
 }
 
+function check_docker_log_file_for_sigkill {
+    sudo journalctl --no-pager -u ${CONTAINER_ENGINE}.service | grep "signal 9"
+}
+
 function filter_out_expected_critical {
     # $1: file
     # Filter out expected critical log messages that we do not want to fail the
@@ -79,6 +83,10 @@ if check_fluentd_log_file_for_level $fluentd_log_file error >/dev/null; then
     echo >> $fluentd_error_summary_file
 fi
 
+if check_docker_log_file_for_sigkill >/dev/null; then
+    any_critical=1
+    echo "(critical) Found containers killed using signal 9 (SIGKILL) in docker logs."
+fi
 
 if [[ $any_critical -eq 1 ]]; then
     echo "Found critical log messages - failing job."