systemd: handle running container without systemd unit

MariaDB bootstrap has a phase where the first MariaDB container
is running with Galera bootstrap - after a check that WSREP
is synced is successful - we restart the container.

The bootstrap container is named mariadb and running with
docker_restart_policy: "no" - the restarted container should be running
in systemd.

Before this patch the code created a systemd unit but it was initially
stopped - so stopping was always a success - and the container would be
killed with SIGKILL on removal (which obviously breaks MariaDB).

This patch also improves docker/systemd stops by waiting for real
unit/container stop and adds failing CI for containers that are
killed with signal 9.

Closes-Bug: #2029613

Change-Id: I0a03e509ce228a50e081fcab44d2b4831251190c

2 files changed, 9 insertions, 0 deletions

diff --git a/tests/check-logs.sh b/tests/check-logs.sh
index 42a8a4541..2da6b5596 100755
--- a/tests/check-logs.sh
+++ b/tests/check-logs.sh
@@ -22,6 +22,10 @@ function check_fluentd_log_file_for_level {
     sudo egrep "\[$2\]:" $1
 }
 
+function check_docker_log_file_for_sigkill {
+    sudo journalctl --no-pager -u ${CONTAINER_ENGINE}.service | grep "signal 9"
+}
+
 function filter_out_expected_critical {
     # $1: file
     # Filter out expected critical log messages that we do not want to fail the
@@ -79,6 +83,10 @@ if check_fluentd_log_file_for_level $fluentd_log_file error >/dev/null; then
     echo >> $fluentd_error_summary_file
 fi
 
+if check_docker_log_file_for_sigkill >/dev/null; then
+    any_critical=1
+    echo "(critical) Found containers killed using signal 9 (SIGKILL) in docker logs."
+fi
 
 if [[ $any_critical -eq 1 ]]; then
     echo "Found critical log messages - failing job."
diff --git a/tests/kolla_docker_tests/test_docker_worker.py b/tests/kolla_docker_tests/test_docker_worker.py
index 2c5efd716..d85c2255d 100644
--- a/tests/kolla_docker_tests/test_docker_worker.py
+++ b/tests/kolla_docker_tests/test_docker_worker.py
@@ -536,6 +536,7 @@ class TestContainer(base.BaseTestCase):
                                     'action': 'stop_container',
                                     'restart_policy': 'no'})
         self.dw.dc.containers.return_value = self.fake_data['containers']
+        self.dw.systemd.check_unit_file.return_value = False
         self.dw.stop_container()
 
         self.assertTrue(self.dw.changed)