From 0743a9bf4b8bdce0aec8ee8640574ce3ee6a6b35 Mon Sep 17 00:00:00 2001
From: Doug Szumski <doug@stackhpc.com>
Date: Thu, 7 Jan 2021 17:14:15 +0000
Subject: Remove Monasca Log Transformer

Historically Monasca Log Transformer has been for log
standardisation and processing. For example, logs from different
sources may use slightly different error levels such as WARN, 5,
or WARNING. Monasca Log Transformer is a place where these could
be 'squashed' into a single error level to simplify log searches
based on labels such as these.

However, in Kolla Ansible, we do this processing in Fluentd so
that the simpler Fluentd -> Elastic -> Kibana pipeline also
benefits. This helps to avoid spreading out log parsing
configuration over many services, with the Fluentd Monasca output
plugin being yet another potential place for processing (which
should be avoided). It therefore makes sense to remove this
service entirely, and squash any existing configuration which
can't be moved to Fluentd into the Log Perister service. I.e.
by removing this pipeline, we don't loose any functionality,
we encourage log processing to take place in Fluentd, or at least
outside of Monasca, and we make significant gains in efficiency
by removing a topic from Kafka which contains a copy of all logs
in transit.

Finally, users forwarding logs from outside the control plane,
eg. from tenant instances, should be encouraged to process the
logs at the point of sending using whichever framework they are
forwarding them with. This makes sense, because all Logstash
configuration in Monasca is only accessible by control plane
admins. A user can't typically do any processing inside Monasca,
with or without this change.

Change-Id: I65c76d0d1cd488725e4233b7e75a11d03866095c
---
 tools/kolla-ansible | 7 +++++++
 1 file changed, 7 insertions(+)

(limited to 'tools')

diff --git a/tools/kolla-ansible b/tools/kolla-ansible
index 56e675e8a..bca3869f0 100755
--- a/tools/kolla-ansible
+++ b/tools/kolla-ansible
@@ -133,6 +133,7 @@ Commands:
     mariadb_backup       Take a backup of MariaDB databases
                              --full (default)
                              --incremental
+    monasca_cleanup      Remove unused containers for the Monasca service
     bootstrap-servers    Bootstrap servers with kolla deploy dependencies
     destroy              Destroy Kolla containers, volumes and host configuration
                              --include-images to also destroy Kolla images
@@ -175,6 +176,7 @@ prechecks
 check
 mariadb_recovery
 mariadb_backup
+monasca_cleanup
 bootstrap-servers
 destroy
 deploy
@@ -359,6 +361,11 @@ case "$1" in
         EXTRA_OPTS="$EXTRA_OPTS -e kolla_action=backup -e mariadb_backup_type=${BACKUP_TYPE}"
         PLAYBOOK="${BASEDIR}/ansible/mariadb_backup.yml"
         ;;
+(monasca_cleanup)
+        ACTION="Cleanup unused Monasca services"
+        EXTRA_OPTS="$EXTRA_OPTS -e kolla_action=cleanup"
+        PLAYBOOK="${BASEDIR}/ansible/monasca_cleanup.yml"
+        ;;
 (destroy)
         ACTION="Destroy Kolla containers, volumes and host configuration"
         PLAYBOOK="${BASEDIR}/ansible/destroy.yml"
-- 
cgit v1.2.3