Specification for additions for Prometheus monitoring

Describes the proposed approach to introduce Prometheus monitoring
for Kolla-Ansible deployment.

Partially-Implements: blueprint prometheus
Change-Id: I54616925abb76bda7efa2f2ae962d4e43f097551

1 files changed, 382 insertions, 0 deletions

diff --git a/specs/prometheus.rst b/specs/prometheus.rst
new file mode 100644
index 000000000..bb8719c0b
--- /dev/null
+++ b/specs/prometheus.rst
@@ -0,0 +1,382 @@
+=====================
+Prometheus Monitoring
+=====================
+
+https://blueprints.launchpad.net/kolla-ansible/+spec/prometheus
+
+One of the challenges faced by Kolla-Ansible operators, particularly of large
+deployments, is monitoring the behavior and performance of the nodes.  To help
+address this concern, it is proposed that Kolla-Ansible use Prometheus [1]_ as a
+framework for introducing monitoring capabilities.
+
+Prometheus is a widely adopted and supported tool for monitoring, capable of
+monitoring both system and service level characteristics.  Many projects have
+existing support for exporting data in Prometheus format either directly from
+the product itself or via a separate exporter [2]_.  This includes several tools
+used as part of the Kolla-Ansible software stack, meaning that with minimal work
+we could provide visibility into the performance characteristics of some of the
+underlying software frameworks.  There are also exporters to do system
+performance monitoring which could provide further visibility into the health of
+the cluster.  Prometheus can also use the OpenStack APIs to automatically
+discover running OpenStack servers such that servers can also easily expose data
+to Prometheus.
+
+Problem description
+===================
+There are three aspects to any useful system monitoring solution using
+Prometheus:
+
+* Exposing the data
+* Configuring and running Prometheus to collect and store data
+* Analyzing the data and reporting problems
+
+Prometheus and existing exporters can help address the first two items above.
+Analysis and reporting is the most complex part of the problem and will require
+additional tools on top of Prometheus.  For example, Grafana can be configured
+to use Prometheus as a data source and can help provide visibility into the data
+collected by Prometheus.
+
+Use Cases
+---------
+1. Query performance characteristics of nodes or other components in
+   Kolla-Ansible software stack (more details about what components can be
+   monitored is given below)
+2. Display dashboard illustrating overall system performance of Kolla-Ansible
+   nodes
+3. Determine high-level status of Kolla-Ansible containers and identify
+   potential issues encountered during deployment
+
+Proposed change
+===============
+
+Exposing the Data
+-----------------
+Prometheus generally works through a pull model where data is scraped at regular
+intervals from data providers.  Therefore, the first step in any Prometheus
+solution is to expose the data so that Prometheus can access it.
+
+Some tools natively expose data in a useful format.  In these cases all that is
+necessary is proper configuration of the tool to ensure the data is exported on
+a known port and configuration of the Kolla-Ansible container to expose the
+relevant port such that Prometheus can access the data.
+
+Most tools, however, use exporters or collectors that run as separate processes
+from the tool itself.  These collect data using exposed APIs and format the data
+in a manner that can be collected by Prometheus.  In these cases, each exporter
+would be run on a separate container from the main process.  This will require
+building of the requisite containers as well as modifications to the
+Kolla-Ansible deployment to run these containers during deployment. Furthermore,
+each exporter requires configuration of the Prometheus server to configure it to
+scrape the data.
+
+Listed below are some of the exporters that already exist for Prometheus that
+are related to components of a typical OpenStack Kolla-Ansible deployment.  This
+is based largely on the list of Prometheus Exporters and Integrations [2]_, and
+links to more information about each exporter can be found there.  Although we
+could choose to expose any of these exporters through Kolla-Ansible, it is not
+expected that we will implement all of these initially.  It is proposed that we
+start with the exporters for which Kolla containers already exist.  cAdvisor is
+also recommended for early implementation since it provides more detailed
+metrics for Docker container performance.  We can investigate and add exporters
+for additional services as time allows, but how far we proceed will depend
+largely on the level of interest amongst Kolla-Ansible developers who might help
+do the work.
+
+Existing Kolla Containers
+^^^^^^^^^^^^^^^^^^^^^^^^^
+The following exporters already have associated Kolla containers (used in
+Kolla-Kubernetes) and therefore should be minimal work to make available for a
+Kolla-Ansible deployment:
+
+* HAProxy
+* MySQL
+* Node Exporter - This exposes basic performance metrics (CPU, memory, IO, etc)
+  on the host itself
+* Prometheus - The Prometheus server contains support for monitoring its own
+  performance without any need for an additional exporter
+
+Other Possible Integrations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+(in no particular order)
+
+* cAdvisor
+* OpenStack discovery - Prometheus contains support for discovering exposed
+  services running on OpenStack instances.  It uses OpenStack client APIs to
+  locate the instances and then can contact these instances that are accessible
+  on the network and load Prometheus data from exporters running on those
+  instances. It is not clear exactly how we would make use of this since we
+  don't know what services might be running on the instances but it could be
+  useful to set up access to a node exporter if it is running.  This not only
+  provides insight into the instances' performance, but also would serve as a
+  template for operators wishing to expose their own exporters from OpenStack
+  instances.
+* Docker - The existing Docker metric support is considered "experimental" and
+  is subject to change so we may not want to use this until the API becomes more
+  stable.
+* Memcached
+* ElasticSearch
+* Fluentd
+* Grafana
+* Kafka
+* InfluxDB
+
+Kolla-Container Exporter
+------------------------
+One  piece of critical instrumentation is notably lacking from the existing
+providers, and that is the ability to determine which Docker containers are
+running on a node.  The existing Docker instrumentation can show how many
+containers are running, but provides no visibility into which containers they
+are.  The cAdvisor exporter also exposes information about containers (and
+provides more detailed view into specific containers than the built-in Docker
+metrics), but the high-level state of the container is still not available.
+Determining which containers may have failed to start or are in the 'restarting'
+state is one of the first troubleshooting steps of a broken Kolla-Ansible
+deployment, so this is a significant shortcoming.  Therefore, it is proposed
+that a simple Prometheus collector be implemented that exposes this data to
+Prometheus.
+
+Initially this collector will be quite simple, but more functionality can be
+added if and when we find more critical data missing from the existing set of
+exporters or when additional health checking becomes available for Kolla-Ansible
+containers [3]_.  The key metric exposed by this collector is a gauge called
+kolla_containers and has two labels, name and state which refer respectively to
+the name of the container (e.g. cinder_volume) and the container state (e.g.
+running).  Since the collector runs on each node, Prometheus will also
+automatically add an implied label, instance, that indicates which node the
+container is running on.  The value of the gauge is either 0 or 1 (1 indicating
+the container with that name is in the indicated state).
+
+A few examples of useful queries on this data include:
+
+* Total number of Kolla-Ansible containers across all nodes:
+  ``sum(kolla_containers)``
+* Number of containers in each state on each node: ``sum(kolla_containers) by
+  (instance)``
+* Number of containers in each state for a given service. For example, for
+  cinder: ``sum(kolla_containers{name=~'cinder_.*'}) by (state)``
+* A list of containers not in a normal (running) state:
+  ``kolla_containers{state!="running"}``
+
+This is just a sample list and other queries can be constructed to provide more
+specific data.
+
+The Kolla-Container collector uses the docker api to query this data and
+connects via the unix socket.  It will use Python docker module to connect to
+docker and the Prometheus_client module to expose this data in Prometheus
+format.  It will filter the docker containers based on container label to only
+expose statistics for Kolla-Ansible containers.  Additionally, the collector
+should expose certain standard metrics exposed by most collectors such as the
+scrape duration which represents the performance characteristics of the
+collector itself.
+
+As with other collectors, this will run in its own docker container deployed via
+the standard Ansible deployment.
+
+Running Prometheus
+------------------
+Prometheus itself will run inside a container on each node in the existing
+Kolla-Ansible monitoring inventory group.  A Prometheus container already exists
+in the Kolla repository (initially provided for Kolla-Kubernetes) and this
+container can be used in Kolla-Ansible deployment as well.
+
+Additions will be required to the Kolla-Ansible deployment process to run this
+container.  Since this monitoring tool is useful in determining the state of
+deployment and analyzing problems that may occur during deployment, the
+container should be started as early as possible during deployment.  Although
+Prometheus could be started even earlier, it is proposed that the Prometheus
+deployment role be applied just after the MariaDB role since the Prometheus
+MySQL exporter requires database user creation to function.
+
+We should also expose Prometheus via HAProxy so that Prometheus data can be
+queried using the virtual IP that is used to access other OpenStack APIs and
+browser UIs.  This also will require modifications to the existing HAProxy
+configuration template in the Kolla-Ansible repository.
+
+In the initial implementation, Prometheus will use local data storage for its
+metrics.  This means that Prometheus data is not HA and there will be data
+retention limits.  Each Prometheus server container will pull metrics
+independently from the exporters and therefore the data may be different between
+Prometheus servers.  In a future version (or if developer involvement and time
+allow), it may be worth considering using external storage solutions to increase
+capacity and allow for HA storage, such as can be provided using InfluxDB and
+Influx-Relay as described at [4]_.
+
+Data Analysis and Reporting
+---------------------------
+The Prometheus server can be directly queried to display and graph any of the
+metrics collected by the server.  However, with the addition of Grafana, the
+information may be organized into dashboards that collect multiple datapoints
+into a single page and present them in a manner that is more useful to the
+operator consuming this data.  In order to integrate with Grafana, Prometheus
+would need to be defined as a datasource using the Grafana provisioning
+framework.  Once that is done an operator can create or import dashboards that
+make use of this data.
+
+It would also be possible to define one or more default, preloaded dashboards
+for Grafana to display the information deemed most useful for Kolla-Ansible
+deployment monitoring.  Grafana also has plugins that provide diagrams [5]_ that
+could help visualize the state of the Kolla-Ansible deployment.  The amount of
+work that can be done in this area will depend upon the level of developer
+interest and involvement in the project.
+
+The addition of the data exported by the proposed Kolla-Container Exporter
+provides a useful tool for checking the state of a Kolla-Ansible deployment.  By
+analyzing the data from this exporter, a tool can provide high-level deployment
+status.  This functionality should be provided via a new status command within
+the kolla-ansible command (or via a CLI if one is introduced [6]_).  Information
+to be displayed will include:
+
+* If Prometheus is not running or cannot be contacted, the status will indicate
+  as such.  This could indicate that Prometheus is disabled, that deployment has
+  not yet been initiated, or that deployment failed before the Prometheus
+  container was started.  In this case, no further information can be provided.
+* Nodes on which the Kolla-Container collector are not running should be
+  highlighted since other information cannot be obtained on those nodes.  This
+  will require correlating the instances on which the kolla_containers metric is
+  exposed against the list of inventory hosts.  This could indicate a problem
+  with the collector or with deployment of the collector, or it might just
+  indicate that deployment has not yet proceeded to the point where the
+  collector has been started.
+* Kolla-Ansible containers that are not in the running state should be listed.
+  For example, containers in a restarting state may represent a misconfiguration
+  of the cluster and should be identified.
+* Other health statistics: on a normally running cluster, some basic statistics
+  can be provided to help identify potential problems.  The set of statistics
+  should include such details as the total number of running Kolla-Ansible
+  containers on each system (an unexpectedly low number on one or more systems
+  might indicate a problem).  Other details can be added in the future as deemed
+  necessary.
+* Optional arguments could limit the output to a specific host, inventory group,
+  or service.
+
+Another common use of Prometheus is the use of a Prometheus Alertmanager which
+is capable of sending alerts in cases where problems occur or predefined
+thresholds are exceeded.  However, there are a number of complications regarding
+the configuration and running of the Alertmanager, and the details are therefore
+left for a future blueprint.
+
+Configuration
+-------------
+As with all optional services in Kolla-Ansible, Prometheus deployment should be
+controlled by Kolla-Ansible variables.  A high level enable_prometheus variable
+should control whether Prometheus is used at all.  Additionally, additional
+variables can be used to control individual exporters.  For example,
+enable_prometheus_haproxy could be used to enable/disable the HAProxy exporter
+to Prometheus.  By default Prometheus should be enabled and exporters should be
+enabled if both Prometheus and the associated service are enabled.
+
+Limitations
+-----------
+At it's core, Prometheus gathers numerical statistics about exposed services,
+and provides a robust query language that allow an operator to query,
+manipulate, and graph this data.  However, collecting and exposing this data is
+really only half of any system monitoring solution.  Operators may not
+understand the inner workings of the system enough for this data to be useful
+without interpretation.  Prometheus can provide a lot of detailed data, but it
+is not ideal for looking at a complex system and determining at a glance whether
+it is running normally.  Initial integrations with Grafana and with a
+kolla-ansible (or CLI) status command will provide useful data, but may prove
+insufficient for many situations. However, even without more detailed analysis
+tools, some benefit can still be drawn from merely collecting and storing the
+data in Prometheus.  Knowledgeable operators can perform their own analysis as
+long as the raw data is available.  Also, having the raw data available allows
+us to incrementally improve on the complex problem of analysis and reporting
+over subsequent releases.
+
+Security Impact
+---------------
+A detailed analysis of the security model of Prometheus and its impact can be
+found at [7]_.  In general, Prometheus considers collected metrics to be
+insecure data accessible to anybody with access to the HTTP API.  For this
+reason, Prometheus should only be exposed on the internal network interface and
+VIP address and not exposed externally.  Operators who want to access Prometheus
+data via the external network can access the data via the Grafana integration
+which adds an additional security layer and requires a password to access any
+data.
+
+Performance Impact
+------------------
+Enabling Prometheus monitoring will have some impact on system performance.  It
+adds a number of additional containers including one for each exporter and for
+Prometheus itself.  Furthermore, the Prometheus server performs periodic
+endpoint scraping where it queries each provider for the latest metrics.  The
+impact of this data gathering will vary by exporter.  Although the impact of any
+one exporter should be negligible, it's possible that in combination they might
+have a measurable impact on the system.
+
+Any potential risk to performance may be mitigated in several ways.  Each
+exporter should be able to be enabled or disabled independently through
+Kolla-Ansible properties so if an exporter is found to have a significant
+detrimental impact it may be disabled. In order to help determine any potential
+impact, Prometheus provides metrics for monitoring its own performance, and most
+exporters also include performance metrics for the exporters themselves.
+
+Alternatives
+------------
+There are a number of possible alternatives to Prometheus for collecting,
+maintaining, and exposing performance metrics.  Some of the primary options are
+discussed at [8]_.  Another potential monitoring solution is Monasca which
+provides a centralized service for both tenant and control plane monitoring.
+Prometheus is more widely adopted and supported than many of the alternatives
+and has rich support for many of the tools already used in the Kolla-Ansible
+software stack.  It's integration with Grafana provides an additional advantage
+over some of the alternate solutions.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+  Mark Giles (mark-giles)
+
+Milestones
+----------
+Target Milestone for completion: Rocky 1
+
+Work Items
+----------
+1. Prometheus server configuration for Kolla-Ansible
+2. Ansible deployment of existing Prometheus server container
+3. Configuration of HAProxy to handle Prometheus server
+4. Implement Kolla-Container Exporter
+5. kolla-ansible (or CLI) status command to display Kolla-Container Exporter
+   results
+6. Integration with Grafana
+7. Implement Grafana dashboard(s) to provide visualization of Kolla-Ansible
+   cluster behavior
+8. Exporters (see below)
+
+For each exporter, the following work items exist:
+
+1. Create a Docker image for the exporter
+2. Depending on the exporter, it may be necessary to modify settings for the
+   monitored service's container to properly expose any necessary APIs
+3. Implement Ansible deployment of the container
+4. Modify Prometheus server configuration to scrape data from the exporter
+5. (Optional) Implement or enhance Grafana dashboard(s) as appropriate.
+
+The MySQL exporter in particular will require additional work:
+
+6. Ansible definition to create Prometheus database user
+
+Testing
+=======
+The existing gate checks will be used to ensure successful deployment.  Behavior
+of the newly exposed functionality will require manual testing.
+
+Documentation Impact
+====================
+A new documentation reference page should be created for "Prometheus in Kolla".
+This page will document how to enable or disable Prometheus and/or individual
+exporters as well as how to access the exposed data.
+
+References
+==========
+.. [1] https://prometheus.io
+.. [2] https://prometheus.io/docs/instrumenting/exporters/
+.. [3] https://blueprints.launchpad.net/kolla/+spec/container-health-check
+.. [4] https://docs.openstack.org/developer/performance-docs/methodologies/monitoring/influxha.html
+.. [5] https://grafana.com/plugins/jdbranham-diagram-panel
+.. [6] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128561.html
+.. [7] https://prometheus.io/docs/operating/security/
+.. [8] https://prometheus.io/docs/introduction/comparison/