prometheus: Support overriding address of scrape targets

In many cases we use the kolla_address filter to look up the IP address
of the current host or another host on a particular network interface.
This filter uses the host's facts to determine the IP, meaning that we
must have gathered facts for the host, even if it is outside of a
requested --limit. This is a limitation, since it requires that all
hosts must be reachable, even if we are not directly configuring them.

Most instances of this cross-host fact referencing involve a controller,
since they host clustered services. The only instance found to affect
compute nodes is in the prometheus role, where Prometheus server needs
to know the IP address of all targets in its scrape configs.

If we are able to specify the address of the scrape targets as a static
variable such as a host variable, then facts would not be required for
compute nodes outside of the --limit.

Removing the requirement to have facts for all compute nodes has
benefits for performance (gathering facts for all hosts can take a long
time) and fault tolerance (we can operate when some compute hosts are
unreachable).

This change modifies the kolla_address filter to accept an optional
override_var argument which can be used to specify the name of a host
variable that may override the returned IP address. This is used in the
Prometheus server configuration to allow specifying the IP address used
by Prometheus server when collecting metrics from exporter using
a 'prometheus_target_address' host variable. If specified, this takes
precedence over the API interface address currently used. This makes it
possible to statically override prometheus_target_address and avoid the
cross-host fact reference.

This is not a complete solution because it is not yet possible to skip
the cross-host fact gathering step.

Partial-Bug: #2041860
Change-Id: I207ca56362de00d8ec578333eab9e1a72e7bcd19

1 files changed, 13 insertions, 0 deletions

diff --git a/doc/source/reference/logging-and-monitoring/prometheus-guide.rst b/doc/source/reference/logging-and-monitoring/prometheus-guide.rst
index 30966c1cd..c8fc066ec 100644
--- a/doc/source/reference/logging-and-monitoring/prometheus-guide.rst
+++ b/doc/source/reference/logging-and-monitoring/prometheus-guide.rst
@@ -243,3 +243,16 @@ For example:
       labels:
         instance: host1
 
+Target address
+~~~~~~~~~~~~~~
+
+By default, Prometheus server uses the IP of the API interface of scrape
+targets when collecting metrics. This may be overridden by setting
+``prometheus_target_address`` as a host variable. The value of this host
+variable must be a valid IPv4 or IPv6 address.
+
+Prometheus server is one of the few instances where we need to know IP
+addresses of all other hosts in the cloud. Being able to specify these via
+``prometheus_target_address`` allows us to operate when facts are not available
+for all hosts. This could be due to some hosts being unreachable or having
+previously failed.