Add support for multiple ceph files

This patch is adding a feature for an option to copy different
ceph configuration files and corresponding keyrings for cinder,
glance, manila, gnocchi and nova services.

This is especially useful when the deployment uses availability
zones as below example.

  - Individual compute can read/write to individual ceph
    cluster in same AZ.
  - Cinder can write to several ceph clusters in several AZs.
  - Glance can use multistore and upload images to
    several ceph clusters in several AZs at once.

Change-Id: Ie4d8ab5a3df748137835cae1c943b9180cd10eb1

1 files changed, 328 insertions, 120 deletions

diff --git a/doc/source/reference/storage/external-ceph-guide.rst b/doc/source/reference/storage/external-ceph-guide.rst
index e88db8136..71d0fb5a0 100644
--- a/doc/source/reference/storage/external-ceph-guide.rst
+++ b/doc/source/reference/storage/external-ceph-guide.rst
@@ -44,46 +44,98 @@ Glance
 Ceph RBD can be used as a storage backend for Glance images. Configuring Glance
 for Ceph includes the following steps:
 
-#. Enable Glance Ceph backend in ``globals.yml``:
+* Enable Glance Ceph backend in ``globals.yml``:
 
-   .. code-block:: yaml
+  .. code-block:: yaml
 
-      glance_backend_ceph: "yes"
+     glance_backend_ceph: "yes"
 
-#. Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
+* Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
 
-   * ``ceph_glance_keyring`` (default: ``ceph.client.glance.keyring``)
-   * ``ceph_glance_user`` (default: ``glance``)
-   * ``ceph_glance_pool_name`` (default: ``images``)
+  * ``ceph_glance_keyring`` (default: ``client.glance.keyring``)
+  * ``ceph_glance_user`` (default: ``glance``)
+  * ``ceph_glance_pool_name`` (default: ``images``)
 
-#. Copy Ceph configuration file to ``/etc/kolla/config/glance/ceph.conf``
+* Copy Ceph configuration file to ``/etc/kolla/config/glance/ceph.conf``
 
-   .. path /etc/kolla/config/glance/ceph.conf
-   .. code-block:: ini
+  .. path /etc/kolla/config/glance/ceph.conf
+  .. code-block:: ini
 
-      [global]
-      fsid = 1d89fec3-325a-4963-a950-c4afedd37fe3
-      mon_initial_members = ceph-0
-      mon_host = 192.168.0.56
-      auth_cluster_required = cephx
-      auth_service_required = cephx
-      auth_client_required = cephx
+     [global]
+     fsid = 1d89fec3-325a-4963-a950-c4afedd37fe3
+     keyring = /etc/ceph/ceph.client.glance.keyring
+     mon_initial_members = ceph-0
+     mon_host = 192.168.0.56
+     auth_cluster_required = cephx
+     auth_service_required = cephx
+     auth_client_required = cephx
 
-#. Copy Ceph keyring to ``/etc/kolla/config/glance/<ceph_glance_keyring>``
+* Copy Ceph keyring to ``/etc/kolla/config/glance/ceph.<ceph_glance_keyring>``
 
-#. For copy-on-write set following in ``/etc/kolla/config/glance.conf``:
+To configure multiple Ceph backends with Glance, which is useful
+for multistore:
 
-   .. path /etc/kolla/config/glance.conf
-   .. code-block:: ini
+* Copy the Ceph configuration files into ``/etc/kolla/config/glance/`` using
+  different names for each
 
-      [GLOBAL]
-      show_image_direct_url = True
+  ``/etc/kolla/config/glance/ceph.conf``
+
+  .. path /etc/kolla/config/glance/ceph.conf
+  .. code-block:: ini
+
+     [global]
+     fsid = 1d89fec3-325a-4963-a950-c4afedd37fe3
+     keyring = /etc/ceph/ceph.client.glance.keyring
+     mon_initial_members = ceph-0
+     mon_host = 192.168.0.56
+     auth_cluster_required = cephx
+     auth_service_required = cephx
+     auth_client_required = cephx
+
+  ``/etc/kolla/config/glance/rbd1.conf``
+
+  .. path /etc/kolla/config/glance/rbd1.conf
+  .. code-block:: ini
+
+     [global]
+     fsid = dbfea068-89ca-4d04-bba0-1b8a56c3abc8
+     keyring = /etc/ceph/rbd1.client.glance.keyring
+     mon_initial_members = ceph-0
+     mon_host = 192.10.0.100
+     auth_cluster_required = cephx
+     auth_service_required = cephx
+     auth_client_required = cephx
+
+* Declare Ceph backends in ``globals.yml``
+
+  .. code-block:: yaml
+
+     glance_ceph_backends:
+       - name: "rbd"
+         type: "rbd"
+         cluster: "ceph"
+         enabled: "{{ glance_backend_ceph | bool }}"
+       - name: "another-rbd"
+         type: "rbd"
+         cluster: "rbd1"
+         enabled: "{{ glance_backend_ceph | bool }}"
+
+* Copy Ceph keyring to ``/etc/kolla/config/glance/ceph.<ceph_glance_keyring>``
+  and analogously to ``/etc/kolla/config/glance/rbd1.<ceph_glance_keyring>``
+
+* For copy-on-write set following in ``/etc/kolla/config/glance.conf``:
+
+  .. path /etc/kolla/config/glance.conf
+  .. code-block:: ini
+
+     [GLOBAL]
+     show_image_direct_url = True
 
 .. warning::
 
-    ``show_image_direct_url`` can present a security risk if using more
-    than just Ceph as Glance backend(s). Please see
-    :glance-doc:`Glance show_image_direct_url <configuration/glance_api.html#DEFAULT.show_image_direct_url>`
+   ``show_image_direct_url`` can present a security risk if using more
+   than just Ceph as Glance backend(s). Please see
+   :glance-doc:`Glance show_image_direct_url <configuration/glance_api.html#DEFAULT.show_image_direct_url>`
 
 Cinder
 ------
@@ -91,76 +143,146 @@ Cinder
 Ceph RBD can be used as a storage backend for Cinder volumes. Configuring
 Cinder for Ceph includes following steps:
 
-#. When using external Ceph, there may be no nodes defined in the storage
-   group.  This will cause Cinder and related services relying on this group to
-   fail.  In this case, operator should add some nodes to the storage group,
-   all the nodes where ``cinder-volume`` and ``cinder-backup`` will run:
+* When using external Ceph, there may be no nodes defined in the storage
+  group.  This will cause Cinder and related services relying on this group to
+  fail.  In this case, operator should add some nodes to the storage group,
+  all the nodes where ``cinder-volume`` and ``cinder-backup`` will run:
+
+  .. code-block:: ini
+
+     [storage]
+     control01
+
+* Enable Cinder Ceph backend in ``globals.yml``:
+
+  .. code-block:: yaml
+
+     cinder_backend_ceph: "yes"
+
+* Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
+
+  * ``ceph_cinder_keyring`` (default: ``client.cinder.keyring``)
+  * ``ceph_cinder_user`` (default: ``cinder``)
+  * ``ceph_cinder_pool_name`` (default: ``volumes``)
+  * ``ceph_cinder_backup_keyring``
+    (default: ``client.cinder-backup.keyring``)
+  * ``ceph_cinder_backup_user`` (default: ``cinder-backup``)
+  * ``ceph_cinder_backup_pool_name`` (default: ``backups``)
+
+* Copy Ceph configuration file to ``/etc/kolla/config/cinder/ceph.conf``
+
+  Separate configuration options can be configured for
+  cinder-volume and cinder-backup by adding ceph.conf files to
+  ``/etc/kolla/config/cinder/cinder-volume`` and
+  ``/etc/kolla/config/cinder/cinder-backup`` respectively. They
+  will be merged with ``/etc/kolla/config/cinder/ceph.conf``.
+
+* Copy Ceph keyring files to:
 
-   .. code-block:: ini
+  * ``/etc/kolla/config/cinder/cinder-volume/ceph.<ceph_cinder_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-backup/ceph.<ceph_cinder_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-backup/ceph.
+    <ceph_cinder_backup_keyring>``
 
-      [storage]
-      control01
+.. note::
+
+   ``cinder-backup`` requires two keyrings for accessing volumes
+   and backup pool.
+
+To configure ``multiple Ceph backends`` with Cinder, which is useful for
+the use with availability zones:
+
+* Copy their Ceph configuration files into ``/etc/kolla/config/cinder/`` using
+  different names for each
+
+  ``/etc/kolla/config/cinder/ceph.conf``
+
+  .. path /etc/kolla/config/cinder/ceph.conf
+  .. code-block:: ini
 
-#. Enable Cinder Ceph backend in ``globals.yml``:
+     [global]
+     fsid = 1d89fec3-325a-4963-a950-c4afedd37fe3
+     mon_initial_members = ceph-0
+     mon_host = 192.168.0.56
+     auth_cluster_required = cephx
+     auth_service_required = cephx
+     auth_client_required = cephx
 
-   .. code-block:: yaml
+  ``/etc/kolla/config/cinder/rbd2.conf``
 
-      cinder_backend_ceph: "yes"
+  .. path /etc/kolla/config/cinder/rbd2.conf
+  .. code-block:: ini
 
-#. Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
+     [global]
+     fsid = dbfea068-89ca-4d04-bba0-1b8a56c3abc8
+     mon_initial_members = ceph-0
+     mon_host = 192.10.0.100
+     auth_cluster_required = cephx
+     auth_service_required = cephx
+     auth_client_required = cephx
 
-   * ``ceph_cinder_keyring`` (default: ``ceph.client.cinder.keyring``)
-   * ``ceph_cinder_user`` (default: ``cinder``)
-   * ``ceph_cinder_pool_name`` (default: ``volumes``)
-   * ``ceph_cinder_backup_keyring``
-     (default: ``ceph.client.cinder-backup.keyring``)
-   * ``ceph_cinder_backup_user`` (default: ``cinder-backup``)
-   * ``ceph_cinder_backup_pool_name`` (default: ``backups``)
+* Declare Ceph backends in ``globals.yml``
 
-#. Copy Ceph configuration file to ``/etc/kolla/config/cinder/ceph.conf``
+  .. code-block:: yaml
 
-   Separate configuration options can be configured for
-   cinder-volume and cinder-backup by adding ceph.conf files to
-   ``/etc/kolla/config/cinder/cinder-volume`` and
-   ``/etc/kolla/config/cinder/cinder-backup`` respectively. They
-   will be merged with ``/etc/kolla/config/cinder/ceph.conf``.
+     cinder_ceph_backends:
+       - name: "rbd-1"
+         cluster: "ceph"
+         enabled: "{{ cinder_backend_ceph | bool }}"
+       - name: "rbd-2"
+         cluster: "rbd2"
+         availability_zone: "az2"
+         enabled: "{{ cinder_backend_ceph | bool }}"
 
-#. Copy Ceph keyring files to:
+* Copy Ceph keyring files for all Ceph backends:
 
-   * ``/etc/kolla/config/cinder/cinder-volume/<ceph_cinder_keyring>``
-   * ``/etc/kolla/config/cinder/cinder-backup/<ceph_cinder_keyring>``
-   * ``/etc/kolla/config/cinder/cinder-backup/<ceph_cinder_backup_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-volume/ceph.<ceph_cinder_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-backup/ceph.<ceph_cinder_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-backup/ceph.
+    <ceph_cinder_backup_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-volume/rbd2.<ceph_cinder_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-backup/rbd2.<ceph_cinder_keyring>``
+  * ``/etc/kolla/config/cinder/cinder-backup/rbd2.
+    <ceph_cinder_backup_keyring>``
 
 .. note::
 
-    ``cinder-backup`` requires two keyrings for accessing volumes
-    and backup pool.
+   ``cinder-backup`` requires two keyrings for accessing volumes
+   and backup pool.
 
 Nova must also be configured to allow access to Cinder volumes:
 
-#. Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
+* Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
 
-   * ``ceph_cinder_keyring`` (default: ``ceph.client.cinder.keyring``)
+  * ``ceph_cinder_keyring`` (default: ``client.cinder.keyring``)
 
-#. Copy Ceph keyring file(s) to:
+* Copy Ceph keyring file(s) to:
 
-   * ``/etc/kolla/config/nova/<ceph_cinder_keyring>``
+  * ``/etc/kolla/config/nova/ceph.<ceph_cinder_keyring>``
+
+To configure ``different Ceph backend`` for nova-compute host, which
+is useful for the use with availability zones:
+
+* Copy Ceph keyring file to:
+
+  * ``/etc/kolla/config/nova/<hostname>/ceph.<ceph_cinder_keyring>``
 
 If ``zun`` is enabled, and you wish to use cinder volumes with zun,
 it must also be configured to allow access to Cinder volumes:
 
-#. Enable Cinder Ceph backend for Zun in ``globals.yml``:
+* Enable Cinder Ceph backend for Zun in ``globals.yml``:
 
-   .. code-block:: yaml
+  .. code-block:: yaml
 
-      zun_configure_for_cinder_ceph: "yes"
+     zun_configure_for_cinder_ceph: "yes"
 
-#. Copy Ceph configuration file to:
-   * ``/etc/kolla/config/zun/zun-compute/ceph.conf``
+* Copy Ceph configuration file to:
 
-#. Copy Ceph keyring file(s) to:
+  * ``/etc/kolla/config/zun/zun-compute/ceph.conf``
 
-   * ``/etc/kolla/config/zun/zun-compute/<ceph_cinder_keyring>``
+* Copy Ceph keyring file(s) to:
+
+  * ``/etc/kolla/config/zun/zun-compute/ceph.<ceph_cinder_keyring>``
 
 
 Nova
@@ -173,29 +295,38 @@ not need to be copied between hypervisors.
 
 Configuring Nova for Ceph includes following steps:
 
-#. Enable Nova Ceph backend in ``globals.yml``:
+* Enable Nova Ceph backend in ``globals.yml``:
+
+  .. code-block:: yaml
 
-   .. code-block:: yaml
+     nova_backend_ceph: "yes"
 
-      nova_backend_ceph: "yes"
+* Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
 
-#. Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
+  * ``ceph_nova_keyring`` (by default it's the same as
+    ``ceph_cinder_keyring``)
+  * ``ceph_nova_user`` (by default it's the same as ``ceph_cinder_user``)
+  * ``ceph_nova_pool_name`` (default: ``vms``)
 
-   * ``ceph_nova_keyring`` (by default it's the same as
-     ``ceph_cinder_keyring``)
-   * ``ceph_nova_user`` (by default it's the same as ``ceph_cinder_user``)
-   * ``ceph_nova_pool_name`` (default: ``vms``)
+* Copy Ceph configuration file to ``/etc/kolla/config/nova/ceph.conf``
+* Copy Ceph keyring file(s) to:
 
-#. Copy Ceph configuration file to ``/etc/kolla/config/nova/ceph.conf``
-#. Copy Ceph keyring file(s) to:
+  * ``/etc/kolla/config/nova/ceph.<ceph_nova_keyring>``
 
-   * ``/etc/kolla/config/nova/<ceph_nova_keyring>``
+  .. note::
 
-   .. note::
+     If you are using a Ceph deployment tool that generates separate Ceph
+     keys for Cinder and Nova, you will need to override
+     ``ceph_nova_keyring`` and ``ceph_nova_user`` to match.
 
-      If you are using a Ceph deployment tool that generates separate Ceph
-      keys for Cinder and Nova, you will need to override
-      ``ceph_nova_keyring`` and ``ceph_nova_user`` to match.
+To configure ``different Ceph backend`` for nova-compute host, which
+is useful for the use with availability zones:
+
+* Copy Ceph configuration file to ``/etc/kolla/config/nova/
+  <hostname>/ceph.conf``
+* Copy Ceph keyring file(s) to:
+
+  * ``/etc/kolla/config/nova/<hostname>/ceph.<ceph_nova_keyring>``
 
 Gnocchi
 -------
@@ -203,21 +334,25 @@ Gnocchi
 Ceph object storage can be used as a storage backend for Gnocchi metrics.
 Configuring Gnocchi for Ceph includes following steps:
 
-#. Enable Gnocchi Ceph backend in ``globals.yml``:
+* Enable Gnocchi Ceph backend in ``globals.yml``:
 
-   .. code-block:: yaml
+  .. code-block:: yaml
 
-      gnocchi_backend_storage: "ceph"
+     gnocchi_backend_storage: "ceph"
 
-#. Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
+* Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
 
-   * ``ceph_gnocchi_keyring``
-     (default: ``ceph.client.gnocchi.keyring``)
-   * ``ceph_gnocchi_user`` (default: ``gnocchi``)
-   * ``ceph_gnocchi_pool_name`` (default: ``gnocchi``)
+  * ``ceph_gnocchi_keyring``
+    (default: ``client.gnocchi.keyring``)
+  * ``ceph_gnocchi_user`` (default: ``gnocchi``)
+  * ``ceph_gnocchi_pool_name`` (default: ``gnocchi``)
+  * ``ceph_gnocchi_conf``
+    (default: ``ceph.conf``)
 
-#. Copy Ceph configuration file to ``/etc/kolla/config/gnocchi/ceph.conf``
-#. Copy Ceph keyring to ``/etc/kolla/config/gnocchi/<ceph_gnocchi_keyring>``
+* Copy Ceph configuration file to
+  ``/etc/kolla/config/gnocchi/<ceph_gnocchi_conf>``
+* Copy Ceph keyring to
+  ``/etc/kolla/config/gnocchi/ceph.<ceph_gnocchi_keyring>``
 
 Manila
 ------
@@ -225,32 +360,105 @@ Manila
 CephFS can be used as a storage backend for Manila shares. Configuring Manila
 for Ceph includes following steps:
 
-#. Enable Manila Ceph backend in ``globals.yml``:
-
-   .. code-block:: yaml
-
-      enable_manila_backend_cephfs_native: "yes"
-
-#. Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
-
-   * ``ceph_manila_keyring`` (default: ``ceph.client.manila.keyring``)
-   * ``ceph_manila_user`` (default: ``manila``)
-
-   .. note::
-
-      Required Ceph identity caps for manila user are documented in
-      :manila-doc:`CephFS Native driver <admin/cephfs_driver.html#authorizing-the-driver-to-communicate-with-ceph>`.
-
-#. Copy Ceph configuration file to ``/etc/kolla/config/manila/ceph.conf``
-#. Copy Ceph keyring to ``/etc/kolla/config/manila/<ceph_manila_keyring>``
-
-#. If using multiple filesystems (Ceph Pacific+), set
-   ``manila_cephfs_filesystem_name`` in ``/etc/kolla/globals.yml`` to the
-   name of the Ceph filesystem Manila should use.
-   By default, Manila will use the first filesystem returned by
-   the ``ceph fs volume ls`` command.
-
-#. Setup Manila in the usual way
+* Enable Manila Ceph backend in ``globals.yml``:
+
+  .. code-block:: yaml
+
+     enable_manila_backend_cephfs_native: "yes"
+
+* Configure Ceph authentication details in ``/etc/kolla/globals.yml``:
+
+  * ``ceph_manila_keyring`` (default: ``client.manila.keyring``)
+  * ``ceph_manila_user`` (default: ``manila``)
+
+  .. note::
+
+     Required Ceph identity caps for manila user are documented in
+     :manila-doc:`CephFS Native driver <admin/cephfs_driver.html#authorizing-the-driver-to-communicate-with-ceph>`.
+
+* Copy Ceph configuration file to ``/etc/kolla/config/manila/ceph.conf``
+* Copy Ceph keyring to ``/etc/kolla/config/manila/ceph.<ceph_manila_keyring>``
+
+To configure ``multiple Ceph backends`` with Manila, which is useful for
+the use with availability zones:
+
+* Copy their Ceph configuration files into ``/etc/kolla/config/manila/`` using
+  different names for each
+
+  ``/etc/kolla/config/manila/ceph.conf``
+
+  .. path /etc/kolla/config/manila/ceph.conf
+  .. code-block:: ini
+
+     [global]
+     fsid = 1d89fec3-325a-4963-a950-c4afedd37fe3
+     mon_initial_members = ceph-0
+     mon_host = 192.168.0.56
+     auth_cluster_required = cephx
+     auth_service_required = cephx
+     auth_client_required = cephx
+
+  ``/etc/kolla/config/manila/rbd2.conf``
+
+  .. path /etc/kolla/config/manila/rbd2.conf
+  .. code-block:: ini
+
+     [global]
+     fsid = dbfea068-89ca-4d04-bba0-1b8a56c3abc8
+     mon_initial_members = ceph-0
+     mon_host = 192.10.0.100
+     auth_cluster_required = cephx
+     auth_service_required = cephx
+     auth_client_required = cephx
+
+* Declare Ceph backends in ``globals.yml``
+
+  .. code-block:: yaml
+
+     manila_ceph_backends:
+       - name: "cephfsnative1"
+         share_name: "CEPHFS1"
+         driver: "cephfsnative"
+         cluster: "ceph"
+         enabled: "{{ enable_manila_backend_cephfs_native | bool }}"
+         protocols:
+           - "CEPHFS"
+       - name: "cephfsnative2"
+         share_name: "CEPHFS2"
+         driver: "cephfsnative"
+         cluster: "rbd2"
+         enabled: "{{ enable_manila_backend_cephfs_native | bool }}"
+         protocols:
+           - "CEPHFS"
+       - name: "cephfsnfs1"
+         share_name: "CEPHFSNFS1"
+         driver: "cephfsnfs"
+         cluster: "ceph1"
+         enabled: "{{ enable_manila_backend_cephfs_nfs | bool }}"
+         protocols:
+           - "NFS"
+           - "CIFS"
+       - name: "cephfsnfs2"
+         share_name: "CEPHFSNFS2"
+         driver: "cephfsnfs"
+         cluster: "rbd2"
+         enabled: "{{ enable_manila_backend_cephfs_nfs | bool }}"
+         protocols:
+           - "NFS"
+           - "CIFS"
+
+* Copy Ceph keyring files for all Ceph backends:
+
+  * ``/etc/kolla/config/manila/manila-share/ceph.<ceph_manila_keyring>``
+  * ``/etc/kolla/config/manila/manila-share/rbd2.<ceph_manila_keyring>``
+
+* If using multiple filesystems (Ceph Pacific+), set
+  ``manila_cephfs_filesystem_name`` in ``/etc/kolla/globals.yml`` to the
+  name of the Ceph filesystem Manila should use.
+  By default, Manila will use the first filesystem returned by
+  the ``ceph fs volume ls`` command.
+
+* Setup Manila in the usual way
 
 For more details on the rest of the Manila setup, such as creating the share
 type ``default_share_type``, please see :doc:`Manila in Kolla <manila-guide>`.
@@ -335,6 +543,6 @@ When configuring Zun with Cinder volumes, kolla-ansible installs some
 Ceph client packages on zun-compute hosts. You can set the version
 of the Ceph packages installed by,
 
-#. Configuring Ceph version details in ``/etc/kolla/globals.yml``:
+* Configuring Ceph version details in ``/etc/kolla/globals.yml``:
 
-   * ``ceph_version`` (default: ``pacific``)
+  * ``ceph_version`` (default: ``pacific``)