diff options
| -rw-r--r-- | README.md | 216 | ||||
| -rw-r--r-- | devenv/README.md | 143 | ||||
| -rw-r--r-- | devenv/kollacluster.yaml | 109 | ||||
| -rw-r--r-- | devenv/kollanet.png | bin | 0 -> 26014 bytes | |||
| -rw-r--r-- | devenv/kollanode.yaml | 199 | ||||
| -rw-r--r-- | devenv/local.yaml.example | 3 |
6 files changed, 502 insertions, 168 deletions
@@ -1,30 +1,36 @@ -Kolla -===== +Kolla Overview +============== -The Kolla project is part of the OpenStack TripleO effort, focused on -deploying OpenStack environments using [Kubernetes][] and [Docker][] -containers. +The Kolla project is part of the OpenStack [TripleO][] effort, focused +on deploying OpenStack services using [Docker][] containers. The initial +system [spec][] provides additional details of Kolla and the use cases +it addresses. -[kubernetes]: https://github.com/GoogleCloudPlatform/kubernetes -[docker]: http://docker.com/ +[TripleO]: https://wiki.openstack.org/wiki/TripleO +[Docker]: http://docker.com/ +[spec]: https://github.com/stackforge/kolla/blob/master/specs/containerize-openstack.rst Getting Started =============== -Kubernetes deployment on bare metal is a complex topic which is beyond the -scope of this project at this time. The developers require a development test -environment. As a result, one of the developers has created a Heat based -deployment tool that can be -found [here](https://github.com/larsks/heat-kubernetes). +Deployment on bare metal is a complex topic which is beyond the scope of +the project at this time. An environment to simplify the deployment of a +single or multi-node Kolla cluster is required for development purposes. +As a result, a [Heat template][] has been created for deploying a Kolla +cluster to an existing OpenStack cloud. +[Heat template]: (https://github.com/stackforge/kolla/blob/master/devenv/README.md) -Build Docker Images -------------------- +Docker Images +------------- -Images are built by the Kolla project maintainers. It is possible to build -unique images with specific changes, but these would end up in a personal -namespace. Read the docs directory image workflow documentation for more -details. +The [Docker images][] are built by the Kolla project maintainers. A detailed +process for contributing to the images can be found [here][]. Images reside +in the Docker Hub [Kollaglue repo][]. + +[here]: https://github.com/stackforge/kolla/blob/master/docs/image-building.md +[Docker images]: https://docs.docker.com/userguide/dockerimages/ +[Kollaglue repo]: https://registry.hub.docker.com/repos/kollaglue/ The Kolla developers build images in the kollaglue namespace for the following services: @@ -45,161 +51,35 @@ $ sudo docker search kollaglue ``` A list of the upstream built docker images will be shown. -Use Kubernetes to Deploy OpenStack ----------------------------------- - -At this point, we believe the key features for a minimum viable feature set -are implemented. This includes the capability to launch virtual machines in -Nova. One key fact is that networking may not entirely work properly yet -until Neutron is finished, so the virtual machines may not actually behave -as expected for an end user deployment. - -Two options exist for those without an existing Kubernetes environment: - -The upstream Kubernetes community provides instructions for running Kubernetes -using Vagrant, available from: -https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/vagrant.md - -The Kolla developers develop Kolla in OpenStack, using Heat to provision the -necessary servers and other resources. If you are familiar with Heat and -have a correctly configured environment available, this allows deployment -of a working Kubernetes cluster automatically. The Heat templates are -available from https://github.com/larsks/heat-kubernetes/. The templates -require at least Heat 2014.1.3 (earlier versions have a bug that will prevent -the templates from working). - -Here are some simple steps to get things rolling using the Heat templates: - -1. Clone the repository: - ``` - git clone https://github.com/larsks/heat-kubernetes/ - cd heat-kubernetes - ``` - -2. Create an appropriate image by running the get_image.sh script in this - repository. This will generate an image called `fedora-20-k8s.qcow2`. - Upload this image to Glance. You can also obtain an appropriate image from - https://fedorapeople.org/groups/heat/kolla/fedora-20-k8s.qcow2 - -3. Create a file `local.yaml` with settings appropriate to your OpenStack - environment. It should look something like: - ``` - parameters: - server_image: fedora-20-k8s - ssh_key_name: sdake - - - dns_nameserver: 8.8.8.8 - external_network_id: 6e7e7701-46a0-49c0-9f06-ac5abc79d6ae - number_of_minions: 1 - server_flavor: m1.large - ``` - You *must* provide settings for external_network_id and ssh_key_name; these - are local to your environment. You will probably also need to provide - a value for server_image, which should be the name (or UUID) of a Fedora 20 - cloud image or derivative. - -4. `heat stack-create -f kubecluster.yaml -e local.yaml my-kube-cluster` - -5. Determine the ip addresses of your cluster hosts by running: - ``` - heat output-show my-kube-cluster kube_minions_external - ``` - -6. Connect to the minion node with `ssh fedora@${minion-ip}` - -7. On the minion node: - ``` - minion$ git clone http://github.com/stackforge/kolla - minion$ cd kolla - minion$ ./tools/start - ``` - -Debugging -========== -A few commands for debugging the system. - -``` -$ sudo docker images -``` -Lists all images that have been pulled from the upstream kollaglue repository -thus far. This can be run on the minion during the `./start` operation to -check on the download progress. - - -``` -$ sudo docker ps -a -``` -This will show all processes that docker has started. Removing the `-a` will -show only active processes. This can be run on the minion during the `./start` -operation to check that the containers are orchestrated. - - -``` -$ sudo docker logs <containerid> -``` -This shows the logging output of each service in a container. The containerid -can be obtained via the `docker ps` operation. This can be run on the minion -during the `./start` operation to debug the container. - -``` -$ kubectl get pods -``` -This lists all pods of which Kubernetes is aware. This can be run on the -master or minion. - -``` -$ sudo systemctl restart kube-apiserver -$ sudo systemctl restart kube-scheduler -``` -This command is needed on the master after heat finishes the creation of the -Kubernetes system (ie: my-kube-cluster is in CREATE_COMPLETE state). This is -just a workaround for a bug in kubernetes that should be fixed soon. - - -``` -$ journalctl -f -l -xn -u kube-apiserver -u etcd -u kube-scheduler -``` -This shows log output on the server for the various daemons and can be filed -in bug reports in the upstream launchpad tracker. +Directories +=========== -``` -$ journalctl -f -l -xn -u kubelet.service -u kube-proxy -u docker -``` -This shows log output on the minion for the various daemons and can be filed -in bug reports in the upstream launchpad tracker. +* docker - contains artifacts for use with docker build to build appropriate + images +* k8s - contains service and pod configuration information for Kubernetes +* tools - contains different tools for interacting with Kolla +* devenv - A collection of tools and resources for managing a Kolla + development environment. -``` -$ telnet minion_ip 3306 -``` -This shows that the Mariadb service is running on the minions. Output should -appear as follows +Getting Involved +================ -``` -$ telnet 10.0.0.4 3306 -Trying 10.0.0.4... -Connected to 10.0.0.4. -Escape character is '^]'. +Need a feature? Find a bug? Let us know! Contributions are much appreciated +and should follow the standard [Gerrit workflow][]. -5.5.39-MariaDB-wsrep -``` +- We communicate using the #tripleo irc channel. +- File bugs, blueprints, track releases, etc on [Launchpad][]. +- Attend weekly [meetings][]. +- Contribute [code][] -If the connection closes before mysql responds then the proxy is not properly -connecting to the database. This can be seen by using jounalctl and watching -for a connection error on the node that you can't connect to mysql through. +[Gerrit workflow]: https://wiki.openstack.org/wiki/Gerrit_Workflow +[Launchpad]: https://launchpad.net/kolla +[meetings]: https://wiki.openstack.org/wiki/Meetings/Kolla +[code]: https://github.com/stackforge/kolla -``` -$ journalctl -f -l -xn -u kube-proxy -``` -If you can conect though one and not the other there's probably a problem with -the overlay network. Double check that you're tunning kernel 3.16+ because -vxlan support is required. If you kernel version is good try restarting -openvswitch on both nodes. This has usually fixed the connection issues. +Contributors +============ -Directories -=========== +Check out who's [contributing][]. -* docker - contains artifacts for use with docker build to build appropriate - images -* k8s - contains service and pod configuration information for Kubernetes -* tools - contains different tools for interacting with Kolla +[contributing]: https://github.com/stackforge/kolla/graphs/contributors diff --git a/devenv/README.md b/devenv/README.md new file mode 100644 index 000000000..406b60e42 --- /dev/null +++ b/devenv/README.md @@ -0,0 +1,143 @@ +A Kolla Cluster with Heat +========================= + +These [Heat][] templates will deploy an *N*-node [Kolla][] cluster, +where *N* is the value of the `number_of_nodes` parameter you +specify when creating the stack. + +Kolla has recently undergone a considerable design change. The details +of the design change is addressed in this [spec][]. As part of the +design change, containers share pid and networking namespaces with +the Docker host. Therefore, containers no longer connect to a docker0 +bridge and have separate networking from the host. As a result, Kolla +networking has a configuration similar to: + + + +Sharing pid and networking namespaces is detailed in the +[super privileged containers][] concept. + +The Kolla cluster is based on Fedora 21, and makes use of the +[pkilambi/docker][] [COPR][] repository for Docker packages. This +is because Kolla requires a newer version of Docker not currently +packaged in Fedora 21. + +These templates are designed to work with the Icehouse or Juno +versions of Heat. If using Icehouse Heat, this [patch][] is +required to correct a bug with template validation when using the +"Fn::Join" function). + +[heat]: https://wiki.openstack.org/wiki/Heat +[kolla]: https://launchpad.net/kolla +[pkilambi/docker]: https://copr.fedoraproject.org/coprs/pkilambi/docker +[copr]: https://copr.fedoraproject.org/ +[spec]: https://review.openstack.org/#/c/153798/ +[super privileged containers]: http://sdake.io/2015/01/28/an-atomic-upgrade-process-for-openstack-compute-nodes/ +[patch]: https://review.openstack.org/#/c/121139/ + +Create the Glance Image +======================= + +After cloning the project, run the get-image.sh script from the project's +devenv directory: + + $ ./get-image.sh + +The script will create a Fedora 21 image with the required modifications. + +Copy the image to your Glance image store: + + $ glance image-create --name "fedora-21-x86_64" \ + --file /var/lib/libvirt/images/fedora-21-x86_64 \ + --disk-format qcow2 --container-format bare \ + --is-public True --progress + +Create the Stack +================ + +Copy local.yaml.example to local.yaml and edit the contents to match +your deployment environment. Here is an example of a customized +local.yaml: + + parameters: + ssh_key_name: admin-key + external_network_id: 028d70dd-67b8-4901-8bdd-0c62b06cce2d + dns_nameserver: 192.168.200.1 + +Review the parameters section of kollacluster.yaml for a full list of +configuration options. **Note:** You must provide values for: + +- `ssh_key_name` +- `external_network_id` + +And then create the stack, referencing that environment file: + + $ heat stack-create -f kollacluster.yaml -e local.yaml kolla-cluster + +Access the Kolla Nodes +====================== + +You can get the ip address of the Kolla nodes using the `heat +output-show` command: + + $ heat output-show kolla-cluster kolla_node_external_ip + "192.168.200.86" + +You can ssh into that server as the `fedora` user: + + $ ssh fedora@192.168.200.86 + +And once logged in you can run Docker commands, etc: + + $ sudo docker images + +Debugging +========== +A few commands for debugging the system. + +``` +$ sudo docker images +``` +Lists all images that have been pulled from the upstream kollaglue repository +thus far. This can be run on the node during the `./start` operation to +check on the download progress. + +``` +$ sudo docker ps -a +``` +This will show all processes that docker has started. Removing the `-a` will +show only active processes. This can be run on the node during the `./start` +operation to check that the containers are orchestrated. + +``` +$ sudo docker logs <containerid> +``` +This shows the logging output of each service in a container. The containerid +can be obtained via the `docker ps` operation. This can be run on the node +during the `./start` operation to debug the container. + +``` +$ sudo systemctl restart docker +``` +Restarts the Docker service on the node. + +``` +$ journalctl -f -l -xn -u docker +``` +This shows log output on the server for the docker daemon and can be filed +in bug reports in the upstream launchpad tracker. + +``` +$ telnet <NODE_IP> 3306 +``` +You can use telnet to test connectivity to a container. This example demonstrates +the Mariadb service is running on the node. Output should appear as follows + +``` +$ telnet 10.0.0.4 3306 +Trying 10.0.0.4... +Connected to 10.0.0.4. +Escape character is '^]'. + +5.5.39-MariaDB-wsrep +``` diff --git a/devenv/kollacluster.yaml b/devenv/kollacluster.yaml new file mode 100644 index 000000000..0c8c65a95 --- /dev/null +++ b/devenv/kollacluster.yaml @@ -0,0 +1,109 @@ +heat_template_version: 2013-05-23 + +description: > + This template will boot a Kolla cluster with one or more + nodes (as specified by the number_of_nodes parameter, + which defaults to "1"). + +parameters: + # + # REQUIRED PARAMETERS + # + ssh_key_name: + type: string + description: name of ssh key to be provisioned on the Nova instances + + external_network_id: + type: string + description: uuid of a network to use for floating ip addresses + # + # OPTIONAL PARAMETERS + # + server_image: + type: string + default: fedora-21-x86_64 + description: glance image used to boot the Nova instance + + server_flavor: + type: string + default: m1.small + description: flavor to use when booting the Nova instance + + dns_nameserver: + type: string + description: address of a dns nameserver reachable in your environment + default: 8.8.8.8 + + number_of_nodes: + type: string + description: how many kolla nodes to spawn + default: 1 + + fixed_network_cidr: + type: string + description: network range for fixed ip network + default: 10.0.0.0/24 + +resources: + + ###################################################################### + # + # network resources. allocate a network and router for our server. + # it would also be possible to take advantage of existing network + # resources (and have the deployer provide network and subnet ids, + # etc, as parameters), but I wanted to minmize the amount of + # configuration necessary to make this go. + fixed_network: + type: "OS::Neutron::Net" + + # This is the subnet on which we will deploy our server eth0. + fixed_subnet: + type: "OS::Neutron::Subnet" + properties: + cidr: {get_param: fixed_network_cidr} + network_id: + get_resource: fixed_network + dns_nameservers: + - get_param: dns_nameserver + + # create a router attached to the external network provided as a + # parameter to this stack. + extrouter: + type: "OS::Neutron::Router" + properties: + external_gateway_info: + network: + get_param: external_network_id + + # attached fixed_subnet to our extrouter router. + extrouter_inside: + type: "OS::Neutron::RouterInterface" + properties: + router_id: + get_resource: extrouter + subnet_id: + get_resource: + fixed_subnet + + kolla_nodes: + type: "OS::Heat::ResourceGroup" + depends_on: + - extrouter_inside + properties: + count: {get_param: number_of_nodes} + resource_def: + type: kollanode.yaml + properties: + ssh_key_name: {get_param: ssh_key_name} + server_image: {get_param: server_image} + server_flavor: {get_param: server_flavor} + fixed_network_id: {get_resource: fixed_network} + fixed_subnet_id: {get_resource: fixed_subnet} + external_network_id: {get_param: external_network_id} +outputs: + + kolla_node_internal_ip: + value: {get_attr: [kolla_nodes, kolla_node_ip_eth0]} + + kolla_node_external_ip: + value: {get_attr: [kolla_nodes, kolla_node_external_ip]} diff --git a/devenv/kollanet.png b/devenv/kollanet.png Binary files differnew file mode 100644 index 000000000..cfb0c6014 --- /dev/null +++ b/devenv/kollanet.png diff --git a/devenv/kollanode.yaml b/devenv/kollanode.yaml new file mode 100644 index 000000000..915c53a01 --- /dev/null +++ b/devenv/kollanode.yaml @@ -0,0 +1,199 @@ +heat_template_version: 2013-05-23 + +description: > + This is a nested stack that defines a single Kolla node, + based on a Fedora 21 cloud image. This stack is included by + a ResourceGroup resource in the parent template (kollacluster.yaml). + +parameters: + + server_image: + type: string + default: fedora-21-x86_64 + description: glance image used to boot the server + + server_flavor: + type: string + default: m1.small + description: flavor to use when booting the server + + ssh_key_name: + type: string + description: name of ssh key to be provisioned on our server + + external_network_id: + type: string + description: uuid of a network to use for kolla host floating ip addresses + + # The following are all generated in the parent template. + fixed_network_id: + type: string + description: Network from which to allocate fixed addresses. + fixed_subnet_id: + type: string + description: Subnet from which to allocate fixed addresses. + +resources: + + node_wait_handle: + type: "AWS::CloudFormation::WaitConditionHandle" + + node_wait_condition: + type: "AWS::CloudFormation::WaitCondition" + depends_on: + - kolla_node + properties: + Handle: + get_resource: node_wait_handle + Timeout: "6000" + + ###################################################################### + # + # security groups. we need to permit network traffic of various + # sorts. + # + secgroup_base: + type: "OS::Neutron::SecurityGroup" + properties: + rules: + - protocol: icmp + - protocol: tcp + port_range_min: 22 + port_range_max: 22 + + # Use by eth1 to permit all traffic to instances. + # Let the Neutron container apply security to this traffic. + secgroup_all_open: + type: "OS::Neutron::SecurityGroup" + properties: + rules: + - protocol: icmp + - protocol: tcp + - protocol: udp + + secgroup_kolla: + type: "OS::Neutron::SecurityGroup" + properties: + rules: + - protocol: tcp + port_range_min: 8773 + port_range_max: 8776 + - protocol: tcp + port_range_min: 6080 + port_range_max: 6080 + - protocol: tcp + port_range_min: 6081 + port_range_max: 6081 + - protocol: tcp + port_range_min: 35357 + port_range_max: 35357 + - protocol: tcp + port_range_min: 5000 + port_range_max: 5000 + - protocol: tcp + port_range_min: 9191 + port_range_max: 9191 + - protocol: tcp + port_range_min: 9292 + port_range_max: 9292 + - protocol: tcp + port_range_min: 9696 + port_range_max: 9696 + - protocol: tcp + port_range_min: 80 + port_range_max: 80 + - protocol: tcp + port_range_min: 443 + port_range_max: 443 + - protocol: tcp + port_range_min: 8000 + port_range_max: 8000 + - protocol: tcp + port_range_min: 8004 + port_range_max: 8004 + - protocol: tcp + port_range_min: 8003 + port_range_max: 8003 + - protocol: tcp + port_range_min: 8080 + port_range_max: 8080 + - protocol: tcp + port_range_min: 8777 + port_range_max: 8777 + + kolla_node: + type: "OS::Nova::Server" + properties: + image: + get_param: server_image + flavor: + get_param: server_flavor + key_name: + get_param: ssh_key_name + user_data_format: RAW + user_data: + str_replace: + template: | + #!/bin/sh + + yum -y upgrade + + yum -y remove NetworkManager + chkconfig network on + + # enable dnf command + yum -y install dnf dnf-plugins-core + + # Docker packages + dnf -y copr enable pkilambi/docker + dnf -y install docker + + # Start Docker + systemctl enable docker + systemctl start docker + + cfn-signal -e0 --data 'OK' -r 'Setup complete' '$WAIT_HANDLE' + params: + "$WAIT_HANDLE": + get_resource: node_wait_handle + networks: + - port: + get_resource: kolla_node_eth0 + - port: + get_resource: kolla_node_eth1 + + kolla_node_eth0: + type: "OS::Neutron::Port" + properties: + network_id: + get_param: fixed_network_id + security_groups: + - get_resource: secgroup_base + - get_resource: secgroup_kolla + fixed_ips: + - subnet_id: + get_param: fixed_subnet_id + + kolla_node_eth1: + type: "OS::Neutron::Port" + properties: + network_id: + get_param: external_network_id + security_groups: + - get_resource: secgroup_all_open + + kolla_node_floating: + type: "OS::Neutron::FloatingIP" + properties: + floating_network_id: + get_param: external_network_id + port_id: + get_resource: kolla_node_eth0 + +outputs: + + kolla_node_ip_eth0: + value: {get_attr: [kolla_node_eth0, fixed_ips, 0, ip_address]} + + kolla_node_external_ip: + value: {get_attr: [kolla_node_floating, floating_ip_address]} diff --git a/devenv/local.yaml.example b/devenv/local.yaml.example new file mode 100644 index 000000000..b642526e2 --- /dev/null +++ b/devenv/local.yaml.example @@ -0,0 +1,3 @@ +parameters: + ssh_key_name: <YOUR_NOVA_KEYPAIR> + external_network_id: <NEUTRON_EXTERNAL_NET_ID> |
