summaryrefslogtreecommitdiff
path: root/kolla/tests/test_build.py
AgeCommit message (Collapse)AuthorFilesLines
2020-04-04Fix --skip-existing and --skip-parentsMark Goddard1-1/+71
There are currently a few corner cases with the --skip-existing and --skip-parents flags. Examples: 1. kolla-build --type source --skip-parents monasca-grafana This does not try to build the monasca-grafana image. This appears to be an issue when the image is a grandchild. 2. kolla-build --type source --skip-parents This will build all images. Expect it to skip parents. 3. kolla-build --type source --skip-existing This will build all images. Expect it to skip existing images. The filter_images method does quite a lot, including handling whether images are buildable, regex/profile matching, and which images to skip. The matching and skipping parts are done in a single pass, which can lead to some weird effects due to dependencies between the images and their statuses. Also, skipping is only currently applied when there is a regex/profile filter. This change splits out the matching and skipping into two separate passes. In the first pass, we mark all buildable images that match the filter as matched. In the second, we iterate over matched images, applying the skip existing and skip parents rules. Change-Id: I2f895aea0cc59d808129e9fc636af0890196af33 Closes-Bug: #1867614 Related-Bug: #1810979 (cherry picked from commit dfddcce3a5bc98cbd177c984fd9d0a8bab51d40c)
2019-10-16handle push error properlyMarcin Juszkiewicz1-0/+40
If there was some error during pushing then Kolla was greeting with "all went fine" message anyway: INFO:kolla.common.utils.aodh-api:Trying to push the image ERROR:kolla.common.utils.aodh-api:Get http://10.101.16.1:5000/v2/: dial tcp 10.101.16.1:5000: connect: no route to host INFO:kolla.common.utils.aodh-api:Pushed successfully This patch changes that. Now if there is an error during push then proper exception is raised to PushTask and image is marked with PUSH_ERROR status. This way at the end of build it is easy to spot which images did not got pushed to registry: INFO:kolla.common.utils:=========================== INFO:kolla.common.utils:Images that failed to build INFO:kolla.common.utils:=========================== ERROR:kolla.common.utils:base Failed with status: push_error ERROR:kolla.common.utils:nova-api Failed with status: push_error ERROR:kolla.common.utils:nova-base Failed with status: push_error ERROR:kolla.common.utils:nova-compute Failed with status: push_error ERROR:kolla.common.utils:nova-compute-ironic Failed with status: push_error ERROR:kolla.common.utils:nova-conductor Failed with status: push_error Closes-Bug: #1848019 Change-Id: Id2ab97bf4c0dc3423268a0ea435b56f4a65f7196
2019-09-19Fix retries when pushing imagesMark Goddard1-0/+34
Currently the retry mechanism is broken for pushing because the image state gets set to an error state, and is never reset to 'built'. This prevents the PushTask from setting success to True. This change sets the image state to 'built' if the push succeeds, ensuring it overrides any previous failures. Change-Id: I93fc0e383da8fec6b3ca31f8094321c2a0c3af71 Closes-Bug: #1844697
2019-09-18Remove support for OracleLinuxMark Goddard1-1/+1
We have agreed to remove support for Oracle Linux. http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006896.html Change-Id: I961dfba1aeef2c509ecdaf376dfe6928926ffb39 Depends-On: https://review.opendev.org/682858
2019-07-21Change how build process treats unbuildable imagesMarcin Juszkiewicz1-4/+11
We have a list of images which are unbuildable for distro/buildtype/arch combos. This patch renames it from SKIPPED_IMAGES to UNBUILDABLE_IMAGES because 'skipped' images is something else in our code. Now all unbuildable images are marked as such and so are their children. At the end of build there is information provided about skipped images (due to --skip-parents or --skip-existing options) and then information about not buildable images. Both list should only contain entries related to requested build. Change-Id: If9b521339f564e483cba03d52e7c4eba271821a5
2019-06-06Debian/Ubuntu: move to Python 3 for source imagesMarcin Juszkiewicz1-2/+2
Commit 43b74ccc157c0b50138c3785ed91dab504895571 enabled use of Python 3 based packages but not switched to use Python 3. Some of images still contain Python 2. There are two reasons: - Ceph (ceph-common depends on Py2) - python3-ldappool on Ubuntu 18.04 In Ceph situation Py3 packages were added. For second one we can not do anything - Py2 dependency got dropped in Ubuntu 18.10 version. Removed neutron-server-plugin-networking-infoblox due to being not maintained. Once https://review.opendev.org/#/c/657578/ get merged someone may revert that part. Implements: blueprint debian-ubuntu-python3 Depends-on: Ie2a1077f7def0743f1403341985e2109aa490026 Change-Id: Ibfe0c2b8be98db56c61f74fb0247488ab3749ef4
2019-04-24Move to opendevZhongShengping1-1/+1
1.Use opendev.org instead of git.openstack.org. 2.Use review.opendev.org instead of review.openstack.org. You can see the discussion below: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html Change-Id: I377ea63827b6f3995ee275fdc466c1b2100eafc7
2019-03-12Add distro_package_manager configurationAlex Schultz1-0/+46
Add configuration to allow the switch between dnf/yum or provide an alternative package manager. Change-Id: I913769fcf5875a12b4561958cf7c90b74cd116ea
2019-03-12replace 'distro_python3' variable with 'distro_python_version'Marcin Juszkiewicz1-10/+30
RHEL7 derived distros use Python 2.7 RHEL8 uses Python 3.6 Ubuntu 18.04 uses Python 3.6, but we haven't enabled it yet. Debian:buster uses Python 3.7, but we haven't enabled it yet. We have several places in code where "/lib/pythonX.Y/site-packages" is used so we need to know value of X.Y rather than py2/py3 switch. How to check for Python 2: {% if distro_python_version == '2.7' %} Breaks: Iad6b70b433a0dd1b0f8ae6790fd280594517661a Change-Id: Ie24f109af19ae0438c21e0d43cb3cc947812b4ce
2019-01-16Add base_package_typeAlex Schultz1-0/+18
Currently in all the Dockerfile.j2 files, we are doing base distro if conditions for package names when installing via a binary. Rather than duplicating these base_distro in [<distros>], let's provide a single value for determining the binary installation method. This change adds a base_package_type which we can use to lookup if 'rpm' or if 'deb' rather than relying on specific distro names. Change-Id: I8a0c6c8fbc32a8cfa6932fddf28a449fceda3d49
2019-01-16Add distro_python3 optionAlex Schultz1-0/+20
As we add support for python3 on RHEL based systems, we need to add a flag to key off of in the Dockerfiles to swap out the python3 package names. This change adds a distro_python3 configuration option that can be defined to force python3 or python2 package names. If this option is not provided, we set this to True for RHEL 8 based distros and false for other distros. Change-Id: I4028991bad92c0e8e21066cc4173c06ce5eba393 Related-Blueprint: python3-support
2018-06-30Ability to build container against docker swarmVictor Coutellier1-5/+5
Add a decode=True option during call to dockerAPI in order to build container against recent docker swarm. Closes-Bug: #1779460 Change-Id: I48f041e7ed9dd5a5f410b4eaa081f6f013119d17
2018-05-16Fix image push for python3Michal Nasiadka1-2/+3
Rework push_image function to use docker client push function with decode=True, to decode automatically to dict. Change-Id: I28d38e968c3809a1074347271e49dc07dc861f11 Closes-Bug: 1771514
2018-04-20Add network_mode for image buildingAlex Schultz1-4/+21
Add the ability to specify a different network_mode when running the image builds. Under the covers the default is to use bridge networking but if that is unavailable or a user would like to specify a different networking method they can now do so using the [DEFAULT]\network_mode configuration. Change-Id: I1d2398273b57ca4d52bbfacbc896ffecc58bc54b
2018-02-03Remove the unsupported 'insecure_registry' for >=docker 3.0.0Jianghua Wang1-1/+12
The argument of 'insecure_registry' for image push has been removed by this commit (in docker 3.0.0): https://github.com/docker/docker-py/commit/b180b8770a265e33099bd6da76c3e556a1028491 Actually this argument was marked as deprecated and unfunctional before this removing. This commit is to remove this parameter at invoking the push() function when docker version >= 3.0.0. Change-Id: Ifa7d304d08a4073dcb1dff751d5a443c112cd377 Closes-Bug: #1746703
2017-08-03Fix for git clone issue during kolla image buildJiri Prokes1-0/+22
When the first build attempt fail for any reason (e.g. short network outage, proxy outage,...) then the second attempt always fail for git source type. Git can't clone repository because destination directory exists and is not empty. Current fix checks destination directory and if it exists then it is removed before cloning. Change-Id: I949140c49a64baea579d61047e3b2f1240da2771 Closes-Bug: #1706369
2017-07-16Merge "Add bandit check to all python code"Jenkins1-2/+2
2017-07-06Add bandit check to all python codeEduardo Gonzalez1-2/+2
Bandit was not running in all directories due missing -r parameter. Also, this change fix bandit errors. * Add nosec to subprocess module * Change /tmp/releases to .releases * Change /tmp/ to tmp/, tmp/ is not checked and in unittest is not an issue, simpler that mocking tempfile and make more complex tests. Change-Id: I42bafa4707a9a4d1faeb01cb9b5a843f56280deb
2017-07-05Fix mock orderMartin André1-3/+3
The mocks were in the wrong order. The only reason the tests were passing so far was because the calls to mocks were not checked, the mocks were only used to prevent the tested function to make calls to actual resources. Change-Id: If7ea23ac20e3a0d584e02e31d9d60ba914efa8aa
2017-07-05Add new --skip-existing option to kolla-buildMartin André1-1/+5
This new option allows to skip images that are already present in the docker cache. Change-Id: I202c70864f8aa86abd882240beb7f6525963e625 Closes-Bug: #1694302
2017-06-21Upgrade from docker-py to dockerHongbin Lu1-8/+8
The pypi package 'docker-py' [1] has been renamed to 'docker' [2]. It is better to move to the new 'docker' package because the old package will be deprecated and all the new features will go into the new package only. Package 'docker' has been added to requirements [3]. The old package 'docker-py' is still allowed to be in the global requirements during the transition period but it should be removed after all or most of the projects finsih the migration. [1] https://pypi.python.org/pypi/docker-py [2] https://pypi.python.org/pypi/docker [3] https://review.openstack.org/#/c/423715/ Change-Id: I3ffd8bb6461e8b7edacc5e833754f00c4249b854
2017-06-02kolla-build --list-dependencies use json, not pprintSteve Baker1-3/+4
kolla-build --list-dependencies uses pprint, which prints almost-but-not-quite json. This makes it hard for other tools to parse its output. This change switches to a json output with an indent of 2. If there is interest in a default output which is friendly to both humans and software, YAML might be more appropriate. Change-Id: Icaead5a62a2d6510965524f68fb57dcf7494e618 Closes-Bug: #1694888
2017-04-18Ensure tests run in isolationMartin André1-5/+16
Previously, the images were copied locally in the tests as an attempt to ensure test isolation but the parent image was still pointing to the global object. This commit updates the images to point to the local copies. Rename the images to have more descriptive names and fix an issue in test_skip_parents where the regex parameter was passed as a string while it should be an array, that appeared once the images had their correct parents. Also bypass docker client memoization to further minimize the risk of breaking test isolation. Change-Id: Id7df645284f260cd615188db2efe889c9cf8115d Closes-Bug: #1683314
2017-03-30Merge "Allow the use of previously built parent images"Jenkins1-3/+20
2017-03-29Allow the use of previously built parent imagesPaul Bourke1-3/+20
Consider the case where you have a set of tagged / released images to production, and you want to apply a small hotfix, security patch, etc. to one of them, say keystone-ssh. Rebuilding keystone-ssh will cause a rebuild of base, openstack-base, and keystone-base, potentially pulling in updated packages / new dependencies, unless you have access to the exact machine they were previously built on. Ideally you want to use the exact parent images that are in production. This patch adds a new build argument '--skip-parents', which will result in only the image(s) specified by the regex and their children to be built. Parents are expected to exist either on machine or in a registry. Change-Id: I79d5f0b422f48d2dc36ae85dfa21668cf8177837 Implements: blueprint image-build-depth
2017-03-23Using assertIsNone() instead of assertEqual(None)lingyongxu1-1/+1
Following OpenStack Style Guidelines: [1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises [H203] Unit test assertions tend to give better messages for more specific assertions. As a result, assertIsNone(...) is preferred over assertEqual(None, ...) and assertIs(..., None) Change-Id: If18685d75a847571c62dd1424c7b71433bbd4ce4
2017-03-01Add work-dir command-line optionJavier Pena1-0/+7
The working directory was previously generated as a temporary one, but there may be cases when we want to specify a directory, for example if we want to generate the Dockerfiles and then do some parsing on them. Change-Id: Ic332328913924bcb3d0dfa4d473e796050fc14b3
2017-03-01Merge "Code coverage improvement for kolla"Jenkins1-3/+89
2017-02-17Fix test_build_rpm_setup failure after changed rpm_setup_configJeffrey Zhang1-0/+1
Change-Id: Ieea2e9008e9bacdc4d1ecb8594a9297c92fffe78 Closes-Bug: #1665674
2017-01-25Code coverage improvement for kollaSurya Prakash Singh1-3/+89
This patch added the unit tests which improved the coverage upto 82% for kolla deliverable with respect to previous coverage report. Change-Id: I0ecb067b5b42ebda8d0a4d77811c842fc7647d1b Implemented: blueprint coverage-increment-for-kolla
2017-01-16Py3 and py2 implementation for kolla/image moduleSurya Prakash Singh1-0/+5
This commit addresses the changes in kolla/image directory for better openstack python coding style. unittest also added for corresponding changes. Reference:- https://wiki.openstack.org/wiki/Python3 Change-Id: I8be60ebd214ac8ab400cc633f8e6005fb8760c24 Partially-Implements: blueprint support-python-35
2016-11-30Merge "Add neutron vpnaas code into neutron-server container"Jenkins1-2/+7
2016-11-15Add neutron vpnaas code into neutron-server containerJeffrey Zhang1-2/+7
* Add neutron vpnaas source in neutron-server container. * Fix the test_build.py to support mutiple plugin configuration. Change-Id: I2778b7fbf23207576c21a3e7fa1b9a399c228cf3 Closes-Bug: #1641063
2016-11-12Extract kolla exceptions into a single exception.py filelijing1-1/+2
Change-Id: I03388360b6ac735d5a6f380a7956cd91ab449131
2016-11-06Merge "Fix plugin parsing for base images"Jenkins1-0/+18
2016-11-04Fix plugin parsing for base imagesPaul Bourke1-0/+18
If we try to add a plugin to a base image, e.g. [neutron-base-plugin-neutron-fwaas] type = git location = https://github.com/openstack/neutron-fwaas reference = master The build will fail as it parses the above as both "neutron-base-plugin-neutron-fwaas" and "base-plugin-neutron-fwaas". Update the regex to ensure the image name is anchored to the beginning of the conf section. Change-Id: I4099ebd0d25cd28a9bb3bdc4fcfbf311ce22a3fd Closes-Bug: #1638900
2016-11-03Remove Fedora supportChristian Berendt1-1/+1
Closes-bug: #1616387 Change-Id: Id97f88b9baa3d48d33ce120962450a374282d044
2016-09-14Add iter function to change list into iteratorgong yong sheng1-5/+21
Change-Id: Ia141a54b1bc0b1742a396b7029ee0b56ead648db Closes-bug: 1623288
2016-08-06Fix inconsistencies in git urlRobin Naundorf1-1/+1
Github is just a mirror of the OpenStack git-repo. Changed from Github to OpenStack git url wherever possible. Change-Id: I7941ef86967de4efe7f23ff9fb11ec86c793901e
2016-06-21Stop using a global logger for all the thingsJoshua Harlow1-4/+2
Instead of using the same logger for all the things which makes it incredibly hard to figure out what the images and there builds are doing; instead have it so that each image gets its own logger which is then used by tasks that process that image. Each images logs now will (optionally) go to a logs directory that can be more easily read and looked over and leaves the main logger for informational/task kind of output only. Closes-Bug: #1586478 Change-Id: I939e073a2bc5ed8b5a1e969888fc743d80f468c9
2016-06-07Make build.py importable libJeffrey Zhang1-5/+6
First step to break it down is to make it importable lib rather than self-contained script. Change-Id: Ibbc924c2849cc79006046f62f31a1909ce959066 Partially-implements: bp build-refactor
2016-06-02Use an image object, recorder object and status constantsJoshua Harlow1-26/+22
Instead of using raw dicts and passing data around via dictionaries (which makes it really hard to figure out what is in those dictionaries at any point) prefer to use objects. That way people can actually understand what the object is supposed to be, vs guessing and/or having to decipher its usage. The same goes for raw string constants, prefer using named constants instead. Closes-Bug: #1586475 Change-Id: Ide179dc6593c50696d47a2d3d4cd000f343855d4
2016-06-02Use a task model and dumb workersJoshua Harlow1-24/+35
Instead of having smart workers that need to know about the full logic of how they operate and what they will do which makes it hard to reuse parts of those workers switch the model so that workers are dumb and they just run tasks which themselves can request additional tasks to run when succesful. This makes it more consistent when tasks are to be retried and also makes it easier to know which task is running and when. Less specialized workers == good. Closes-Bug: #1586474 Change-Id: Ie7b7303b8bf2a80b3e26b6d4ffc27787f8c794d6
2016-05-31Don't always unpack image build results tuple, as it may be NoneMark Goddard1-0/+27
Change-Id: I21b1327da1de1bd3cc3018ba37715b0a19c1dbef Closes-Bug: #1587501
2016-02-28Add timeout to requests.get() in kolla/cmd/build.pyVikram Hosakote1-0/+22
When kolla-build is running, if there are network issues or if the source's location (like http://tarballs.openstack.org) fails to respond due to high number of concurrent requests, kolla-build just hangs/blocks indefinitely. This patch set resolves this issue by adding a timeout of 120 seconds for requests.get() in kolla/cmd/build.py, adds a unit test for it in kolla/tests/test_build.py and also the "timeout" argument in kolla/common/config.py. Change-Id: I7c8745a20b9bd1c3f5d6a55c72a794f16fd7e513 Closes-Bug: #1548614
2016-02-15fix the custom profile raise exception issueJeffrey Zhang1-9/+53
Closes-Bug: #1536272 Change-Id: I6c91e522923eade16ba23711e6014e6b4b8cd3e0
2016-02-12Fix image plugin functionality for oslo.configPaul Bourke1-0/+24
The new oslo.config style parsing for kolla-build.conf stopped plugins from working. This patch fixes it and adds a unit test. Co-Authored-By: Jeffrey Zhang <zhang.lei.fly@gmail.com> Closes-Bug: 1534556 Change-Id: I135797c733ae0cae040a009c364073769b19e4eb
2016-02-04Move the install base type check to kolla-buildAllen Gao1-0/+26
The install type is converted in kolla-build, so it will never fail in dockerfile, move the check to kolla-build just above the install type converting. TrivialFix Co-Authored-By: Jeffrey Zhang <jeffrey.zhang@99cloud.net> Change-Id: I1500d3b47e909f94ea9f68c5245297733f63a70b
2016-01-18Pick up proxy buildargs from environmentPaul Bourke1-0/+38
We currently support any buildarg via --build-args. This patch picks up the default supported proxy buildargs if set in the user's environment, to allow for more transparent proxy support. The --build-args will take precendence. DocImpact Implements blueprint: better-proxy-support Change-Id: I084e5d1cc8f9993d65167878f9983ad58a68e86e
2016-01-18Expose the docker build_arg to build.pyJeffrey Zhang1-1/+19
With this implement, we can add variables at building stage. For example, add HTTP_PROXY and NO_PROXY when needed like below. build.py --build-args \ HTTP_PROXY:http://127.0.0.1:8080,NO_PROXY:127.0.0.1 More info about build_arg, pls check[0] [0] https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables-build-arg DocImpact Implements: bp add-buildargs Change-Id: I29ed7f536670fef59d697603dc562a69d90743c9
generated by cgit v1.2.3 (git 2.39.1) at 2026-11-05 22:16:31 +0000