aboutsummaryrefslogtreecommitdiff
path: root/docker/fluentd
AgeCommit message (Collapse)AuthorFilesLines
2024-11-15Add mechanism for patching files in containersMichal Arbet1-0/+2
This patch adds a way to patch files in a Docker image built by Kolla. This is very useful for several reasons, specifically: - Custom modifications - The stable branch of some library has a fix but no pip package has been released - Eliminates the need to package your own pip packages - Eliminates the need to invent your own versioning to prevent upstream versioning - Eliminates the need to manage a pip server - In other words, it eliminates the need to get a wheel into the image and install it manually using any method not previously mentioned It is also highly desirable because, although Kolla can replace the source for a service with a custom URL for a tarball or its own Git repo, it cannot do this for dependencies pulled from pip. I would also like to point out that this is a feature with its own code path and works only if the user "inserts" a patch into the folder patches/docker-image/something.patch and creates an analogous series file for patch source code. Simply said, this code will never interfere with the upstream build process since this feature is not intended for use in upstream. It is rather meant for downstream users who know what they are doing. Now they just have an option to patch their images. Everything works on all layers of the Docker image and stores a report of applied patches which can then be seen in /etc. This mechanism is similar as debian patch quilt. Change-Id: I61d0790c5d4d070b7ea9e8c99c0a76ff5d22bf9d
2023-11-30bump td-agent lts from v4 to v5Sven Kieske3-17/+10
notice that the td-agent command got renamed to fluentd. see also the upgrade notes at[0]. for more details, refer to the changelog[1]. [0]: https://web.archive.org/web/20231006084137/https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5 [1]: https://github.com/fluent/fluent-package-builder/blob/b4b7017c09d78f2c404db62612733b40f69668a6/CHANGELOG.md#release-v500---20230729 Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/901509 Change-Id: I5e95363c6c180e330945fe3d98e97c7049bbd6f7 Signed-off-by: Sven Kieske <kieske@osism.tech>
2023-11-29fluentd: Add labels for transition to v5Michal Nasiadka1-3/+4
Change-Id: Ibe0eb9ada69db3ef8042306fdec44bfda1dd9ce0
2022-12-07Merge "Drop elasticsearch, kibana and logstash"Zuul1-6/+0
2022-12-06Drop elasticsearch, kibana and logstashMichal Nasiadka1-6/+0
Support has been dropped in kolla-ansible, the upgrade path is opensearch. Change-Id: I6f32aacb4e0b7dd1eb7206e02425b16f5beae4f0
2022-12-06Remove monasca, kafka, storm and zookeeper.Michal Nasiadka1-17/+0
Monasca, kafka, storm and zookeeper have been removed in kolla-ansible [1],[2]. [1]: https://review.opendev.org/c/openstack/kolla-ansible/+/861392 [2]: https://review.opendev.org/c/openstack/kolla-ansible/+/865029 Change-Id: I0958a9353bc4585c4f65a1d316328a0e80b0b8b7
2022-10-10Fix writable rootwrap/privsep configRadosław Piliszek1-1/+1
Fixes a hypothetical security issue related to privilege escalation via rootwrap/privsep. A potential vulnerable service could previously allow writes to its rootwrap/privsep config and thus allow for more commands to be run with root privileges via rootwrap/privsep. For a succesful attack, this would also require the service to allow to run arbitrary commands via rootwrap/privsep. Thus far, no such vulnerabilities have been reported and thus this fix is simply strengthening the container images against such an issue in the future. Change-Id: I92c81c77e6a16570a108cde8031f7977930fb02a Closes-Bug: #1874298
2022-04-21flatten images a bitMarcin Juszkiewicz1-9/+7
As we have one type of images now some RUN calls could be merged so we will have less layers in resulting images. Change-Id: I5178c58fbd8c65efe825dc249c0f1368ef0fe8e0
2022-04-11Adjust permissions on _extend_start files.Tim Shearer1-1/+1
Explicitly set the permissions on the kolla-toolbox kolla_extend_start file. Also, since all extend_start files are sourced rather than executed, the executable bits are now cleared throughout the project. Change-Id: I5c2deb4a2e33575d57c852089f856a9acc6818d0
2022-04-11drop infra_image_prefixMarcin Juszkiewicz1-1/+1
We have one install_type only now so that infra stuff is not needed anymore. Change-Id: I60d50ba7699d46d001eaef365fb84b0ce8e9b262
2022-03-11Unify curl use (customisation options)Maksim Malchuk1-3/+6
This change adds and unifies an ability to override hardcoded URLs for air-gapped environments via the usual blocks mechanism. Also, this change replaces ENV with ARG instruction for the variables used only in building images, and uses bash variable expansion where it possible. This change is a continuation of the I46b77978926fc2b578a68d1aaa944b2198af0685 Change-Id: I1fbad333b1bc95484e6f4c5145d5936a0e2db84f Signed-off-by: Maksim Malchuk <maksim.malchuk@gmail.com>
2022-01-27Futureproof the uninstallation of offending es gemsRadosław Piliszek1-3/+3
Uninstall based on version range. Change-Id: I353ef9b6b8da0e726c17ad2e06995d3690731e1d
2022-01-27Unpin td-agent and cap elasticsearch gemDr. Jens Harbott1-3/+8
With [0] the version of td-agent being installed was pinned, which isn't sustainable in the long run, so we drop the pin. Latest version of the elasticsearch gem no longer works with older (OSS) versions of Elasticsearch. This is fixed by downgrading the version of the elasticsearch gems. [0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885 Related-Bug: 1930867 Closes-Bug: 1954759 Signed-off-by: Dr. Jens Harbott <harbott@osism.tech> Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823155 Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f
2021-12-29Clean up fluentd image recipeRadosław Piliszek1-66/+15
Since Xena, there is no arch supported other than x86_64 and aarch64. Additionally, the conditional for x86_64 Debian does not seem to be needed now with current td-agent. Moreover, drop two plugins that are relevant only to fluentd 0.12, one of which (parser) causes fluentd 0.12 (!) to be installed. Modern fluentd has both built-in. Finally, drop the labels as they are no longer useful. Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823093 Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823094 Change-Id: I7326cd8fcfa459dedcb6404e5287f0f5aeeed6cb
2021-12-28Add fluentd grok-parser plugin on CentOS on aarch64Radosław Piliszek1-1/+1
This conditional was missed in the original patch moving aarch64 to td-agent. [1] After this change, other plugins will have newer versions installed as well, as it happens in other image flavours. [1] I86ecdc1ac45dfd76b94a34d1b837f7c0bc975dcd Closes-Bug: #1955889 Change-Id: I10ff098b3356eb02ad4a1006b988812356768fec
2021-06-15Pin td-agent to 4.0.* to fix missing logsDoug Szumski1-1/+2
The recent inclusion of Fluentd 1.12.x in Centos/Ubuntu packages causes a significant number of logs to fail to appear in the Elasticsearch or Monasca output plugins. The logs appear to sit in a local buffer, and are only flushed when Fluentd is restarted. The same issue persists in the most recent release (1.13.0), so for now we pin the td-agent package to 4.0.* which should restrict Fluentd to the 1.11.x release where we don't see this issue. Backport to Victoria. Change-Id: Iefcdd3100b7e3c5320bc5f1286a18251bdeab885 Closes-Bug: #1930867
2021-01-06Upgrade the Monasca Fluentd output pluginDoug Szumski1-1/+1
From Victoria onwards a conflict is seen between the rest-client library used by the Monasca Fluentd output plugin and the openssl library. Specifically, a more recent release of openssl has dropped a default which is referenced by the 1.8 release of the rest-client library. In this commit we bump the Fluentd output plugin release to pull in a more recent version of the rest-client library which can handle this change. Change-Id: Iae5402ea40c9f856b1787fdf82cc2389ffe0be03 Closes-Bug: 1910382
2020-12-02Unify curl optionsMichał Nasiadka1-1/+1
Currently we use couple of curl options throughout Dockerfiles, this change adds all common options to curlrc (-sSLf) and removes usage of those in Dockerfiles. Change-Id: I46b77978926fc2b578a68d1aaa944b2198af0685
2020-07-27Upgrade Fluentd Monasca pluginDoug Szumski1-1/+1
Switch to YAJI JSON parser as suggested in [1]. [1] https://docs.fluentd.org/quickstart/faq#i-got-encoding-error-inside-plugin-how-to-fix-it Closes-Bug: #1889065 Change-Id: I2ecd3b029b956e000feaaa9e10a169521ba53d66
2020-07-22Merge "fluentd: migrate to td-agent 4 where possible"Zuul1-5/+5
2020-07-20fluentd: migrate to td-agent 4 where possibleMarcin Juszkiewicz1-5/+5
td-agent 4 got released for all distributions we support. For both x86-64 and aarch64 architectures. Change-Id: I86ecdc1ac45dfd76b94a34d1b837f7c0bc975dcd
2020-07-09introduce 'infra_image_prefix' for infra imagesMarcin Juszkiewicz1-1/+1
Defaults to 'image_prefix' for now but shows which images gets their names changed soon. Change-Id: I0608e8f62f28d6667b4c8753c47553e4cbf75503
2020-06-15Install Fluentd Prometheus plugin out of the boxDoug Szumski1-0/+1
This plugin [1,2] supports a number of use cases which are likely to be useful out of the box to users enabling either Monasca or Prometheus: 1. As a system admin I want to monitor the status of Fluentd to ensure that it is functioning normally. 2. As a system admin I want to know the time to response for calls made to endpoints defined in HAProxy. [1] https://docs.fluentd.org/deployment/monitoring-prometheus [2] https://github.com/fluent/fluent-plugin-prometheus Change-Id: I9790cd6c9d142a4a3ced6d5c9a9af621c3892eb0
2020-05-16Upgrade Monasca Fluentd output pluginDoug Szumski1-2/+2
As part of the support for using the unified Monasca API for logs, the Fluentd Monasca output plugin has been upgraded. Any custom Fluentd configuration which forwards logs to Monasca will need to be updated. Change-Id: I7deefec590cd3bfc774b47867d968afe1bd69d99
2020-03-01Add block labels to all DockerfilesChristian Berendt1-0/+2
Change-Id: I9692dda817ef134d647247431565e1b58cf9da41
2019-12-17docker: fix 'Empty continuation line found' warningsMarcin Juszkiewicz1-3/+3
Whitespaces suxx ;D Change-Id: I4bfd437c914680a5dcbc67dfe9b0a9a74e2eb35d
2019-11-29Enable repos only when neededMarcin Juszkiewicz1-0/+2
Disable external repositories by default and enable only when needed. Depends-on: https://review.opendev.org/696480 Implements: blueprint repos-off-by-default Change-Id: Icf2a8397a8349e0fe849d88d160409fd234480a9
2019-10-02Fluentd: There is available td-agent pkg for BusterViktor Michalek1-0/+6
Thanks a lot to Masahiro Nakagawa https://github.com/treasure-data/omnibus-td-agent/issues/209#issuecomment-530340475 Change-Id: I9c4993436844d45bc9df2815659fa7c420910b01
2019-09-25Debian: switch fluentd to td-agent on x86_64Radosław Piliszek1-7/+8
This patch uses the newly added td-agent repo on Debian. It also makes code more robust regarding different distro handling. We now have td-agent everywhere on x86_64. Change-Id: I4d2449d7d100c68084f937ef9981596fc1625375 Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-09-04Change fluentd to td-agent on CentOSMichal Nasiadka4-55/+54
Fluentd packages in CentOS Opstools SIG repos is very old, because of old-ish ruby (2.0) packaged with CentOS 7. On the other hand - TreasureData packages for fluentd are packaged with embedded ruby as dependencies - so this change moves to those. Thanks to that we are able to move from fluentd 0.12 to 1.4.2. Still we need ruby-devel for non-td-agent distros, because we download monasca output plugin as source and build it during installation. * Bump up td.repo to version 3 * Merge sudoers into one file * Change fluentd plugin install macro to reflect changes Depends-On: https://review.opendev.org/679209/ Depends-On: https://review.opendev.org/679885/ Change-Id: I40a81103317ac044d9311b2505593056cf313bd2
2019-08-22Merge "Add fluentd_binary and fluentd_version labels"Zuul1-0/+5
2019-08-19fluentd: there is no 'td-agent' for Debian/x86-64Marcin Juszkiewicz1-11/+7
As there is no 'td-agent' package image build fails on 'x86-64'. We have to wait for TD Agent 4. Change-Id: I03d38628eee0f0823a58cba3a70df22f29022956
2019-08-19Add fluentd_binary and fluentd_version labelsMichal Nasiadka1-0/+5
In order to orchestrate a smooth migration of fluentd to newer td-agent on CentOS x86_64, a label that points to fluentd binary and version are needed. Depends-On: https://review.opendev.org/677144/ Change-Id: Ie2257870d87c5b3a9b0572f8fd35085a3f95fedb
2019-08-19Unbreak CI: fluentd and TripleO dependencyRadosław Piliszek1-2/+0
fluentd - we are not deploying kubernetes, no need for plugins TripleO - CI currently broken (to reverse, FIXME in place) Change-Id: I203f70176af7f2b53398ddeb5c21e6234a45e268 Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-06-28Bump Monasca Fluentd output pluginDoug Szumski1-1/+1
This resolves an issue where the previous version of the plugin would fail to re-send buffered logs if a post to the Monasca Log API failed. With this new version of the plugin, logs are correctly buffered and resent. Change-Id: Ic1d93a8ea5f2571edfacf50809d51a8b77959a73
2019-05-29Use td-agent-gem to install fluentd monasca plugin on ubuntuMark Goddard1-1/+1
If monasca is enabled on ubuntu, we see the following error in fluentd: [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown output plugin 'monasca'. Run 'gem search -rd fluent-plugin' to find plugins" This is because we use 'fluent-gem install' to install the gem, which works on CentOS but on Ubuntu it is necessary to use 'td-agent-gem install'. Change-Id: I515f9764ca93a41a871e520696193a9298856d08 Closes-Bug: #1830147
2019-05-28Merge "Install fluentd gem on debian/ubuntu for non x86_64"Zuul2-5/+23
2019-05-23Bump Monasca Fluentd plugin versionDoug Szumski1-1/+1
Bump version to include support for setting log message field name. Change-Id: Ib3953b49247bb36ac8ce25578c05c6bcb2a1905a Partial-Bug: #1830184
2019-03-28Install fluentd gem on debian/ubuntu for non x86_64Pedro Alvarez2-5/+23
The package used to install fluentd on debian/ubuntu is not available for architectures other than x86_64. For those cases install fluentd as a gem until the package 'td-agent' is available for more architectures. Change-Id: Ic1b25b37d06b429a112f1418a79d385d505bb1e9
2019-01-17Use base_package_typeAlex Schultz1-7/+7
This change updates the docker files to use base_package_type instead of doing specific distro checks for the rhel/deb generic cases. The base_distro is still available and is used when a specific distro needs a customization but if the differences are purely rpm vs deb, then the base_package_type can be used. Change-Id: I8d720bb185df65a0178061ccf20b1ab2265da2c5
2018-10-11Build and install Fluentd Monasca pluginDoug Szumski1-0/+16
This is a simple output plugin for Fluentd which can forward logs to the Monasca Log API. Partially-Implements: blueprint monasca-containers Change-Id: I599554052c3cb99ebcbcbc4d7ea1aeb57c573d2e
2018-01-05Fluentd plugins rpm installation on CentOSJuan Badia Payno1-24/+12
Currently, fluentd plugins are installed via gem install, this may causes some issues when gems are updated upstream. This patch uses the plugins packaged on CentOS opstools repository. The debian/ubuntu image has not been modified Change-Id: Ib32ceaae0d9fa56c6ec25a47ae6f44259d3b6e6d
2017-11-24Clean fluentd_plugins_install blockMartin Mágr1-16/+36
fluentd_plugins_install should contain only plugin installation operations, so that the override does not have to handle irrelevant operations Change-Id: I8f810d194a1e638683df1f445c187b8d98629730
2017-11-14Fix the Ubuntu Fluentd imagechenxing1-3/+1
For ubuntu, we actually use fluentd 0.14.22 and ruby 2.1.10 now. Thus, we should use the specific gem version for the plugins: - fluent-plugin-rewrite-tag-filter:2.0.0 According to the README.md of this plugin[1] It shows: ---------------------------------------------------------- | fluent-plugin-rewrite-tag-filter | Fluentd | Ruby | |----------------------------------|------------|--------| | >= 2.0.0 | >= v0.14.2 | >= 2.1 | | < 2.0.0 | >= v0.12.0 | >= 1.9 | ---------------------------------------------------------- - fluent-plugin-grok-parser:2.1.4 When using the fluent-plugin-rewrite-tag-filter 2.0.0 to fix this bug, another bug[1] which is the same as this one[2] was triggered. According to https://github.com/fluent/fluent-plugin-grok-parser, it shows: --------------------------------------------------- | fluent-plugin-grok-parser | fluentd | Ruby | |---------------------------|------------|--------| | >= 1.0.0 | >= v0.14.0 | >= 2.1 | | < 1.0.0 | >= v0.12.0 | >= 1.9 | --------------------------------------------------- We actually use fluentd 0.14.22 now, but the version of fluent-plugin-grok-parser is still pinned in 0.3.1. So we need to upgrade the fluent-plugin-grok-parser to 2.1.4. [1] https://github.com/fluent/fluent-plugin-rewrite-tag-filter [2] http://paste.openstack.org/show/626145/ [3] https://github.com/fluent/fluent-plugin-grok-parser/issues/32 backport: pike Change-Id: I47985113fe732569d640a262ca832c6edc8c2bb6 Partial-Bug: #1730664
2017-11-06Fix CentOS 7 fluentd imageJuan Badia Payno1-1/+1
Use specific gem version for the plugins: - fluent-plugin-rewrite-tag-filter - fluent-plugin-elasticsearch Change-Id: Icc5b8972e7eca01767cdc9321ee82d1fcde187d8 Closes-Bug: 1730346
2017-10-16Revert "Fluentd: add fluent-plugin-concat:1.0.0 gem"Eduardo Gonzalez1-1/+1
This reverts commit 83baad304d359e17a5be3ad1e92f7cd31dc129e0. The plugin is not used in any deployment tool. Was introduced to make multiline but finally standard multiline was used. This change removes an unused plugin. Change-Id: Ie38a876d56846ebdec02fa05707e9d22a02788e6
2017-10-13Fluentd: enable forward logs over SSL with authenticationJuan Badia Payno1-2/+2
On the fluent Docker added the fluent-plugin-secure-forward, so fluentd will be able to forward logs over SSL with authentication Closes-Bug: #1723408 Change-Id: I231ebc1052dac93fb9420dcfb10595a3647c35ce
2017-08-29Merge "Fluentd: add fluent-plugin-concat:1.0.0 gem"Jenkins1-1/+1
2017-08-28fix image build error of fluentdjiangpch1-1/+1
the newest public_suffix gem required ruby>2.1.0, but the ruby on centos' release is 2.0.0, we should install public_suffix==2.0.5 manual to avoid the ruby version not matched error. Closes-Bug: #1713416 Change-Id: I725294da206a73b759027d3123d140bddafcfdfb
2017-08-17Fluentd: add fluent-plugin-concat:1.0.0 gemBertrand Lallau1-1/+1
Actual Fluentd config doesn't manage multiline log pattern like: 2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [req-34ea3f23-f780-4059-b911-2788ea4af7b2 - - - - -] Option "firewall_driver" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. This log will generate 1 Fluentd records for each line (3 records), this is really a wrong behaviour, and made Kibana interface unusable. This patch add fluent-plugin-concat in order to concatenate multiline log separated in multiple events. Change-Id: I83e583605b49b7c421109d662de1291f38bd5c07 Closes-Bug: #1711399
generated by cgit v1.2.3 (git 2.39.1) at 2026-11-05 21:59:49 +0000