metwork-framework / resources Goto Github PK

For the moment, we only build a master branch (and integration branch which is kind of "pre-master"). We also build "PRs" before acceptance which is great.

Now, we need to prepare our build system to releases branches.

• modify the guess_version.sh script for release_* branches
• generate all integration-levels>=4 .drone.yml files with cookiecutter in resources repository
• write a maintenance script to create a branch from master for all integration-levels>=4 repositories
• introduce this new branch in all integration-levels>=4 .drone.yml files

Then:

• define the release policy (for release tags) in that branch
• test the released RPMs
• document

http://software.clapper.org/daemonize/

https://medium.com/mergify/announcing-the-new-mergify-engine-7bb2eceeb778

probably breaks existing branch protection

probably with a sum of processes filtered like it's done in kill_remaining_processes.py or something like that

depending on the integration-level of each repo, various configurations should be forced:

• labels
• branch protection
  [...]

• centos6
• centos7
• centos8
  • systemd support
• fedora29
  • systemd support
• fedora30
  • systemd support
• mageia x
  • find how to generate repo metadatas with genhdlist2
• opensuse y
  • test the yum repository with zypper

When this test is run, nothing is explicitely done to load the tested layer.
So the test can fail on a library which is provided by the layer itself or by one of his dependencies.
This usually does not happen because many layers are optionally loaded as dependencies of default@mfext (which is loaded when the test is run).
But we don't want to load by default every layers from every available addon.
So it would be a good thing to change the test to make sure that the tested layer is loaded and avoid false alarms on extra dependencies.

for example:

• we make a PR for mfserv
• on CI, we build/test the PR for the mfserv module

we should execute also integrations tests on the corresponding PR

warning: not an easy task (with drone)

New feature:

• When a new Priority: * label is set, old Priority: * labels are removed (if necessary)
• When a new Status: * label is set, old Status: * labels are removed (if necessary)

(see CONTRIBUTING.md guide)

ping @thebaptiste

so it's difficult, error-prone to rename a plugin:

• the directory name
• the name in [general] section
• the .layerapi2_label file content

this is also blocking the "plugins.swap" feature on mfserv

I think we want to keep the .layerapi2_label file content as a reference.

So we have to:

• drop name key in [general] section
• do changes to never use the directory name as a plugin name

• simpler and cleaner rewrite: opinionated_configparser (use envtpl)
• add opinionated_configparser in mfext (python3)
• add opinionated_configparser in mfext (python2)
• better __ini_to_env using it
• python2 version of opinionated_configparser
• change usage in mfext:
  • config.py
  • layers/layer9_python3_misc/0100_mfutil/mfutil/plugins.py
  • remove old parameter use_envtpl
• change usage in mfadmin
• change usage in mfsysmon
• change usage in mfbase:
  • adm/__make_nginx_conf
  • remove old parameter use_envtpl
• change usage in mfserv:
  • adm/__make_nginx_conf
  • adm/_plugins.is_dangerous
  • adm/_make_circus_conf
  • remove old parameter use_envtpl
• change usage in mfdata:
  • plugins/switch/main.py
  • layers/layer1_python3/0300_acquisition/acquisition/configargparse_confparser.py
  • layers/layer1_python3/0100_directory_observer/requirements.txt
  • layers/layer1_python3/0100_directory_observer/directory_observer/directory_observer.py
  • adm/_make_directory_observer_conf
  • adm/_make_switch_conf
  • adm/_make_circus_conf
  • adm/before_start_directory_observer
  • remove old parameter use_envtpl
• remove old __ini_to_env
• remove configparser_extended

from the private repository

https://github.com/thefab/mfsysmon

https://github.com/git-chglog/git-chglog

To take an example with mfserv:

• when a plugin is executed with circus, the code is run inside the plugin_env but we add the plugin_dir to PYTHONPATH (we can change this but this is the default configuration) and in some cases (not configured by default), we can also add app_dir to PYTHONPATH
• when a plugin is executed through plugin_wrapper (cron, continuous integration...), we don't add plugin_dir or app_dir to PYTHONPATH
• when a plugin is executed in an interactive way inside plugin_env, we don't add plugin_dir or app_dir to PYTHONPATH

We have to define a unique way to do all these things. The main entry points should be plugin_wrapper. We have to rewrite it to add some options. Then, plugin_env and _make_circus_conf should use it and not write a specific code.

• write a better plugin_wrapper with an option (true by default) to add current plugin dir to pythonpath and an option (false by default) to add an app dir to pythonpath
• change _make_circus_conf in mfdata, mfserv to use this new plugin_wrapper instead of custom code
• change plugin_env in mfext to use this new plugin_wrapper instead of custom code

And last:

• [x] change layerapi2 behaviour in mfext to add lib to PYTHONPATH by default
  (reverted change because of build problems)
• add lib to PYTHONPATH by default in plugin_wrapper

maybe we can introduce some helpers and warning to avoid this

first ideas :

• publish configuration generation time an env var
• refuse mfxxx.stop/mfxxx.start/mfxxx.status is the config.ini is newer that the current loaded environnement (metwork-framework/mfext#252)
• [ ] add a message in the prompt to say "hey ! the environnement was changed, restart this terminal"

see https://github.com/metwork-framework/mflog

Thanks to @thebaptiste last change about layer packaging names, we can now think about better "metwork metapackage names".

Now:

• we do yum install metwork-mfext, we install (nearly) all mfext layers and the symbolic link /opt/metwork-mfext => /opt/metwork-mfext-trunk (not really a big deal)
• (but) when we do yum install metwork-mfsysmon (because of metwork-mfsysmon => metwork-mfext dependency), we install (nearly) all mfext layers (for this module, we need only little things)

I think we can change this now to have (for example):

• metwork-mfext-minimal-{BRANCH}: install mfext minimal layers (root, core, default, python3_core, python3, python, default)
• metwork-mfext-minimal: install mfext minimal layers + symbolic link /opt/metwork-mfext => /opt/metwork-mfext-{BRANCH}
• metwork-mfext-full-{BRANCH}: install all mfext layers
• metwork-mfext-full: install all mfext layers + symbolic link /opt/metwork-mfext => /opt/metwork-mfext-{BRANCH}
• metwork-mfext: alias of metwork-mfext-full
• metwork-mfext-scientific-{BRANCH} and metwork-mfext-scientific (install scientific, python3_scientific)
• metwork-mfext-devtools-{BRANCH} and metwork-mfext-devtools (install python3_devtools, devtools, python3_devtools_jupyter, python3_scientific, scientific)

And keep only these meta-packages.

Of course, we keep all newly created packages: metwork-mfext-layer-*

For other modules, we have to describe better the root dependencies. For example, the metwork-mfsysmon package has a .layerapi2_dependencies with only root@mfcom. It should be:

• root@mfcom
• python3@mfcom
• monitoring@mfext
• python3_circus@mfext

And it should be automatically read in _metwork.spec file.

So when we would do: yum install metwork-mfsysmon, it would install:

• metwork-mfcom-layer-root
• metwork-mfcom-layer-python3
• metwork-mfext-layer-monitoring
• metwork-mfext-layer-python3_circus

and (because of dependencies):

• metwork-mfext-layer-python3 (because of metwork-mfcom-layer-python3)
• metwork-mfext-layer-root (because of all metwork-mfext-layer-* packages)
• metwork-mfext-layer-python3_core (because of metwork-mfext-layer-python3)

metwork-mfext-layer-default and metwork-mfext-layer-python are missing in this idea. Maybe, we have to change something about them.

An idea would be:

• we rename default into default@mfext (in mfext) and we change its dependencies to limit them to mfext components
• we create a default@mfcom layer with mfcom components (FIXME: what about python dependency => same idea) and default@mfext
• we rename default into default@mfadmin (in mfadmin) with mfadmin components and default@mfcom
• we change the main profile to load default@mfext and default@mfcom (if relevant) and default@${MODULE_LOWERCASE} (if relevant) instead of default
• we create (if not existing) a default@${MODULE_LOWERCASE} layer for all modules with ${MODULE_LOWERCASE} components and default@mfcom as dependencies :
• mfdata
• mfserv
• mfbase
• mfsysmon

Do the same idea for python layer (we want to drop it later but not now) :

• we rename python into python@mfext (in mfext) and we change its dependencies to limit them to mfext components
• we create a python@mfcom layer with mfcom components and python@mfext
• we create (if not existing) a python@${MODULE_LOWERCASE} layer in each module having python2 or python3 layers with ${MODULE_LOWERCASE} python{METWORK_PYTHON_MODE} components and python@mfcom as dependencies :
• mfdata
• mfserv
• mfbase
• mfadmin
• in mfext, under wrappers, we modify wrappers using python layer and Makefile :
  (python ==> python@mfext,-python@mfcom,-python@${MODULE_LOWERCASE})
• replace dynamically {METWORK_PYTHON_MODE} during rpm dependencies check (with the 3 hardcoded value)

.drone.yml and maybe other files

• MODULE
• MODULE_HOME
• MODULE_LOWERCASE
• MODULE_RUNTIME_SUFFIX
• MODULE_RUNTIME_HOME
• MODULE_RUNTIME_USER
• MODULE_VERSION

maybe we should prefix them to get:

• MFMODULE
• MFMODULE_HOME
• [...]

Because of metwork-framework/mfserv#281 and metwork-framework/mfext#571, we propose a big log refactoring for metwork.

Changes:

• develop https://github.com/metwork-framework/log_proxy
• add log_proxy to mfext in layer0_core
• introduce a template system for modules config.ini
• migration mfserv config.ini in new templated system: config.ini.custom
• migration mfdata config.ini in new templated system: config.ini.custom
• migration mfbase config.ini in new templated system: config.ini.custom
• migration mfsysmon config.ini in new templated system: config.ini.custom
• migration mfadmin config.ini in new templated system: config.ini.custom
• provide a log_proxy global configuration for all modules
  • number of files to keep
  • maximum size of files
  • maximum age of files
  • (for plugins) split stdout/stderr in different files (yes/no)
  • (for plugins) split multiple workers in different files (yes/no)
• provide a plugin configuration for overriding global configuration for:
  • split stdout/stderr in different files (yes/no)
  • split multiple workers in different files (yes/no)
• replace std_redirect in all circus workers by corresponding tool in log_proxy
  • in mfext templates
  • in mfserv
  • in  mfdata
  • in mfbase
  • in mfsysmon
  • in mfadmin
• add log_proxy usage in crontab for mfserv
• add log_proxy usage in crontab for mfdata
• add log_proxy usage in crontab for mfbase
• add log_proxy usage in crontab for mfsysmon
• add log_proxy usage in crontab for mfadmin
• deal with nginx: https://stackoverflow.com/questions/22541333/have-nginx-access-log-and-error-log-log-to-stdout-and-stderr-of-master-process
  • stdout/stderr for nginx (in mfserv)
  • custom log maximum size in config (in mfserv)
  • stdout/stderr for nginx (in mfbase)
  • stdout/stderr for nginx (in mfdata)
• stop to use logrotate copytruncate feature
• add https://github.com/metwork-framework/jsonsyslog2elasticsearch in mfext
• remove https://github.com/metwork-framework/jsonlog2elasticsearch from mfext
• deprecate https://github.com/metwork-framework/jsonlog2elasticsearch
• changes in mflog to replace json_logs.log feature with syslog to jsonsyslog2elasticsearch
• new syslog configuration for mflog in common profile
• same change for nodejs lib (work started in metwork-framework/mfserv#309)

... since my commits with a very long comment

the current implementation of conf_monitor is written in bash and is cpu consumming
(because we build all configuration files at each iteration)

we should rewrite it better and in python

• better conf_monitor in mfdata
• better conf_monitor in mfserv
• better nginx reload (by using direct signals because of circusctl seems to be unreliable)
• better circus reload (by using direct signals because of circusctl seems to be unreliable)

between "pr ready to merge" status, github branch protections and mergifybot

like the number of workers for example

• set in the common profile (if MFMODULE != "MFEXT"): export LOGPROXY_LOG_DIRECTORY="${MFMODULE_RUNTIME_HOME}/log"
• change all log_proxy usages to remove the absolute log path in:
  • mfext (templates)
  • mfserv (including plugin templates)
  • mfbase (including plugin templates)
  • mfdata (including plugin templates)
  • mfadmin
  • mfsysmon

• It should not be allowed to install a single layer of a module (this is meaningless)
• Some rpms are aliases of others. We can suppress most (and maybe all) of these rpms by using the key word "Provides" in spec file
• When uninstalling all rpm of a metwork module, there are often warnings because the rpm layer-root-*, which removes all metwork files, is sometimes uninstalled before other rpms, destroying the files of these other rpms before they are uninstalled

All these points can be corrected by evolutions of _metwork.spec from mfext module
The first and third points should be corrected as follows :

• all single layer rpms from a module must have the meta rpm "minimal" in his dependencies and must not have the "layer root" rpm in his dependencies
• the rpm "minimal" must have the "layer root" rpm in his dependencies
  Doing so, the "layer root" rpm should be the last rpm uninstalled and the installation of a single layer rpm should at least install the "minimal" module

• generic build
• systemd support
• documentation
• CI tests