flyingcircusio / batou_ext Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 2.0 545 KB

A library of batou extensions.

License: Other

Python 92.18% PHP 2.48% Shell 1.83% C 0.47% Nix 3.05%

batou_ext's People

Contributors

Stargazers

Watchers

Forkers

frlan sweh

batou_ext's Issues

fcio.Provision: Error if service user is missing

With missing service user in environment-section I'm getting this traceback

 % ./batou appenv-run fcio provision production
Traceback (most recent call last):
  File "/my/project/.batou/567b4dd6/bin/fcio", line 33, in <module>
    sys.exit(load_entry_point('batou-ext==0.1.dev0', 'console_scripts', 'fcio')())
  File "/my/project/.batou/567b4dd6/lib/python3.9/site-packages/batou_ext/fcio.py", line 315, in main
    return args.func(**func_args)
  File "/my/project/.batou/567b4dd6/lib/python3.9/site-packages/batou_ext/fcio.py", line 309, in <lambda>
    p.set_defaults(func=lambda **kw: Provision(**kw).apply())
  File "/my/project/.batou/567b4dd6/lib/python3.9/site-packages/batou_ext/fcio.py", line 261, in apply
    pprint(self.api.apply(calls))
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 1116, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 1455, in __request
    request = dumps(params, methodname, encoding=self.__encoding,
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 975, in dumps
    data = m.dumps(params)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 508, in dumps
    dump(v, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 530, in __dump
    f(self, value, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 583, in dump_array
    dump(v, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 530, in __dump
    f(self, value, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 601, in dump_struct
    dump(v, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 530, in __dump
    f(self, value, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 534, in dump_nil
    raise TypeError("cannot marshal None unless allow_none is enabled")
TypeError: cannot marshal None unless allow_none is enabled

SymlinkAndCleanup - make cleanup asynchronous and

At least one customer has been bitten by the fact that cleanups may take a very long time.

The cleanup IMHO should spawn a systemd-run job for the rm -rf and limit IOPS to something like 100 iops/s to ensure the cleanup job doesn't overload the system.

batou_ext.cron.SystemdTimer: Allow to supply multiple onCalendar expressions

Systemd allows to supply timers with multiple values for OnCalendar. This will trigger the service every time any timer matches. The SystemdTimer class does, however, only supports supplying a single string value.

NixOS seems to be able to handle multiple timer units. Local testing suggests that supplying a list should be able to generate a valid systemd timer:

systemd.timers.test_timer = {
    # …
    timerConfig.OnCalendar = ["*-*-01 01:00:00" "*-*-2 02:00:00"];
}

# …

[Timer]
OnCalendar=*-*-01 01:00:00
OnCalendar=*-*-02 02:00:00

An alternative would be to create multiple timers/services doing the same thing. But as each timer
is marked as Persistent, my understanding is that such jobs could be triggered multiple times after a system failure.

batou_ext.rabbitmq.User: Check and update password if needed

batou_ext.rabbitmq.User should check, whether a given password for a user is set and set in case of not.

mail.Mailhog: Change on configuration does not restart service

When changing configuration of mailhog -- e.g. to make usage of feature introduced with 5432201 -- the service is not restarted automaticity

nixos.NixOSModule: Build failure is not -reissues on re-run

If NixOSModule runs into a build/switch problem on target VM, it is not retried on the next run – not correctly applied changes might go out of view (from deployment point of view)

nix.rebuild - defer rebuild to the end

Could be useful deferred to the end, @dpausp experiences half failed builds sometimes

Cronjob name clash not detected

Given two cron jobs in the same component:

        self += batou_ext.cron.CronJob(
            "maintenance",
            command="command a", …)
        self += batou_ext.cron.CronJob(
            "maintenance",
            command="command b', …)

They both have the same name, hence the script will be overwritten. I think this should be prevented somehow.

Starting up services configured with `batou.lib.service.Service` via `systemctl start` may not be necessary on NixOS

To be precise, I'm talking about the following code-snippet:

https://github.com/flyingcircusio/batou_ext/blob/2.4.2/src/batou_ext/nix.py#L298-L305

I'm not sure if this is actually needed:

each service generated like this has a wantedBy = [ "multi-user.target" ];:

https://github.com/flyingcircusio/fc-nixos/blob/11b6b49aaaa04145496a208cf23d1d7515dab0b2/nixos/platform/systemd.nix#L74-L78

inactive services that are wanted by the multi-user.target are started on switch-to-configuration.pl: See https://nixos.org/manual/nixos/stable/#sec-unit-handling (multi-user.target gets started on a switch which starts up inactive units).
a rebuild causing a switch is done on the nixos platform implementation: https://github.com/flyingcircusio/batou_ext/blob/2.4.2/src/batou_ext/nix.py#L290

The reason why I'm bringing this up is a practical one: in some cases you want to deploy services without starting them (which is effectively forced by the current implementation).

An example is a bunch of queue consumers that must not be started before a certain component is finished. What I tried was to temporarily mask the services during a deployment, however then systemctl is-active returns false causing the component to attempt starting up the unit which fails because the unit got masked before.

For context, see also FC-31563.

cc @zagy

s3.Bucket: Show bucket name that was ment to be created

If you are defining a lot of buckets or there is some issue with one of the buckets to be enssured, that adding the information about bucket name to output would be nice.

Postgres.Extension: Predict with missing database not working

Code:
self += batou_ext.postgres.Extension('postgis', db=self.dbname)

Results in:

xxx00 > DBBackend > Extension('postgis')
ERROR: sudo -u postgres psql -d dbname -qtAX -c "SELECT extname FROM pg_extension WHERE extname = 'postgis';"
Return code: 2
STDOUT

STDERR
psql: FATAL:  database "dbname" does not exist

if database 'dbname' does not yet exists (e.g. on bootstrapping)

`HTPasswd` requires external library `passlib`

The batou_ext.htpasswd module imports passlib.hash.sha512_crypt.

Unfortunately:

passlib is not an install requirement in setup.py
It fails with the newest version from pypi: Exception: No module named 'passlib.hash.sha512_crypt'; 'passlib.hash' is not a package

Importing passlib.hash instead of passlib.hash.sha512_crypt does the trick.

config.CustomizeYaml: Unsupported usage of yaml.load()

A deployment using CustomizeYaml giving this warning:

/deployment/.batou/3537682d/lib/python3.7/site-packages/batou_ext/config.py:96: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  data = yaml.load(f)

SSHKeyPair: issues with existing keys on a machine

On one VM we had two files in ~/.ssh: ~/.ssh/id_ed25519 & ~/.ssh/id_ed25519.pub.
I wanted to deploy an SSH keypair to said machine and basically did

[component:sshkeypair]
id_ed25519 = <redacted>

Now, ~/.ssh/id_ed25519 had the correct private key, but ~/.ssh/id_ed25519.pub had an old public key.

Interestingly, this lead to SSH failing with "Permission denied (public key)" while cloning a git repository. The issue could be fixed by updating the .pub file, though I'm pretty sure that removing it would've solved the issue as well.

This is counter-intuitive and should be changed IMHO. I'd suggest that when updating the private key, the corresponding pubkey should be removed or always being derived from the private key (i.e. ssh-keygen -y -f ~/.ssh/id_ed25519). To me, this seems like a more sensible default than what we currently have, so I'd explicitly suggest to make this the default given how SSH behaves if pubkey and private key don't match.

Jenkins version update: allow unicode encoding

batou_ext/src/batou_ext/jenkins.py

Line 30 in 7eac817

stdout = stdout.decode("ascii")

In cases where the remote branch name includes a unicode symbol (e.g. german umlauts) this line raises an unexpected error:

Traceback (most recent call last):
  File "<redacted>/./.appenv/9ac02331/bin/jenkins", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "<redacted>/.appenv/9ac02331/lib/python3.11/site-packages/batou_ext/jenkins.py", line 137, in main
    return args.func(**func_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/.appenv/9ac02331/lib/python3.11/site-packages/batou_ext/jenkins.py", line 104, in set_versions
    vu()
  File "<redacted>/.appenv/9ac02331/lib/python3.11/site-packages/batou_ext/jenkins.py", line 51, in __call__
    self.update(service, version)
  File "<redacted>/.appenv/9ac02331/lib/python3.11/site-packages/batou_ext/jenkins.py", line 65, in update
    func(service, version, args)
  File "<redacted>/.appenv/9ac02331/lib/python3.11/site-packages/batou_ext/jenkins.py", line 68, in update_git
    resolved = git_resolve(self.config.get(service, "url"), version)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<redacted>/.appenv/9ac02331/lib/python3.11/site-packages/batou_ext/jenkins.py", line 30, in git_resolve
    stdout = stdout.decode("ascii")
             ^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 85: ordinal not in range(128)

RabbitMQ: ErlangCookie cannot be updated

When changing content of .erlang.cookie btou cannot write the file:

    with open(self.path, "wb") as target:
PermissionError: [Errno 13] Permission denied: '/srv/xxx/.erlang.cookie'

.erlang.cookie is created with -r--------

RabbitMQ: ErlangCookie cannot handle prediction mode

If there is no .erlang.cookie-file already created, the ErlangCookie crases

ERROR: Traceback (most recent call last):
  File "/my.batou/.appenv/80e88ae5/lib/python3.10/site-packages/batou/remote_core.py", line 402, in <module>
  File "/my.batou/.appenv/80e88ae5/lib/python3.10/site-packages/batou/remote_core.py", line 366, in deploy
  File "/my.batou/.appenv/80e88ae5/lib/python3.10/site-packages/batou/remote_core.py", line 178, in deploy
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou/component.py", line 327, in deploy
    sub_component.deploy(predict_only)
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou/component.py", line 339, in deploy
    call_with_optional_args(
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou/utils.py", line 524, in call_with_optional_args
    return func(**call_kw)
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou_ext/rabbitmq.py", line 43, in verify
    with open(self.path, 'r') as target:
FileNotFoundError: [Errno 2] No such file or directory: '/remote/.erlang.cookie'

S3: Better error handling when not able to connect to s3 gateway

Something like this is not helpful:

oks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/srv/s-gitlab/deployment/.appenv/e2e91147/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/srv/s-gitlab/deployment/.appenv/e2e91147/lib/python3.10/site-packages/botocore/handlers.py", line 279, in validate_bucket_name
    if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
TypeError: expected string or bytes-like object

nix.PurgePackage: Add documentation

There is no documentation for PurgePackage available.

ssh.SSHKeyPair: Deletes local SSH keys when using local as deployment target

        # ED25519
        if self.id_ed25519:
            self += File(
                "~/.ssh/id_ed25519",
                content="{}\n".format(self.id_ed25519),
                mode=0o600,
                sensitive_data=True,
            )

Mailhog: Allow Mailhog to store mails outside of the docker container for persitency

With PR #65 we introduced the option to change the general storage engine for Mailhog.
When using maildir, it should be possible to store mails outside the docker container to keep them, even the container is destroyed. However, this will need also a strategy to clean this folder up at some point.

git.GitCheckout: Allow to use clobber=True

In most cases we can assume that the git checkout used for git.GitCheckout should be clean. Therefore setting clobber=True got batou.lib.git.Clone() might prevent unwanted surprises.

ssh.SSHKeyPair: Allow to set name of key

Currently the component can only either deploy id_ed25519(.pub) or id_rsa(.pub). For deploying multiple SSH-keys it would be nice if this could be configurable (optional)

fcio.Provision: Roles are not sorted correctly -> changes not correct detected

When running fcio provision --diff <environment> the list of assigned roles is in some cases not correct sorted. This is causing a message like this:

myvm00
    classes             : ['role::docker', 'role::external_net', 'role::webgateway'] → ['role::docker', 'role::webgateway', 'role::external_net']

Even though there is actually no role update needed.

batou_ext.nix.UserEnv: Add a way to deinstall a UserEnv

Installing a package like this

        self += batou_ext.nix.UserEnv(
                "django",
                packages=[
                    "gcc",
                     … 
                    "yarn",
                    "zip",
                ],
                shellInit="# additional shell init")

will cause a package like django-1.bb2a973726635812db18475991c37f382495943257f0d77faaf80f0f4a7cce4b can be found via nix-env -q. To ensure it's deinstalled later you have to use the exact name at batou_ext.nix.PurgePackage().

It would be nice if there would be an easier way to do so.

redis.Redis: Allow to configure providing name

Usecase:

Using multiple instances of Redis inside a project

batou.ssl.Certificate: Ensure alternative_names are given as tuple (not a string only)

When using ssl.Certifcate() to get an Lets Encrypt certificate it might happen alternative_names is not set correctly to a tuple of strings (list of domains).

Given, that at this example self.alternative_name is a string with one FQDN only …

Good call:

    self.cert = batou_ext.ssl.Certificate(
          ....
          alternative_names=(self.alternative_name,),
          ....
      )

Wrong call:

    self.cert = batou_ext.ssl.Certificate(
          ....
          alternative_names=(self.alternative_name),
          ....
      )

As the default of alternative_names is an empty tupel, this error might only occur on specific configurations (like e.g. production deployment) during runtime of the deployment itself.

We should ensure that alternative_names is a really a tuple of strings.

document transition from self signed to valid certificate

After deploying a new site for the first time, a self signed cert is created.

Please document

that a self signed cert is created in the first place (yes, it is documented in the source code )
how to manually trigger the generation of the LE cert (ie goto deployment/work/nginx/.../ and execute the bash script)
document when - without manual intervention - a new cert is created ( -> cronjob )

geo.GeoIPDatabase: Make download and extracting step more rebust against failures

With current implementation it might happen that a broken download is corrupting active used one

oci.Container: Add explicit monitoring of container

I think it would be handy to allow the component to set up a dedicated service check for the health of the container.

fcio: Allow to set a RG to maintenance for xx time

Could be something like

./appenv run fcio set-maintenance --duration 1h production

Ideas for improvements of the NixOS module components

This issue gathers a few ideas I had in mind about batou_ext.nixos.NixOSModule. We talked about the location topic already, but I'd like to have these written down somewhere.

Location of `.nix` files

The actual Nix expression is now in /srv/s-service/deployment/component rather than /etc/local/nixos. This is not only unexpected (until now one could be sure that everything from AppOps is in /etc/local/nixos), it's also harder to grep for certain patterns in the NixOS configuration on the live system.

JSON vs Nix

I'm wondering if we should rather transform the entire tree into JSON rather than Nix:

Python already has support for (pretty)printing JSON which would save us a lot of code to maintain.
- It's probably sufficient to write custom encoders for e.g. Host, Address, etc..
Nix has a builtins.fromJSON and I'd expect it to run quicker than the Nix evaluator given that the JSON parser is pure C++ and far simpler than the one for Nix.

`component` vs `_module.args`

The attributes from a component are handed in like this:

{ pkgs, lib, config # from config._module.args
, component # from the Nix file importing this specific file
, ...
}:

The problem I see here is that the arguments in the attribute-set that each module function gets is the value of the option _module.args, we however inject a custom value and bypass the module system which seems unintuitive for people being used to write NixOS modules. Also, we potentially shadow values in _module.args because of //. Generally I'm not sure if we're hitting issues eventually with that now that we leave the path of what's supported and what the module system itself expects.

A common pattern is to define a tree with custom values in config, i.e. config.mydeployment.<...>. Perhaps we want to do something similar here? I.e. config.component.nginx.listen_port where nginx is a component for instance.

oci.Container: User story for one shoots

In some cases we only want to run a one-shoot with a given container -- e.g. for a database migration or a converting job of an pdf/image. It would be nice if this can be done without defining 2 times registry, container name, env-variables etc.

RabbitMQ: Permissions fails in prediction mode without propper user

ERROR: rabbitmqctl -q list_user_permissions myuser
    Return code: 69
STDOUT

STDERR
Error:
{:no_such_user, "myuser"}

batou_ext.cron.CronJob: Explicit log if an timeout occurs

It would be nice to make it more visible whether a CronJob run into a timeout.

From timout manpage:

If the command times out, and --preserve-status is not set, then
exit with status 124.  Otherwise, exit with the status of
COMMAND.  If no signal is specified, send the TERM signal upon
timeout.  The TERM signal kills any process that does not block
or catch that signal.  It may be necessary to use the KILL (9)
signal, since this signal cannot be caught, in which case the
exit status is 128+9 rather than 124.

Consider a better versioning schema and using actual releases.

The upgrade to batou 2.3 is a bit confusing/painful with respect to batou_ext.

I think we should consider adding real releases and a proper versioning scheme.

I suggest that we:

use the same minor version as batou when releasing for specific batou versions e.g. batou_ext 2.3.X is compatible with batou 2.3.X
make a batou dependency so that for batou_ext==x.y.z it requires batou<=x.y+1

Does that sound reasonable?

fcio.Provision: Allow to provision VM inside different RG

When having a parent RG, it's possible to use the same API credentials to provision VM in child rg.
The provision method should be capable of handling such a scenario

ScanHost.update breaks as ssh-keyscan changed its API

ssh-keyscan now returns 1 (previously 0) when no keys are found

cf openssh/openssh-portable@c2c18a3

This broke my build on Travis when I updated to focal (Ubuntu 20.04).

Ubuntu 18.04

jugmac00@odin:~$ ssh-keyscan -6 github.com
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
jugmac00@odin:~$ echo $?
0

Ubuntu 20.04

jugmac00@heimdall:~$ ssh-keyscan -6 github.com
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
jugmac00@heimdall:~$ echo $?
1

P.S.: This is certainly no Ubuntu issue, I just used those distros as they provide ssh-keyscan versions before and after the above linked API change.

PostgreSQL password must not contain dollar sign

Setting the password for a PostgreSQL user is done via subprocess.Popen with the parameter shell=True. So the call is using a shell which interprets the dollar sign as signal of a requested variable interpolation.

Example:

Using abc$efg as the password for a postgres user
leads to the following command issued: sudo -u postgres psql -d postgres -c "ALTER ROLE \\"username\\" WITH ENCRYPTED PASSWORD 'abc$efg' "
What actually sent to PostgreSQL is: ALTER ROLE "username" WITH ENCRYPTED PASSWORD 'abc' (as there is no variable named efg in the environment.)

This leads to passwords which can be way shorter than expected.
When using in combination with a .pgpass file no log-in is possible because the passwords stored in .pgpass probably does not match with the password stored in PostgreSQL.

Document meaning of `checkWarning` and `checkCritical` of the CronJob component

The current docstring is not really helpful for non-involved users of the library.

s3.Download: Print URL tried for downloading in case of an failure

in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
ERROR: Unexpected exception

is not very helpful when debugging an error.

nixos.NixOSModules: Mark generated_context.nix as sensitve

This file might container passwords we shall not see diffs for in Jenkins etc.

git.Checkout(): Don't run SSH Scanhost when git url is https://

Was cloning a repo from github via it's https://-address:

testvm > Backend > GitCheckout > ScanHost('github.com') > File('/srv/test/.ssh') > Directory('.ssh')
testvm > Backend > GitCheckout > ScanHost('github.com') > File('/srv/test/.ssh') > Mode('.ssh')
testvm > Backend > GitCheckout > ScanHost('github.com')

I think we should not perform the ScanHost-part if there is a https-URL present

fcio.Provision: --diff is handling wrong VM name not correct

In cases of there is a typo in host: the -–diff subcommand is not handling the error correct.

Here I had I typo within VM name so i didn't belonged to RG anymore.

% ./batou appenv-run fcio provision –diff staging                              
Running unclean installation from requirements.txt
Ensuring unclean install ...
Traceback (most recent call last):
  File ".batou/unclean/bin/fcio", line 11, in <module>
    load_entry_point('batou-ext==0.1.dev0', 'console_scripts', 'fcio')()
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 309, in main
    return args.func(**func_args)
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 303, in <lambda>
    p.set_defaults(func=lambda **kw: Provision(**kw).apply())
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 230, in apply
    diff = self.get_diff(self.get_currently_provisioned_vms(), vms)
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 173, in get_currently_provisioned_vms
    return self.api.query('virtualmachine')
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1170, in single_request
    return self.parse_response(resp)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1342, in parse_response
    return u.close()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32500: 'application error'>

SSL `CertificateCheck` broken with Batou >= 2.1.

flyingcircusio/batou#65 prevents from setting unknown attributes to components.

When enabling a certificate check on a let's encrypt certificate this leads to the instantiation of CertificateCheck which calls ServiceCheck with a name kw-argument which is not defined on Service. This breaks now a customer deployment.

batou_ext/src/batou_ext/ssl.py

Lines 245 to 254 in c13a461

 self += batou.lib.nagios.ServiceCheck( 

 self.expand( 

 'https://{{component.public_name}} certificate valid?'), 

 name=self.expand('ssl_cert_{{component.public_name}}'), 

 command='check_http', 

 args=self.expand( 

 '-H {{component.public_name}} ' 

 '-p {{component.port}} ' 

 '-S --sni ' 

 '-C {{component.warning_days}},{{component.critical_days}}'))

fcio.Provison: Failing when a host is not having componets assigned

Scenario:
One want to provision VM without actually assigning components at the very moment (e.g. on initial bootstrapping of project)

When not having a component-section at all:

 ./appenv run fcio provision --diff staging
Traceback (most recent call last):
  File "mybatou/./.appenv/6e86ce17/bin/fcio", line 33, in <module>
    sys.exit(load_entry_point('batou-ext==0.1.dev0', 'console_scripts', 'fcio')())
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 315, in main
    return args.func(**func_args)
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 309, in <lambda>
    p.set_defaults(func=lambda **kw: Provision(**kw).apply())
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 181, in apply
    self.environment_ = self.load_env()
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 156, in load_env
    environment.load()
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 187, in load
    self.load_hosts(config)
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 289, in load_hosts
    self._load_hosts_multi_section(config)
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 325, in _load_hosts_multi_section
    config[section].as_list("components"))
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 44, in as_list
    result = self[option]
KeyError: 'components'

If the section is empty:

./appenv run fcio provision --diff staging
ERROR: Missing component
 Component: 
      Host: myhost01
ERROR: Missing component
 Component: 
      Host: myhost02

Prediction for not existing checkout impossible

Bug: You are not able to test a new GitCheckout, without deploying it.

Reproduction: Add a new batou_ext.git.GitCheckout to the deployment and test it with the -P prediction option. The deployment prediction fails, because of a (correctly) missing folder.

Expectation: A prediction doesn't fail, because the component could not change the file system.

Traceback:

host > CRS > GitCheckout > Directory('work/crs/prepared-0bd51ff806c68e2a54c4d60ca13f731c5355696d') > SyncDirectory('prepared-0bd51ff806c68e2a54c4d60ca13f731c5355696d')
ERROR: rsync -rclnv --exclude '.git' /srv/user/deployment/work/crs/checkout/ /srv/user/deployment/work/crs/prepared-0bd51ff806c68e2a54c4d60ca13f731c5355696d
    Return code: 23
STDOUT
sending incremental file list

sent 19 bytes  received 12 bytes  62.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

STDERR
rsync: [sender] change_dir "/srv/user/deployment/work/crs/checkout" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]


ERROR: Unexpected exception
        Traceback (most recent call last):
        File "/Deployment/.appenv/2a69e431/lib/python3.11/site-packages/batou/deploy.py", line 371, in main
    getattr(deployment, step)()
        File "/Deployment/.appenv/2a69e431/lib/python3.11/site-packages/batou/deploy.py", line 317, in deploy
    self.loop.run_until_complete(asyncio.gather(*pending))
        File "/User/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
        File "/Deployment/.appenv/2a69e431/lib/python3.11/site-packages/batou/deploy.py", line 272, in _deploy_component
    await self.loop.run_in_executor(
        File "/User/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Deployment/.appenv/2a69e431/lib/python3.11/site-packages/batou/host.py", line 196, in deploy_component
    self.rpc.deploy(component, predict_only)
        File "/Deployment/.appenv/2a69e431/lib/python3.11/site-packages/batou/host.py", line 97, in call
    raise RuntimeError(
        RuntimeError: host.domain: Remote exception encountered.
======================================================================= DEPLOYMENT PREDICTION FAILED (during deploy) ========================================================================

fcio.Provision: Implicit creation of service user is not setting its description

Setting a description for a service user is now enforced. The implicit creation via provisioning is not setting it.

redis.Redis: Support customizable provide name

Currently the name within the components provides itself is hard coded.

    def configure(self):
        assert self.password
        self.provide("redis", self)

I think either a boolean switch to turn off providing for providing it via inheritance or allow to define the strng would be a good idea.

fcio.Provision: Cannot provision VM without assigned components

% .batou/bin/fcio provision –diff production
 ERROR: Missing component
       Component:
            Host: myhost10

VM configuration:

 [host:myhost10]
 data-cores = 1
 data-ram = 4
 data-disk = 60
 data-description = backend
 components =

	self += batou.lib.nagios.ServiceCheck(
	self.expand(
	'https://{{component.public_name}} certificate valid?'),
	name=self.expand('ssl_cert_{{component.public_name}}'),
	command='check_http',
	args=self.expand(
	'-H {{component.public_name}} '
	'-p {{component.port}} '
	'-S --sni '
	'-C {{component.warning_days}},{{component.critical_days}}'))

flyingcircusio / batou_ext Goto Github PK

batou_ext's People

Contributors

Stargazers

Watchers

Forkers

batou_ext's Issues

Location of .nix files

JSON vs Nix

component vs _module.args

Recommend Projects

Recommend Topics

Recommend Org

Jobs

Location of `.nix` files

`component` vs `_module.args`