garywiz / chaperone

Lightweight process-tree manager for Docker-like containers

Chaperone

Chaperone is a lean init-style startup manager for Docker-like containers. It runs as a single lightweight full-featured process which runs at the root of a docker container tree and provides all of the following functionality, plus much more:

• Monitoring for all processes in the container, automatically shutting down the container when the last process exits.
• A complete, configurable syslog facility built in and provided on /dev/log so daemons and other services can have output captured. Configurable to handle log-file rotation, duplication to stdout/stderr, and full Linux logging facility, severity support. No syslog daemon is required in your container.
• The ability to start up system services in dependency order, with options for per-service environment variables, restart options, and stdout/stderr capture either to the log service or stdout.
• A built-in cron scheduling service.
• Emulation of systemd notifications (sd_notify) so services can post ready and status notifications to chaperone.
• Process monitoring and zombie elimination, along with organized system shutdown to assure all daemons shut-down gracefully.
• The ability to have an optional controlling process, specified on the docker command line, to simplify creating containers which have development mode vs. production mode.
• Complete configuration using a chaperone.d directory which can be located in various places, and even allows different configurations within the container, triggered based upon which user is selected at start-up.
• Default behavior designed out-of-the-box to work with simple Docker containers for quick start-up for lean containers.
• More...

If you want to try it out quickly, the best place to start is on the chaperone-docker repository page. There is a quick section called "Try it out" that uses images available now on Docker Hub.

For full details of features and usage: see the documentation.

There is some debate about whether docker containers should be transformed into complete systems (so-called "fat containers"). However, it is clear that many containers contain one or more services to provide a single "composite feature", but that such containers need a special, more streamlined approach to managing a number of common daemons.

Chaperone is the best answer I've come up with so far, and was inspired by The Phusion baseimage-docker approach. However, unlike the Phusion image, it does not require adding daemons for logging, system services (such as runit). Chaperone is designed to be self-contained.

Status

Chaperone is now stable and ready for production. If you are currently starting up your container services with Bash scripts, Chaperone is probably a much better choice.

Full status is now part of the documentation.

Downloading and Installing

The easiest way to install `chaperoneis usingpip`` from the https://pypi.python.org/pypi/chaperone package:

# Ubuntu or debian prerequisites...
apt-get install python3-pip

# chaperone installation (may be all you need)
pip3 install chaperone

License

Copyright (c) 2015, Gary J. Wisniewski [email protected]

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

chaperone's Issues

Incompatibility with PyYAML 6.0

Hello

chaperone fails with

Configuration Error: load() missing 1 required positional argument: 'Loader'

To fix, I forced the version PyYAML == 5.2

Regards

Project status

Is this project dead?

Shutdown/cleanup script

Is there a way to make chaperone run a script with cleanup stuff before it quits? My use case is: I have a vpn service running in a docker container that sets some firewall rules and I want to make sure those get cleared when the container/chaperone stops (for whatever reason), something like the functionality provided by SysV run levels... Thanks!

pid changes?

haproxy supports lightweight restarts which generate a new pid and take over the work of the previous process.

In this case, the active PID changes, but I've seen chaperone not updating the active pid when given a pidfile.

(sometimes it seems to honor the change, othertimes not)

Any clue on a timer what chaperone looks at the pid file?

build failure due to undocumented dependency

On both ubuntu:14.04 and debian:jessie, installing chaperone requires both python3-pip and also the build-essential and python3-dev packages which python3-pip recommends, but does not require.

As a result, this fails:

from ubuntu:14.04

ENV DEBIAN_FRONTEND noninteractive
ENV DEBIAN_PRIORITY critical
ENV DEBCONF_NOWARNINGS yes

RUN apt-get update \
    && apt-get upgrade -y

RUN apt-get install --no-install-recommends -y \
    python3-pip

RUN pip3 install chaperone

In the interests of building small containers, using apt-get --no-install-recommends -y is typically a good idea, but here it means that the other two packages need to be explicitly installed (or apt-get install -y python3-pip needs to be in a separate RUN statement to other packages that will be installed, which uses an extra filesystem layer for little advantage).

Reasonable people could argue, but consider stating the dependency in the README instructions. And if not, then at least this issue might point out the problem for someone.

Python 2 support

Would you be interested in accepting PR with Python 2 support?

Task was destroyed but it is pending!

After sending mysqld a SIGTERM singal to shut it down,
chaperone detects this but an error message occurs.

mysqld.conf for mariadb/mysql official Dockerhub image:

# mysqld.conf

settings: {
  env_set: {
    "MYSQL_PID_FILE":  "/run/mysqld/mysqld.pid"
  },
  env_inherit: [
    "MYSQL_ROOT_PASSWORD",
    "MYSQL_DATABASE",
    "MYSQL_USER",
    "MYSQL_PASSWORD",
    "MYSQL_ALLOW_EMPTY_PASSWORD",
    "MYSQL_RANDOM_ROOT_PASSWORD",
    "MYSQL_ONETIME_PASSWORD"
  ]
}

mysql1.service: {
  type: forking,
  command: "mysqld",
  enabled: false,
  uid: mysql,
  gid: mysql,
  service_groups: database,
  pidfile: "$(MYSQL_PID_FILE)",
}

# setup script starts also mysqld
mysql.service: {
  type: simple,
  command: "/docker-entrypoint.sh mysqld",
    # Dockerfile entrypoint + default cmd
  uid: root,
  gid: root,
  pidfile: "$(MYSQL_PID_FILE)",
  process_timeout: 60,
  restart: true,
  enabled: true,
  stderr: "log",
  service_groups: database,
}

$ docker exec test killall -s SIGTERM mysqld

Docker log:

Dec  6 12:33:13 e1588f1e7257 mysql[152]: 2016-12-06 12:33:13 140467036891072 [Note] mysqld (mysqld 10.1.19-MariaDB-1~jessie) starting as process 9 ...
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: REAP pid=9,status=0
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: no child processes present
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: mysql.service exit status for pid=9 is '<ProcStatus exit_status=0>'
Dec  6 12:35:00 e1588f1e7257 chaperone[1]: Final termination phase.
Task was destroyed but it is pending!
task: <Task pending coro=<_monitor_service() running at /usr/local/lib/python3.4/dist-packages/chaperone/cproc/pt/simple.py:29> wait_for=<Future pending cb=[Task._wakeup()]> cb=[SubProcess.add_pending.<locals>.<lambda>() at /usr/local/lib/python3.4/dist-packages/chaperone/cproc/subproc.py:631]>

The mysql process should have been restarted by chaperone,
but it terminated completely and the Docker container stopped.

One failed cron job execution stops any further execution

If only one cron fail than all further execution are ignored:

2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: REAP pid=49,status=256
2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: REAP pid=0,c=0
2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: executor.service exit status for pid=49 is '<ProcStatus exit_status=1>'
2016-12-30 16:30:20 | Dec 30 16:30:20 fdffabd9114a chaperone[1]: executor.service terminated abnormally with <ProcStatus exit_status=1>
2016-12-30 16:31:00 | Dec 30 16:31:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:31:00 | Dec 30 16:31:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.
2016-12-30 16:32:00 | Dec 30 16:32:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:32:00 | Dec 30 16:32:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.
2016-12-30 16:33:00 | Dec 30 16:33:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:33:00 | Dec 30 16:33:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.
2016-12-30 16:34:00 | Dec 30 16:34:00 fdffabd9114a chaperone[1]: cron service executor.service running CMD ( /executor.sh )
2016-12-30 16:34:00 | Dec 30 16:34:00 fdffabd9114a chaperone[1]: service executor.service already started.  further starts ignored.

Transfer ownership

@garywiz would you be willing to transfer ownership of this repo to me, or create an organization for it and add some more people as maintainers? Would be nice to maintain it without having a separate fork.

Allow to specify udp port in syslog_host

Hi,
i want to use chaperone to relay syslog messages to logstash (that not running as root, so can't bind to port < 1024).
So can you allow to specify the port where the syslog messages are sent in the syslog_hostoption.
Ex:

syslog_host: "logstash:1514"

Thanks

exitkills doesn't pass the error code

It would be useful if the error code of the process is passed as the error code of chaperone if exitkills is enabled for a process.

chaperone/chaperone/cproc/subproc.py

Lines 560 to 572 in 9ff2c3a

	def process_exit(self, code):
	self.returncode = code
	if self._exit_event:
	self._exit_event.set()
	self._exit_event = None
	if self.exit_kills:
	self.logwarn("{0} terminated with exit_kills enabled", self.service.name);
	# Since we're dead, and the system is going away, disable any process management
	self._proc = None
	self.pid = None
	self._kill_system();

[Question] Is this project actively maintained?

I'm looking at supervisord alternatives and stumbled upon Chaperone. It looks great and is exactly what I need.

However, there has been no activity after 2016. Not to sound rude, but I plan to use this on production, and would like to know if the community is interested in actively maintaining this project.

How should I call a runtime startup script?

It's common practice in docker containers to have some script that gets run at run time to do things like take values from environment values and construct the odd config file with them before the main daemon process(es) get run. This commonly gets shoe-horned into the CMD or ENTRYPOINT directive.

It seems in keeping with the philosophy of chaperone that this should probably hang off the chaperone config, and also I suspect not doing it that way would mean that the wrong process would be PID 1, and responsible for zombies. I'm still just finding my way here, but I'm not seeing a really elegant way to do it with chaperone. What am I missing?

Maybe I could create a one-shot service, and set up some before/after rules. Is there a better way?

add docs for telchap

telchat is hinted at, but no detailed docs on telchap that I could find.

(outside telchap help)

Chaperone with cron job got crazy on DST change

If cron job is configured and DST change occurs Chaperone starts executing job in loop regardless cron configuration.

Log:

Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: Switching all chaperone logging to /dev/log
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: chaperone version 0.3.9, ready.
start
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service scheduled using interval spec '* * * * *'
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: periodic_long_running.service attempting start '/bin/echo EXECUTION'...
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: REAP pid=8,status=0
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: no child processes present
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: cron service periodic_long_running.service running CMD ( echo 'EXECUTION' )
Oct 29 02:58:50 d3a4d1f36a3a chaperone[1]: service periodic_long_running.service enabled, queueing start request

Test case:
chaperone-dst-bug.tar.gz

To run test case. Execute attached archive and run ./run_test.sh. Script is using sudo command to change system datetime.

Disabling time sync may be required. (Note that Virtual Box Guest additions synchronizes your time with host clock)

Env:
Ubuntu 14.04

Allow to add the PRI field in front of logging message

Hi,

actually we can use the extendedoption to prefixes every output syslog line with the facility and priority.
The problem is that it can be difficult and time consuming to parse this fields to convert them to numeric levels.
So can you add an option to prefix every output syslog line with the "standard" PRI field (surround by <> like defined in RFC3164)
Ex:

<13> Jun 15 02:09:33 su [27]: pam_unix(su:session): session opened for user root by (uid=1000)

Thanks

Python 3.7 compatibility: asyncio.async and async keyword

Chaperone version 0.3.9 does not work with Python 3.7+ because it uses asyncio.async and async is now a keyword.

type forking and exit_fills true

Hi,

i want to start a service that fork and want to kill the system if the service (which have a pid) die.
For example for my postfix service i use:

postfix.service: {
  type: forking,
  service_groups: IDLE,
  command: "postfix start",
  pidfile: /var/spool/postfix/pid/master.pid,
  after: "postgrey.service",
  exit_kills: true,
}

The problem is that the postfix start command start the postfix master process but return when it has fork the master process. So chaperone see that the postfix start end and kill the whole system (because of the exit_kills with value true) even if the pid is valid and the forking process is running.

EDIT the logs i have:

May 16 19:39:07 7f454733d6e4 chaperone[1]: system will be killed when '/usr/sbin/postfix' exits
May 16 19:39:07 7f454733d6e4 postfix-script[321]: starting the Postfix mail system
May 16 19:39:07 7f454733d6e4 master[323]: daemon started -- version 3.0.4, configuration /etc/postfix
May 16 19:39:07 7f454733d6e4 chaperone[1]: REAP pid=257,status=0
May 16 19:39:07 7f454733d6e4 chaperone[1]: REAP pid=0,status=0
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service exit status for pid=257 is '<ProcStatus exit_status=0>'
May 16 19:39:07 7f454733d6e4 chaperone[1]: Request made to kill system.
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service waiting for PID file: /var/spool/postfix/pid/master.pid
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service successfully started
May 16 19:39:07 7f454733d6e4 chaperone[1]: postfix.service notified waiters upon completion
May 16 19:39:15 7f454733d6e4 master[323]: terminating on signal 15
May 16 19:39:15 7f454733d6e4 chaperone[1]: no child processes present
May 16 19:39:15 7f454733d6e4 chaperone[1]: Final termination phase.

Syslog implementation does remove utf-8 characters

Scenario:

• syslog functionality enabled by default
• service writes utf-8 characters to stdout/stderr, for example "request processed in 100 µs"
• chaperone/Python does remove the utf-8 charcater => "request processed in 100 s"

Config:

settings: {
  env_set: {
    'LANG': 'en_US.UTF-8',
    'LC_CTYPE': '$(LANG)'
  }
}

test.service: {
  command: '/display-utf8.sh',
}

console.logging: {
  enabled: true,
  stdout: true
}

display-utf8.sh:

#!/bin/bash

echo "++++TEST++++"
echo "request processed in 100 µs"
echo "++++/TEST++++"

Config validation support?

Hi!
does chaperone support validation of config and exit accordingly? My use case would be to test chaperone config at container build time and fail container build if something is wrong, without actually starting chaperone.

thanks!

Clarification: console, syslog

Does console encompass all logs, including from syslog?

console.logging: {
  file: '/var/log/chaperone-syslog.log'
}

Would it be also possible to enable or disable logging to stdout globally?
For debugging, it is helpful but in production it would unnecessarily strain the server.

Edit: Is the stdout/stderr of cron type services logged, too?
Or will this be logged to a default cron log file (/var/log/cron)?

Trailing commas at end of block

The chaperone documentation examples use trailing commas for entries that come last in a block:

nginx.service: {
  [...]
  env_set: {
    NGINX_PID_FILE: "$(NGINX_PID_FILE:-/run/nginx.pid)"
  },
  service_group: server,
}

However, this results in an error when chaperone starts:

Configuration Error: extra keys not allowed @ data['nginx.service']['service_group']

Removing the traililng comma for the last entry in block resolves the issue and chaperone starts correctly:

[...]
  service_group: server
}

chaperone python3.2 issues :(

hi,

benn trying to install chaperone with python3.2 (debian7), it clearly fails :(
error logs:

    no previously-included directories found matching 'documentation/_build'
Installing collected packages: chaperone, docopt, PyYAML, voluptuous, aiocron, setproctitle, croniter, trollius, python-dateutil, six
  Running setup.py install for chaperone
      File "/usr/local/lib/python3.2/dist-packages/chaperone/cutil/servers.py", line 46
        self.server = yield from self._create_server()
                               ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cutil/syslog_handlers.py", line 130
        (transport, protocol) = yield from connect
                                         ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cutil/notify.py", line 49
        yield from self.run()
                 ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/commands.py", line 45
        result = yield from self.do_exec(opts, protocol.owner.controller)
                          ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/subproc.py", line 24
        data = yield from stream.readline()
                        ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/process_manager.py", line 182
        yield from asyncio.sleep(0.1)
                 ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/pt/forking.py", line 11
        result = yield from self.timed_wait(self.process_timeout, self._exit_timeout)
                          ^
    SyntaxError: invalid syntax

      File "/usr/local/lib/python3.2/dist-packages/chaperone/cproc/pt/cron.py", line 74
        yield from super().start()
                 ^
    SyntaxError: invalid syntax

this seems to be related to chaned appeared in python3.3:
https://docs.python.org/3/whatsnew/3.3.html

by any chance, is there any way to make chaperone compatible with python3.2 ?

thanks anyway for this awesome piece of work :)