merll / docker-map

Managing Docker images, containers, and their dependencies in Python.

Docker-Map

Managing Docker images, containers, and their dependencies in Python.

Project: https://github.com/merll/docker-map

Docs: https://docker-map.readthedocs.io/en/latest/

Overview

This package provides tools for building Docker images, create containers, connect dependent resources, and run them in development as well as production environments.

The library builds on functionality of the Docker Remote API client for Python, docker-py. Its main target is to reduce the repetitive and error-prone code that is required for creating and connecting containers in a non-trivial stack. It can be used standalone for custom orchestration or for enhancing available deployment / remote execution utilities (see Docker-Fabric, Salt Container-Map).

Containers and their dependencies are configured object-based, through Python dictionaries, or YAML files.

Building images

Writing Dockerfiles is not hard. However, it only allows for using variable context to a limited extent. For example, you may want to re-define directory paths in your project, without having to adjust it in multiple places; or you keep frequently reoccurring tasks (e.g. creating system user accounts) in your Dockerfile, and would like to use templates rather than copy & paste.

Dockerfiles

A DockerFile object generates a Dockerfile, that can either be saved locally or sent off to Docker through the remote API. Supports common commands such as addfile (ADD) or run, but also formats CMD and ENTRYPOINT appropriately for running a shell or exec command.

Docker file context

DockerContext generates a Docker context tarball, that can be sent to the remote API. Its main purpose is to add files from DockerFile automatically, so that the Dockerfile and the context tarball are consistent.

Creating, connecting, and running containers

This package reduces repetitions of names and paths in API commands, by introducing the following main features:

• Automatically create, configure, and assign shared volumes.
• Automatically update containers if their shared volumes are inconsistent, their image, or their configuration has been updated.
• Use alias names instead of paths to bind host volumes to container shares.
• Automatically create and start containers when their dependent containers are started.

Container configuration

ContainerConfiguration objects keep the elements of a configured container. Their main elements are:

• image: Docker image to base the container on (default is identical to container name).
• clients: Optional list of clients to run the identical container configuration on.
• instances: Can generate multiple instances of a container with varying host mappings; by default there is one main instance of each container.
• shares: Volumes that are simply shared by the container, only for the purpose of keeping data separate from the container instance, or for linking the entire container to another.
• binds: Host volume mappings. Uses alias names instead of directory paths.
• uses: Can be names of other containers, or volumes shared by another volume through attaches. Has the same effect as the volumes_from argument in the API, but using alias names and automatically resolving these to paths.
• links: For container linking. Container names are translated to instance name on the map.
• attaches: Generates a separate container for the purpose of sharing data with another one, assigns file system permissions as set in permissions and user. This makes configuration of sockets very easy.
• exposes: Configures port bindings for linked containers and on host interfaces.
• exec_commands: Launches commands on containers after they have been created and started.
• create_options and host_config provide the possibility to add further keyword arguments such as command or entrypoint, which are passed through to the docker-py client.

Container maps

ContainerMap objects contain three sets of elements:

1. Container names, each associated with a ContainerConfiguration.
2. Volumes, mapping shared directory paths to alias names.
3. Host shares, mapping host directory paths to alias names.

Clients, as defined in a ContainerConfiguration, can also be set globally on map level.

ContainerConfiguration instances and their elements can be created and used in a dictionary-like or attribute syntax, e.g. container_map.containers.container_name.uses or container_map.containers['container_name']['uses']. Volume aliases are stored in container_map.volumes and host binds in container_map.host; they support the same syntax variations as containers.

Client configuration

ClientConfiguration objects allow for a host-specific management of parameters, such as service URL and timeout. For example, the interfaces property translates the exposes setting for a configuration on each host into a port binding argument with the local address.

Combining the elements

MappingDockerClient applies one or multiple ContainerMap instances to one or multiple Docker clients. A container on the map can easily be created with all its dependencies by running client.create('container_name').

Running the container can be as easy as client.start('container_name') or can be enhanced with custom parameters such as client.start('container_name', expose={80: 80}).

If all configuration is stored on the map, creation and start are combined in client.startup('container_name').

docker-map's Issues

Validate Names

Due to a change in docker, container names can no longer include an underscore. A legacy name with an underscore leads to an invalid hostname (merll/docker-fabric#14) error that is hard to trace down. I suggest an assertion in the constructor for maps/containers (and/or their configs) that validates names and raises an informative exception.

Errors when using a client not based on docker-py

Problem occurs in Docker-Fabric, but is caused by a check if the client is a subclass of docker.client.Client. This check should probably be replaced with something more flexible in order to support non-API clients such as the CLI.

python 3.x support

Traceback (most recent call last):
  File "test_docker.py", line 3, in <module>
    from dockermap.build.dockerfile import DockerFile
  File "/venv/lib/python3.2/site-packages/dockermap/build/dockerfile.py", line 10, in <module>
    from .buffer import DockerStringBuffer
  File "/venv/lib/python3.2/site-packages/dockermap/build/buffer.py", line 5, in <module>
    import cStringIO
ImportError: No module named cStringIO

swallows errors during build_from_file

For example

(Pdb) 
> /Users/dylanjay/Projects/download-cache/eggs/docker_map-0.3.0-py2.7.egg/dockermap/map/base.py(59)_docker_log_stream()
-> for e in response:
(Pdb) output
{u'errorDetail': {u'message': u'Cannot create container with more than 127 parents'}, u'error': u'Cannot create container with more than 127 parents'}
(Pdb) 

CLI Response Issues

Unlike the API, the sudo CLI sometimes returns spurious data. For example, my LDAP password is about to expire so I'm actually seeing:

[172.24.6.11] sudo: docker version --format="{{json .}}"
[172.24.6.11] out: sudo password:
[172.24.6.11] out: Your password will expire in 4 day(s).
[172.24.6.11] out: {"Client":{"Platform":{"Name":""}, ...

Presumably, other notifications could be sent this way as well. Perhaps all CLI parsing code (e.g. dockermap/client/cli.py", line 347, in parse_version_output) should be wrapped with a decorator that catches errors and writes the full output to the terminal/logging. I only discovered the issue by manually suppressing the quiet kwarg on the docker version call. If I wasn't familiar with docker-map, this would have been an extremely difficult issue to diagnose.

Option to run docker commands using sudo

The discussion started on docker-fabric. TL;DR

• This article argues that sudo docker (with a very restrictive sudoers line and an alias to docker) is a better security practice than adding users to the docker group.
• This approach also makes it easy to manage docker execution privileges from LDAP since sudoers supports LDAP groups.
• @merll indicated that the key constraint is the way docker-map interacts with docker -- using the docker-py API client through socat. docker-map would need to issue shell commands instead.

Since these commands would need to be issued by docker-map, I'm moving the ticket here.

Docker Networking Support

According to the documentation on links, the feature has been deprecated in favor of a new networking option. Have you looked at this new feature to determine whether (and how) it's supported by docker-map (and docker-fabric)?

Please provide imports necessary in docs

It took me 15 minutes to figure out which parts of dockermap to import to re-create your examples in the introduction. Could you add the correct import statements so that getting started is easier?

No way to set environment variables on the container

container configuration like

'db': { # Configure container creation and startup
'image': 'postgres:9.4.4',
'exposes': {
5432: 5432
},
'binds': ['dbdata', "dbinit"],
'Env': {
"PGDATA": "/dbdata"
}
},

would need the PGDATA environment variable set in the container that is started but after some debugging i realized that the vvariable was being completely ignored. i tried changing it to environment too but that was being ignored too.

Hyphen in container names

Got confused and thought I had reported it in a different issue when it was actually new. I ran into a linking error and it turns out that it was related to #15. I changed the container name to eliminate a hyphen the problem resolved itself. If you go back to the same thread referenced in #15, it looks like there's a toxic interaction between EVN variables (no hyphen) and network names (no underscore) that's working its way through the docker consciousness.

I'm not sure if there's a local fix -- except raise a useful exception -- until they get the conflict sorted out internally.

CLI args formating

This is a (sanitized, and wrapped for readability) create command generated by client.cli.get_cmd:

 docker create 
--name="graylog_map.graylog" 
--links="(u'graylog_map.mongo', 'mongo')" 
--links="(u'graylog_map.elasticsearch', 'elasticsearch')" 
--hostname="graylog_map.graylog" 
--env="GRAYLOG_PASSWORD_SECRET=<pepper>" 
--env="GRAYLOG_ROOT_PASSWORD_SHA2=<password>" 
--env="GRAYLOG_REST_TRANSPORT_URI=https://<url>:12900" 
--volume="/data/graylog/data:/usr/share/graylog/data/journal:rw" 
--volume="/data/graylog/config:/usr/share/graylog/data/config:rw" 
graylog2/server:2.0.3-2

Here are the issues (and resolutions) I've identified so far:

• links is actually link
  • in client.cli.KWARGS_MAP add 'links': 'link'
• the value for link is not collapsed correctly and should be <name>:<name>
  • The code block below resolves a couple issues: recursive calls for e.g. a list of tuples and cases where the kw (i.e. the outer flags) are None

def _transform_kwargs(ka):
    if ka is None:
        yield
    for key, value in iteritems(ka):
        cmd_arg = KWARG_MAP.get(key, key.replace('_', '-'))
        if value is None:
            continue
        for response in _transform_kwarg(cmd_arg, value):
            yield response


def _transform_kwarg(cmd_arg, value):
    if isinstance(value, list):
        for vi in value:
            for response in _transform_kwarg(cmd_arg, vi):
                yield response
    elif isinstance(value, dict):
        for ki, vi in iteritems(value):
            yield _mapping_format(cmd_arg, ki, vi)
    elif isinstance(value, tuple):
        yield _mapping_format(cmd_arg, value[0], value[1])
    elif isinstance(value, bool):
        yield _arg_format(cmd_arg, 'true' if value else 'false')
    else:
        yield _quoted_arg_format(cmd_arg, value)

• graylog_map.graylog is not a valid hostname. As mentioned here valid hostnames need to be sanitized to [a-zA-Z0-9\-]+ or left out. Replacing non-alphanumeric characters with hyphens could be a valid option. For my testing, I used a code block to skip it.

Wait timeout for starting containers

In some cases containers fail to start, if they link to containers that are still in the process of starting up. One occurrence is described in a Docker-Fabric issue. An optional startup delay should be added to the container configuration.

CLI: multiple values for keyword argument 'cmd' when using 'exec_commands'

I'm trying to run a build script using exec_commands. I'm getting the following error:

Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/data/apex-server/build_deploy/src/fabfile.py", line 612, in app_setup
    apex.workflow_add(config=config, tag=tag)
  File "/data/apex-server/build_deploy/src/helpers/apex.py", line 66, in workflow_add
    workflow_containers_add(config, tag)
  File "/data/apex-server/build_deploy/src/helpers/apex.py", line 402, in workflow_containers_add
    docker.startup_required_containers(docker_map)
  File "/data/apex-server/build_deploy/src/helpers/docker.py", line 125, in startup_required_containers
    container_fabric_inst.startup(container)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 224, in startup
    return self.run_actions('startup', container, instances=instances, map_name=map_name, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 110, in run_actions
    results.extend(runner.run_actions(*actions))
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/__init__.py", line 94, in run_actions
    res = c_method(action_config, container_name, **action.extra_data)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 54, in exec_container_commands
    self.exec_commands(config, c_name, run_cmds=config_cmds)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 37, in exec_commands
    e_id = client.exec_create(**ec_kwargs)['Id']
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/cli.py", line 122, in exec_create
    cmd_str = self._out.get_cmd('exec_create', *args, **kwargs)
TypeError: get_cmd() got multiple values for keyword argument 'cmd'

I've seen something similar before and assumed it was something wrong with my legacy code. This time I seem to be using 100% internal functionality.

CLI: exec_commands issue

Assume this is another issue related to #16

...
  File "/data/apex-server/build_deploy/src/helpers/docker.py", line 125, in startup_required_containers
    container_fabric_inst.startup(container)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 224, in startup
    return self.run_actions('startup', container, instances=instances, map_name=map_name, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 110, in run_actions
    results.extend(runner.run_actions(*actions))
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/__init__.py", line 94, in run_actions
    res = c_method(action_config, container_name, **action.extra_data)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 54, in exec_container_commands
    self.exec_commands(config, c_name, run_cmds=config_cmds)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/runner/cmd.py", line 37, in exec_commands
    e_id = client.exec_create(**ec_kwargs)['Id']
TypeError: 'NoneType' object has no attribute '__getitem__'

... and for reference, this is how we're calling it:

    # because we don't add users to the docker group, we need to use_sudo
    docker_client = docker_fabric(use_sudo=True)
    # we inject the map directly rather than use env
    container_fabric_inst = container_fabric(docker_client=docker_client, container_maps=docker_map)
    for container, config in docker_map.containers:
        container_fabric_inst.startup(container)