zhouyangyu / toil_container

:whale: A python package with container support for Toil pipelines.

Toil Container

A python package with container support for Toil pipelines.

Check the example! This package was built to support the cookiecutter-toil repository.

Features

• 📦   Easy Installation

    pip install toil_container
  

• 🐳   Container System Calls

  docker_call and singularity_call are functions that run containerized commands with the same calling signature. Be default the exit code is returned, however you can get the stdout with check_output=True. You can also set the env, cwd, volumes and working_dir for the container call. working_dir is used as the /tmp directory inside the container.

  from toil_container import docker_call
  from toil_container import singularity_call
  
  cmd = ["cowsay", "hello world"]
  status = docker_call("docker/whalesay", cmd)
  output = docker_call("docker/whalesay", cmd, check_output=True)
  
  status = singularity_call("docker://docker/whalesay", cmd)
  output = singularity_call("docker://docker/whalesay", cmd, check_output=True)

• 🛳   Container Job Class

  ContainerJob is a Toil Job Class with a call method that executes commands with either Docker, Singularity or Subprocess depending on image availability. Check out this simple whalesay example! The Job must be constructed with an options argument of the type argparse.Namespace that may have the following attributes:

  attribute	action	description
  options.docker	use docker	name or path to image
  options.singularity	use singularity	name or path to image
  options.workDir	set as container /tmp	path to work directory
  options.volumes	volumes to be mounted	list of src, dst tuples

• 📘   Container Parser With Short Toil Options

  ContainerArgumentParser adds the --docker, --singularity and --volumes arguments to the options namespace. This parser only prints the required toil arguments when using --help. However, the full list of toil rocketry is printed with --help-toil. If you don't need the container options but want to use --help-toil use ToilShortArgumentParser.

   whalesay.py --help
  
       usage: whalesay [-h] [-v] [--help-toil] [TOIL OPTIONAL ARGS] jobStore
  
        optional arguments:
        -h, --help            show this help message and exit
        --help-toil           print help with full list of Toil arguments and exit
  
        container arguments:
        --docker              name/path of the docker image available in daemon
        --singularity         name/path of the singularity image available in deamon
        --volumes             tuples of (local path, absolute container path)
  
        toil arguments:
        TOIL OPTIONAL ARGS    see --help-toil for a full list of toil parameters
        jobStore              the location of the job store for the workflow [REQUIRED]
  

Usage

whalesay.py is an example that runs a toil pipeline with the famous whalesay docker container. The pipeline can now be executed with either docker, singularity or subprocess.

# whalesay.py
from toil_container import ContainerJob
from toil_container import ContainerArgumentParser


class WhaleSayJob(ContainerJob):

    def run(self, fileStore):
        """Run `cowsay` with Docker, Singularity or Subprocess."""
        msg = self.call(["cowsay", self.options.msg], check_output=True)
        fileStore.logToMaster(msg)


def main():
    parser = ContainerArgumentParser()
    parser.add_argument("-m", "--msg", default="Hello from the ocean!")
    options = parser.parse_args()
    job = WhaleSayJob(options=options)
    ContainerJob.Runner.startToil(job, options)


if __name__ == "__main__":
    main()

Then run:

# run with docker
whalesay.py jobstore -m 'hello world' --docker docker/whalesay

# run with singularity
whalesay.py jobstore -m 'hello world' --singularity docker://docker/whalesay

# if cowsay is available in the environment
whalesay.py jobstore -m 'hello world'

If you want to convert a docker image into a singularity image instead of using the docker:// prefix, check docker2singularity, and use -m '/shared-fs-path /shared-fs-path' to make sure your shared file system is mounted inside the singularity image.

Contributing

Contributions are welcome, and they are greatly appreciated, check our contributing guidelines! Make sure you add your name to the contributors list:

• 🐋   Juan S. Medina @jsmedmar
• 🐴   Juan E. Arango @juanesarango
• 🐒   Max F. Levine @mflevine
• 🐼   Joe Zhou @zhouyangyu

Credits

• This repo was inspired by toil's implementation of a Docker Call and toil_vg interface of Singularity Calls.
• This package was initiated with Cookiecutter and the audreyr/cookiecutter-pypackage project template.