user-cont / colin Goto Github PK

We should:

• update the readme
• make it pretty and compact

Hi,

Fedora container guidelines say: "run or usage | Either provides an Atomic run line, or a human readable example of container execution"

Tests enforce both at the same time.

• configurable via CLI
• enable user to select output format (doesn't mean you should implement all of them)
  • machine readable (e.g. json)
  • human readable
    • short (provide design first, this needs brainstorming)
    • verbose (provide design first, this needs brainstorming)

Update documentation before we can ship it:
Ideas:

• Provide lists of checks for ecosystem
• Show how it can be used as python module

Please add a few more sentences to readme so that people know what this is:

• Mention that our initial plan is to validate containers against different ecosystems (Red Hat Container Catalogue, Fedora Infra (and container guidelines), CentOS (?), Atomic Container Best Practices) and clearly state our priorities.
• Describe the technical details, architecture and planned features a little bit.
• TODO list.

This check means, that after package installation
yum/dnf clean all is called.

We have to go through layers and find whether dnf/yum install foobar and dnf/yum clean all is called in the same layer.
Input for check is an image.
Output of the check is True, whether dnf clean all is called in the same layer
and False in opposite case.

Ladies and gentlemen, I'm pretty sure you know this, but the time has come and we need as much tests for colin as possible. Ideally, reuse things from conu as much as possible.

• unit tests
• integration tests (+ #56 )
• functional/acceptance tests -- we should verify that when colin is invoked via CLI, it checks correctly
• release tests - make sure that whatever we have in PyPI can be installed and works well

In case config files does not exist, colin should not failed with traceback,
but with error message like: no config file is specified.

UPDATED

$ colin -c fedora fedora:27
Traceback (most recent call last):
  File "/usr/local/bin/colin", line 11, in <module>
    load_entry_point('colin==0.0.1', 'console_scripts', 'colin')()
  File "/usr/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/colin/cli/colin.py", line 14, in cli
    config_name=config)
  File "/usr/local/lib/python3.6/site-packages/colin/core/colin.py", line 22, in run
    target_type=target.target_type)
  File "/usr/local/lib/python3.6/site-packages/colin/core/config/config.py", line 38, in get_checks
    severity=severity)
  File "/usr/local/lib/python3.6/site-packages/colin/core/config/config.py", line 99, in _get_check_files
    for g in self._get_check_groups(group):
  File "/usr/local/lib/python3.6/site-packages/colin/core/config/config.py", line 80, in _get_check_groups
    groups = [g for g in self.config_dict]
AttributeError: 'Config' object has no attribute 'config_dict'

Do not report missing 'io.k8s.description' if 'description' is present. If one exists, the build system copies it to the other. The same applies for the other way, missing description and present io.k8s.description. No need to have both in Dockerfiles.

Red Hat and Fedora container images are built using atomic-reactor, and while the 'architecture' label is required to be present in the built images, in fact it is added automatically by atomic-reactor.

In other words, the Dockerfile in the git repository serves as input to atomic-reactor, which adds a set of labels before building. 'architecture' is one of those labels.

I'd suggest removing the requirement to have 'architecture' present in the Dockerfile for Red Hat and Fedora container images.

• colin should check whether an image, or container has a help file
  • the name can be help.1 or README.md
  • in the past it also could have been help.md

https://fedoraproject.org/wiki/Container:Guidelines#Help_File

#75 (comment)

Figure out a way of loading checks from all dirs.

Users should be able to point colin to additional directories where checks could live.

#79 was initial issues.
The command was:
$ colin check --json fedora.json -r fedora fedora:27

The code is here:

$ colin check --json fedora.json -r fedora fedora:27
LABELS:
nok:failed:maintainer_label_required
   -> Label 'maintainer' has to be specified.
   -> The name and email of the maintainer (usually the submitter).
   -> https://fedoraproject.org/wiki/Container:Guidelines#LABELS
[..snip..]
nok:warning:io.openshift.expose-services_label
   -> Label 'io.openshift.expose-services' has to be specified.
   -> port:service pairs separated with comma, e.g. "8080:http,8443:https"
   -> ?????
Failed to mount OverlayFS device.
mount: only root can use "--options" option

16:38:17.688 colin.py          ERROR  An error occurred: ColinException("Problem with mounting filesystem with atomic. (Command '['atomic', 'mount', 'fedora:27', '/tmp/conuwzyyvd3g']' returned non-zero exit status 1.)",)
Error: Problem with mounting filesystem with atomic. (Command '['atomic', 'mount', 'fedora:27', '/tmp/conuwzyyvd3g']' returned non-zero exit status 1.)

@TomasTomecek proposed to provide the image content not via mount, instead using docker create -name c $image; docker export c | tar -xf) so that we don't need root.

This is something I deeply care about that why I'm such a PITA about it.

After I tried colin recently, I was overwhelmed by the output: there was too much info for me and it wasn't easy to read. One suggestion I have here is to color only lines such as nok:failed:bzcomponent_deprecated and not the description.

The other thing I'm thinking about is the line itself; would it be better to have something like this?

failed   Deprecated label BZComponent is used.

Since we have almighty @jpopelka on our team now, I have an idea: Jirko, would you be so kind and played with colin a bit a tell us your opinion about the output and your user experience?

Franto, I understand if you are now like "what the funk is this guy's problem", but as I said, I would love if colin had an awesome output which suits everyone and the tool is such a pleasure to be used. I think we are heading there, I just want to make sure we get there. Happy Friday!

Contributors should know, how to contribute and add new tests.

Let's start thinking about tests. Initially, just copy the filesystem structure for testing (and make targets) from conu.

This is coming from an existing test: #94 (diff)

Ideally colin would be able to accept ruleset config directly, without the need of creating files in filesystem.

I propose to use commands:

$ colin check
$ colin list-checks
$ colin list-configurations

How do we name configurations? Targets maybe?

TODO:

• use yaml.safe_load to deserialize ruleset file
• update dependencies
• update documentation

We can leave present rulesets in json.

If there is a label defined in base image and the dockerfile does not overwrite it, the resulting image may contain incorrect metadata.

This is a check only for dockerfiles.

List of labels:
summary
description
io.k8s.description (same as description)
io.k8s.display-name
io.openshift.tags
com.redhat.component
name
version

I fear that this check may be false positive in some scenarios (when the layered image contains only minor change which does not invalidate the metadata).

Probably the most comprehensive logic to parse image names is here: https://github.com/projectatomic/atomic-reactor/blob/master/atomic_reactor/util.py#L57

Rulesets can easily live outside of colin. One of the requirements to marry them happily afterwards is to check whether ruleset is being ran against the right version of colin, so...

• add version field inside ruleset
• colin should check whether its version is good enough for the selected ruleset

Colin was executed with
/usr/bin/colin -c redhat registry.access.redhat.com/rhscl/mariadb-101-rhel7 --json artifacts/colin.json

colin version: 0.0.2

Colin traceback from ansible playbook:

  "Traceback (most recent call last):", 
        "  File \"/usr/bin/colin\", line 11, in <module>", 
        "    load_entry_point('colin==0.0.2', 'console_scripts', 'colin')()", 
        "  File \"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py\", line 480, in load_entry_point", 
        "    return get_distribution(dist).load_entry_point(group, name)", 
        "  File \"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py\", line 2693, in load_entry_point", 
        "    return ep.load()", 
        "  File \"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py\", line 2324, in load", 
        "    return self.resolve()", 
        "  File \"/usr/lib/python2.7/site-packages/pkg_resources/__init__.py\", line 2330, in resolve", 
        "    module = __import__(self.module_name, fromlist=['__name__'], level=0)", 
        "  File \"/usr/lib/python2.7/site-packages/colin/cli/colin.py\", line 22, in <module>", 
        "    from colin.checks.abstract.abstract_check import AbstractCheck", 
        "ImportError: No module named checks.abstract.abstract_check"

we should have a dedicated set of images for integration testing of colin

• placed in tests/, e.g. tests/data/
• defined by dockerfiles
• built as part of integration testing process

In my world, the best tools provide excellent verbose/debugging output. Let's make colin the best tool.

This should be able to be specified via CLI. It should also pick logs from conu.

Should these be 2 modes? (e.g. -v, -vv?)

• migrate all dockerfile checks from MTF over here

When the argument is specified do:

• show debug logs
• print full tracebacks #35
• pick debug logs from conu

We need to provide the solution allowing:

• Have one definition for one label (description, links, ..)
• Allow using the same definition for the image, container and dockerfile
• Allow specifying target type for the check in ruleset JSON.

Related to #78

When we are in changing of the ruleset file structure, what about some metadata/versioning.
(Or move check groups from the root level to be able to extend the scheme.)

I just installed colin on Fedora 28 system. Every colin command fails for me with The 'enum34' distribution was not found and is required by conu.

$ colin -h
Traceback (most recent call last):
  File "/usr/bin/colin", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3101, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 574, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 892, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 778, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'enum34' distribution was not found and is required by conu

However, I already have python2-enum34-1.1.6-4.fc28.noarch installed. Installing enum34 using below command fixes it:

$ pip3 install --user enum34

Version of various packages on my Fedora 28 system:

$ rpm -qa | grep colin
colin-0.0.4-3.fc28.noarch
python3-colin-0.0.4-3.fc28.noarch

$ which colin
/usr/bin/colin

$ rpm -qa | grep conu
python3-conu-0.3.1-1.fc28.noarch

$ rpm -qa | grep docker
docker-1.13.1-51.git4032bd5.fc28.x86_64
python3-docker-3.2.1-1.fc28.noarch
docker-common-1.13.1-51.git4032bd5.fc28.x86_64
python3-dockerfile-parse-0.0.10-1.fc28.noarch
docker-rhel-push-plugin-1.13.1-51.git4032bd5.fc28.x86_64
python3-docker-pycreds-0.2.2-2.fc28.noarch

Should this be documented somewhere? Or am I doing something wrong?

This is a detailed plan for GSoC project -- Fedora Dockerfile linter leveraging Container Best Practices.

• Fedora Container Guidelines support
  • First thing to implement is to add new checks inside colin which verify that the provided container image fulfils Fedora Container Guidelines
  • Ideally, the tool would implemented everything but it's possible that some rules might be hard to automate.
  • The output from colin should be easy to ready and meant to be pasted inside bugzilla when a review for container image is requested.
  • When working on this part, please open a new issue where we, as upstream maintainers, discuss:
    • List of rules to implement.
    • Implementation details.
  • Once the design is approved, feel free to continue with implementation.
• Project Atomic Container Best Practices support
  • Select at least 5 important rules from the best practices and implement automated checks inside colin for those (implement 5 more as a stretch goal).
  • The checks should be diverse, please discuss your plans with us first.
• Fedora deployment
  • Once colin is running smoothly locally and is able to check selected container images, we can start preparing for integration inside Fedora Infrastructure.
  • Open an issue at selected Fedora trackers (Fedora-infra, Fedora QE) what we want to do and how can we achieve that.
  • Once the infrastructure team approves our plans, we can realize them.
• Report
  • Great work needs marketing.
  • After running it in production, write a report on fedora-devel (and other mailing lists) on what colin found in container images built inside Fedora infrastructure with a summary.

Let's say we a have a MaintainerCheck. It would make things easier if we were able to create a ruleset where we would be able to override arguments, such as reference_url inside this ruleset. This will help us defining internal ruleset pointing to internal resources.

I fear that we might need to restructure the config quite a bit in order to enable this. Or maybe not, how about something like:

{
  "labels": {
    "required": [
      ...
        ]
      }
    ],
    "optional": [
      ...
    ],
    "extra-args": {
      "check-name": {
        "foo": "bar"
      }
    }
},

Each check has filename and name property in the implemented class.

There are two questions:

• Which one should be in the config file?
• If the filename has to be same as the name property?

There is a need to correctly load the config files.

In conu 0.4.0 we are planning to introduce a breaking change in .get_metadata method. We should make sure that colin works with the new changes.

We've got a report internally that $subject does not work. Please investigate and ideally create a test case for this.

I am pretty sure the "WARN:Label 'maintainer' has to be specified." is a
false warning, unless I am blind and can not see the typo in my Dockerfile.

CC @fkluknav

Edit: I think I get it know, the message is wrong. We check whether the content matches a certain regex.

• new class to check that a label is NOT present
• release, architecture labels
• a bunch of other labels from https://github.com/projectatomic/ContainerApplicationGenericLabels/blob/master/vendor/redhat/labels.md

This is a low-prio RFE, mainly added here just to track the problem.

Let's collaborate on a 0.1.0 release definition of colin. Please edit this post when adding more items.

• colin is packaged as RPM #64
• colin can be installed at least from copr https://copr.fedorainfracloud.org/coprs/phracek/colin/ see the latest build https://copr.fedorainfracloud.org/coprs/phracek/colin/builds/ #64
• documentation is up #69
• all checks are documented http://colin.readthedocs.io/en/latest/list_of_checks.html
• we have some unit and integration tests and they are passing #63
• CI is set up (this is a tough one but we need it)
• we have pretty output #65

stretch goal:

• we have contributing.md #71
• colin is part of Fedora

What do you think?

Based on #132 (comment), I installed colin with pip2. When I try to do colin check it fails with load_config() got an unexpected keyword argument 'config_dict'.

Sample output from run with --debug:

$ sudo colin check --debug -f /usr/share/colin/rulesets/fedora.json registry.fedoraproject.org/fedora:27
11:57:32.683 colin.py          DEBUG  Checking started.
11:57:32.683 target.py         DEBUG  Finding target 'registry.fedoraproject.org/fedora:27'.
11:57:32.683 backend.py        INFO   conu has initiated, welcome to the party!
11:57:32.683 backend.py        DEBUG  conu version: 0.3.1
11:57:32.698 __init__.py       INFO   docker environment info: 'Client:\n Version:         1.13.1\n API version:     1.26\n Package version: docker-1.13.1-51.git4032bd5.fc28.x86_64\n Go version:      go1.10\n Git commit:      c301b04-unsupported\n Built:           Wed Mar 28 13:53:29 2018\n OS/Arch:         linux/amd64\n\nServer:\n Version:         1.13.1\n API version:     1.26 (minimum version 1.12)\n Package version: docker-1.13.1-51.git4032bd5.fc28.x86_64\n Go version:      go1.10\n Git commit:      c301b04-unsupported\n Built:           Wed Mar 28 13:53:29 2018\n OS/Arch:         linux/amd64\n Experimental:    false\n'
11:57:32.699 colin.py          ERROR  An error occurred: TypeError("load_config() got an unexpected keyword argument 'config_dict'",)
Traceback (most recent call last):
  File "/bin/colin", line 11, in <module>
    load_entry_point('colin==0.0.4', 'console_scripts', 'colin')()
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/colin/cli/colin.py", line 79, in check
    tags=tag)
  File "/usr/lib/python2.7/site-packages/colin/core/colin.py", line 45, in run
    logging_level=logging_level)
  File "/usr/lib/python2.7/site-packages/colin/core/target.py", line 54, in __init__
    self.instance = Target._get_target_instance(target, logging_level=logging_level)
  File "/usr/lib/python2.7/site-packages/colin/core/target.py", line 80, in _get_target_instance
    with DockerBackend(logging_level=logging_level) as backend:
  File "/usr/lib/python2.7/site-packages/conu/backend/docker/backend.py", line 75, in __init__
    self.d = get_client()
  File "/usr/lib/python2.7/site-packages/conu/backend/docker/client.py", line 37, in get_client
    client = docker.APIClient(version="auto")  # >= 2
  File "/usr/lib/python2.7/site-packages/docker/api/client.py", line 110, in __init__
    config_dict=self._general_configs
TypeError: load_config() got an unexpected keyword argument 'config_dict'

Similar results when trying to run a check against a running container:

$ docker ps
CONTAINER ID        IMAGE                                  COMMAND             CREATED             STATUS              PORTS               NAMES
c224fd48add9        registry.fedoraproject.org/fedora:27   "sleep 300"         4 minutes ago       Up 4 minutes                            some-fedora

$ sudo colin check --debug -f /usr/share/colin/rulesets/default.json some-fedora
11:58:57.780 colin.py          DEBUG  Checking started.
11:58:57.780 target.py         DEBUG  Finding target 'some-fedora'.
11:58:57.780 backend.py        INFO   conu has initiated, welcome to the party!
11:58:57.780 backend.py        DEBUG  conu version: 0.3.1
11:58:57.794 __init__.py       INFO   docker environment info: 'Client:\n Version:         1.13.1\n API version:     1.26\n Package version: docker-1.13.1-51.git4032bd5.fc28.x86_64\n Go version:      go1.10\n Git commit:      c301b04-unsupported\n Built:           Wed Mar 28 13:53:29 2018\n OS/Arch:         linux/amd64\n\nServer:\n Version:         1.13.1\n API version:     1.26 (minimum version 1.12)\n Package version: docker-1.13.1-51.git4032bd5.fc28.x86_64\n Go version:      go1.10\n Git commit:      c301b04-unsupported\n Built:           Wed Mar 28 13:53:29 2018\n OS/Arch:         linux/amd64\n Experimental:    false\n'
11:58:57.795 colin.py          ERROR  An error occurred: TypeError("load_config() got an unexpected keyword argument 'config_dict'",)
Traceback (most recent call last):
  File "/bin/colin", line 11, in <module>
    load_entry_point('colin==0.0.4', 'console_scripts', 'colin')()
  File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/colin/cli/colin.py", line 79, in check
    tags=tag)
  File "/usr/lib/python2.7/site-packages/colin/core/colin.py", line 45, in run
    logging_level=logging_level)
  File "/usr/lib/python2.7/site-packages/colin/core/target.py", line 54, in __init__
    self.instance = Target._get_target_instance(target, logging_level=logging_level)
  File "/usr/lib/python2.7/site-packages/colin/core/target.py", line 80, in _get_target_instance
    with DockerBackend(logging_level=logging_level) as backend:
  File "/usr/lib/python2.7/site-packages/conu/backend/docker/backend.py", line 75, in __init__
    self.d = get_client()
  File "/usr/lib/python2.7/site-packages/conu/backend/docker/client.py", line 37, in get_client
    client = docker.APIClient(version="auto")  # >= 2
  File "/usr/lib/python2.7/site-packages/docker/api/client.py", line 110, in __init__
    config_dict=self._general_configs
TypeError: load_config() got an unexpected keyword argument 'config_dict'

Am I doing something wrong here?

Version info:

$ sudo colin -V
colin, version 0.0.4

$ rpm -q python2-conu
rpm -q python2-conu

$ rpm -q python2-dockerfile-parse
python2-dockerfile-parse-0.0.10-1.fc28.noarch

$ rpm -qa | grep docker
python2-dockerfile-parse-0.0.10-1.fc28.noarch
docker-1.13.1-51.git4032bd5.fc28.x86_64
python3-docker-3.2.1-1.fc28.noarch
docker-common-1.13.1-51.git4032bd5.fc28.x86_64
python2-docker-3.2.1-1.fc28.noarch
docker-rhel-push-plugin-1.13.1-51.git4032bd5.fc28.x86_64
python3-docker-pycreds-0.2.2-2.fc28.noarch
python2-docker-pycreds-0.2.2-2.fc28.noarch

I tried to install and use colin locally but it fails.

$ colin -h
Traceback (most recent call last):
  File "/usr/bin/colin", line 11, in <module>
    load_entry_point('colin==0.0.1', 'console_scripts', 'colin')()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 587, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2800, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2431, in load
    return self.resolve()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2437, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python2.7/site-packages/colin-0.0.1-py2.7.egg/colin/__init__.py", line 1, in <module>
    from .core.colin import run
  File "/usr/lib/python2.7/site-packages/colin-0.0.1-py2.7.egg/colin/core/colin.py", line 1, in <module>
    from colin.core.check_runner import go_through_checks
ImportError: No module named core.check_runner

We would like to start a new group of generic dynamic checks on containers. One of the use cases is to run shell inside and check whether the container is running, e.g.:

$ docker run -t --rm -d --name asd $image sh
$ docker ps | grep asd

We should check for these cases:

• docker run -d -t $image bash
• docker run -d $image sleep infinity && docker exec ...

In order to add colin into Fedora https://bugzilla.redhat.com/show_bug.cgi?id=1572084
we have to update SPEC file accordingly.

See comments in bugzilla

DON'T CLOSE IT yet.

The command was:
$ colin check --json fedora.json -r fedora fedora:27

The code is here:

$ colin check --json fedora.json -r fedora fedora:27
LABELS:
nok:failed:maintainer_label_required
   -> Label 'maintainer' has to be specified.
   -> The name and email of the maintainer (usually the submitter).
   -> https://fedoraproject.org/wiki/Container:Guidelines#LABELS
nok:failed:name_label_required
   -> Label 'name' has to be specified.
   -> Name of the Image or Container.
   -> https://fedoraproject.org/wiki/Container:Guidelines#LABELS
nok:failed:com_redhat_component_label_required
   -> Label 'com.redhat.component' has to be specified.
   -> The Bugzilla component name where bugs against this container should be reported by users.
   -> https://fedoraproject.org/wiki/Container:Guidelines#LABELS
[..snip..]
nok:warning:io.openshift.expose-services_label
   -> Label 'io.openshift.expose-services' has to be specified.
   -> port:service pairs separated with comma, e.g. "8080:http,8443:https"
   -> ?????
Failed to mount OverlayFS device.
mount: only root can use "--options" option

16:38:17.688 colin.py          ERROR  An error occurred: ColinException("Problem with mounting filesystem with atomic. (Command '['atomic', 'mount', 'fedora:27', '/tmp/conuwzyyvd3g']' returned non-zero exit status 1.)",)
Error: Problem with mounting filesystem with atomic. (Command '['atomic', 'mount', 'fedora:27', '/tmp/conuwzyyvd3g']' returned non-zero exit status 1.)

We're running colin like this:

["colin", "-f", "ruleset.json", "-t", "optional", "registry.access.redhat.com/rhscl/postgresql-96-rhel7", "--json", "./artifacts/optional-results.json"]

There was en error in the automation when the artifacts dir did not exist. I would expect colin failing with an error message that results can't be written to the file because the directory does not exist. That did not happened, instead colin failed like this:

"stderr": "Error: No such file or directory"

That's hard to figure out.

For users who would like to play with colin, it would be nice
to write a section like installation how to install colin locally.

https://strawpoll.com/k5pr2pyz

I have no idea what should I do.

• running colin from git #35
• running colin from a standard installation
• checking containers
• checking images
• selecting config
• update asciinema before release, per @lachmanfrantisek