app-sre / sretoolbox

Set of libraries commonly used by multiple SRE projects

SRE Toolbox

Set of libraries commonly used by multiple SRE projects:

• container.Image: class for container image inspection.
• container.Skopeo: wrapper around Skopeo.
• utils.replace_values: deep replace of object values according to values map.
• utils.retry: decorator to add resilience to function calls.

Install

From PyPI:

$ pip install sretoolbox

From source:

$ python setup.py install

Use

Just import the library you want. Example:

>>> from sretoolbox import container
>>> image = container.Image('fedora')
>>> if image:
...     print('Image exists!')
...
Image exists!
>>>

Develop

Install the development requirements:

$  make develop

Run the code checks and tests:

$  make check

Release

Edit the VERSION file and change the new version. Submit a pull request to master. When it is merged, create a tag and push it to app-sre/sretoolbox.

This will trigger a CI job that will publish the package on pypi.

License

The default license of the code in this repository is [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0). That applies for most of the code here, as they were written from scratch, but exceptions exist. In any case, each module carries the corresponding licensing information.

sretoolbox's Issues

License of retry decorator

While reviewing the code of

sretoolbox/sretoolbox/utils/retry.py

Line 12 in c0def7d

# source: https://www.calazan.com/retry-decorator-for-python-3/

I visited the URL at https://www.calazan.com/retry-decorator-for-python-3/ and found that it was mentioning this:

Below is a nice retry decorator I found on this https://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/ site. I just changed one line of code to make it Python 3 compatible (basically had to change the print statement to a function).

And on https://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/

Code / License
This code is also on github at: https://github.com/saltycrane/retry-decorator It is BSD licensed.

And a link to https://github.com/saltycrane/retry-decorator/blob/master/LICENSE

This is a BSD-3-Clause. It would make sense to add this in the repo?

License for usage as a library?

Hi,
I was wondering if you may consider using a license that would help reusing your fine code as a library in Apache-licensed code that I co-maintain at https://github.com/nexB/container-inspector and https://github.com/nexB/scancode-toolkit and https://github.com/nexB/scancode.io/
Something like an LGPL or an Apache license would work better than a GPL for library usage.

Thank you for you kind consideration!

_parse_image_url() doesn't support image digests

A recent deploy of a Hive operator failed with:

Image does not exist: quay.io/****/pagerduty-operator-registry@sha256:bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e

The image exists:

docker pull quay.io/****/pagerduty-operator-registry@sha256:bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e
quay.io/****/pagerduty-operator-registry@sha256:bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e: Pulling from ****/pagerduty-operator-registry
Digest: sha256:bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e
Status: Image is up to date for quay.io/****/pagerduty-operator-registry@sha256:bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e
quay.io/****/pagerduty-operator-registry@sha256:bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e

13:58:43 [2021-04-16 17:58:43] [ERROR] [saasherder.py:_check_image:543] - [saas-pagerduty-operator/pagerduty-operator] https://github.com/openshift/pagerduty-operator/blob/master/hack/olm-registry/olm-artifacts-template.yaml: Image does not exist: quay.io/****/pagerduty-operator-registry@sha256:bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e
13:58:43 Traceback (most recent call last):
13:58:43   File "/usr/local/bin/qontract-reconcile", line 33, in <module>
13:58:43     sys.exit(load_entry_point('reconcile==0.2.2', 'console_scripts', 'qontract-reconcile')())
13:58:43   File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 829, in __call__
13:58:43     return self.main(*args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 782, in main
13:58:43     rv = self.invoke(ctx)
13:58:43   File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1259, in invoke
13:58:43     return _process_result(sub_ctx.command.invoke(sub_ctx))
13:58:43   File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 1066, in invoke
13:58:43     return ctx.invoke(self.callback, **ctx.params)
13:58:43   File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/core.py", line 610, in invoke
13:58:43     return callback(*args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/utils/environ.py", line 15, in f_environ
13:58:43     f(*args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/utils/environ.py", line 15, in f_environ
13:58:43     f(*args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/utils/binary.py", line 18, in f_binary
13:58:43     f(*args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/utils/binary.py", line 60, in f_binary_version
13:58:43     f(*args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/click-7.1.2-py3.6.egg/click/decorators.py", line 21, in new_func
13:58:43     return f(get_current_context(), *args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/cli.py", line 758, in openshift_saas_deploy
13:58:43     saas_file_name, env_name, gitlab_project_id)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/cli.py", line 399, in run_integration
13:58:43     func_container.run(dry_run, *args, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/utils/defer.py", line 15, in func_wrapper
13:58:43     return func(*args, defer=defer, **kwargs)
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/openshift_saas_deploy.py", line 69, in run
13:58:43     if not saasherder.validate_promotions():
13:58:43   File "/usr/local/lib/python3.6/site-packages/reconcile-0.2.2-py3.6.egg/reconcile/utils/saasherder.py", line 928, in validate_promotions
13:58:43     commit_sha = item['commit_sha']
13:58:43 TypeError: 'NoneType' object is not subscriptable

SREP haven't hit this issue yet because on cluster operators are deployed via selector sync set inside the OC template. Saasherder doesn't validate the CatalogSource images inside the selector sync set here. This is different for Hive operators as we deploy the CatalogSource directly in the OC template.

It looks like that method is the issue.

Parsing a digest returns the following structure:

{'scheme': 'docker://', 'registry': 'docker.io', 'port_colon': None, 'port': None, 'registry_slash': None, 'repository': 'library', 'repo_slash': None, 'image': 'sha256', 'tag_colon': ':', 'tag': 'bc1ed82a75f2ca160225b8281c50b7074e7678c2a1f61b1fb298e545b455925e'}

cc/ @2uasimojo @BumbleFeng @apahim

Cache registry token

We acquire a new token on every request:

https://github.com/app-sre/sretoolbox/blob/master/sretoolbox/container/image.py#L355

We should probably make a more efficient use of it.