eduardocerqueira / janitor

cli tool to perform automatic clean up in Openstack based on whitelist file

Janitor

Janitor is a Linux component to perform quota management tasks to an Openstack tenant.

It is not an official Openstack's tool, and Janitor's code is wrapping Openstack python client libraries:

• python-openstackclient to manage authentication and as general API
• python-neutronclient to manage network
• python-novaclient to manage compute nodes and vms
• python-glanceclient to manage cloud images
• python-cinderclient to manage volumes and block storage

Table of content

• Motivation
• Quick links
• Running in docker
• Install and usage
• Setup your dev environment
• Running janitor functional tests
• Contributing
• release pypi package
• Build RPM and release
• DEMO

Quick links

repo	description
Git	source code repository
Copr	RPM build
Quay.io	container image build
test.pypi	pypi repo for testing python package release
pypi	pypi repo for prod python package release

Motivation

I have been using OpenStack continuously since 2014 as a resource to run workloads with the main purpose of testing software. Manage a healthy Openstack tenant quota is vital when working with CI/CD and a large number of automated pipelines which run 24/7. A long time ago I and my team were having a lot of headaches with one of internal Openstack the cluster which at that time had a very limited quota and workload capacity but was important for the test workflow. With the constant false-positive for test failing due to lack of public/floating IP, storage/volume, and even virtual CPU when provisioning new VMs, we got the idea of implementing Janitor, a super simple script/codes to automate the clean-up task of deleting left-over virtual machines as well network and volume from an Openstack tenant, all managed by API and remote calls.

Install and Usage

from container:

docker pull quay.io/ecerquei/janitor
docker run --rm -it --network host --name janitor ecerquei/janitor /bin/bash

# copy your files
docker cp openrc.sh janitor:/home/janitor
docker cp whitelist.txt janitor:/home/janitor

bash-4.4$ janitor --help
bash-4.4$ janitor openstack --openrc openrc.sh --whitelist whitelist.txt --keystone v3

see container

from pip:

• https://test.pypi.org/project/janitor-osp/
• https://pypi.org/project/janitor-osp/

python3 -m venv venv
source venv/bin/activate
pip install pip --upgrade
pip install -r https://raw.githubusercontent.com/eduardocerqueira/janitor/master/requirements/production.txt
python3 -m pip install janitor --upgrade

For Fedora:

1. from Copr

dnf install dnf-plugin-copr
dnf copr enable eduardocerqueira/janitor
dnf install janitor

or if you prefer you can install the repo from https://copr.fedorainfracloud.org/coprs/eduardocerqueira/janitor/ and maybe is needed to disable gpgcheck=0

[eduardocerqueira-janitor]
name=Copr repo for janitor owned by eduardocerqueira
baseurl=https://copr-be.cloud.fedoraproject.org/results/eduardocerqueira/janitor/epel-7-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=0
gpgkey=https://copr-be.cloud.fedoraproject.org/results/eduardocerqueira/janitor/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1

2. from local build::

make rpm
dnf install rpmbuild/RPMS/x86_64/janitor-0.2-0.x86_64.rpm

for CentOS:

requires openstack repo. check the latest repo https://wiki.openstack.org/wiki/Release_Naming

sudo yum install centos-release-openstack-newton.noarch
sudo yum install install rpmbuild/RPMS/x86_64/janitor-0.2-0.x86_64.rpm

using Janitor to clean up left-over virtual machines and release floating ips for Openstack tenant:

source ~/mytenant-openrc.sh
janitor openstack --whitelist /tmp/whitelist.txt --keystone v3

# or passing the path for your openrc file
janitor openstack --openrc /tmp/mytenant-openrc.sh --whitelist /tmp/whitelist.txt --keystone v3

listing history for your janitor:

janitor history

local dev environment

Requirements:

• Linux OS tested on Fedora and CentOS
• Python3
• packages:

# install packages needed to interact with make, build RPM, make release to Copr, and generate doc
sudo dnf install redhat-rpm-config rpm-build python3-devel gcc python3-devel python3-pip python3-wheel python3-setuptools, python3-sphinx copr-cli

# prep your python env
git clone [email protected]:eduardocerqueira/janitor.git
cd janitor
python3 -m venv venv
source venv/bin/activate
pip install -r requirements/devel.txt

# check janitor installation
python janitor/cli.py --help
openstack --openrc /home/ecerquei/osp/janitor-openrc.sh --whitelist /home/ecerquei/osp/janitor-whitelist.txt --keystone v3

see running janitor functional tests

also you can explore the make tasks running make.

when running make doc the generated doc can be access at : file:///home/user/git/janitor/docs/build/html/index.html

Contributing

Any idea, suggestions and pacthes are welcome to this project! Fork the project, make your code change, run the test locally to ensure your changes are not breaking any functionality, remember to run the code static analysis before submitting your PR and if needed open issues or discussion on this project.

# make flake8

running in docker

image is available in quay janitor repository

docker pull quay.io/ecerquei/janitor
docker run --rm -it --network host --name janitor ecerquei/janitor /bin/bash

# running janitor inside the container
bash-4.4$ janitor --help

to build a new docker image:

docker build -f Dockerfile . -t ecerquei/janitor

pushing new container to quay

# terminal 1
docker run --rm -it --network host --name janitor ecerquei/janitor /bin/bash

# terminal 2
docker ps -l # and copy the container id 
docker commit aa20639aa618 quay.io/ecerquei/janitor
docker push quay.io/ecerquei/janitor

release pypi package

ensure you have permission, and the API token saved at ~/.pypirc to publish new release to pypi.

1. increment the build updating the var VERSION=0.3 from Makefile
2. make rpm ( needed to generate the new version file )
3. run commands below:

source venv/bin/activate
python3 -m build

# for testpypi
python3 -m twine upload --repository testpypi dist/*

# for pypi
python3 -m twine upload dist/*

package will be available at:

• https://test.pypi.org/project/janitor-osp/#history
• https://pypi.org/project/janitor-osp/#history

Build RPM and release

$ make

Usage: make <target> where <target> is one of

clean     clean temp files from local workspace
doc       generate sphinx documentation and man pages
test      run functional/unit tests locally
tarball   generate tarball of project
rpm       build source codes and generate rpm file
srpm      generate SRPM file
build     generate srpm and send to build in copr
all       clean test doc rpm
flake8    check Python style based on flake8

Running from your local machine, you can generate your own RPM running:

$ make rpm

if your environment is properly setup you should have your RPM at: /home/user/git/janitor/rpmbuild/RPMS/x86_64/janitor-0.0.2-0.x86_64.rpm

janitor is being built on Fedora Copr https://copr.fedorainfracloud.org/coprs/eduardocerqueira/janitor/builds/

running a new build you need to check your ~/.config/copr-fedora file and run:

make build

Before starting the release process, check your account permissions in Copr.

$ make srpm

1. copy rpmbuild/SRPMS/janitor-0.0.2-0.src.rpm to janitor/copr
2. push janitor/copr to github

copr-cli will be used, installed by sudo yum/dnf install copr-cli and configure it.

Request as Builder for projects janitor, wait until admin approves.

$ copr-cli build janitor https://github.com/eduardocerqueira/janitor/raw/master/copr/janitor-0.0.2-0.src.rpm

Go and grab a cup of tea or coffee, the release build will be come out soon ::

# tag based builds: `https://copr.fedorainfracloud.org/coprs/eduardocerqueira/janitor/builds/`

Copr and RPM release helpful Links

• https://fedorahosted.org/copr/wiki/HowToEnableRepo
• http://fedoraproject.org/wiki/Infrastructure/fedorapeople.org#Accessing_Your_fedorapeople.org_Space
• https://fedorahosted.org/copr/wiki/UserDocs#CanIgiveaccesstomyrepotomyteammate
• https://copr.fedoraproject.org/api/

DEMO

Running the program without parameters:

[ecerquei@dev ~]$ janitor
Usage: janitor [OPTIONS] COMMAND [ARGS]...

  Janitor clean-up the left-over in your Openstack tenant to know more check
  documentation running: man janitor

Options:
  --help  Show this message and exit.

Commands:
  history    show janitor history
  openstack  clean-up VM and public IP from Openstack tenant

Running the program with with parameters to make clean-up:

[ecerquei@dev ~]$ janitor openstack --openrc /home/ecerquei/git/janitor/tests/test-openrc.sh --whitelist /home/ecerquei/git/janitor/tests/whitelist.txt --keystone v3
+---------------------+--------------+------------------+-----------------------------+--------------------------------------------+--------+----------------------+
|      TIMESTAMP      |    ACTION    |       NAME       |             IPs             |                   IMAGE                    | FLAVOR |   CREATED AT (UTC)   |
+---------------------+--------------+------------------+-----------------------------+--------------------------------------------+--------+----------------------+
| 2017-07-13 15:00:57 |   deleted    |    QA-DB-100     |        172.16.91.195        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:56:38Z |
| 2017-07-13 15:00:57 |   deleted    |    DEV-WEB-10    |        172.16.91.194        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:56:24Z |
| 2017-07-13 15:00:57 |   deleted    |    PROD-APP1     |        172.16.91.193        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:56:10Z |
| 2017-07-13 15:00:57 |   deleted    | test_123_456_789 |        172.16.91.192        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:55:58Z |
| 2017-07-13 15:00:57 |   deleted    |    test123456    |        172.16.91.191        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:55:47Z |
| 2017-07-13 15:00:57 | in whitelist |  jslave_centos   | 172.16.91.173, 10.8.187.160 |            paws-jslave-centos7             |   3    | 2017-07-13T01:02:58Z |
| 2017-07-13 15:00:57 | in whitelist | jslave_fedora25  | 172.16.91.172, 10.8.187.207 |            paws-jslave-fedora25            |   3    | 2017-07-13T01:02:08Z |
+---------------------+--------------+------------------+-----------------------------+--------------------------------------------+--------+----------------------+
+---------------------+---------+--------------+
|      TIMESTAMP      |  ACTION | FLOATING IP  |
+---------------------+---------+--------------+
| 2017-07-13 15:00:57 | deleted | 10.8.180.199 |
+---------------------+---------+--------------+

+---------------------+---------+--------------------------------------+
|      TIMESTAMP      |  ACTION |              VOLUME ID               |
+---------------------+---------+--------------------------------------+
| 2017-07-13 15:00:57 | deleted | 358be8d1-6d4a-4db7-973f-8369d4ff86f7 |
+---------------------+---------+--------------------------------------+	

Running the program with parameter to print history file:

[ecerquei@dev ~]$ janitor history
+---------------------+--------------+------------------+-----------------------------+--------------------------------------------+--------+----------------------+
|      TIMESTAMP      |    ACTION    |       NAME       |             IPs             |                   IMAGE                    | FLAVOR |   CREATED AT (UTC)   |
+---------------------+--------------+------------------+-----------------------------+--------------------------------------------+--------+----------------------+
| 2017-07-13 15:00:57 |   deleted    |    QA-DB-100     |        172.16.91.195        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:56:38Z |
| 2017-07-13 15:00:57 |   deleted    |    DEV-WEB-10    |        172.16.91.194        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:56:24Z |
| 2017-07-13 15:00:57 |   deleted    |    PROD-APP1     |        172.16.91.193        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:56:10Z |
| 2017-07-13 15:00:57 |   deleted    | test_123_456_789 |        172.16.91.192        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:55:58Z |
| 2017-07-13 15:00:57 |   deleted    |    test123456    |        172.16.91.191        | CentOS-Atomic-Host-7.20160130-GenericCloud |   2    | 2017-07-13T18:55:47Z |
| 2017-07-13 15:00:57 | in whitelist |  jslave_centos   | 172.16.91.173, 10.8.187.160 |            paws-jslave-centos7             |   3    | 2017-07-13T01:02:58Z |
| 2017-07-13 15:00:57 | in whitelist | jslave_fedora25  | 172.16.91.172, 10.8.187.207 |            paws-jslave-fedora25            |   3    | 2017-07-13T01:02:08Z |
+---------------------+--------------+------------------+-----------------------------+--------------------------------------------+--------+----------------------+
+---------------------+---------+--------------+
|      TIMESTAMP      |  ACTION | FLOATING IP  |
+---------------------+---------+--------------+
| 2017-07-13 15:00:57 | deleted | 10.8.180.199 |
+---------------------+---------+--------------+

+---------------------+---------+--------------------------------------+
|      TIMESTAMP      |  ACTION |              VOLUME ID               |
+---------------------+---------+--------------------------------------+
| 2017-07-13 15:00:57 | deleted | 358be8d1-6d4a-4db7-973f-8369d4ff86f7 |
+---------------------+---------+--------------------------------------+	

issues

Running on CentOS7 even having EPEL and centos-release-openstack-newton I got this error in one of my CentOS server.

[root@centos7 yum.repos.d]# janitor
Traceback (most recent call last):
  File "/usr/bin/janitor", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3038, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3022, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3051, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 659, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 672, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 862, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (requests 2.11.1 (/usr/lib/python2.7/site-packages), Requirement.parse('requests>=2.14.2'), set(['python-keystoneclient']))

It is happening because python-requests lib is needed on version requests>=2.14.2 required by python-keystoneclient when I ran:

yum provides python-requests

the latest version I got was 2.11 that still don't satisfacts.

python-requests-2.6.0-1.el7_1.noarch : HTTP library, written in Python, for human beings
Repo        : base



python-requests-2.10.0-1.el7.noarch : HTTP library, written in Python, for human beings
Repo        : centos-openstack-newton



python2-requests-2.11.1-1.el7.noarch : HTTP library, written in Python, for human beings
Repo        : centos-openstack-newton
Matched from:
Provides    : python-requests = 2.11.1-1.el7



python2-requests-2.11.1-1.el7.noarch : HTTP library, written in Python, for human beings
Repo        : @centos-openstack-newton
Matched from:
Provides    : python-requests = 2.11.1-1.el7

The trick here was, forcing reinstall requests using pip:

pip install requests --upgrade

janitor's Issues

upgrade to python3

upgrade to python3

version and build timestamp

Add version and build timestamp so user can run: janitor version or --version and print version and build timestamp.

• nice to have: show what is the latest version available

tests and lint

• unit and functional tests
• jenkinsfile, travis or github jobs for validating PR
• lint/black pre-commit hook

Openstack vm/instance without image associated - TypeError: string indices must be integers

when an instance/vm doesn't have the image associated Janitor is throwing this error and exit(1). It needs to be treated and set properly in case the instance/vm does not have the image ( that it can happens )

13:16:46 + python janitor/cli.py openstack --openrc /var/lib/jenkins/secure_creds/openrc.sh --whitelist ../insights-tests/jenkins/jobs/janitor_whitelist.txt
13:16:50 Traceback (most recent call last):
13:16:50   File "janitor/cli.py", line 59, in <module>
13:16:50     janitor()
13:16:50   File "janitor/cli.py", line 55, in janitor
13:16:50     cli(prog_name='janitor')
13:16:50   File "/usr/lib/python2.7/site-packages/click/core.py", line 722, in __call__
13:16:50     return self.main(*args, **kwargs)
13:16:50   File "/usr/lib/python2.7/site-packages/click/core.py", line 697, in main
13:16:50     rv = self.invoke(ctx)
13:16:50   File "/usr/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
13:16:50     return _process_result(sub_ctx.command.invoke(sub_ctx))
13:16:50   File "/usr/lib/python2.7/site-packages/click/core.py", line 895, in invoke
13:16:50     return ctx.invoke(self.callback, **ctx.params)
13:16:50   File "/usr/lib/python2.7/site-packages/click/core.py", line 535, in invoke
13:16:50     return callback(*args, **kwargs)
13:16:50   File "janitor/cli.py", line 39, in openstack
13:16:50     vms = openstack.get_all_instances()
13:16:50   File "/var/lib/jenkins/jobs/janitor/workspace/janitor/janitor/provider/openstack.py", line 91, in get_all_instances
13:16:50     image = self.glance.images.get(vm.image['id'])
13:16:50 TypeError: string indices must be integers

loading to indicating janitor is working

Add loading or progress bar to indicate janitor is working.
it can be done using python-click lib that has already been used by Janitor