GithubHelp home page GithubHelp logo

ansible-community / ara Goto Github PK

View Code? Open in Web Editor NEW
1.8K 65.0 166.0 37.14 MB

ARA Records Ansible and makes it easier to understand and troubleshoot.

Home Page: https://ara.recordsansible.org

License: GNU General Public License v3.0

Python 73.48% CSS 2.28% HTML 22.05% Shell 1.82% JavaScript 0.38%
ansible ara python api reporting django django-rest-framework interface

ara's Introduction

ARA Records Ansible

ARA Records Ansible and makes it easier to understand and troubleshoot.

logo

It is another recursive acronym with a focus on simplicity.

About ara

ara provides Ansible reporting by recording ansible and ansible-playbook commands regardless of how and where they run:

  • from most Linux distributions and even on Mac OS (as long as python >= 3.8 is available)
  • from tools that run Ansible like ansible-(pull|test|runner|navigator), AWX & Automation Controller (Tower), Molecule and Semaphore
  • from a terminal, a script or by hand
  • from a laptop, desktop, server, virtual machine, container or execution environment
  • from CI/CD platforms such as Jenkins, Rundeck and Zuul
  • from git forges like GitHub, GitLab, Gitea & Forgejo

The recorded results are available via an included CLI, a REST API as well as a self-hosted, local-first web reporting interface.

demo.mp4

How it works

ARA Records Ansible results to SQLite, MySQL and PostgreSQL databases with a standard callback plugin.

This plugin gathers data as Ansible runs and sends it to a Django REST API server:

recording-workflow

Requirements

  • Any recent Linux distribution or Mac OS with python >=3.8 available
  • The ara package (containing the Ansible plugins) must be installed for the same python interpreter as Ansible itself

Getting started

For production use, consider learning about best practices, enabling authentication and ignoring what doesn't need to be recorded.

Recording playbooks without an API server

ara records to a local sqlite database by default and does not require a persistent server:

# Install ansible (or ansible-core) with ara (including API server dependencies)
python3 -m pip install --user ansible "ara[server]"

# Configure Ansible to enable ara
export ANSIBLE_CALLBACK_PLUGINS="$(python3 -m ara.setup.callback_plugins)"

# Run an Ansible playbook as usual
ansible-playbook playbook.yml

# Check out the CLI
ara playbook list
ara host list

# or the UI at http://127.0.0.1:8000
ara-manage runserver

getting-started

Recording playbooks with an API server

The server includes a REST API as well a web reporting interface.

Consider running one to aggregate playbook runs from different tools, jobs or servers into a single dashboard that can be shared with friends.

Get started with the ara_api role or with the container images published by the project on DockerHub and quay.io:

# Create a directory for a volume to store settings and a sqlite database
mkdir -p ~/.ara/server

# Start an API server with docker from the image on DockerHub:
docker run --name api-server --detach --tty \
  --volume ~/.ara/server:/opt/ara -p 8000:8000 \
  docker.io/recordsansible/ara-api:latest

# or with podman from the image on quay.io:
podman run --name api-server --detach --tty \
  --volume ~/.ara/server:/opt/ara -p 8000:8000 \
  quay.io/recordsansible/ara-api:latest

Once the server is running, ara must be installed and configured to send data to it:

# Install ansible (or ansible-core) with ara (excluding API server dependencies)
python3 -m pip install --user ansible ara

# Configure Ansible to enable ara
export ANSIBLE_CALLBACK_PLUGINS="$(python3 -m ara.setup.callback_plugins)"

# Set up the ara callback to know where the API server is located
export ARA_API_CLIENT="http"
export ARA_API_SERVER="http://127.0.0.1:8000"

# Run an Ansible playbook as usual
ansible-playbook playbook.yml

# Check out the CLI
ara playbook list
ara host list

# Or browse http://127.0.0.1:8000 (running from the container)

Live demo

A live demo is deployed with the ara Ansible collection from Ansible Galaxy.

It is available at https://demo.recordsansible.org.

Documentation and changelog

Documentation for installing, configuring, running and using ara is available on ara.readthedocs.io.

Common issues may be resolved by reading the troubleshooting guide.

Changelog and release notes are available within the repository's git tags as well as the documentation.

Community and getting help

Contributing

Contributions to the project are welcome and appreciated !

Get started with the contributor's documentation.

Authors

Code contributions to the project can be viewed from the git log or on GitHub.

The ara parrot logo was designed and contributed by Jason E. Rist.

Copyright

Copyright (c) 2023 The ARA Records Ansible authors

ARA Records Ansible is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

ARA Records Ansible is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with ARA Records Ansible.  If not, see <http://www.gnu.org/licenses/>.

ara's People

Contributors

ansible-zuul[bot] avatar apollo13 avatar arrfab avatar berendt avatar code-review-doctor avatar cosemansbert avatar danopt avatar devon-mar avatar dmsimard avatar evrardjp avatar flowerysong avatar gaby avatar goneri avatar helmo avatar hille721 avatar jayv3e avatar jhampson-dbre avatar keuko avatar laurentdumont avatar mattp- avatar phemmer avatar rh-gvincent avatar ricardom17 avatar sshnaidm avatar thulium-drake avatar tomsiewert avatar tristancacqueray avatar turgon37 avatar zxaos avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ara's Issues

ara_api role: gunicorn fails to start if selinux is enforcing

See:

type=AVC msg=audit(1559060052.066:203): avc:  denied  { execute } for  pid=32057 comm="(gunicorn)" name="gunicorn" dev="vda1" ino=528956 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0
type=AVC msg=audit(1559060259.586:238): avc:  denied  { execute } for  pid=872 comm="(gunicorn)" name="gunicorn" dev="vda1" ino=528956 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=1
type=AVC msg=audit(1559060259.586:239): avc:  denied  { read open } for  pid=872 comm="(gunicorn)" path="/root/.ara/virtualenv/bin/gunicorn" dev="vda1" ino=528956 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=1
type=AVC msg=audit(1559060259.586:240): avc:  denied  { execute_no_trans } for  pid=872 comm="(gunicorn)" path="/root/.ara/virtualenv/bin/gunicorn" dev="vda1" ino=528956 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=1
type=AVC msg=audit(1559060259.598:241): avc:  denied  { ioctl } for  pid=872 comm="gunicorn" path="/root/.ara/virtualenv/pyvenv.cfg" dev="vda1" ino=397404 ioctlcmd=0x5401 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=1
type=AVC msg=audit(1559060260.003:242): avc:  denied  { read } for  pid=875 comm="gunicorn" name="settings.yaml" dev="vda1" ino=397411 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:admin_home_t:s0 tclass=file permissive=1
type=AVC msg=audit(1559060260.003:243): avc:  denied  { open } for  pid=875 comm="gunicorn" path="/root/.ara/server/settings.yaml" dev="vda1" ino=397411 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:admin_home_t:s0 tclass=file permissive=1
type=AVC msg=audit(1559060260.003:244): avc:  denied  { ioctl } for  pid=875 comm="gunicorn" path="/root/.ara/server/settings.yaml" dev="vda1" ino=397411 ioctlcmd=0x5401 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:admin_home_t:s0 tclass=file permissive=1

This used to be addressed by shipping a selinux policy file but somehow got lost during the transition from ara-infra to ara.

Could/Should there be an ARA callback plugin in upstream Ansible ?

Would an ARA callback plugin in upstream Ansible be useful ?

The current implementation of the callback plugin requires at a minimum the ARA HTTP API client which requires ARA to be installed.
When using the default offline API client, it requires the server dependencies to be installed as well.

So, that said, I don't think the callback in it's current shape is a good fit for being included upstream.
However, since the http API client is nothing much more than a wrapper around python requests, there /could/ be a generic and self-contained implementation of the callback plugin.

Confusing behavior of env vars ARA_DIR and ARA_LOG_FILE

Migrated from https://storyboard.openstack.org/#!/story/2005197

@riccardomurri reported:

The current documentation for environmental variable ARA_DIR states:

ARA_DIR: Base directory where ARA will store itโ€™s log file and sqlite database, unless specified otherwise.

ARA_LOG_FILE: Path to the logfile to store ARA logs in.

However, it is not clear how the two are combined: I would expect that Ara does os.path.join(ARA_DIR, ARA_LOG_FILE). Instead, it apparently does the equivalent of ARA_DIR + '/' + os.path.abspath(ARA_LOG_FILE), as the following experiments show:

1.

...
elasticluster[15012] DEBUG - ARA_DIR='.'
elasticluster[15012] DEBUG - ARA_LOG_FILE='/home/***/elasticluster/ara.log'
...
 [WARNING]: ... Unable to configure handler 'ara_file': [Errno 2] No such file or directory: '/tmp/elasticluster.P9hCdo.d/home/***/elasticluster/ara.log'

2.

...
elasticluster[15178] DEBUG - ARA_DIR='/home/***/elasticluster'
elasticluster[15178] DEBUG - ARA_LOG_FILE='ara.log'
...
 [WARNING]: ... Unable to configure handler 'ara_file': [Errno 2] No such file or directory: '/home/***/elasticluster/home/***/elasticluster/ara.log'

3.

2019-03-11 11:43:58 monia gc3.elasticluster[15264] DEBUG - ARA_DIR='/'
2019-03-11 11:43:58 monia gc3.elasticluster[15264] DEBUG - ARA_LOG_FILE='/home/***/elasticluster/ara.log'
...
 [WARNING]: [Errno 13] Permission denied: u'/logging.yml'

This last combination seems to work w.r.t. writing to the correct log file, but it fails because of other files that are assumed to be present and/or created in ARA_DIR -- which should not be there in the first place, as the documentation for ARA_DIR only mentions it being the "base directory" for logs and the SQLite database..

Add support for recording diffs

Diffs are available from the callback but they are currently discarded:

https://github.com/ansible/ansible/blob/7b7d6a1fef42eed7e144a7c391817a0a8263c3ae/lib/ansible/plugins/callback/__init__.py#L113-L115

We can retrieve the diff before it is discarded around here:

The diff printed by Ansible looks like this:

TASK [template] **********************************************************************************************************************
--- before: /tmp/foo
+++ after: /home/fedora/.ansible/tmp/ansible-local-25447850rn27v/tmpu1ngsu7a/foo.j2
@@ -1 +1 @@
-this is a roary
+this is a roar

And this is the format in which the data is available from the callback:

[{'before_header': '/tmp/foo', 'before': 'this is a roary\n', 'after_header': '/home/fedora/.ansible/tmp/ansible-local-25447850rn27v/tmpu1ngsu7a/foo.j2', 'after': 'this is a roar\n'}]

It seems like this would be an addition to the result model. A diff column which would default to null. From a web client perspective, the diff could be displayed if there is one.

delete old Ansible runs from Ara

Is there an easy way to delete old data in ARA? I noticed our has Ansible runs going back to 2017 so I would like to prune the old data and maybe keep a year. I'm also hoping deleting odl data might speed it up abit.

Is there a simple SQL query I can use or even a script ?

Add tooling to prune the database of old records

Users might not want to keep all records around until the end of time. It might be relevant to keep them for 30 days, 60 days, a year, etc.

The API does not yet support searching by dates (see #30) so that would need to be addressed first.

We need to determine what this would look like in practice. A django-admin command ? A command in the (eventual) CLI client ?

Add support for Django dynamic database routing (ex: sqlite middleware from 0.x)

In ARA 0.x, we have a custom WSGI middleware which makes the web application load a database dynamically based on the URL where the request came from.

This feature is most notably used by the OpenStack CI environment which leverages Zuul and Ansible to run more than a million playbooks a month, see: https://superuser.openstack.org/articles/scaling-ara-ansible/

In a nutshell, it maps an URL (like http://logserver/some/path/ara-report) to a location on the filesystem (such as /srv/logs/some/path/ara-report) and configures the backend to use the database found in /srv/logs/some/path/ara-report/ansible.sqlite.

This is done with the help of a webserver configuration which routes /ara-report/* to the middleware.

Looking at how this might work with the new Django backend, their documentation explicitly recommends against altering settings during runtime.

There are different approaches using a mixture of Django middlewares and database routers (like this or that).
What is problematic with the approach of using middlewares and routers is that the databases should be configured in the settings prior to launching the backend server.

The official Django docs for multi-database usage has different examples like routing write requests to a master server and read requests to replicas.
This does not really help us since we do not know the location of the databases ahead of time.

So it seems like we'd need a custom Django database backend which extends the sqlite engine but allows us to define the database location dynamically after settings have been loaded.

Any help, suggestions and ideas would be much appreciated for this issue.

Add client-side implementation of labels

A label is a concept exclusive to ara and it is meant to be applied to playbooks by users in order to organize playbooks in logical groups.

Some example use cases:

  • Labels called dev, staging and production could be applied to playbooks in order to group playbooks targetting specific environments.
  • Labels called pre, run and post could be used to identify which phase of a deployment the playbook ran for

Labels are simple alphanumerical strings so you could have labels targetting a bug number, a reference number, etc.

This later allows users to search for playbooks matching that label.

While labels are implemented in the API, there is no way for users to easily use them yet. I've been thinking about how this would look like in practice and it might end up being an Ansible module that looks like this:

# Would create the label if it doesn't exist
# Description is optional but can be provided
# Appends the label to playbook.labels for state.present
# Removes the label from playbook.labels for state.absent
- name: Label this playbook
  ara_playbook_label:
    name: "{{ item }}"
    state: present
  loop:
    - pre-run
    - "{{ ansible_os_distribution }}"
    - "{{ lookup('env','USER') }}"

Failures with non-ascii characters in filesystem paths

Noticed when running ansible integration tests:

+ ansible-playbook test_connection.yml -i ../connection_paramiko_ssh/test_connection.inventory -e target_hosts=paramiko_ssh -e action_prefix= -e local_tmp=/tmp/ansible-local -e remote_tmp=/tmp/ansible-remote
Failed to post on /api/v1/playbooks: {'ansible_version': '2.9.0.dev0', 'arguments': {'version': None, 'verbosity': 0, 'ask_pass': False, 'private_key_file': None, 'remote_user': None, 'connection': 'smart', 'timeout': 10, 'ssh_common_args': '', 'sftp_extra_args': '', 'scp_extra_args': '', 'ssh_extra_args': '', 'force_handlers': True, 'flush_cache': False, 'become': False, 'become_method': 'sudo', 'become_user': None, 'become_ask_pass': False, 'tags': ('all',), 'skip_tags': (), 'check': False, 'syntax': False, 'diff': False, 'inventory': ('/root/.ansible/test/tmp/connection_paramiko_ssh-dd2zu_78-\xc5\xd1\u015a\xcc\u03b2\u0141\xc8/test/integration/targets/connection_paramiko_ssh/test_connection.inventory',), 'listhosts': False, 'subset': None, 'extra_vars': "Not saved by ARA as configured by 'ignored_arguments'", 'vault_ids': (), 'ask_vault_pass': False, 'vault_password_files': (), 'forks': 5, 'module_path': None, 'listtasks': False, 'listtags': False, 'step': False, 'start_at_task': None, 'args': ('test_connection.yml',)}, 'status': 'running', 'path': '/root/.ansible/test/tmp/connection_paramiko_ssh-dd2zu_78-\udcc3\udc85\udcc3\udc91\udcc5\udc9a\udcc3\udc8c\udcce\udcb2\udcc5\udc81\udcc3\udc88/test/integration/targets/connection/test_connection.yml'}
Failed to post on /api/v1/playbooks: {'ansible_version': '2.9.0.dev0', 'arguments': {'version': None, 'verbosity': 0, 'ask_pass': False, 'private_key_file': None, 'remote_user': None, 'connection': 'smart', 'timeout': 10, 'ssh_common_args': '', 'sftp_extra_args': '', 'scp_extra_args': '', 'ssh_extra_args': '', 'force_handlers': True, 'flush_cache': False, 'become': False, 'become_method': 'sudo', 'become_user': None, 'become_ask_pass': False, 'tags': ('all',), 'skip_tags': (), 'check': False, 'syntax': False, 'diff': False, 'inventory': ('/root/.ansible/test/tmp/connection_paramiko_ssh-dd2zu_78-\xc5\xd1\u015a\xcc\u03b2\u0141\xc8/test/integration/targets/connection_paramiko_ssh/test_connection.inventory',), 'listhosts': False, 'subset': None, 'extra_vars': "Not saved by ARA as configured by 'ignored_arguments'", 'vault_ids': (), 'ask_vault_pass': False, 'vault_password_files': (), 'forks': 5, 'module_path': None, 'listtasks': False, 'listtags': False, 'step': False, 'start_at_task': None, 'args': ('test_connection.yml',)}, 'status': 'running', 'path': '/root/.ansible/test/tmp/connection_paramiko_ssh-dd2zu_78-\udcc3\udc85\udcc3\udc91\udcc5\udc9a\udcc3\udc8c\udcce\udcb2\udcc5\udc81\udcc3\udc88/test/integration/targets/connection/test_connection.yml'}
 [WARNING]: Failure using method (v2_playbook_on_start) in callback plugin
(<ansible.plugins.callback./usr/local/lib/python3.6/dist-
packages/ara/plugins/callback/ara_default.CallbackModule object at
0x7f0c8a53f048>): Expecting value: line 1 column 1 (char 0)


PLAY [paramiko_ssh] ************************************************************
 [WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin
(<ansible.plugins.callback./usr/local/lib/python3.6/dist-
packages/ara/plugins/callback/ara_default.CallbackModule object at
0x7f0c8a53f048>): 'ascii' codec can't encode characters in position 57-63:
ordinal not in range(128)
2019-05-23 23:07:33,671 ERROR django.request: Internal Server Error: /api/v1/playbooks
Traceback (most recent call last):
  File "/root/virtualenv/lib64/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/viewsets.py", line 116, in view
    return self.dispatch(request, *args, **kwargs)
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/mixins.py", line 26, in perform_create
    serializer.save()
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/serializers.py", line 214, in save
    self.instance = self.create(validated_data)
  File "/root/virtualenv/lib64/python3.7/site-packages/rest_framework/serializers.py", line 943, in create
    instance = ModelClass._default_manager.create(**validated_data)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/query.py", line 422, in create
    obj.save(force_insert=True, using=self.db)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/base.py", line 741, in save
    force_update=force_update, update_fields=update_fields)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/base.py", line 779, in save_base
    force_update, using, update_fields,
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/base.py", line 870, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/base.py", line 908, in _do_insert
    using=using, raw=raw)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/models/sql/compiler.py", line 1335, in execute_sql
    cursor.execute(sql, params)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/root/virtualenv/lib64/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 383, in execute
    return Database.Cursor.execute(self, query, params)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 48-61: surrogates not allowed
[23/May/2019 23:07:33] "POST /api/v1/playbooks HTTP/1.1" 500 27

Add documention for using ARA with AWX/Tower

ARA and AWX/Tower aren't mutually exclusive. They can be used individually or together depending on the user's needs and use cases.

In theory, we would be able to install ara in the default virtualenv but that doesn't work and is documented in ansible/awx#1737

What had worked for me in the past would be to create a new virtualenv, install ara and ansible in it and then make awx/tower use that virtualenv for job templates after setting the callback plugin in the settings interface.

Screencaps from an Ansible meetup presentation:
Screenshot from 2019-04-24 10-13-23

...

Screenshot from 2019-04-24 10-13-32

An Ansible collection for ARA

Ansible is moving forward with an approach called Ansible collections, see "The Future of Ansible Content Delivery".

ARA has a lot of Ansible content and it might be worthwhile to consider packaging everything in an Ansible collection:

Docs:

Let's use this issue to discuss how we might do this.

Differentiate between a run that is incomplete vs in-progress

Migrated from https://storyboard.openstack.org/#!/story/2000852

When browsing ara, a playbook run that hasn't been completed will show as incomplete.
This is the current behavior regardless if the playbook execution has been interrupted (i.e, ctrl+c) or if the playbook is in progress and has not yet finished running.

I don't think ara has a reliable way to tell if the playbook has been interrupted or if it's just waiting for the next task to be recorded.
We could perhaps set some sort of timeout where a playbook will show as incomplete if a task hasn't been reported in 'n' seconds but I'm not convinced this is a good solution since tasks can take anything from under 1 seconds to hours if users wanted to.

Maybe there is a way, let's try and figure out something.

psycopg2.OperationalError when using ARA 0.x with PostgreSQL

Migrated from https://storyboard.openstack.org/#!/story/2005492

Sergei Volin reported issues trying to run ARA with PostgreSQL:

ansible@debian9:~/ansible/out$ grep OperationalError ansible.log | cat -n
1 2019-04-18 18:13:22,641 p=50426 u=uansible | fatal: [l1]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
2 (psycopg2.OperationalError) SSL error: decryption failed or bad record mac [SQL: SELECT tasks.id AS
3 2019-04-18 18:13:25,421 p=50426 u=uansible | fatal: [l2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'0d78ff1d-c3f0-454a-9b46-2e24fa39accb', 'key_1': u'l2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
4 object at 0x7f09731a1950>): (psycopg2.OperationalError) server closed the connection unexpectedly This
5 2019-04-18 18:41:04,521 p=50665 u=uansible | fatal: [l2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'f0f32eab-8a2c-40e3-98d0-d77a54111bad', 'key_1': u'l2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
6 object at 0x7ff8c514d950>): (psycopg2.OperationalError) server closed the connection unexpectedly This
7 2019-04-18 18:56:18,571 p=50898 u=uansible | fatal: [l2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'bfd7130b-76e6-4140-8b31-7e628da58612', 'key_1': u'l2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
8 object at 0x7f66406e0950>): (psycopg2.OperationalError) server closed the connection unexpectedly This
9 2019-04-18 18:57:30,896 p=51013 u=uansible | fatal: [l2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'5b626cec-08a9-4444-b457-336385796dfd', 'key_1': u'l2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
10 object at 0x7f247c0e6950>): (psycopg2.OperationalError) server closed the connection unexpectedly This
11 2019-04-18 18:59:29,083 p=51149 u=uansible | fatal: [l2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'd6f6ab7b-5380-4bac-b6fa-a61531b35c1a', 'key_1': u'l2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
12 object at 0x7f06ce8e8950>): (psycopg2.OperationalError) server closed the connection unexpectedly This
13 2019-04-18 18:59:57,440 p=51262 u=uansible | fatal: [l2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'7f576582-7da8-41cd-95a3-5e4eaf4036a0', 'key_1': u'l2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
14 2019-04-18 19:08:49,911 p=52450 u=uansible | fatal: [j1]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'252b17ff-253c-4a6e-8c46-d9cf49e58f72', 'key_1': u'j1_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
15 object at 0x7f88d830f5d0>): (psycopg2.OperationalError) server closed the connection unexpectedly This
16 2019-04-18 19:08:50,081 p=52450 u=uansible | fatal: [j2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: INSERT INTO data (id, playbook_id, key, value, type) VALUES (%(id)s, %(playbook_id)s, %(key)s, %(value)s, %(type)s)]\n[parameters: {'type': u'json', 'value': <psycopg2.extensions.Binary object at 0x7f88cdc87198>, 'playbook_id': u'252b17ff-253c-4a6e-8c46-d9cf49e58f72', 'id': 'ce10c92b-e944-41de-bf07-1094586ac35f', 'key': u'j2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
17 2019-04-18 19:42:04,311 p=52990 u=uansible | fatal: [j1]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'24645bcc-8a4f-43f1-8788-fdcd529ad62a', 'key_1': u'j1_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
18 object at 0x7f2806dacf90>): (psycopg2.OperationalError) SSL SYSCALL error: EOF detected [SQL: SELECT tasks.id AS
19 2019-04-18 19:42:04,387 p=52990 u=uansible | fatal: [j2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'24645bcc-8a4f-43f1-8788-fdcd529ad62a', 'key_1': u'j2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
20 2019-04-18 19:55:39,624 p=53357 u=uansible | fatal: [j2]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) server closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n\n[SQL: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type \nFROM data \nWHERE data.key = %(key_1)s AND data.playbook_id = %(playbook_id_1)s]\n[parameters: {'playbook_id_1': u'7b398a6e-0eb0-4f2a-8a3e-5154f564ec57', 'key_1': u'j2_facts'}]\n(Background on this error at: http://sqlalche.me/e/e3q8)"}
21 object at 0x7f392dd54f90>): (psycopg2.OperationalError) SSL SYSCALL error: EOF detected [SQL: SELECT tasks.id AS
22 2019-04-18 19:55:39,727 p=53357 u=uansible | fatal: [j1]: FAILED! => {"changed": false, "msg": "Data not recorded in ARA: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n\n[SQL: INSERT INTO data (id, playbook_id, key, value, type) VALUES (%(id)s, %(playbook_id)s, %(key)s, %(value)s, %(type)s)]\n[parameters: {'type': u'json', 'value': <psycopg2.extensions.Binary object at 0x7f391f713350>, 'playbook_id': u'7b398a6e-0eb0-4f2a-8a3e-5154f564ec5

Excerpt from posgresql-9.6-main.log file:

2019-04-18 18:13:20.317 MSK [50433] ara@ara ::1(48976)LOG: duration: 0.283 ms statement: SELECT data.id AS data_id, data.playbook_id AS data_playbook_id, data.key AS data_key, data.value AS data_value, data.type AS data_type FROM data WHERE data.key = 'l1_facts' AND data.playbook_id = '0d78ff1d-c3f0-454a-9b46-2e24fa39accb'
2019-04-18 18:13:20.322 MSK [50433] ara@ara ::1(48976)LOG: duration: 0.484 ms statement: INSERT INTO data (id, playbook_id, key, value, type) VALUES ('ff50ead9-79c3-42e1-8d5a-5b9939572f5e', '0d78ff1d-c3f0-454a-9b46-2e24fa39accb', 'l1_facts', '\x789ced5b796fda4abbff<...>'::bytea, 'json')
2019-04-18 18:13:20.343 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.051 ms statement: BEGIN
2019-04-18 18:13:20.343 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.445 ms statement: SELECT t.oid, typarray FROM pg_type t JOIN pg_namespace ns ON typnamespace = ns.oid WHERE typname = 'hstore';
2019-04-18 18:13:20.343 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.011 ms statement: ROLLBACK
2019-04-18 18:13:20.344 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.007 ms statement: BEGIN
2019-04-18 18:13:20.344 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.052 ms statement: select version()
2019-04-18 18:13:20.344 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.045 ms statement: select current_schema()
2019-04-18 18:13:20.345 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.016 ms statement: show transaction isolation level
2019-04-18 18:13:20.345 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.070 ms statement: SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
2019-04-18 18:13:20.346 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.029 ms statement: SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
2019-04-18 18:13:20.346 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.027 ms statement: SELECT 'x' AS some_label
2019-04-18 18:13:20.346 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.009 ms statement: ROLLBACK
2019-04-18 18:13:20.346 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.007 ms statement: BEGIN
2019-04-18 18:13:20.346 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.013 ms statement: show standard_conforming_strings
2019-04-18 18:13:20.346 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.008 ms statement: ROLLBACK
2019-04-18 18:13:20.347 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.007 ms statement: BEGIN
2019-04-18 18:13:20.347 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.361 ms statement: SELECT c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = 'public' AND c.relkind in ('r', 'p')
2019-04-18 18:13:20.347 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.011 ms statement: ROLLBACK
2019-04-18 18:13:20.350 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.033 ms statement: BEGIN
2019-04-18 18:13:20.350 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.167 ms statement: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname='alembic_version'
2019-04-18 18:13:20.350 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.094 ms statement: SELECT alembic_version.version_num FROM alembic_version
2019-04-18 18:13:20.351 MSK [50507] ara@ara ::1(48986)LOG: duration: 0.012 ms statement: ROLLBACK
2019-04-18 18:13:22.633 MSK [50433] ara@ara ::1(48976)LOG: duration: 2311.093 ms statement: COMMIT
2019-04-18 18:13:22.633 MSK [50433] ara@ara ::1(48976)LOG: SSL error: decryption failed or bad record mac
2019-04-18 18:13:22.633 MSK [50433] ara@ara ::1(48976)LOG: could not receive data from client: Connection reset by peer
2019-04-18 18:13:22.649 MSK [50506] ara@ara ::1(48984)LOG: could not receive data from client: Connection reset by peer
2019-04-18 18:13:22.665 MSK [50507] ara@ara ::1(48986)LOG: could not receive data from client: Connection reset by peer

Installed packages:

user@debian9:/var/log/postgresql# pip freeze
alembic==1.0.8
ansible==2.7.6
ansible-cmdb==1.30
ara==0.16.3
Babel==2.6.0
beautifulsoup4==4.5.3
chardet==2.3.0
chrome-gnome-shell==0.0.0
Click==7.0
cliff==2.14.1
cmd2==0.8.9
contextlib2==0.5.5
cryptography==1.7.1
debtcollector==1.21.0
enum34==1.1.6
extras==1.0.0
fixtures==3.0.0
Flask==1.0.2
Flask-Migrate==2.4.0
Flask-Script==2.0.6
Flask-SQLAlchemy==2.3.2
Frozen-Flask==0.15
funcsigs==1.0.2
html5lib==0.999999999
httplib2==0.9.2
idna==2.2
ipaddress==1.0.17
iso8601==0.1.12
itsdangerous==1.1.0
Jinja2==2.10
jmespath==0.9.0
jsonxs==0.6
junit-xml==1.8
junos-eznc==2.2.0
jxmlease==1.0.1
keyring==10.1
keyrings.alt==1.3
linecache2==1.0.0
lxml==3.7.1
Mako==1.0.7
MarkupSafe==0.23
monotonic==1.5
msgpack==0.6.1
ncclient==0.6.3
netifaces==0.10.9
ntlm-auth==1.2.0
numpy==1.12.1
oslo.i18n==3.23.1
oslo.serialization==2.28.2
oslo.utils==3.40.3
paramiko==2.0.0
pbr==5.1.3
prettytable==0.7.2
psycopg2==2.7.7
pyasn1==0.1.9
pycrypto==2.6.1
pyfakefs==3.5.8
Pygments==2.3.1
pygobject==3.22.0
pyOpenSSL==16.2.0
pyparsing==2.3.1
pyperclip==1.7.0
pyserial==3.4
python-dateutil==2.8.0
python-editor==1.0.4
python-mimeparse==1.6.0
python-subunit==1.3.0
pytz==2018.9
pywinrm==0.3.0
pyxdg==0.25
PyYAML==3.12
requests==2.12.4
requests-ntlm==1.1.0
scour==0.32
scp==0.13.0
SecretStorage==2.3.1
selectors2==2.0.1
simplejson==3.16.1
six==1.10.0
SQLAlchemy==1.3.1
stevedore==1.30.1
subprocess32==3.5.3
testtools==2.3.0
traceback2==1.4.0
unicodecsv==0.14.1
unittest2==1.1.0
urllib3==1.19.1
ushlex==0.99.1
wcwidth==0.1.7
webencodings==0.5
Werkzeug==0.14.1
wrapt==1.11.1
XStatic==1.0.2
XStatic-Bootstrap-SCSS==3.3.7.1
XStatic-DataTables==1.10.15.1
XStatic-jQuery==3.3.1.1
XStatic-Patternfly==3.21.0.1
XStatic-Patternfly-Bootstrap-Treeview==2.1.3.2
ansible@debian9:~/ansible/out$ python
Python 2.7.13 (default, Sep 26 2018, 18:42:22)
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.version
'2.6.0'
>>> sqlite3.sqlite_version
'3.16.2'

A non-django way of modifying settings.py

While we can modify settings.py through a module on the python path ( as described : https://docs.djangoproject.com/en/dev/topics/settings/), I don't beleive ARA is aimed as standalone server (that happens to use Django).

Would it make sense to have a CLI or ENV variable type configuration that is more "simplistic". Or if there is and I missed it? point it out.

I encountered this while trying to create separate server and client - I needed to provide ALLOWED_HOSTS. While possible, I felt this was alot of python configuration ( not code ) that had to be understood.

API: Add support for sorting results

What component is this about ?

API, API clients

What is the idea ?

When querying the API, we should be able to request a sorted list of results based on different fields, for example:

  • Sort by start date
  • Sort by end date
  • Sort by duration
  • Sort by status

django-rest-framework provides filtering and ordering (docs) but we should also consider whether or not we need to implement things at the django model level like suggested in #66.

Wrong playbook execution time

I have installed ARA using the playbooks provided and everything is fine, except that the execution time is wrong. I have added TIME_ZONE: Europe/Paris in ARA API settings.yaml and it seems that the start time taken into account is in GMT whereas the end time is in local time. ARA API server and the computer playing the playbook are set up in the same french timezone.
Screenshot_2019-09-06 ARA Records Ansible
Screenshot_2019-09-06 https ara-api ecritel net

slow loading of tasks

Hi

When I click on a TASK and then onto an entry on the "status" column, it takes ages to load. I have the same setup on an older version of ARA and its much faster.

Database in stored in AWS RDS.

Versions.
ara (0.16.4)
ansible (2.5.5)

Move default configuration template generation out of settings.py

The configuration template is currently created out of settings.py:

ara/ara/server/settings.py

Lines 202 to 234 in d56c68f

# TODO: Split this out to a CLI command (django-admin command ?)
# Ensure default base configuration/data directory exists
if not os.path.isdir(BASE_DIR):
logger.info(f"Creating data & configuration directory: {BASE_DIR}")
os.makedirs(BASE_DIR, mode=0o700)
if not os.path.exists(DEFAULT_SETTINGS) and "ARA_SETTINGS" not in os.environ:
SETTINGS = dict(
BASE_DIR=BASE_DIR,
ALLOWED_HOSTS=ALLOWED_HOSTS,
CORS_ORIGIN_WHITELIST=CORS_ORIGIN_WHITELIST,
CORS_ORIGIN_ALLOW_ALL=CORS_ORIGIN_ALLOW_ALL,
SECRET_KEY=SECRET_KEY,
DATABASES=DATABASES,
DEBUG=DEBUG,
LOG_LEVEL=LOG_LEVEL,
LOGGING=LOGGING,
READ_LOGIN_REQUIRED=READ_LOGIN_REQUIRED,
WRITE_LOGIN_REQUIRED=WRITE_LOGIN_REQUIRED,
)
with open(DEFAULT_SETTINGS, "w+") as settings_file:
comment = f"""
---
# This is a default settings template generated by ARA.
# To use a settings file such as this one, you need to export the
# ARA_SETTINGS environment variable like so:
# $ export ARA_SETTINGS="{DEFAULT_SETTINGS}"
"""
logger.info(f"Writing default settings to {DEFAULT_SETTINGS}")
settings_file.write(textwrap.dedent(comment))
yaml.dump({"default": SETTINGS}, settings_file, default_flow_style=False)

This feature is useful but would be better suited as a CLI command so, amongst other things, it doesn't run every time settings.py is loaded.

Document a way to use ARA with Travis or other non persistent CI systems

As probably anyone knowns, Travis does not have any embedded support to store build files, its only output is the build log. Users are expected to upload files to other systems during the build process, most of them do not do any such activity.

As use of Ansible on Travis got more popular it would be really useful to document a solution for making ARA reports available for builds made on such a limited CI. Having this on a sample git repository it would be great as people could look at how is implemented and replicate it to their own repos.

I am inclined to believe that the static approach is more likely to work because the database approach does not rule-out the need to a web-server, making the setup even more inconvenient. Maybe AWS S3 upload could be found ok?

zsh: no matches found: ara[server]

What component is this about ?

Not sure. Am just trying to install ara.

What is your ARA installation like ?

Currently not successfully installed.

Am trying to install ARA on Ubuntu 18.04.

What is happening ?

I ran the following:

python3 -m venv ~/.ara/virtualenv
source ~/.ara/virtualenv/bin/activate
pip install ansible ara[server]

Which resulted in no matches found: ara[server].

What should be happening ?

Not sure, but I'm guessing not what did happen.

Ansible 2.8: Regression because of new sentinel class representing None or []

Example trace from ara 1.0:

[WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible.plugins.callback./home/dmsimard/dev/git/ansible-community/ara/ara/plugins/callback/ara_default.CallbackModule object at 0x7fb58bb36a90>): Object of type type is not
JSON serializable

Callback Exception: 
  File "/home/dmsimard/dev/git/ansible-community/ara/.tox/ansible-integration/lib/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 333, in send_callback
    method(*new_args, **kwargs)
   File "/home/dmsimard/dev/git/ansible-community/ara/ara/plugins/callback/ara_default.py", line 232, in v2_playbook_on_task_start
    handler=handler,
   File "/home/dmsimard/dev/git/ansible-community/ara/ara/clients/http.py", line 104, in post
    return self._request("post", endpoint, **kwargs)
   File "/home/dmsimard/dev/git/ansible-community/ara/ara/clients/http.py", line 82, in _request
    response = func(url, **kwargs)
   File "/home/dmsimard/dev/git/ansible-community/ara/ara/clients/http.py", line 63, in post
    return self._request("post", url, data=json.dumps(payload))
   File "/usr/lib64/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
   File "/usr/lib64/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
   File "/usr/lib64/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
   File "/usr/lib64/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '

Or screenshot from 0.x:
Screenshot

This is because the structure of the task object has changed.

In 2.7:

(Pdb) task._attributes
{'args': {'name': '{{ item }}', 'ansible_host': '127.0.0.1', 'ansible_connection': 'local'}, 'action': 'add_host', 'async_val': 0, 'changed_when': [], 'delay': 5, 'delegate_to': None, 'delegate_facts': None, 'failed_when': [], 'loop': ['host1', 'host2', 'host3'], 'loop_control': None, 'notify': None, 'poll': 10, 'register': None, 'retries': 3, 'until': [], 'loop_with': 'items', 'name': 'Add fake hosts in inventory', 'connection': None, 'port': None, 'remote_user': None, 'vars': {}, 'module_defaults': None, 'environment': None, 'no_log': None, 'run_once': None, 'ignore_errors': None, 'ignore_unreachable': None, 'check_mode': None, 'diff': None, 'any_errors_fatal': None, 'debugger': None, 'when': [], 'tags': [], 'become': None, 'become_method': None, 'become_user': None, 'become_flags': None}
(Pdb) task._attributes["tags"]
[]

In 2.8 (notice all the ansible.utils.sentinel.Sentinel):

(Pdb) task._attributes
{'args': {'name': '{{ item }}', 'ansible_host': '127.0.0.1', 'ansible_connection': 'local'}, 'action': 'add_host', 'async_val': <class 'ansible.utils.sentinel.Sentinel'>, 'changed_when': <class 'ansible.utils.sentinel.Sentinel'>, 'delay': <class 'ansible.utils.sentinel.Sentinel'>, 'delegate_to': <class 'ansible.utils.sentinel.Sentinel'>, 'delegate_facts': <class 'ansible.utils.sentinel.Sentinel'>, 'failed_when': <class 'ansible.utils.sentinel.Sentinel'>, 'loop': ['host1', 'host2', 'host3'], 'loop_control': <class 'ansible.utils.sentinel.Sentinel'>, 'notify': <class 'ansible.utils.sentinel.Sentinel'>, 'poll': <class 'ansible.utils.sentinel.Sentinel'>, 'register': <class 'ansible.utils.sentinel.Sentinel'>, 'retries': <class 'ansible.utils.sentinel.Sentinel'>, 'until': <class 'ansible.utils.sentinel.Sentinel'>, 'loop_with': 'items', 'name': 'Add fake hosts in inventory', 'connection': <class 'ansible.utils.sentinel.Sentinel'>, 'port': <class 'ansible.utils.sentinel.Sentinel'>, 'remote_user': <class 'ansible.utils.sentinel.Sentinel'>, 'vars': {}, 'module_defaults': <class 'ansible.utils.sentinel.Sentinel'>, 'environment': <class 'ansible.utils.sentinel.Sentinel'>, 'no_log': <class 'ansible.utils.sentinel.Sentinel'>, 'run_once': <class 'ansible.utils.sentinel.Sentinel'>, 'ignore_errors': <class 'ansible.utils.sentinel.Sentinel'>, 'ignore_unreachable': <class 'ansible.utils.sentinel.Sentinel'>, 'check_mode': <class 'ansible.utils.sentinel.Sentinel'>, 'diff': <class 'ansible.utils.sentinel.Sentinel'>, 'any_errors_fatal': <class 'ansible.utils.sentinel.Sentinel'>, 'debugger': <class 'ansible.utils.sentinel.Sentinel'>, 'when': <class 'ansible.utils.sentinel.Sentinel'>, 'tags': <class 'ansible.utils.sentinel.Sentinel'>, 'become': <class 'ansible.utils.sentinel.Sentinel'>, 'become_method': <class 'ansible.utils.sentinel.Sentinel'>, 'become_user': <class 'ansible.utils.sentinel.Sentinel'>, 'become_flags': <class 'ansible.utils.sentinel.Sentinel'>, 'collections': <class 'ansible.utils.sentinel.Sentinel'>}
(Pdb) task._attributes["tags"]
<class 'ansible.utils.sentinel.Sentinel'>

After discussing this with upstream in #ansible-devel, it turns out that this new Sentinel object is used to indicate that something has not been set (to allow a distinction between unset and set to None).

We can use something like from ansible.utils.sentinel import Sentinel; if foo is Sentinel to determine if the value is set and otherwise coerce to a value we can send to the database like None or an empty list, for example.

ara_record stucks with MariaDB and Postgresql

ara_record stucks with MariaDB and Postgresql when running playbook for a few inventory hosts and fork>=2
With fork = 1 ara_record works just fine.
Normaly a user wants this parameter be much >>1!!
So putting it to 1 can not be regarded as a solution. It's only an indication of a problem with DB more mature than SQLite.
It demands more testing of ARA with other DB.

Ara v1.0 feaure list

Hi

Ara v1.0 is not feature complete yet

Do you have a list of features that work in Ara v1.0 as compared to the latest v0.xx ?

What features are still missing from Ara v1.0?

pytz.exceptions.UnknownTimeZoneError: None

Problem

Running ara on Ubuntu 18.04 fails with nginx and gunicorn.
When opening the URL via nginx, I get a Internal Server Error and the following exception in the log: https://gist.github.com/rndmh3ro/2117d0d5dc286f38fe849788709ad42e

However when opening ara on port 3000 it works.

Installed with the playbook:

- name: Install ARA with default settings
  hosts: all
  gather_facts: yes
  vars:
    ara_api_frontend_server: nginx
    ara_api_wsgi_server: gunicorn
    ara_api_fqdn: 172.24.126.194
    ara_api_allowed_hosts:
      - localhost
  roles:
    - ara_api
    - ara_web

Resolution

I don't know. I tried fiddling with the TIME_ZONE parameter but that didn't help.

Additional data

ARA-release: 1.0.0b1

OS:

VERSION="18.04.2 LTS (Bionic Beaver)"

Timezone:

(virtualenv) ubuntu:~/ara-1.0.0b1/playbooks$ cat /etc/timezone
Europe/Berlin

Running ara_api playbook fails on install_ara

Problem

Installing ara on Ubuntu 18.04 with:

> ubuntu:~/ara-1.0.0b1/playbooks$ cat hosts
127.0.0.1
> ansible-playbook -i hosts ara_web.yaml -K

fails with:

TASK [ara_api : Prepare git repository for ara] **********************************************************************
changed: [127.0.0.1]

TASK [ara_api : Install ara] *****************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Failed to find required executable virtualenv in paths: /usr
/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"}
        to retry, use: --limit @/home/basti/ara-1.0.0b1/playbooks/ara_api.retry

PLAY RECAP ***********************************************************************************************************
127.0.0.1                  : ok=12   changed=5    unreachable=0    failed=1

Resolution

Change the install ara task to look like this.
Since Python 3 is required for ara and on Ubuntu the binary virtualenv does not exist, I guess this should be fine. However I did not test in on other operating systems. I can provide a PR for it.

- name: Install ara
  pip:
    name: "{{ ara_api_source_checkout }}[server]"
    state: present
    virtualenv: "{{ ara_api_venv | bool | ternary(ara_api_venv_path, omit) }}"
    #virtualenv_python: python3
    virtualenv_command: "/usr/bin/python3 -m venv"

Additional data

ARA-release: 1.0.0b1

OS:

VERSION="18.04.2 LTS (Bionic Beaver)"

Python:

ubuntu:~/ara-1.0.0b1/playbooks$ python --version
Python 2.7.15rc1
ubuntu:~/ara-1.0.0b1/playbooks$ which python
/usr/bin/python

ansible_user undefined variable on gunicorn-task

Problem

Installing ara on Ubuntu 18.04 with:

> ubuntu:~/ara-1.0.0b1/playbooks$ cat hosts
127.0.0.1
> ansible-playbook -i hosts ara_web.yaml -K

fails with:

TASK [ara_api : Install gunicorn] ************************************************************************************
ok: [127.0.0.1]

TASK [ara_api : Set up systemd unit file for gunicorn to run the ARA API] ********************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "AnsibleUndefinedVariable: 'ansible_user' is undefined"}

PLAY RECAP ***********************************************************************************************************
127.0.0.1                  : ok=25   changed=1    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0

Resolution

One also could either define ansible_user as a variable or use ansible_user_id in the gunicorn service-template.

Additional data

ARA-release: 1.0.0b1

OS:

VERSION="18.04.2 LTS (Bionic Beaver)"

Python:

ubuntu:~/ara-1.0.0b1/playbooks$ python --version
Python 2.7.15rc1
ubuntu:~/ara-1.0.0b1/playbooks$ which python
/usr/bin/python

Internal Server Error in browser

I often get the error in the browser "Internal Server Error". If I view the other pages in the browser, they load fine, so its only 1 page that has the issue. e.g. https://ara/reports/list/4.html

[ root@ara-20:~]$ pip list
ansible (2.3.2.0)
ara (0.13.0)

Should i just upgrade ARA version, or is this a known issue?

The error logs show

[2019-05-09 13:20:24,624] ERROR in app: Exception on /reports/ [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/site-packages/ara/views/reports.py", line 66, in report_list
    stats=stats)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 134, in render_template
    context, ctx.app)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 116, in _render
    rv = template.render(context)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/ara/templates/report_list.html", line 1, in top-level template code
    {% extends "layout.html" %}
  File "/usr/local/lib/python2.7/site-packages/ara/templates/layout.html", line 6, in top-level template code
    {% block content -%}
  File "/usr/local/lib/python2.7/site-packages/ara/templates/report_list.html", line 10, in block "content"
    {% include "report.html" -%}
  File "/usr/local/lib/python2.7/site-packages/ara/templates/report.html", line 22, in top-level template code
    <a href="#" data-toggle="modal" data-target="#file_modal" data-load="{{ playbook.file.id }}/" title="{{ playbook.path }}">{{ playbook.path | basename }}</a>
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 430, in getattr
    return getattr(obj, attribute)
  File "/usr/local/lib/python2.7/site-packages/ara/models.py", line 195, in file
    .filter(File.is_playbook)).one()
  File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2953, in one
    raise orm_exc.NoResultFound("No row was found for one()")
NoResultFound: No row was found for one()
[pid: 20661|app: 0|req: 228290/228300] 10.6.0.18 () {54 vars in 892 bytes} [Thu May  9 13:20:24 2019] GET /reports/ => generated 291 bytes in 413 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)

Add table type for ara_record

It could be useful to record data with 2 dimensions, using either csv, or json with a list of homogeneous records.
It can then be displayed using the same UI as host table, task table, etc, with all existing features like sorting.

The ultimate goal is to display a table from similar data accumulated from different hosts. For example, show uptime and free memory for each hosts in a single table, that can be sorted and filtered.
In an ideal world, ara_record will be called on each host, and the module should be able to aggregate those records. This seems to require new parameters for the module, such as append_to_table along with a new type.
But as a start, there could be an simple table type, which accepts a 2 dimensional list. The first record are the headers, and the other one are the data.

This basic start will give the opportunity to test the rendering of that table.

I don't really expect to have a fully compliant CSV renderer, except if some data are generated by a program itself, which is rare.

Add parameter to ignore file path patterns to avoid saving matched files

The callback will retrieve files that are loaded by Ansible throughout the execution of the playbook. This could contain files that users might not want to save, either because they are sensitive or are simply not relevant.

The implementation would be somewhat similar to that we already do with ignored_arguments and ignored_facts for ignoring specified playbook parameters and host facts.

Cannot get it running on Ubuntu 18.04 LTS

Hi,

Ubuntu 18.04 LTS comes with Python 3 by default but Ara has problems. Please see below.

dincer@bozhane:~$ python3 -m ara.setup.action_plugins
/usr/local/lib/python3.6/dist-packages/ara/plugins/action

dincer@bozhane:~$ python3 -m ara.setup.callback_plugins
/usr/local/lib/python3.6/dist-packages/ara/plugins/callback

dincer@bozhane:~$ grep plugins /etc/ansible/ansible.cfg 
## Ansible ships with some plugins that require whitelisting,
## Custom plugins should not need this unless plugin author specifies it.
# enable callback plugins, they can output to stdout but cannot be 'stdout' type.
action_plugins     = /usr/local/lib/python3.6/dist-packages/ara/plugins/action
#become_plugins     = /usr/share/ansible/plugins/become
#cache_plugins      = /usr/share/ansible/plugins/cache
callback_plugins   = /usr/local/lib/python3.6/dist-packages/ara/plugins/callback
#connection_plugins = /usr/share/ansible/plugins/connection
#lookup_plugins     = /usr/share/ansible/plugins/lookup
#inventory_plugins  = /usr/share/ansible/plugins/inventory
#vars_plugins       = /usr/share/ansible/plugins/vars
#filter_plugins     = /usr/share/ansible/plugins/filter
#test_plugins       = /usr/share/ansible/plugins/test
#terminal_plugins   = /usr/share/ansible/plugins/terminal
#strategy_plugins   = /usr/share/ansible/plugins/strategy
# enable inventory plugins, default: 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml'
#enable_plugins = host_list, virtualbox, yaml, constructed

dincer@bozhane:~$ ansible localhost -m ping
 [WARNING]: Skipping plugin (/usr/local/lib/python3.6/dist-packages/ara/plugins/callback/ara_default.py) as it seems to be invalid: No module named ara.clients

localhost | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

dincer@bozhane:~$ ara-manage runserver
Performing system checks...

Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x7f8bddbe9598>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/utils/autoreload.py", line 228, in wrapper
    fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/core/management/commands/runserver.py", line 124, in inner_run
    self.check(display_num_errors=True)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 359, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 346, in _run_checks
    return checks.run_checks(**kwargs)
  File "/usr/lib/python3/dist-packages/django/core/checks/registry.py", line 81, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/lib/python3/dist-packages/django/core/checks/urls.py", line 16, in check_url_config
    return check_resolver(resolver)
  File "/usr/lib/python3/dist-packages/django/core/checks/urls.py", line 26, in check_resolver
    return check_method()
  File "/usr/lib/python3/dist-packages/django/urls/resolvers.py", line 256, in check
    for pattern in self.url_patterns:
  File "/usr/lib/python3/dist-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3/dist-packages/django/urls/resolvers.py", line 407, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/lib/python3/dist-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3/dist-packages/django/urls/resolvers.py", line 400, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/dist-packages/ara/server/urls.py", line 22, in <module>
    from django.urls import include, path
ImportError: cannot import name 'include'

pep517 makes tox confused when both pyproject.toml and setup.py are present

Got this error today:

$ tox -e ansible-integration
ansible-integration create: /home/dmsimard/dev/git/ansible-community/ara/.tox/ansible-integration
ansible-integration installdeps: ansible
ansible-integration develop-inst: /home/dmsimard/dev/git/ansible-community/ara
ERROR: invocation failed (exit code 1), logfile: /home/dmsimard/dev/git/ansible-community/ara/.tox/ansible-integration/log/ansible-integration-2.log
ERROR: actionid: ansible-integration
msg: developpkg
cmdargs: ['/home/dmsimard/dev/git/ansible-community/ara/.tox/ansible-integration/bin/pip', 'install', '-U', '-e', '/home/dmsimard/dev/git/ansible-community/ara', '-c/dev/null']

Obtaining file:///home/dmsimard/dev/git/ansible-community/ara
ERROR: Error installing 'file:///home/dmsimard/dev/git/ansible-community/ara': editable mode is not supported for pyproject.toml-style projects. pip is processing this project as pyproject.toml-style because it has a pyproject.toml file. Since the project has a setup.py and the pyproject.toml has no "build-backend" key for the "build_system" value, you may pass --no-use-pep517 to opt out of pyproject.toml-style processing. See PEP 517 for details on pyproject.toml-style projects.

Add support for API authentication in the clients

The API can require authentication for reads and writes and the client must be able to authenticate against the API.

We will need to do the plumbing so that the different components are able to authenticate:

This work is in progress in https://review.openstack.org/#/c/651569/

ara_api role on fedora could not detect package manager

The use of of auto make it bug on my fedora 30
TASK [ara_api : Get list of installed packages] **********************************************************************************************************************************************************************************************
task path: /home/YC588F8N/ara/ara-feature-1.0/roles/ara_api/tasks/pre-requirements.yaml:32
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: YC588F8N
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /tmp/ansible-tmp-1558101442.807949-53175802335807" && echo ansible-tmp-1558101442.807949-53175802335807="echo /tmp/ansible-tmp-1558101442.807949-53175802335807" ) && sleep 0'
Using module file /home/X/.ara/virtualenv/lib64/python3.7/site-packages/ansible/modules/packaging/os/package_facts.py
<127.0.0.1> PUT /home/X/.ansible/tmp/ansible-local-4634b0t1_ln9/tmprp6ve_5_ TO /tmp/ansible-tmp-1558101442.807949-53175802335807/AnsiballZ_package_facts.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-tmp-1558101442.807949-53175802335807/ /tmp/ansible-tmp-1558101442.807949-53175802335807/AnsiballZ_package_facts.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/X/.ara/virtualenv/bin/python3 /tmp/ansible-tmp-1558101442.807949-53175802335807/AnsiballZ_package_facts.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-tmp-1558101442.807949-53175802335807/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"manager": [
"auto"
],
"strategy": "first"
}
},
"msg": "Could not detect a supported package manager from the following list: ['portage', 'pkg', 'apt', 'rpm']"
}

Add default callback and ara_record documentation to the docs

ansible-doc -t callback ara_default works because we have documentation in the callback plugin:

DOCUMENTATION = """
callback: ara
callback_type: notification
requirements:
- ara
short_description: Sends playbook execution data to the ARA API internally or over HTTP
description:
- Sends playbook execution data to the ARA API internally or over HTTP
options:
api_client:
description: The client to use for communicating with the API
default: offline
env:
- name: ARA_API_CLIENT
ini:
- section: ara
key: api_client
choices: ['offline', 'http']
api_server:
description: When using the HTTP client, the base URL to the ARA API server
default: http://127.0.0.1:8000
env:
- name: ARA_API_SERVER
ini:
- section: ara
key: api_server
api_timeout:
description: Timeout, in seconds, before giving up on HTTP requests
default: 30
env:
- name: ARA_API_TIMEOUT
ini:
- section: ara
key: api_timeout
ignored_facts:
description: List of host facts that will not be saved by ARA
type: list
default: ["ansible_env"]
env:
- name: ARA_IGNORED_FACTS
ini:
- section: ara
key: ignored_facts
ignored_arguments:
description: List of Ansible arguments that will not be saved by ARA
type: list
default: ["extra_vars"]
env:
- name: ARA_IGNORED_ARGUMENTS
ini:
- section: ara
key: ignored_arguments
"""

Same thing for ara_record:

DOCUMENTATION = """
---
module: ara_record
short_description: Ansible module to record persistent data with ARA.
version_added: "2.0"
author: "David Moreau-Simard <[email protected]>"
description:
- Ansible module to record persistent data with ARA.
options:
playbook_id:
description:
- id of the playbook to write the key to
- if not set, the module will use the ongoing playbook's id
required: false
key:
description:
- Name of the key to write data to
required: true
value:
description:
- Value of the key written to
required: true
type:
description:
- Type of the key
choices: [text, url, json, list, dict]
default: text
requirements:
- "python >= 3.5"
- "ara >= 1.0.0"
"""
EXAMPLES = """
- name: Associate specific data to a key for a playbook
ara_record:
key: "foo"
value: "bar"
- name: Associate data to a playbook that previously ran
ara_record:
playbook_id: 21
key: logs
value: "{{ lookup('file', '/var/log/ansible.log') }}"
type: text
- name: Retrieve the git version of the development repository
shell: cd dev && git rev-parse HEAD
register: git_version
delegate_to: localhost
- name: Record and register the git version of the playbooks
ara_record:
key: "git_version"
value: "{{ git_version.stdout }}"
register: version
- name: Print recorded data
debug:
msg: "{{ version.playbook_id }} - {{ version.key }}: {{ version.value }}
# Write data with a type (otherwise defaults to "text")
# This changes the behavior on how the value is presented in the web interface
- name: Record different formats of things
ara_record:
key: "{{ item.key }}"
value: "{{ item.value }}"
type: "{{ item.type }}"
with_items:
- { key: "log", value: "error", type: "text" }
- { key: "website", value: "http://domain.tld", type: "url" }
- { key: "data", value: "{ 'key': 'value' }", type: "json" }
- { key: "somelist", value: ['one', 'two'], type: "list" }
- { key: "somedict", value: {'key': 'value' }, type: "dict" }
"""
RETURN = """
playbook:
description: ID of the playbook the data was recorded in
returned: on success
type: int
sample: 1
key:
description: Key where the record is saved
returned: on success
type: str
sample: log_url
value:
description: Value of the key
returned: on success
type: complex
sample: http://logurl
type:
description: Type of the key
returned: on success
type: string
sample: url
created:
description: Date the record was created (ISO-8601)
returned: on success
type: str
sample: 2018-11-15T17:27:41.597234Z
updated:
description: Date the record was updated (ISO-8601)
returned: on success
type: str
sample: 2018-11-15T17:27:41.597265Z
"""

However, that doesn't get us pretty rendered pages on ara.readthedocs.io like the ones from upstream Ansible: https://docs.ansible.com/ansible/latest/plugins/callback/slack.html

Upstream uses plugin_formatter.py and the plugin.rst.j2 template to render them.

It is not immediately clear if we can (or should) re-use those as-is but we need docs for the callback and ara_record to be on the docs website.

Add documentation for playbook names

It's possible to give playbook names by supplying the ara_playbook_name Ansible variable.

The intent of the feature is to allow users to give their playbook user friendly names that would display in-lieu of the playbook path by default.
For example, you might have 3 different playbooks named "pre" "run" and "post".

The name is a searchable field in the API and users could therefore search for all the playbooks named "run", for example.

Add postgresql integration tests

We're only testing with sqlite right now which was good enough but we need a minimum of coverage on mysql and postgresql too.

Unable to connect to MySQL with SSL enabled

Unable to connect to MySQL database with SSL enabled.

[exception]
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 684, in _read_packet
packet.check_error()
File "/usr/local/lib/python3.7/site-packages/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python3.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (9002, 'SSL connection is required. Please specify SSL options and retry.\x00')

Added "ssl" parameter to the connection string but it doesn't appear the value is recognized as a dictionary. Tried passing in the database via cfg file and environment variable. Both generate same error message.

[ara]
database = "mysql+pymysql://ara:password@mysqlinstance/ara?ssl={'ca': 'pemfile'}
or
export ARA_DATABASE="mysql+pymysql://ara:password@mysqlinstance/ara?ssl={'ca': 'pemfile'}"

File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 330, in _create_ssl_ctx
ca = sslp.get('ca')
AttributeError: 'str' object has no attribute 'get'

Unable to use the ara 1.x callback under python2.7

What component is this about ?

Ara callback plugin

What is your ARA installation like ?

Ansible 2.7.5
Python 2.7.5
Ara 1.1.0

What is happening ?

If I want to run a playbook with the Ara callback plugin, I get the following error:

Skipping plugin (/usr/lib/python2.7/site-packages/ara/plugins/callback/ara_default.py) as it seems to be invalid: No module named ara.clients

What should be happening ?

Callback plugin should be running and reporting

Slow loading of tasks in ara 0.x web interface

Hi

When I click on a TASK and then onto an entry on the "status" column, it takes ages to load. I have the same setup on an older version of ARA and its much faster.

Database in stored in AWS RDS.

Versions.
ara (0.16.4)
ansible (2.5.5)

Add support for recording Ansible adhoc commands

ansible-playbook commands have always been recorded by ara if the callback plugin was loaded.

By default, callback plugins are not enabled for ansible adhoc commands but they can be enabled by setting bin_ansible_callbacks in ansible.cfg or the ANSIBLE_LOAD_CALLBACK_PLUGINS env variable to True.

However, enabling this at time of writing does not work because there is no v2_playbook_on_start callback hook fired, it goes straight to v2_play_on_start so we have no playbook to tie the data back into:

$ ansible localhost -m setup -vvv
# [...]

 [WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin (<ansible.plugins.callback./home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.CallbackModule object at 0x7f3d283eb0f0>): 'NoneType'
object is not subscriptable

Callback Exception: 
  File "/home/fedora/.ara/virtualenv/lib64/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 375, in send_callback
    method(*new_args, **kwargs)
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 173, in v2_playbook_on_play_start
    "/api/v1/plays", name=play.name, status="running", uuid=play._uuid, playbook=self.playbook["id"]

META: ran handlers
 [WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible.plugins.callback./home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.CallbackModule object at 0x7f3d283eb0f0>): 'NoneType'
object is not subscriptable

Callback Exception: 
  File "/home/fedora/.ara/virtualenv/lib64/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 375, in send_callback
    method(*new_args, **kwargs)
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 194, in v2_playbook_on_task_start
    path = self.playbook["path"]

# [...]
localhost | SUCCESS => {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "10.88.0.1",
            "172.24.0.11"
        ],
       # [...]
        "gather_subset": [
            "all"
        ],
        "module_setup": true
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "fact_path": "/etc/ansible/facts.d",
            "filter": "*",
            "gather_subset": [
                "all"
            ],
            "gather_timeout": 10
        }
    }
}
 [WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback./home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.CallbackModule object at 0x7f3d283eb0f0>): 'NoneType' object is not
subscriptable

Callback Exception: 
  File "/home/fedora/.ara/virtualenv/lib64/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 375, in send_callback
    method(*new_args, **kwargs)
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 216, in v2_runner_on_ok
    self._load_result(result, "ok", **kwargs)
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 288, in _load_result
    host = self._get_or_create_host(result._host.get_name())
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 280, in _get_or_create_host
    return self.client.post("/api/v1/hosts", name=host, alias=host_alias, playbook=self.playbook["id"])

META: ran handlers
META: ran handlers
 [WARNING]: Failure using method (v2_playbook_on_stats) in callback plugin (<ansible.plugins.callback./home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.CallbackModule object at 0x7f3d283eb0f0>): 'NoneType' object
is not subscriptable

Callback Exception: 
  File "/home/fedora/.ara/virtualenv/lib64/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 375, in send_callback
    method(*new_args, **kwargs)
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 231, in v2_playbook_on_stats
    self._load_stats(stats)
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 335, in _load_stats
    host = self._get_or_create_host(hostname)
   File "/home/fedora/.ara/virtualenv/lib/python3.7/site-packages/ara/plugins/callback/ara_default.py", line 280, in _get_or_create_host
    return self.client.post("/api/v1/hosts", name=host, alias=host_alias, playbook=self.playbook["id"])

We need to investigate what's the best course of action to support this.
For example, if there is no playbook, should v2_play_on_start create a "fake" one ?

Provide standalone packages for the API, the clients and the plugins

ara-server, ara-plugins and ara-clients were standalone packages on PyPi before we pulled everything back into a single repository in order to simplify things and reduce the overhead from managing multiple repositories.

If possible, it would be relevant to provide standalone packages so that users don't end up installing things they don't need.

It's worth noting that ARA 1.0 is already far more lightweight than 0.x in terms of dependencies and as a result I don't see this as a blocker right now.

20 packages for 1.0a4:

ara==1.0.0.0a4
certifi==2019.3.9
chardet==3.0.4
Click==7.0
Django==2.2
django-cors-headers==2.5.3
django-filter==2.1.0
djangorestframework==3.9.3
dynaconf==2.0.2
idna==2.8
pbr==5.2.0
python-box==3.4.0
python-dotenv==0.10.1
pytz==2019.1
PyYAML==5.1
requests==2.21.0
sqlparse==0.3.0
toml==0.10.0
urllib3==1.24.2
whitenoise==4.1.2

65 packages for 0.16.3:

alembic==1.0.10
ansible==2.7.10
ara==0.16.3
asn1crypto==0.24.0
attrs==19.1.0
Babel==2.6.0
bcrypt==3.1.6
cffi==1.12.3
Click==7.0
cliff==2.14.1
cmd2==0.9.12
colorama==0.4.1
cryptography==2.6.1
debtcollector==1.21.0
extras==1.0.0
fixtures==3.0.0
Flask==1.0.2
Flask-Migrate==2.4.0
Flask-Script==2.0.6
Flask-SQLAlchemy==2.4.0
Frozen-Flask==0.15
iso8601==0.1.12
itsdangerous==1.1.0
Jinja2==2.10.1
junit-xml==1.8
linecache2==1.0.0
Mako==1.0.9
MarkupSafe==1.1.1
msgpack==0.6.1
netaddr==0.7.19
netifaces==0.10.9
oslo.i18n==3.23.1
oslo.serialization==2.29.0
oslo.utils==3.41.0
paramiko==2.4.2
pbr==5.2.0
prettytable==0.7.2
pyasn1==0.4.5
pycparser==2.19
pyfakefs==3.5.8
Pygments==2.3.1
PyNaCl==1.3.0
pyparsing==2.4.0
pyperclip==1.7.0
python-dateutil==2.8.0
python-editor==1.0.4
python-mimeparse==1.6.0
python-subunit==1.3.0
pytz==2019.1
PyYAML==5.1
six==1.12.0
SQLAlchemy==1.3.3
stevedore==1.30.1
testtools==2.3.0
traceback2==1.4.0
unittest2==1.1.0
wcwidth==0.1.7
Werkzeug==0.15.2
wrapt==1.11.1
XStatic==1.0.2
XStatic-Bootstrap-SCSS==3.3.7.1
XStatic-DataTables==1.10.15.1
XStatic-jQuery==3.3.1.1
XStatic-Patternfly==3.21.0.1
XStatic-Patternfly-Bootstrap-Treeview==2.1.3.2

ara-manage runserver silently hangs when Postgres dependency is missing

What component is this about ?

API

What is your ARA installation like ?

Installed from PyPi

What is happening ?

If ara-manage runserver is invoked while the db engine set to Postgres but without the psycopg2 or psycopg2-binary package installed, it fails to start, but doesn't print any information about why it's not working.

Output:

2019-08-07 20:30:30,855 INFO django.utils.autoreload: Watching for file changes with StatReloader

What should be happening ?

Something closer to what happens if ara-manage is called without arguments:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 20, in <module>
    import psycopg2 as Database
ModuleNotFoundError: No module named 'psycopg2'

During handling of the above exception, another exception occurred:
...

Measuring and improving api server performance

ansible-role-ara_api currently supports a few different deployment options:

  • with or without gunicorn to launch the django wsgi application for the API server
  • support for nginx in front of gunicorn
  • sqlite as the default database backend
  • support for mysql and postgresql

All of these options are already integration tested and we could leverage these tests to benchmark the performance of each option. The data would be interesting but also valuable in finding issues and improvement opportunities.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.