gunnery / gunnery

Remote task execution tool

Gunnery

Gunnery is a multipurpose task execution tool for distributed systems with web-based interface.

If your application is divided into multiple servers, you are probably connecting to them via ssh and executing over and over the same commands. Clearing caches, restarting services, backups, checking health. Wouldn't it be cool if you could do that from browser or smartphone? Gunnery is here for you!

Features

• Support for a wide variety of tools
  Thanks to simple design it's possible to integrate with tools like capistrano, ant, phing, fabric, make, or puppet
• Designed for distributed systems
  Handles multi-environment applications with many servers
• Usable for deployment, service control, backups
  Every command executed in shell can be turned into a Gunnery task
• Secure remote execution
  Certificate based authentication provides secure access to your network
• Web-based interface
  Clear, responsive interface pleases eye and enables usage on mobile devices
• User notifications
  Team members will be notified when tasks are executed
• Permission system
  Create custom user groups and limit their access to specific environments or tasks

Screenshots

Documentation

Step by step install instructions, and usage notes are available in documentation.

Feedback

Please submit feedback, bugs, feature requests here.

Contribute

Vagrant configuration is available for easy development, included Puppet rules will build complete environment. Read more

gunnery's Issues

Deploy Guide

Event that Readme states "Supports capistrano..." there are no guides how to deploy using capistrano. It would be nice to have some at least notes how to deploy with Gunnery.

Thanks

Wiki detail

Great Work!
It will be great if there are guidelines on what to put in <% ... %> for the configs.
Even better if a docker image can be created! 👍

Vagrant Box outdated

I ran into the issue that when I ran vagrant up with a fresh git clone nothing would provision. I would get a ton of errors with ubuntu links that would 404 in terminal. I am assuming that the vm Box that is currently in the Vagrantfile is outdated which is causing the issues.

I decided to change the box that I would run on to ubuntu/trusty64 and everything seems to be fine. Would updating the box to a newer version of ubuntu be something considered to help fix the vagrant?

Forget it

My mistake

Buggy sidebar on iPad

Error on sending email

Even after I setup google SMTP or AWS SMS i got this error:

Traceback (most recent call last):
  File "/root/.virtualenvs/dev_webdeploy2/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/root/.virtualenvs/dev_webdeploy2/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/repos/webdeploy2/backend/tasks.py", line 274, in run
    message.send()
  File "/root/.virtualenvs/dev_webdeploy2/local/lib/python2.7/site-packages/django/core/mail/message.py", line 276, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/root/.virtualenvs/dev_webdeploy2/local/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 87, in send_messages
    new_conn_created = self.open()
  File "/root/.virtualenvs/dev_webdeploy2/local/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 51, in open
    self.connection.starttls()
  File "/usr/lib/python2.7/smtplib.py", line 644, in starttls
    raise SMTPException("STARTTLS extension not supported by server.")
SMTPException: STARTTLS extension not supported by server.

hide h1 icons on all pages for small mobile devices

Add hidden-xs class to FA icons in h1 elements on all pages.

Submodule 'puppet-module-python.git'

Hi,
I am getting this error while cloning the repo
"fatal: repository 'https://github.com/puppetmodules/puppet-module-python.git/' not found
Clone of 'https://github.com/puppetmodules/puppet-module-python.git' into submodule path 'puppet/modules/python' failed"

Turns out https://github.com/puppetmodules/puppet-module-python has been deleted or made private, is there an alternative to above module?

Add pagination and searching

Lists of users, servers and executions should have pagination and search feature.

• universal implementation - should be reusable for lists of other models
• search input should work for multiple model fields

email notifications settings

Hi! The documentation does not describe how to set email notifications. I am far from python, so there are some difficulties in adjustment. You can add this in the documentation?

Better documentation

I'd really like to see some better documentation about setting this up and getting it running. If you're not familiar with the db layout, or any python apps, it's a real nightmare navigating how to install it.

Abort running task

User needs to have an option to abort running task.

• Add "Abort" button in top right corner
• Confirmation dialog should appear after clicking on it (simple js)
• Find way to kill celery task along with its ssh process
• Task status should be set to "Failed"

User timezone setting

• add timezone field to CustomUser model
• add select field to user profile form in settings
• apply user timezone settings whenever date or time is displayed

Please sync your develop branch with your master

I'd like to contribute but your develop branch is out of sync with master.
In docs you say we need to create branch from develop but it's old.
Please @Eyjafjallajokull update develop so I can add some PR's .:D

Phing ? docu, tutorial...

i also like to give this a try, but like other mentioned, the doc does not give me any hint, howto i.e. use phing with it. it would be nice to have a complete tutorial from a clean setup (without vagrant/puppet) to a simple basic project and some tasks. i.e. how do i say: if this task failes, stop and don´t do any further tasks... ?

Code clean-up

Hi @Eyjafjallajokull
I know you want to do some re factoring, are you ok to do some code clean-ups first?

My propositions:

1. If you want to follow PEP 8 some changes/fixes needed.
2. Some unused import/variables.
3. General JS clean up(not necessary now, can cause potential issues but you know I like good JS :)
4. Some general fixes.

If you ok with that can prepare PR's to all of them.
It's not rocket science, I just want to use some PyCharm features(code inspections report).
Please let me know what do you think.

Add Create Own SSH Certificate Feature.

Thanks for this great solution.

Due to our org system security policy.
Our org need to regenerate all these ssh certificate after one month.

So If there can add our own ssh cert that will be great.

Hope that can consider this advice. thanks.

conn error

TestConnectionTask server_id=5: <type 'exceptions.TypeError'> cannot concatenate 'str' and 'long' objects
Traceback (most recent call last):
  File "backend/tasks.py", line 240, in run
    transport = self.create_transport(server_id)
  File "backend/tasks.py", line 259, in create_transport
    transport = ssh.SSHTransport(server)
  File "backend/ssh.py", line 26, in __init__
    self.secure_files = SecureFileStorage(self.server.environment_id)
  File "backend/securefile.py", line 12, in __init__
    'private_key': PrivateKey(uid),
  File "backend/securefile.py", line 33, in __init__
    name_hash = md5(settings.SECRET_KEY + self.prefix + uid).hexdigest()
TypeError: cannot concatenate 'str' and 'long' objects
[2018-01-22 09:16:55,656: INFO/MainProcess] Task backend.tasks.TestConnectionTask[5afd2b89-afee-45ac-bb9e-66ab800ef0b8] succeeded in 0.183372716001s: (False, u'Unknown error')

Getting started

I would really like to test this as it seems a useful app but following the steps (either shortcut or longer installation) both leave me at the same spot with seemingly not reading the settings properly. It complains about SECRET_KEY being empty and DATABASE/ENGINE settings etc. so seems to be reading from /var/gunnery/virtualenv/production/local/lib/python2.7/site-packages/django/conf/ when I believe it should read from /var/gunnery/gunnery/gunnery/settings?

Basically, following the steps always fails at the "python manage.py syncdb" step.

The instructions I personally find really poor, unless this is only meant for people who have a deep understanding of all the various apps used for it to work? In a previous issue about poor documentation one reply is "For now, I recommend looking at puppet rules, they will get you running application in minutes.". I don't think anyone could get a running app in minutes on this, especially without knowing puppet really well also.

For now I have to give up as I have spent almost a whole day on this which is a shame, but until following the instructions works, I cannot spend more time on it. I hope you can tell me something really obvious I am doing wrong but if I am, it is not in the docs.

Schedule

It would be awesome if task execution could be either scheduled from the web interface or executed from console and scheduled with cron.

Capistrano recipe administration

This tool is great, but it would be great to have capistrano recipes administration on Gunnery, something like Webistrano. Right now, is not clear for me on how to deploy recipes to servers and deploy.
Cheers!

Add user groups and advanced permissions management

Setup guidelines create init.d for uwsgi with incorrect path to gunnery.ini

The setup directions at http://gunnery.readthedocs.org/en/latest/install.html write gunnery.ini to /etc/uwsgi/apps-enabled but the init.d script for uwsgi points to sites-enabled.

The dir needs to be renamed in the docs.

Password authentication to servers

There should be an option to choose between password and key authentication.

• Add radio button for selecting method
• Find a secure way to store passwords
• Extend ssh classes

Webhook triggered deployment (for e.g. travis-ci etc.)

I think adding webhook support for automated triggering of deployments will be a great addition to this already awesome tool. This way CI services like travis, jenkins etc. can easily be integrated creating a fully automated development to deployment chain .
For a webhook per task, sample Workflow could be:
[Code commit to a VCS e.g. github] => [github makes webhook call to CI] => [CI runs the tests and validates the build[ => [CI makes webhook call to gunnery ]=> [Gunnery verifies the call and triggers respective task]

Similar use cases can be made for automated back-ups etc.
I will be more than happy to work on it. Your thoughts??

400 Bad Request

hey,

just ran the install on a ubuntu 12.04 and a 13.10 box ( aws ec2 instance )

and i am getting a 400 bad request.

does it run on 8080 or 80 ?

Relation "django_content_type" does not exist

When I spin up the VM and go to the localhost I am immediately greeted with this error

relation "django_content_type" does not exist
LINE 1: ..."."app_label", "django_content_type"."model" FROM "django_co...

Environment:


Request Method: GET
Request URL: http://localhost:8080/account/login/?next=/

Django Version: 1.6.7
Python Version: 2.7.6
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django_extensions',
 'timezone_field',
 'guardian',
 'crispy_forms',
 'djcelery',
 'south',
 'core',
 'task',
 'backend',
 'account',
 'event',
 'debug_toolbar')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.contrib.auth.backends.ModelBackend',
 'guardian.backends.ObjectPermissionBackend',
 'core.middleware.CurrentDepartment',
 'debug_toolbar.middleware.DebugToolbarMiddleware')


Traceback:
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  137.                 response = response.render()
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/template/response.py" in render
  105.             self.content = self.rendered_content
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/template/response.py" in rendered_content
  81.         context = self.resolve_context(self.context_data)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/template/response.py" in resolve_context
  159.         return RequestContext(self._request, context, current_app=self._current_app)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/debug_toolbar/panels/templates/panel.py" in _request_context__init__
  55.         context = processor(request)
File "./core/context_processors.py" in sidebar
  8.     departments = get_objects_for_user(request.user, 'core.view_department')
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/guardian/shortcuts.py" in get_objects_for_user
  354.                 permission__codename=codename)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/models/manager.py" in get
  151.         return self.get_queryset().get(*args, **kwargs)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/models/query.py" in get
  304.         num = len(clone)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/models/query.py" in __len__
  77.         self._fetch_all()
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/models/query.py" in _fetch_all
  857.             self._result_cache = list(self.iterator())
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/models/query.py" in iterator
  220.         for row in compiler.results_iter():
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in results_iter
  713.         for rows in self.execute_sql(MULTI):
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
  786.         cursor.execute(sql, params)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/debug_toolbar/panels/sql/tracking.py" in execute
  174.         return self._record(self.cursor.execute, sql, params)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/debug_toolbar/panels/sql/tracking.py" in _record
  104.             return method(sql, params)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/backends/util.py" in execute
  69.             return super(CursorDebugWrapper, self).execute(sql, params)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/backends/util.py" in execute
  53.                 return self.cursor.execute(sql, params)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/utils.py" in __exit__
  99.                 six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/var/gunnery/virtualenv/development/local/lib/python2.7/site-packages/django/db/backends/util.py" in execute
  53.                 return self.cursor.execute(sql, params)

Exception Type: ProgrammingError at /account/login/
Exception Value: relation "django_content_type" does not exist
LINE 1: ..."."app_label", "django_content_type"."model" FROM "django_co...

                                                         ^ 

Not sure exactly why this is occurring on a fresh git clone

Thanks for the help

Add default department user setting

• add field to user profile form, it should be only displayed when user has access to multiple departments
• add field to CustomUser model

static files not found

Hello,

I am following http://gunnery.readthedocs.org/en/latest/install.html
documentation to get server up and running. I am able to get it running.

But, css, jquery, bootstrap files are not found http://localhost:8000/static/css/bootstrap.css
Also, nothing happens if I click on "Create first".

what could be the reason / what do i need to change ?

thanks,
Manjiri

No way to add new application?

Using Gunnery+Vagrant setup.

Login, Click Settings -> Applications tab has no options to add anything new.
Other than the Settings page info it just says "no applications yet.".

URL on Vagrant box:
http://localhost:8080/settings/applications/

Special characters result in error

Special characters in the output of commands (such as the checkmarks used by Capifony) result in the task stopping with error (code 1024).

change celery worker & concurrency settings

We have 80 machines that need to execute the command in a task.

And we found out that there only 8 concurrency worker will run in a same time.

I know celery is the key of gunnery task and I found some config in puppet file
https://github.com/gunnery/gunnery/blob/master/puppet/manifests/hieradata/common.yaml

celery::workers : 1
celery::concurrency : 8

We are trying to increase those number after already installed.
Any hint will be great. Thanks.

Clicking on + for "add server" causes Django to silently hang

Clicking on plus sign causes this request to spawn and never finish:
http://10.211.55.7:8080/modal_form/a:/environment/1/server/
It halts whole UWSGI process, no requests can be made, until UWSGI is killed (SIGTERM does nothing). All requests timeout with 504.

Nothing new appears in logs when this happens. Not even nginx request.

I've tried this in development environment, i.e. by running python manage.py runserver and issuing curl request, which causes it to hang too and Python must be killed. Note that by using curl I'm not even logged in. The request doesn't even shop up on the screen.

root@vmdebian7:/var/gunnery/gunnery# python manage.py runserver
Validating models...

0 errors found
June 10, 2014 - 12:43:11
Django version 1.6.2, using settings 'gunnery.settings.production'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[10/Jun/2014 12:43:16] "HEAD / HTTP/1.1" 302 0

Second terminal window:

root@vmdebian7:/var/gunnery/log# curl localhost:8000/ -I
HTTP/1.0 302 FOUND
Date: Tue, 10 Jun 2014 10:43:16 GMT
Server: WSGIServer/0.1 Python/2.7.3
Vary: Cookie
X-Frame-Options: SAMEORIGIN
Content-Type: text/html; charset=utf-8
Location: http://localhost:8000/account/login/?next=/

root@vmdebian7:/var/gunnery/log# curl localhost:8000/modal_form/a:/environment/1/server/ -I
# hangs....

It is a clear installation. I just created Department, Server roles, Application, Environment and clicking on + in chosen Environment causes this.

I'm not really familiar with Django or Python, is there something I can do to nail down this issue?

Also, I don't know if it can be related - issuing service celeryd stop does nothing, it waits. I must ctrl+c the stopping process and issue it again. Workers stop properly on this second try. Always. Doesn't matter if I have one worker or 3 of them. I don't know if clicking on Add server causes some background task to fire... so that's why I'm mentioning this.

root@vmdebian7:/var/gunnery/log# /etc/init.d/celeryd start
celery multi v3.1.7 (Cipater)
> Starting nodes...
        > worker1@vmdebian7: OK
root@vmdebian7:/var/gunnery/log# /etc/init.d/celeryd stop
celery multi v3.1.7 (Cipater)
> Stopping nodes...
        > worker1@vmdebian7: TERM -> 5685
> Waiting for 1 node -> 5685....................................................................................^C
Session terminated, terminating shell... ...terminated.

root@vmdebian7:/var/gunnery/log# /etc/init.d/celeryd stop
celery multi v3.1.7 (Cipater)
> Stopping nodes...
        > worker1@vmdebian7: TERM -> 5685
> Waiting for 1 node -> 5685.....
        > worker1@vmdebian7: OK

VM:
Debian 3.2.0-4-amd64
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [async-threads:30] [kernel-poll:true]
RabbitMQ 2.8.4

If you need anything more, let me know.

Setup guidelines do not explain replacements for nginx configuration

Specifically, the erb for the nginx configuration is:

  <% if @static_only_collected %>
                try_files /gunnery$uri /404;
                <% else %>
                try_files $uri /core$uri /static$uri /<%= @app_name %>$uri /404;
                <% end %>

                <% if @gzip_enable %>
                gzip on;
                gzip_min_length 1100;
                gzip_buffers 4 32k;
                gzip_types text/plain application/x-javascript text/xml text/css;
                gzip_vary on;
                <% end %>

                <% if @cache_headers_enable %>
                expires 30d;
                add_header Cache-Control "public";
                <% end %>

It'd be good to know which of these sections to leave :)

syncdb step should include info that username MUST be in user@host format.

Default username on Vagrant setup when running 'python manage.py syncdb' is 'vagrant'. If you hit enter and take that default you can't login to the web interface since the login page forces [email protected] format.

Setup guidelines do not setup python virtualenv

http://gunnery.readthedocs.org/en/latest/install.html when following the non-puppet version of install steps do not create a virtualenv directory.

I needed to apt-get install python-virtualenv and then virtualenv env to create it. Then my pip install -r requirements/production.txt became sudo -u gunnery env/bin/pip install -r requirements/production.txt.

Fix simple typo: specyfic -> specific

There is a small typo in docs/develop.rst.
Should read specific rather than specyfic.