ome / infrastructure

A repository containing scripts for managing infrastructure

OME Infrastructure

Tools, scripts and other systems infrastructure used to support the work of OME (http://www.openmicroscopy.org/).

If you are interested in this work or require more information, please contact us.

Ansible

OME uses Ansible extensively for deploying production services. Installation and usage instructions are provided in the ansible README.

OpenStack

• Server setup outlines the installation of OpenStack which is running at the University of Dundee.
• Basic usage outlines possible first steps.
• Local setup details the steps to access OpenStack from a client system
• CentOS image creation
• FreeBSD image creation
• Ubuntu image creation
• Windows image creation
• MacOS X image creation
• CI setup details the steps to set up a CI node

GPFS

GPFS.md provides details on the configuration of GPFS that is used at the University of Dundee.

---

For further information, you may want to read the Contributing to OME page.

infrastructure's Issues

Galaxy roles should be prefixed with organisation

The dependencies of galaxy roles have the form openmicroscopy.basedeps, however the playbooks in this repository expect a role name without a prefix (just basedeps). As a result some roles are installed twice, with and without the prefix.

Proposed fix: all playbooks using Galaxy roles should use the prefix.

Add contributing guidelines

https://www.reinteractive.net/posts/167-ansible-real-life-good-practices is a good start

devspace: make port configurable

The name for JENKINS_MASTER is configurable but the port is currently hard-coded as 8443.

devspace: "ensure home ..." doesn't respect check mode

TASK [devspace : ensures /home/omero/.ssh dir exists] **************************
fatal: [adhoc-devspace]: FAILED! => {"failed": true, "msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user. For information on working around this, see https://docs.ansible.com/ansible/become.html#becoming-an-unprivileged-user"}

jenkinsslave: .gitconfig needs to be created

When run for the first time, the task fails with

fatal: [web-qa-staging.openmicroscopy.org]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"dest": "/home/hudson/.gitconfig", "group": "hudson", "owner": "hudson", "src": "gitconfig.j2"}, "module_name": "template"}, "msg": "file (/home/hudson/.gitconfig) is absent, cannot continue", "path": "/home/hudson/.gitconfig", "state": "absent"}

Getting started docs

I recently tried to 'Get Started' running some Ansible examples, with no particular idea what I wanted to achieve, but hoping that getting something working would help me understand / learn something!

However, starting at https://github.com/openmicroscopy/infrastructure/blob/master/ansible/README.md I never managed to get anything working.

Followed instructions at https://github.com/openmicroscopy/infrastructure/blob/master/docs/ansible/installation.md until

Clone the repository containing the inventory, host and group vars files. Ansible will automatically look for host_vars and group_vars directories in the parent directory of the inventory file. This should be located at ../../ansible/inventory such that -i ../../ansible/inventory would be correct.

but didn't know where to look for example repositories. The only "Getting started" example I could see was https://github.com/openmicroscopy/ansible-role-omero-server but this doesn't seem to have the required files mentioned above and there were also no alternative instructions on that page on how to actually get started with the example.

Tried looking at "infrastructure" repo for examples, but don't know what's expected to work. Tried a few things e.g.

$ cd /Users/wmoore/Desktop/INFRASTRUCTURE/infrastructure/ansible 
$ ansible-playbook -u wmoore --ask-become-pass -C -v os-devspace.yml
...
 [WARNING]: Host file not found: ../../ansible/inventory/
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "vm_key_name is required"}

In the end I gave up and went to https://github.com/openmicroscopy/devspace which is probably where I should have started.

To summarise, if the "Getting Started" instructions are designed for newbies to get something running then they need the step-by-step instructions to go a bit further.
Maybe some more high-level intro to the various repos (Infrastructure, Management tools & devspace) would also help, since the relationship & differences between these is still kinda confusing so figuring out where to start is hard.

Role: ansible-role-postgresql-backup

See:

• #279
• https://github.com/openmicroscopy/ansible-role-mysql-backup/blob/master/templates/etc-crond-mysql-backup.j2

The production server playbooks currently have logic for creating a backup cronjob. This is a prime candidate for refactoring into a role. (Also a good entry task for anyone wanting to learn ansible)

Versions of dependencies should be managed in playbooks where possible

At the moment most of our playbooks use the default version (as defined in the role) of dependencies such as Ice and Postgres, which means bumping any of these dependencies means checking all known playbooks and deciding whether to upgrade or limit to the previous version.

I think it would be better to decouple the playbooks, so the roles could always default to the latest supported version (e.g. Ice 3.6, postgres 9.5), and we make it clear that it's the responsibility of the playbook to define the required version if it's not the latest. This makes it easier to upgrade systems independently.

See also ome/omero-install#121

Travis-CI syntax check

Something along the lines of:

for playbook in ansible/*.yml; do
    ansible-playbook --syntax-check $playbook
done

haproxy reload drops connection

RUNNING HANDLER [haproxy : restart haproxy] ************************************

logged me off of ssh to idr-ci

infrastructure needs to support different OSes

At the moment this repo only support CentOS7, what about Ubuntu and http://docs.ansible.com/ansible/apt_module.html? How this will be organized to support the same tasks for different OS? Do we need extra level of hierarchy?

Inventory/groups/tags strategy

Following on from @sbesson's proposal in joshmoore#3 (comment) of appending a tag to all include and role statements, e.g.:

roles:
- { role: common, tags: "common" }
- { role: nginx, tags: "nginx" }
- { role: appserver, tags: appserver }
- { role: database, tags: database }
- { role: memcached, tags: memcahced }

I'd also propose a group per inventory file:

; inventories/production_servers
[web]
4.2.2.1
4.2.2.2

[database]
8.8.8.8

[production:children]
web
database

(See http://rosstuck.com/multistage-environments-with-ansible/) with all inventory files in one directory (per subsystem which for the moment would be ci, prod, idr).

And further, that all os-uod playbooks add the callers name as a group so that someone could, e.g., stop all of their servers with ansible ... 'owner-josh' -a 'shutdown -h now'

sudo: sorry, you must have a tty to run sudo

see #112 (comment)

TASK [upgrade-distpackages : system packages | upgrade] ************************
task path: /Users/ola/OMERO/SYSADMIN/infrastructure/ansible/roles/upgrade-distpackages/tasks/main.yml:4
Monday 12 September 2016  14:46:05 +0100 (0:00:00.363)       0:00:00.403 ****** 
failed: [10.0.51.143] (item=[u'*']) => {"failed": true, "item": ["*"], "module_stderr": "sudo: sorry, you must have a tty to run sudo\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
10.0.51.143                : ok=1    changed=0    unreachable=0    failed=1   

edit ansible.conf and change: pipelining=False and allow_world_readable_tmpfiles=True

roles/system-monitor-agent needs to set `only_from`

The role which deploys Check_MK, ansible/roles/system-monitor-agent/tasks/main.yml should set only_from in the xinetd.d/check_mk cf https://mathias-kettner.de/checkmk_linuxagent.html.

This will need to accept an IP address or hostname (both work), despite the check_mk docs suggesting an IP address.

Change yum package state from latest to present

This will avoid packages being upgraded unless it's explicit. Alternatively as @kennethgillen suggested state could be a variable defaulting to present, and override it for upgrades, though that make things more complicated.

Deployment playbooks should run independently of Provisioning

Deployment playbooks should not require hardware configuration and should therefore work in Docker (the main exception being Docker in Docker which is tested on a VM). It should therefore be possible to set default variables in any roles so that they'll just work on a standard system, but will take account of optional vars for custom configs as provided by the provisioning playbook.

An added benefit is that in the event of a disaster the entire system can be run on any VM, albeit with degraded performance.

Remove docker-advanced docker-storage roles

These have been replaced by the single docker role, but the old roles are still in use by one server. As soon as this server is updated delete the old roles.

Mount /home(s) on a new volume

This will prevent users filling the root partition....

Setup NTP in openstack instances

Fix deprecated syntax in docker role

I think this just needs to be quoted: when: docker_groupmembers is defined

• https://github.com/openmicroscopy/infrastructure/blob/master/ansible/roles/docker/tasks/main.yml#L69
• #149 (comment)

Non-idempotent tasks in omero-server role

#70 Introduced a non-idempotent pair-of tasks: https://github.com/openmicroscopy/infrastructure/blob/de1d03ae64511202401b1d370cc4d8f01c0d6c45/ansible/roles/omero-server/tasks/main.yml#L38

• name: omero | copy bash skeleton files copies .bashrc from /etc/skel/
• name: omero | add to path modifies the same .bashrc

jenkinsslave: install git

This should likely belong to the tasks when setting up a Jenkins slave.

Fix omero-server systemd service dependencies

A problem noticed when rebooting an OMERO.server with a non-local postgres and NFS mount:
Networking is stopped before the omero service, which causes omero to hang. This dependency needs to be fixed.
https://github.com/openmicroscopy/infrastructure/tree/master/ansible/roles/omero-server

Flatten all dictionary variables

Dictionaries can't be merged, which means if you want to change one field you have to specify all fields. Flattening them would make things easier to override. E.g.

docker_dns:
  forwarders: 8.8.8.8:53,8.8.4.4:53
  domain: docker.internal
  etcd: etcd1

It's not possible to override just docker_dns.domain, you have to specify the entire dict.

devspace: ssl directory misformed

Within the nginx container, there's a directory named ssl]:

drwxr-xr-x.  2 root  root    6 Dec  1 09:12 ssl]

Configuration changes don't reach the server

Configuration changes made in Ansible do not seem to be propagated to an installed server. I installed a test server with training-server.yml, and afterwards set

---
  omero_dbpassword: "kukkuu"

and ran the playbook again. The change shows up in a config file, but not in "omero config"

(omego)[omero@omero ~]$ cat config/omero-base.config 
config set omero.db.host localhost
config set omero.db.user omero
config set omero.db.name omero
config set omero.db.pass kukkuu
config set omero.data.dir /OMERO

# Additional custom options
(omego)[omero@omero ~]$ omero config list
omero.data.dir=/OMERO
omero.db.host=localhost
omero.db.name=omero
omero.db.pass=omero
omero.db.user=omero

Looking at roles/omero-server/tasks/main.yml, it seems that only omego install and omego upgrade use this configuration. Maybe there could also be a handler to run omero config set when omero-base.config or omero-additional.config are changed?

Add `become: yes` to ansible/roles/*/handlers/*yml

A few handlers are missing become: yes

Tags

Define a set of tags (any volunteers to decide on an ontology?). Apply them.

nginx log rotation

Override rotation in /etc/logrotate.d/nginx as part of the nginx/web-proxy roles.

Consider setting default filesystem in role lvm-partition

Set the default filesystem lvm_lvfilesystem variable to xfs in roles/lvm-partition

At present there is no default (the role will fail if it's not defined) which means it shouldn't affect existing config, and it's one less thing to have to configure.

`devspace` group should be `devspace-hosts`

This matches docker-hosts

Default value of `lvm_lvopts` fails to create logical volume

The default value of lvm_lvopts which is null - see #11 gives runtime errors of type

fatal: [web-qa-staging.openmicroscopy.org]: FAILED! => {"changed": false, "err": "  Volume group \"None\" not found\n", "failed": true, "invocation": {"module_args": {"lv": "jenkins-workdir", "opts": null, "size": "500m", "vg": "VolGroup00"}, "module_name": "lvol"}, "msg": "Creating logical volume 'jenkins-workdir' failed", "rc": 5}

Removing opts from the lvm-partition role allows to complete the task.

Remove `runningindocker` conditionals

A few roles use separate tasks to control services due to previous problems with systemd in docker. These shouldn't be required any more https://github.com/ome/ome-docker/tree/master/omero-ssh-systemd

Change environment vars to use `{{ }}`

Change environment vars to use {{ }} as reported in #53 (comment)

TASK [upgrade-distpackages : system packages | upgrade] ************************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so that the environment value uses the full variable syntax ('{{upgrade_distpackages}}'). This
feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

devspace: non-home directory failure

Trying to locate devspace at /data/devspace leads to:

TASK [devspace : clone devspace] ***********************************************
fatal: [adhoc-devspace]: FAILED! => {"changed": false, "cmd": "/bin/git clone --origin origin --branch 0.3.4 https://github.com/openmicroscopy/devspace.git /data/devspace", "failed": true, "msg": "fatal: could not create work tree dir '/data/devspace'.: Permission denied", "rc": 128, "stderr": "fatal: could not create work tree dir '/data/devspace'.: Permission denied\n", "stdout": "", "stdout_lines": []}

Remove spacewalk variable

See #137

"openstack-snapshoter" requires jq

This may could/should be encapsulated in a role.

Use suggested json docker daemon config file

See https://docs.docker.com/engine/reference/commandline/dockerd/#/linux-configuration-file

This should be used in the docker role, so the upstream docker.service systemd file can be used unmodified. This should avoid problems with breaking changes which would otherwise require a change to the template, e.g. #90

role per webapp

Rather than having all webapps installed by a single role, each webapp should have its own role so that specific configuration can be applied. In the IDR case, the workflow would be:

• base IDR/OMERO role removes all top_links and then adds "Studies"
• mapr role adds "Genes", "Phenotypes", etc.
• other webapps can do the same.

The same type of extension is needed for the definition of filters (see MT 191)

See: #108

cc: @manics @aleksandra-tarkowska