curationexperts-deprecated / ansible-hydra Goto Github PK

apache?

I'm running the vagrant branch and getting this error

TASK: [vagrant-housekeeping | add keys for capistrano, ubuntu users] ********** 
changed: [default] => (item=[u'deploy', 'https://github.com/acozine.keys'])
changed: [default] => (item=[u'deploy', 'https://github.com/mark-dce.keys'])
failed: [default] => (item=['ubuntu', 'https://github.com/acozine.keys']) => {"failed": true, "item": ["ubuntu", "https://github.com/acozine.keys"]}
msg: Failed to lookup user ubuntu: 'getpwnam(): name not found: ubuntu'
failed: [default] => (item=['ubuntu', 'https://github.com/mark-dce.keys']) => {"failed": true, "item": ["ubuntu", "https://github.com/mark-dce.keys"]}
msg: Failed to lookup user ubuntu: 'getpwnam(): name not found: ubuntu'

FATAL: all hosts have already failed -- aborting

If I delete the ubuntu user from this line, the error clears
https://github.com/acozine/sufia-ansible/blob/vagrant/roles/vagrant-housekeeping/tasks/users_groups_dirs.yml#L30

Consider which roles if any should continue to run with elevated privileges for all tasks.

Advantage of making the entire role run with elevated privileges - code is DRYer.
Disadvantage of this approach - may be confusing when looking at the task list, may lead to new tasks running with elevated privileges when they should not, makes debugging by recreating each task at the command line trickier (the tasks themselves don't show that they are being run / need to be run as root).
Also, when writing new playbooks with existing roles, you need to know that the role needs to be called with become at the role-level - probably worth a comment in the role's main.yml if we keep this structure.

incorporate changes from sufia-prod-ubuntu-vagrant's vagrantfile
include comments to offer new users options for using the vagrantfile

I tried deploying in Vagrant using your sample Vagrantfile. It all got off to a great start, but went off the rails when it got to the task: housekeeping | format volume for /opt.

TASK: [housekeeping | format volume for /opt] *********************************
failed: [default] => {"failed": true}
msg: Device /dev/xvdf not found.

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
          to retry, use: --limit @/Users/erickpeirson/vagrant.retry

default                    : ok=4    changed=2    unreachable=0    failed=1  

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

The entire Vagrant output is here.

Vagrant 1.7.4
Ansible 1.9.3

EPIC

Current CentOS repo is at https://github.com/acozine/sufia-centos - goal is to integrate both into a single repo.

Sudo is deprecated already. Refactor to use become throughout the roles.

Also, consider which roles if any should run with elevated privileges for all tasks.

• Advantage of making the entire role run with elevated privileges - code is DRYer.
• Disadvantage of this approach - may be confusing when looking at the task list, may lead to new tasks running with elevated privileges when they should not, makes debugging by recreating each task at the command line trickier (the tasks themselves don't show that they are being run / need to be run as root).

Thoughts @HackMasterA?

This will support the ability to build dev machines, both remote and in vagrant.

I'm thinking the best way to do this is to split up roles where necessary, possibly using some nested roles in places (e.g where the split is install vs. configure).

I considered using tags but they cannot be filtered from within a playbook (only from the command line) and I want to be able to encapsulate execution to a set of high-level playbooks without requiring a user to know a bunch of different command-line switches (which are easy to accidentally leave out).

Generalize the mount_opt role for a more granular architecture - instead of a single disk at /opt, support mounting multiple disks. Dedicate a disk to fedora-data for easier backups, upgrades, and disaster recovery. May require changes to how and where ansible creates the fedora-data directory.

Currently the java options are set in roles/hydra-stack/install/tasks/fedora.yml. There's a template (roles/hydra-stack/install/templates/tomcat7.j2) but it doesn't use any variables yet.

• include variables in the tomcat7 template
• add default values for the variables

It doesn't really make sense on a vagrant box but if someone wanted to build a dev machine on a non-vagrant box it would be nice to have.

Even though I tested the task last week without issue, I'm now getting this error

TASK: [imagemagick | clone ImageMagick source] ******************************** 
failed: [default] => {"dest": "/opt/install/imagemagick_sources/ImageMagick-6.9.2-7.tar.xz", "failed": true, "response": "HTTP Error 404: Not Found", "state": "absent", "status_code": 404, "url": "http://www.imagemagick.org/download/releases/ImageMagick-6.9.2-7.tar.xz"}
msg: Request failed

So, it's not finding ImageMagick-6.9.2-7. If I look at http://www.imagemagick.org/download/releases/ you can see that a newer release has superseded that: http://www.imagemagick.org/download/releases/ImageMagick-6.9.2-8.7z
There will also be an issue here because the extension has changes from .tar.xz to just .z

Maybe we should consider defaulting to http://www.imagemagick.org/download/ImageMagick.tar.gz (always points to the current release) and letting someone override image_magic_source_url if they really want something different.

We need "/prod" instead of "prod" here:
https://github.com/acozine/sufia-ansible/blob/54fc6d2cb012db269af673453ddaf7251e32709e/roles/hydra-stack/defaults/main.yml#L7

We've decided it would be a good idea to add some basic comments to the main.yml file for each role. Both just to help you figure out where you are, and, when, needed to help give more context about the role:

BASIC VERSION EXAMPLE
https://github.com/curationexperts/ansible-hydra/blob/eed11d39df4cddb9b4d3828410974c7e357c9977/roles/imagemagick/tasks/main.yml#L2-L3

GIVES MORE CONTEXT VERSION
https://github.com/curationexperts/ansible-hydra/blob/5957bb726bd596d01b52dba099e1335384cfe513/roles/mount_opt/tasks/main.yml#L2-L10

ACCEPTANCE
Add comments to the roles that are missing them:

• ansible-hydra/roles/apache-passenger/config/tasks/main.yml
• ansible-hydra/roles/apache-passenger/install/tasks/main.yml
• ansible-hydra/roles/app-config/tasks/main.yml
• ansible-hydra/roles/capistrano_setup/tasks/main.yml
• ansible-hydra/roles/ec2/tasks/main.yml
• ansible-hydra/roles/ffmpeg/tasks/main.yml
• ansible-hydra/roles/hydra-stack/install/tasks/main.yml
• ansible-hydra/roles/hydra-stack/config-capistrano/tasks/main.yml
• ansible-hydra/roles/imagemagick/tasks/main.yml
• ansible-hydra/roles/launch_ec2/tasks/main.yml
• ansible-hydra/roles/mount_opt/tasks/main.yml
• ansible-hydra/roles/openjdk1_8/tasks/main.yml
• ansible-hydra/roles/packages/tasks/main.yml
• ansible-hydra/roles/riiif/tasks/main.yml
• ansible-hydra/roles/ruby/tasks/main.yml
• ansible-hydra/roles/set_timezone/tasks/main.yml
• ansible-hydra/roles/sufia_dependencies/config/tasks/main.yml
• ansible-hydra/roles/sufia_dependencies/install/tasks/main.yml
• ansible-hydra/roles/system_setup/tasks/main.yml
• ansible-hydra/roles/vagrant_setup/tasks/main.yml

Update the "A production-like vagrant box" section of the README.

Get rid of the TODO line - preferably by adding a link to Newfia's internal deploy task (https://github.com/curationexperts/newfia/blob/master/config/deploy/internal_vm.rb) or other information that demonstrates how to deploy your project to your vagrant box using capistrano.

see http://docs.ansible.com/playbooks_best_practices.html#how-to-differentiate-staging-vs-production

I believe this would allow different variables to be set while also enabling current / future dynamic inventory work.

Also helps protect against running plays on the wrong machine/s.

The ec2_vol_1 gets set in the create_ec2 role, but sometimes you get download or other transient errors running configure.ym and you want to restart it without re-running create_ec2. But one of the roles/ec2/templates/snapshot_backup.j2 template references this variable and fails.

https://github.com/curationexperts/ansible-hydra/blob/d00c413c5e0c75318863869926c9dc8fcc6924f7/create_ec2.yml#L12-L13

Would it be possible to just reference hostvars['localhost'].ec2_vol in the template directly? This appears to be the only use of the variable.

Proposal:

1. Create 'defaults' for ec2 and launch_ec2 roles as a way to document the variables used in those roles (people new to the scripts won't be able to see the vaulted files so would have to comb through the tasks to get variable names). Obviously they would be dummy values / won't actually work.
2. Override varibles using group_vars
   • Create group_vars/sample_all (see http://docs.ansible.com/playbooks_best_practices.html#group-and-host-variables) containing a list of variables that are recommended/required to override.
   • Edit README to recommend that this file be copied to group_vars/all and protected by ansible-vault.
   • Delete 'vars' directories from ec2, launch_ec2, and services roles

Advantages:

• single vaulted file makes it easier to get at things
• variable override location is clearly provided
• follows ansible best practices
• 'defaults' continue to exist for each role as documentation and to allow re-usability in future.

Disadvantage:

• not all roles / playbooks require all the vaulted variables, so you would end up typing the ansible-vault password more frequently than needed.

Comments welcome! I'll work on a PR next week.

When installing Solr and Fedora as part of the Hydra stack, ansible should restart Tomcat when everything else is done. Currently this task happens at the end of roles/hydra-stack/install/tasksfedora.yml. If the role were used to upgrade Solr or to install a project that did not use Fedora, the restart wouldn't happen.

Pull the restart into a task file of its own, or else document and close.

In this line:
https://github.com/acozine/sufia-ansible/blob/vagrant/sample_Vagrantfile#L36

I had to use "SATA Controller" instead of "SATAController". I don't know if this was due to a typo in the file or a difference in the version of VirtualBox I'm using.

This should be set as a default variable; ansible convention will cause users to look there for customizations they may want to make.

https://github.com/curationexperts/ansible-hydra/blob/master/roles/hydra-stack/install/tasks/fedora.yml#L5-L7

I've just manually deleted three Instances and their separate opt volumes that don't delete automatically when you terminate the main instance.

Is it worth a playbook that tears down an instance build using launch_ec2?

ISSUE
FITS is a hydra-derivatives dependency and may be used by applications other than Sufia

NOTES
Consider renaming Sufia-dependencies or moving FITS, ImageMagick, and FFmpeg to individual roles that sufia-dependencies can depend on

Currently create_ec2.yml includes the ec2 role, and the ec2 role has a meta dependency on the mount_opt role. Make the code easier to understand by including the mount_opt role directly in create_ec2.yml

Use the deploy user or the default ansible_user (ansible ssh_user)

The current FITS role copies alot of unnecessary files into /usr/local/bin

- name: copy fits to /usr/local/bin
  sudo: yes
  shell: cp {{ install_path }}/fits-{{ fits_version }}/* /usr/local/bin/

It would be nicer to just simlink to a full installation in the opt directory. BUT... the fits.sh script doesn't do a good job of figuring out where it lives when launched as a symlink.

The other possibility would be to install fits to /opt/fits-x.y.z and just add that to the path.

I'm not sure what the best solution is. Possibly just leaving it as-is.

Add info on how to set this up into the README or wiki

I ran the playbook successfully and I appear to have fedora and solr (tomcat) running fine, but there's no web server responding on port 80.

When I try to check the service I get this:

$ sudo service apache2 status
apache2: unrecognized service

EPIC

Currently the sufia-dependencies role contains some items that non-sufia-based hydra heads might also use. For example, FITS.

Refactor to separate project-specific dependencies and make the scripts more user-friendly for installing a variety of hydra heads.

http://edgeguides.rubyonrails.org/upgrading_ruby_on_rails.html#config-secrets-yml
https://help.ubuntu.com/community/EnvironmentVariables

1. make the tone more welcoming
2. add links to anna & alicia's hydra connect slides and notes (include lots of good info on AWS). Even better, incorporate those instructions into the readme directly.

Is this file an orphan?
./roles/app-config/templates/ansible-sudoers.j2

contents:

{{ sudo_user_1 }} ALL=(ALL) NOPASSWD:ALL
{{ sudo_user_2 }}  ALL=(ALL) NOPASSWD:ALL

I can't find any task that references ansible-sudoers.j2 or any place that sudo_user_1 is defined.

Scripts currently have conflicting versions
./roles/ruby/defaults/main.yml - 2.2.0
./roles/ruby/tasks/main.yml - 2.2.2
./roles/apache-passenger/install/defaults/main.yml - 2.2.0

There are known security vulnerabilities in 2.2.x prior to the 2.2.4 release:
https://www.ruby-lang.org/en/news/2015/12/16/unsafe-tainted-string-usage-in-fiddle-and-dl-cve-2015-7551/

• Turn the repo into a variable with newfia as the default.
• Add a variable to replace "vagrant" as the user, so the role will run in any environment.
• Rename the role to "self-deploy" once the role will deploy any codebase in any environment.

install Avahi

ISSUE
The current installer appears to install the latest version of FFMPEG on source forge. This might make it difficult to duplicate configurations over time if the version of FFMPEG gets updated between the time systems are built.

See https://github.com/curationexperts/ansible-hydra/blob/f77e6bdbec81925da2e7fa37aa47bc162e61b212/roles/ffmpeg/tasks/ffmpeg.yml

OPTIONS

1. Let ffmpeg version float as-is and close this ticket
2. Add versioning logic like imagemagick
3. Just use the Ubunutu packaged version (need to check how out of date it is for 14.04 LTS repos)

Need to have add-apt-repository command available (it was missing from my server):

sudo apt-get install software-properties-common python-software-properties

Then see
http://ubuntuhandbook.org/index.php/2015/01/install-openjdk-8-ubuntu-14-04-12-04-lts/

Either make tickets or remove each of the TODO's listed below (grep'ed on 12/4/12)

• ./configure.yml:# TODO: it should work to default to all ec2hosts in this way:
• ./README.md:4. TODO: deploy your capistrano project to your vagrant box. Haven't tried this yet.
• ./roles/app-config/tasks/systems.yml:#TODO: manage env variables for this separately
  ./roles/ec2/tasks/ec2_crons.yml:# TODO
• ./roles/hydra-stack/install/tasks/fedora.yml:# TODO: make Java options configurable at the time of the ansible run
• ./roles/hydra-stack/install/tasks/fedora.yml:# TODO: make this a service called from main.yml

Getting this error in resque:

Unable to execute command "fits.sh -i "/tmp/content-1.tiff20151130-1854-1m0nm6s.tiff"" sh: 1: fits.sh: not found

And this at the command line

Unable to execute command "fits.sh -i "/tmp/content-1.tiff20151130-1854-1m0nm6s.tiff"" sh: 1: fits.sh: not found