jhu-library-applications / catalyst-ansible

Builds and deploys Catalyst (Johns Hopkins University Libraries blacklight-based library catalogue) using Ansible

catalyst-ansible

Ansible project to build and deploy Catalyst, Johns Hopkins University Libraries Blacklight-based library catalog.

This project requires

• Requires Ansible 2.10+ (using import_role)
• Tested with Vagrant 2.2, VirtualBox 6.1, macOS Big Sur

Getting Started

Clone the git repo

git clone [email protected]:jhu-sheridan-libraries/catalyst-ansible.git

Create vault password file

mkdir ~/.ssh/catalyst-ansible
touch ~/.ssh/catalyst-ansible/vault_password_file

Go to lastpass, search for "catalyst-ansible" to retrieve the vault password, and put it in the vault_password_file file

Install the required roles. You may check the requirements.yml file for details.

ansible-galaxy install -r requirements.yml

Create ~/.ansible.ini if it doesn't exist. Put the following in the file:

[cross-project]
remote_user = <jhedid>
login_user  = <jhedid>
login_group = msel-libraryapplications

Create and copy SSH keys

We are using our GitHub SSH keys to deploy and provision the development environment. If you were able to clone the repo above you have setup SSH based login with GitHub. You can find your public key here: https://github.com/<github-username>.keys. Your private key is stored in the ~/.ssh folder.

You will need to add your private key to your ssh agent like this (on macOS):

ssh-add -K ~/.ssh/id_rsa

This article has more detailed instructions for (setting up key forwarding)[https://docs.github.com/en/developers/overview/using-ssh-agent-forwarding] and explains why using key forwarding simplifies the deployment process.

Put the following in ~/.ssh/config and add your JHEDID to the User line:

# --- Catalyst ---
Host catalyst catalyst.library.jhu.edu
        Hostname catalyst.library.jhu.edu
        User <jhedid>
        ForwardAgent yes
        StrictHostKeyChecking no
# ----------------------------

# --- Catalyst Staging ---
Host catalyst-stage catalyst-stage.library.jhu.edu
        Hostname catalyst-stage.library.jhu.edu
        User <jhedid>
        ForwardAgent yes
        StrictHostKeyChecking no
# ----------------------------

# --- Catalyst Test ---
Host catalyst-test catalyst-test.library.jhu.edu
        Hostname catalyst-test.library.jhu.edu
        User <jhedid>
        ForwardAgent yes
        StrictHostKeyChecking no
# ----------------------------

# --- Catalyst Vagrant ---
Host catalyst-dev.test
	   User <jhedid>
     ForwardAgent yes
	   StrictHostKeyChecking no
Host catsolrmain-dev.test
	   User <jhedid>
     ForwardAgent yes
	   StrictHostKeyChecking no
Host catsolrreplica-dev.test
	   User <jhedid>
     ForwardAgent yes
	   StrictHostKeyChecking no
# ------------------

Copy your ssh key to the remote server. You will be prompted to enter your JHED password.

ssh-copy-id -i ~/.ssh/id_rsa catalyst-stage

Verify that you can ssh to stage without login

ssh catalyst-stage

Local Vagrant environment

*NOTE: If using VirtualBox 6.1.28 (at least on a Mac), it is necessary to modify/create the /etc/vbox/networks.conf file such that it has one and only one line in it: * 0.0.0.0/0 ::/0. See: https://forums.virtualbox.org/viewtopic.php?f=7&t=104218#p507770

1. Edit setup.yml to indicate that you are using vagrant:

---
- name: create login user for installation & configuration
  hosts: all

  vars:
    using_vagrant: true

  roles:
  - { role: login-user, tags: ['login-user'] }

2. Run vagrant up

This step will download the VMs and setup the hostnames. If this is your first time downloading the VMs the amount of time this takes will depend on your connection speed. After they have been cached this step takes around 2 minutes.

3. Run ansible-playbook playbooks/catalyst.yml -i inventory/vagrant --extra-vars "app_branch=main”

Create a key on your local development machine with a generic name:

4. Connect to the VPN. The Vagrant environment relies on servers in the test environment.

5. If the playbook is completed successfully you should be able to visit (catalyst-dev.test)[https://catalyst-dev.test] and see the catalyst home page after type thisisunsafe if you are using Chrome. The local environment uses a self-signed certificate for https which Chrome will block. Other browsers will have different methods of getting around this.

6. The other applications that are provisioned and deployed by this repo can also be installed locally:

Solr:

ansible-playbook playbooks/solr.yml -i inventory/vagrant

Services (includes the pull reserves setup):

ansible-playbook playbooks/services.yml -i inventory/vagrant

Horizon Web Service:

ansible-playbook playbooks/horizonws.yml -i inventory/vagrant

This has a manual step currently beause it assumes that catalyst has been installed first:

cd /opt/catalyst/current ; cp -a horizon-servlet /opt/catalyst

Traject:

ansible-playbook playbooks/services_install_traject.yml -i inventory/vagrant

Deployment

The usual tasks of catalyst deployment is to deploy the blacklight-rails app.

Deploy blacklight-rails

To deploy the blacklight app, run the commands list below. Replace <branch_or_tag_name_or_commit_hash> with a release tag, or a branch name. For example, you may use v1.4.14, master, or hotfix/v1.4.11.

You will be prompted to enter a BECOME password. This will be your JHED password.

• To catalyst-prod.library.jhu.edu

ansible-playbook playbooks/catalyst_install.yml -i inventory/prod --extra-vars "app_branch=<branch_or_tag_name_or_commit_hash>" -v -K

• To catalyst-stage.library.jhu.edu

ansible-playbook playbooks/catalyst_install.yml -i inventory/stage --extra-vars "app_branch=<branch_or_tag_name_or_commit_hash>" -v -K

• To catalyst-test.library.jhu.edu

ansible-playbook playbooks/catalyst_install.yml -i inventory/test --extra-vars "app_branch=<branch_or_tag_name_or_commit_hash>" -v -K

• To local dev vms

vagrant up
ansible-playbook playbooks/catalyst_install.yml -i inventory/vagrant --extra-vars "app_branch=<branch_or_tag_name_or_commit_hash> force_default_jhed=<username>" -v

Deploy the web service

This is very rare. Only necessary if you need to update the Java based web service at https://github.com/jhu-sheridan-libraries/horizon-holding-info-servlet

After packaging the war file (See https://github.com/jhu-sheridan-libraries/horizon-holding-info-servlet for instructions), copy the war file in the target directory to blacklight-rails/horizon-servlet/deploy/ws.war

Git commit the new war file. Push it to the remote. Create a new release of Catalyst. Deploy the new release (See steps above).

Then run the ansible playbook playbooks/horizonws_install.yml to release the war file. For example,

• To catalyst-prod.library.jhu.edu

ansible-playbook playbooks/horizonws_install.yml -i inventory/prod -v -K

Deploy the Solr servers

The ansible scripts also include steps to upgrade solr servers. This is only necessary when the solr server needs to be updated. So ignore this section most of the time. Also note that this section is not very well maintained. Use with discretion.

To delete all the data from a core

curl http://localhost:8983/solr/catalyst/update?commit=true -H "Content-Type: text/xml" --data-binary '<delete><query>*:*</query></delete>'

Server Component Upgrades

Upgrade Ruby

We assume that the application has been tested on dev, and either test/stage.

Follow the following checklist

• Update the production inventory, choose ruby version, and app_branch
• Check you have ssh [email protected] works
• Check free space on server (df -h ) | require at least 2G
• Run the catalyst.yml playbook on production

time ansible-playbook -i inventory/prod playbooks/catalyst.yml  --limit=catalyst -v

• This takes 13m to run, 4m of downtime
• Keep an eye on the logs for errors
• Run the cucumber test locally

Note: Fallback plan you can pass in the ruby, and app version to revert.

#Role Back
time ansible-playbook -i inventory/prod playbooks/catalyst.yml   --extra-vars="app_branch=v1.1.4"  --extra-vars="chruby_ruby_version=ruby-2.2.2" --limit=catalyst -v  # 2m23s

For MacOSX 10.13.3 updated the default python which has introduced an issue with python affecting ansible #32499

TASK [jetty : get jetty checksum] *******************************************************************
objc[86359]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[86359]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

As a workaround, adding the 'export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES' to your ~/.profile. A fix mas been merged into the unreleased ansible-2.5 branch.

Deploy solr main and replica

To local dev vagrant vm Update your local Vagrantfile and add catsolrmain-dev and catsolrreplica-dev Update your local inventory/vagrant file and ensure the solr group is present

vagrant up ansible-playbook playbooks/solr.yml -i inventory/vagrant

Deploy to Test ansible-playbook playbooks/solr.yml -i inventory/test -v -K

Changing Service Cron times

To just update the cron configuration for catalyst-traject and catalyst-pull-reserves

time ansible-playbook -i inventory/test  playbooks/services_install_traject.yml  --tags=cron -v

took 21s

Deploy traject

ansible-playbook -i inventory/test playbooks/services_install_traject.yml --tags=services -v -K

catalyst-ansible's Issues

test connectivity between catalyst and hip server

add ability to automatically test connectivity upon configuration. something as simple as curling the homepage and searching for a string would do, but there may be a more ansible-y way to get feedback on the success of our configuration

Shib failing to install

Apparently, the official yum repo for shib (http://download.opensuse.org/repositories/security://shibboleth/CentOS_7/) is having issues wherein most of the mirrors it knows about return 403 or 404 errors when asked for the needed packages.

Originally reported in jhu-library-devops/findit-ansible#10

RFC: "somewhat experimental"

Objective: Determine usefulness / necessity for the following code block in catalyst's /etc/httpd/conf.d/01_catalyst.conf:

  ### TODO: research
  # Let apache serve the pre-compiled .gz version of static assets,
  # if available, and the user-agent can handle it. Set all headers
  # correctly when doing so.
  #
  # SOMEWHAT EXPERIMENTAL. If you think it's causing problems,
  # just remove the following three LocationMatch.
  <LocationMatch "^/assets/.*\.(css|js)$">
    RewriteEngine on

    # Make sure the browser supports gzip encoding before we send it,
    # and that we have a precompiled .gz version.
    RewriteCond %{HTTP:Accept-Encoding} \b(x-)?gzip\b
    RewriteCond %{REQUEST_FILENAME}.gz -s
    RewriteRule ^(.+)$ $1.gz
  </LocationMatch>

  # Make sure Content-Type is set for 'real' type, not gzip,
  # and Content-Encoding is there to tell browser it needs to
  # unzip to get real type.
  #
  # Make sure Vary header is set; while apache docs suggest it
  # ought to be set automatically by our RewriteCond that uses an HTTP
  # header, does not seem to be reliably working.
  <LocationMatch "^/assets/.*\.css\.gz$">
      ForceType text/css
      Header set Content-Encoding gzip
      Header add Vary Accept-Encoding
  </LocationMatch>

  <LocationMatch "^/assets/.*\.js\.gz$">
      ForceType application/javascript
      Header set Content-Encoding gzip
      Header add Vary Accept-Encoding
  </LocationMatch>

Performance on services VM is poor

After installation of JRuby, a java process (apparently JRuby) gets started whenever ansible is doing work on the VM. CPU usage climbs to over 100% before dropping, only to climb again. Ansible task completion is noticeably slower than usual, despite having nothing to do with JRuby itself (e.g. gathering facts or deploying ssh keys. To resolve, I suggest we start by troubleshooting JRuby, as described here: https://github.com/jruby/jruby/wiki/Troubleshooting-Performance

RFC: "test" block in apache config

There's a small block in catalyst's /etc/httpd/conf.d/01_catalyst.conf labeled "TODO: test". Is it needed or helpful?

  ### TODO: test
  # Cache Rails finger-printed assets, as per
  # http://guides.rubyonrails.org/asset_pipeline.html#precompiling-assets
  # Try only match if the asset actually has a fingerprint in it.
  <LocationMatch "^/assets/.*-[0-9a-f]{32}.*$">
  #<Location /assets/>
    # Use of ETag is discouraged when Last-Modified is present
    Header unset ETag
    FileETag None
    # RFC says only cache for 1 year
    ExpiresActive On
    ExpiresDefault "access plus 1 year"
  </LocationMatch>

Catalyst HTTPS redirect

Topic: the HTTPS redirect on catalyst
Both -prod and -stage use the following code in /etc/httpd/conf.d/01_catalyst.conf:

# BEGIN http redirect
  # We run Catalyst/Blacklight in the main virtual host, and we require
  # https access. So this :80 virtual host insists on redirects to https.
  #
  # This could also be put in <Location> directives if it wasn't the whole
  # server that was so restricted.
  #
  # Except Refworks sometimes refuses to make callbacks to https, so we
  # have to allow refworks-formatted exports in http.
  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteCond %{REQUEST_URI} !refworks_marc_txt$
  RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R,L]

# END http redirect

The proposed change/simplification is:

# BEGIN http redirect
  Redirect "/" "https://{{ hostname | default('myapp') }}.{{ domainname | default('test.test') }}"
# END http redirect

Is this ok? Will some rare condition not be fulfilled? Will something be lost?