discoposse / nomad-vagrant-lab

A nifty little 3-node or 6-node local cluster configuration to run HashiCorp Nomad

HashiCorp Nomad - Local Lab Using Vagrant

Accompanying blog for the initial setup: https://discoposse.com/2019/11/21/building-a-hashicorp-nomad-cluster-lab-using-vagrant-and-virtualbox/

Pluralsight course - Getting Started with HashiCorp Nomad: https://app.pluralsight.com/library/courses/hashicorp-nomad-getting-started/table-of-contents

What is this?

A simple 3-node or 6-node lab running Ubuntu servers on VirtualBox and each node runs Consul and Nomad servers which can be configured as a cluster.

Why use this method?

This is a great way to get your feet wet with Nomad in a simplified environment and you also have a chance to mess around with the configurations and commands without risking a cloud (read: money) installation or a production (read: danger!) environment.

Requirements

There are a few things you need to get this going:

• Vagrant

• VirtualBox

How to use the Nomad lab configuration

For 3-node clusters you must rename Vagrantfile.3node to Vagrantfile

For 6-node (two region) clusters you must rename Vagrantfile.6node to Vagrantfile

• Clone this repo (or fork it of you so desire and want to contribute to it)

• Change directory and run a vagrant status to check the dependencies are met

• Run a vagrant up command and watch the magic happen! (spoiler alert: it's not magic, it's technology)

• Each node will able to run Consul and Nomad

To start your Nomad cluster just do this:

• Connect to the first node (either nomad-a-1 or nomad-b-1) using the vagrant up using vagrant ssh <nodename> where <nodename> is the instance name (e.g. nomad-a-1, nomad-b-1).
• Change directory to the /vagrant folder using cd /vagrant
• Launch Nomad using the shell script which is sudo <nodename>.sh where <nodename> is the node you are running on (e.g. sudo launch-a-1.sh)
• Connect to the remaining two nodes (nomad-a-2, nomad-a-3) and repeat the process of changing to the /vagrant folder and running the appropriate launch script

The first node in each of the set of three will begin as the leader. The other two node launch scripts have a nomad server join command to join the cluster with the first node.

Once you're used to the commmands, you can start and stop as much as needed.

Consul is installed but not used for the basic configuration. More to come on that.

Now you're running!

Interacting with the Nomad and Consul cluster

Logging into the systems locally can be done

• You can use some simple commands to get started

nomad node status

• To open the Nomad UI use this command on your local machine

open http://172.16.1.101:4646

nomad-vagrant-lab's Issues

Node Install Scripts Cause Issues On Subsequent Provisioning

Although this may not be an issue in all cases, we should account for these scenarios. When spinning up the Vagrant lab the scripts work just fine. But if we decide to run vagrant provision after the environment is up and running, the scripts throw many errors and cause failures.

Optimize Launch Scripts

The launch scripts work fine except on subsequent executions. We begin to experience errors, etc. which are less than optimal. We need to ensure that we are only killing processes when they are actually already running. This will allow for future potential of streamlining the spin up of auto starting the services at Vagrant provisioning time.

Direction of scripts for spinning up Consul/Nomad clusters

@discoposse What are your thoughts on the direction of the scripts to spin up Consul/Nomad clusters?

Now that I know this repo is for usage of the Pluralsight course as well, is your idea that the user is to login to each node and spin up these services? Which would lead them into further learning potentially? Or does it make sense to have these services up, running and ready for interaction?

Vancouver site for Consul/Nomad Is Wrong

One of the previous commits mistakingly changed Vancouver to also be Toronto. I'll be submitting a PR to resolve this.

Cannot find '/vagrant/nomad-server.hcl' during provisioning

When doing a vagrant up and the shell script provisioning kicks in we are presented with the following error (for each host):

nomad-a-1: cp:
nomad-a-1: cannot stat '/vagrant/nomad-server.hcl'
nomad-a-1: : No such file or directory

The script is failing at line#35

Need to add nohup back to startup script(s)

When the scripts were changed in #4, we removed the nohup in the startup commands. This leaves flowing logging to stdout which makes it less than ideal for executing commands on the nodes. Especially when walking through the Pluralsight course. I will be submitting a PR to add nohup back.

README and Blog Post should be in sync? ;)

So, following the blog post the following instructions seem incorrect:

Connect via ssh to the first node using the command vagrant ssh nomad-a-1
Change to the code folder with the command cd /vagrant
Launch the Nomad script with this command: sudo sh launch nomad-a-1.sh
Connect via ssh to the second node using the command vagrant ssh nomad-a-2
Change to the code folder with the command cd /vagrant
Launch the Nomad script with this command: sudo sh launch nomad-a-2.sh
Connect via ssh to the third node using the command vagrant ssh nomad-a-3
Change to the code folder with the command cd /vagrant
Launch the Nomad script with this command: sudo sh launch nomad-a-3.sh

I believe they should be:

Connect via ssh to the first node using the command vagrant ssh nomad-a-1
Change to the code folder with the command cd /vagrant
Launch the Nomad script with this command: sudo sh launch-a-1.sh
Connect via ssh to the second node using the command vagrant ssh nomad-a-2
Change to the code folder with the command cd /vagrant
Launch the Nomad script with this command: sudo sh launch-a-2.sh
Connect via ssh to the third node using the command vagrant ssh nomad-a-3
Change to the code folder with the command cd /vagrant
Launch the Nomad script with this command: sudo sh launch-a-3.sh

Nomad Cluster Does Not Form Because Of Consul Cluster Not Available

When attempting to spin up Nomad cluster it fails because the backend Consul cluster is not up and running. We need to tweak the spin up script(s) to form the Consul cluster first before forming the Nomad cluster.