jakesidsmith / watfish

Simple development platform with process management & router

watfish

Simple development platform with process management & router

About

Watfish is designed to allow you to easily run multiple processes in various different environments inside the context of a project.

It loads processes from an etc/environments/${env-name}/procfile inside your project directory and assigns them random ports, which are then routed from a url of your choice defined in a wtf.json in your home directory.

All processes are provided with your global environment variables, variables defined in etc/environments/${env-name}/env, and sensitive environments variables (that should not be included in your project), from the wtf.json.

Environment variables can be used within your procfile (e.g. $PORT) and will be substituted when the project is started.

It also looks in env/bin inside your project for suitable commands to run within the project context, so when starting, for example, a python process, it will use the python version that is installed within your virtual environment.

Watfish provides several commands to easily help you setup a project (wtf init), and define sensitive environment variables in your wtf.json (wtf env set, wtf env get, wtf env del).

Install

Run the following command from any terminal

npm install watfish -g

Setup route for project

Run the following command inside your project directory

wtf init

Then follow the on screen instructions

What is the name of the process you would like to route?
web
From what url would you like to route this process?
url.com

Created config:

{
  "routes": {
    "web": "url.com"
  }
}

Is this correct? [y]

Usage

Now you can go ahead and start your project or run arbitrary commands in the project's environment.

wtf start

wtf runtests

Here's some of the commands that are available

  Usage: wtf command [<sub-command>] [options]

  Commands:
    env, e    Change local project environment variables
    init, i   Generate project config (in ~/wtf.json)
    run, r    Run arbitrary commands in the environment
    start, s  Start project processes

  Options:
    --help, -h     Display help and usage info
    --version, -v  Display version
    <command>      Arbitrary command to run in the environment

  Examples:
    wtf init
    wtf env set KEY value
    wtf start
    wtf start --env production
    wtf run manage.py migrate

procfile and env

Procfiles and env files should be stored in a directory named after the environment you want to run them in, inside etc/environments within your project. E.g. project/etc/environments/development/procfile & project/etc/environments/development/env.

By default watfish will run all processes in the development environment, but you can start processes in an environment of your choice by providing the --env argument. E.g. wtf start --env staging.

An example procfile might look like the following.

web: http-server ./ -p :$PORT
watch-js: watchify -t babelify src/index.js -o build/index.js
watch-less: watch-less-do-more -u postcss -i src/index.less -o build/index.less

An example env file might look like the following.

DEBUG=true
ALLOWED_HOSTS=localhost

wtf.json

Routes & sensitive environment variables are loaded from a wtf.json in your home directory.

If this is your first time running watfish on your machine, or for a new project, you can run wtf init to setup a route for this project and create a wtf.json for you.

You can run wtf init for as many projects as you like and additional config will be added to the wtf.json.

Each project will receive its own key within the wtf.json based on the its directory name.

You can also run wtf init for a single project more than once. This will, however, erase any existing config for that project.

You can manually edit the wtf.json to add, remove, or adjust routes & environment variables, or use the provided commands.

An example wtf.json might look like the following.

{
  "project-name": {
    "routes": {
      "web": "sub.domain.com"
    },
    "env": {
      "SECRET_KEY": "********"
    }
  }
}

Quick setup

Initialize the project:

wtf init

Answer questions about routing:

What is the name of the process you would like to route?
web
From what url would you like to route this process?
example.domain.com

Created config:

{
  "routes": {
    "web": "example.domain.com"
  }
}

Is this correct? [y]

wtf.json written to /Users/user/wtf.json

Add a development environment variable:

wtf env set SECRET_KEY my-secret-key

Created config:

{
  "development": {
    "SECRET_KEY": "my-secret-key"
  }
}

Is this correct? [y]

wtf.json written to /Users/user/wtf.json

Add a production environment variable:

wtf env set SECRET_KEY another-secret-key --env production

Created config:

{
  "development": {
    "SECRET_KEY": "my-secret-key"
  },
  "production" : {
    "SECRET_KEY": "another-secret-key"
  }
}

Is this correct? [y]

wtf.json written to /Users/user/wtf.json

This will have created a wtf.json in your home directory with the following contents:

{
  "project": {
    "routes": {
      "web": "example.domain.com"
    },
    "env": {
      "development": {
        "SECRET_KEY": "my-secret-key"
      },
      "production" : {
        "SECRET_KEY": "another-secret-key"
      }
    }
  }
}

watfish's Issues

Ensure ports are not already used

Although we check to find an available port, because this is done asynchronously, some OS's may assign the same port to 2 different processes.

Quoted env vars contain quotes

ALLOWED_HOSTS="*" resolves to '"*"' instead of '*'

https://github.com/JakeSidSmith/watfish/blob/master/src/utils.ts#L185

Should consider: https://www.npmjs.com/package/dotenv

const envConfig = dotenv.parse(fs.readFileSync('.env.override'))

Do not assume the procfile path

Currently, Watfish assumes the path to the Procfile which should be overridable on a per-project basis (or when the command is run).

Exclude processes from running

wtf start --exclude watch-js
wtf start -e watch-js

Display current time HH:mm:ss in logs

Enable with --time -t

Display error when env variables in procfile are missing

E.g. web: manage.py runserver 0.0.0.0:$PORT

Get's parsed as:

{
  command: 'manage.py',
  options: ['runserver', '0.0.0.0:$PORT']
}

Because the routes have not been configured the PORT / url are undefined and causes:

CommandError: "0.0.0.0:" is not a valid port number or address:port pair.

Should display an error if env variables are missing.

Possibly should also generate a PORT for every process, regardless of routes?

Grep logs

Add --grep or similar option to filter logs by process name

Return json for invalid requests to API with content type json

Currently watfish displays something like this:

When requesting an unknown route, but this message should be returned as json if that is the content type that is requested.

Projects race to start new router

If more than 2 projects are running and you kill the project that started the initial router, the remaining projects will race to start a new router. Only one of these will succeed.

To solve this I could either:

• Have the router assign another project to start a new router if it is killed or...
• Try to start a new router in all projects, but gracefully fail and continue running

Start in env roughly matching provided value

E.g. --env dev should match development.
--prod should match production.

Log routes with `http` prefix so they can be clicked in VSCode terminal

Configure secret env variables

It should be possible to use wtf env to set env variables in the global wtf.json that should not be included in the repo itself.

Incompatible with node 4

Cannot locate bash executable

Running /bin/bash build
PID: undefined, Parent PID: 86803

events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: spawn /bin/bash build ENOENT
    at exports._errnoException (util.js:837:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:178:32)
    at onErrorNT (internal/child_process.js:344:16)
    at doNTCallback2 (node.js:429:9)
    at process._tickCallback (node.js:343:17)
    at Function.Module.runMain (module.js:477:11)
    at startup (node.js:117:18)
    at node.js:951:3

Ensure a project cannot be started twice

Running wtf start should check with an existing router if that project is already running. If so the user should receive a warning.

Allow piping start

Currently running wtf start | grep watch-js has no effect

Apps hang if url accessed before web process is fully running