jaeg / hats-worker

A simple interpreter designed to process data sitting in redis.

hats-worker

H - Highly
A - Available
T - Tasking
S - System
A simple interpreter designed to process data sitting in redis.

Each worker in a cluster checks in redis for work to do. If it finds a stopped thread or a dead thread it takes the thread and runs it locally. There are also jobs which instead of continuously running they execute on a cron schedule.

Runtime params

• cluster-name - name of cluster
• worker-name - name of the worker
• redis-address - address to redis server
• redis-password - password for redis server
• scripts - scripts to register
• run-now - run registered scripts on this worker immediately

Getting dependencies

Requires a version of go that supports go.mod

• go get

Get up and running

• Build it
  • make build
• Build and run it
  • make run
• You can get started using an example config as such
  • ./bin/worker --config worker1.config
• Or you can pass in through runtime params
  • ./bin/worker --redis-address=<address> --redis-password=<password> --worker-name=worker1
• Or run through a docker container
  • docker run jaeg/hats-worker:latest --redis-address=<address> --redis-password=<password> --worker-name=worker1
• You can also tell it to load a script on start like this
  • ./bin/worker --redis-address=<address> --redis-password=<password> --worker-name=worker1 --scripts examples/hello.js
  • docker run jaeg/hats-worker:latest --redis-address=<address> --redis-password=<password> --worker-name=worker1 --scripts examples/hello.js

Hatter

Hatter is a tool used to deploy and maintain a HATS cluster. https://github.com/jaeg/hatter

Example Environment to deploy

https://github.com/jaeg/eliza_hats

Javascript implementation

worker's Javascript implementation is based on Otto. Each thread maintains its own scope. When a thread starts it runs the entire script. It then runs init() if present in the source code. If present a thread will call main() after confirming the thread is still running.

Extra Functions Available to scripts

Env

• env.Get(key)
  • returns string or undefined if key does not exist
• env.Set(key,value)
  • returns null or error if there is one
• env.Unset(key)
  • returns null or error if there is one

worker

• worker.Name
  • returns string
• worker.Cluster
  • returns string
• worker.ShuttingDown - It is suggested that if you have code that loops you also check this to make sure the code end cleanly.
  • returns bool

Thread

• thread.Key
  • returns string
• thread.State()
  • returns string
• thread.Status()
  • returns string
• thread.Stopped - It is suggested that if you have code that loops you also check this to make sure the code end cleanly.
  • returns bool
• thread.Disable() - Disables the thread completely
  • returns nothing
• thread.Stop() - Stops the thread causing another node to possibly pick it up
  • returns nothing

Job

• job.Key
  • returns string
• job.State()
  • returns string
• thread.Status()
  • returns string
• thread.Stopped - It is suggested that if you have code that loops you also check this to make sure the code end cleanly.
  • returns bool
• thread.Disable() - Disables the thread completely
  • returns nothing

HTTP

• http.Get(url)
  • returns {body:'',headers:[], status: 200}
• http.Post(url, body)
  • returns {body:'',headers:[], status: 200}
• http.PostForm(url, bodyObject)
  • returns {body:'',headers:[], status: 200}
• http.Put(url, body)
  • returns {body:'',headers:[], status: 200}
• http.Head(url)
  • returns {headers:[], status: 200}
• http.Delete(url)
  • returns {body:'',headers:[], status: 200}

Redis

• redis.Do(method, args....)
  • return [response from redis, error (if one)]

Response

• response.Write(value)
  • Used when a worker is in endpoint mode. Writes to the response body of an http request.
  • returns nothing.
• response.Error(errorString, statusCode)
  • Used when a worker is in endpoint mode. Writes an error in response to the endpoint.
  • returns nothing.
• response.SetContentType(type)
  • Used when a worker is in endpoint mode. Sets the content type header.
  • returns nothing.
• response.SetHeader(key,value)
  • Used when a worker is in endpoint mode. Sets the header specfied in key to the value.
  • returns nothing.

Request

• request.Method
  • returns string
• request.Path
  • returns string
• request.Query
  • returns {:[values]}
• request.Body
  • returns string
• request.GetHeader(key)
  • Used when a worker is in endpoint mode. Gets value from header.
  • returns value.

SQL

• sql.New(connectionString, driverType)
  • supported driver types: mysql
  • returns db interface
• db.Ping()
  • returns an error if it fails to ping the db
• db.Query(query, arguments...)
  • returns rows or nil if error
• db.Exec(query, arguments...)
  • returns number of impacted rows or undefined if error
• db.Close()
  • returns error if any

worker Todo

• - Run a thread from redis.
• - Create thread from file.
• - Stop thread if worker is unhealthy.
• - Stop thread if status is disabled.
• - Stop thread if not the owner of thread.
• - CPU health check based on threshold.
• - Memory health check based on threshold.
• - Recover from critical state when thresholds are met.

Javascript Todo

• - Basic javascript implementation
• - Keep scope inside of thread
• - Redis wrapper
• - Http wrapper
• - worker information i.e. Health, name, cluster
• - Thread information i.e. Delay, State, Status
• - Thread control i.e. Stop thread, disable thread.

hats-worker's Issues

Create an AWS library for the JS implementation

Things to include:

• RDS
• MQ as a possible way to trigger scripts to run and add the ability for scripts to trigger events.
• SES
• S3

Thread Scheduling

Right now threads run continuously with no control over when they run, how often they run, and how fast they loop again.

New Thread fields:

• NumberOfCycles : How many times to loop when a thread is ran. -1 to not stop looping.
• Frequency: Follow cron job formatting.
• CycleDelay: How long to wait between loops.
• DeadSeconds: Amount of time between heartbeats to consider thread dead.

Create SSH library for JS implementation

Handle private keys, be able to ssh into other machines and execute commands.

Add library to allow JS to access environment variables or settings passed to node.

This will be helpful when we need secrets or other information to connect to things such as DBs and don't want them hard coded.

API to store and retrieve files from Hatbox.

We need JS api hooks to allow users to store and retrieve files from a hatbox that is connected to the system.

Support the ability to use "require" in JS

To allow scripts to have dependencies implement a version of "require" that either looks up the code from disk or redis and runs it in the otto vm to get it into memory and usable by other parts of the script.

Handle sigterms to gracefully shutdown and free up scripts that the node might be running.

In an attempt to make this work better with kube we should gracefully shutdown the node and free up in scripts that might be running to better assist in minimizing their downtime.

Implement way for hats-worker to pull source code from hatbox instead of redis.

Not sure if it should keep supporting code being hosted in redis. I think requiring hatbox to run is reasonable given the nature of hats.

Make thread crashes consumable by wart control

We need new fields for Threads. Error date, Error Text. On a script failure put both of these in redis.