GithubHelp home page GithubHelp logo

jaeg / hats-worker Goto Github PK

View Code? Open in Web Editor NEW
1.0 2.0 0.0 6.36 MB

A simple interpreter designed to process data sitting in redis.

License: Apache License 2.0

Go 90.90% Makefile 3.26% Smarty 5.85%
redis javascript golang distributed-systems cluster interpreter

hats-worker's Introduction

hats-worker

H - Highly
A - Available
T - Tasking
S - System
A simple interpreter designed to process data sitting in redis.

Each worker in a cluster checks in redis for work to do. If it finds a stopped thread or a dead thread it takes the thread and runs it locally. There are also jobs which instead of continuously running they execute on a cron schedule.

Runtime params

  • cluster-name - name of cluster
  • worker-name - name of the worker
  • redis-address - address to redis server
  • redis-password - password for redis server
  • scripts - scripts to register
  • run-now - run registered scripts on this worker immediately

Getting dependencies

Requires a version of go that supports go.mod

  • go get

Get up and running

  • Build it
    • make build
  • Build and run it
    • make run
  • You can get started using an example config as such
    • ./bin/worker --config worker1.config
  • Or you can pass in through runtime params
    • ./bin/worker --redis-address=<address> --redis-password=<password> --worker-name=worker1
  • Or run through a docker container
    • docker run jaeg/hats-worker:latest --redis-address=<address> --redis-password=<password> --worker-name=worker1
  • You can also tell it to load a script on start like this
    • ./bin/worker --redis-address=<address> --redis-password=<password> --worker-name=worker1 --scripts examples/hello.js
    • docker run jaeg/hats-worker:latest --redis-address=<address> --redis-password=<password> --worker-name=worker1 --scripts examples/hello.js

Hatter

Hatter is a tool used to deploy and maintain a HATS cluster. https://github.com/jaeg/hatter

Example Environment to deploy

https://github.com/jaeg/eliza_hats

Javascript implementation

worker's Javascript implementation is based on Otto. Each thread maintains its own scope. When a thread starts it runs the entire script. It then runs init() if present in the source code. If present a thread will call main() after confirming the thread is still running.

Extra Functions Available to scripts

Env

  • env.Get(key)
    • returns string or undefined if key does not exist
  • env.Set(key,value)
    • returns null or error if there is one
  • env.Unset(key)
    • returns null or error if there is one

worker

  • worker.Name
    • returns string
  • worker.Cluster
    • returns string
  • worker.ShuttingDown - It is suggested that if you have code that loops you also check this to make sure the code end cleanly.
    • returns bool

Thread

  • thread.Key
    • returns string
  • thread.State()
    • returns string
  • thread.Status()
    • returns string
  • thread.Stopped - It is suggested that if you have code that loops you also check this to make sure the code end cleanly.
    • returns bool
  • thread.Disable() - Disables the thread completely
    • returns nothing
  • thread.Stop() - Stops the thread causing another node to possibly pick it up
    • returns nothing

Job

  • job.Key
    • returns string
  • job.State()
    • returns string
  • thread.Status()
    • returns string
  • thread.Stopped - It is suggested that if you have code that loops you also check this to make sure the code end cleanly.
    • returns bool
  • thread.Disable() - Disables the thread completely
    • returns nothing

HTTP

  • http.Get(url)
    • returns {body:'',headers:[], status: 200}
  • http.Post(url, body)
    • returns {body:'',headers:[], status: 200}
  • http.PostForm(url, bodyObject)
    • returns {body:'',headers:[], status: 200}
  • http.Put(url, body)
    • returns {body:'',headers:[], status: 200}
  • http.Head(url)
    • returns {headers:[], status: 200}
  • http.Delete(url)
    • returns {body:'',headers:[], status: 200}

Redis

  • redis.Do(method, args....)
    • return [response from redis, error (if one)]

Response

  • response.Write(value)
    • Used when a worker is in endpoint mode. Writes to the response body of an http request.
    • returns nothing.
  • response.Error(errorString, statusCode)
    • Used when a worker is in endpoint mode. Writes an error in response to the endpoint.
    • returns nothing.
  • response.SetContentType(type)
    • Used when a worker is in endpoint mode. Sets the content type header.
    • returns nothing.
  • response.SetHeader(key,value)
    • Used when a worker is in endpoint mode. Sets the header specfied in key to the value.
    • returns nothing.

Request

  • request.Method
    • returns string
  • request.Path
    • returns string
  • request.Query
    • returns {:[values]}
  • request.Body
    • returns string
  • request.GetHeader(key)
    • Used when a worker is in endpoint mode. Gets value from header.
    • returns value.

SQL

  • sql.New(connectionString, driverType)
    • supported driver types: mysql
    • returns db interface
  • db.Ping()
    • returns an error if it fails to ping the db
  • db.Query(query, arguments...)
    • returns rows or nil if error
  • db.Exec(query, arguments...)
    • returns number of impacted rows or undefined if error
  • db.Close()
    • returns error if any

worker Todo

  • - Run a thread from redis.
  • - Create thread from file.
  • - Stop thread if worker is unhealthy.
  • - Stop thread if status is disabled.
  • - Stop thread if not the owner of thread.
  • - CPU health check based on threshold.
  • - Memory health check based on threshold.
  • - Recover from critical state when thresholds are met.

Javascript Todo

  • - Basic javascript implementation
  • - Keep scope inside of thread
  • - Redis wrapper
  • - Http wrapper
  • - worker information i.e. Health, name, cluster
  • - Thread information i.e. Delay, State, Status
  • - Thread control i.e. Stop thread, disable thread.

hats-worker's People

Contributors

jaeg avatar

Stargazers

 avatar

Watchers

 avatar  avatar

hats-worker's Issues

Thread Scheduling

Right now threads run continuously with no control over when they run, how often they run, and how fast they loop again.

New Thread fields:

  • NumberOfCycles : How many times to loop when a thread is ran. -1 to not stop looping.
  • Frequency: Follow cron job formatting.
  • CycleDelay: How long to wait between loops.
  • DeadSeconds: Amount of time between heartbeats to consider thread dead.

Support the ability to use "require" in JS

To allow scripts to have dependencies implement a version of "require" that either looks up the code from disk or redis and runs it in the otto vm to get it into memory and usable by other parts of the script.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.