theycallmemac / odin Goto Github PK

A programmable, observable and distributed job orchestration system.

License: MIT License

Makefile 0.60% Go 58.18% Python 4.93% JavaScript 9.84% TypeScript 19.25% CSS 4.47% HTML 1.58% Shell 1.04% Pug 0.11%

go golang python nodejs typescript job-scheduler distributed-systems raft cron observability

odin's Introduction

📖 Overview

Odin is a programmable, observable and distributed job orchestration system which allows for the scheduling, management and unattended background execution of user created tasks on Linux based systems.

Job schedulers by definition are supposed to eliminate toil, a kind of work tied to running a service which is manual, repetitive and most importantly, automatable. While Odin cares about what is to be executed and when it will be executed, Odin is equally concerned with the expected behaviour of your job, which is described entirely by the user’s code.

This observability is achieved through a web facing user interface which displays job logs and metrics. All of this is gathered through the use of Odin Libraries written in Go, Node.js and Python.

Odin can infer the internal state of jobs by leveraging these libraries, which in turn can be used to speed up the debugging process associated with jobs which yield unexpected behaviour.

🔧 Installation

To install Odin please consult the installation guide for a quick walkthrough on setting up the system.

🚀 Usage

You can check out the documentation for using Odin here.

👤 Author

James McDermott

⭐️ Show your support

Give a ⭐️ if this project helped you!

📝 License

odin's People

Contributors

Stargazers

Watchers

Forkers

hamzah-z hiepd triplekill przor3n elijah gspu tawawhite zeta1999 isgasho anarchycrew cnglish ylmcc achillesxu uniun-technology davincidreams yangguoandresolution reconbug iq-scm astrolemonade

odin's Issues

Automate SDK publishing

Is your feature request related to a problem? Please describe.
Publishing new versions of the Odin librarys to the likes of NPM and Pypi should be automated.

Describe the solution you'd like
This is probably best if automated using Github Actions. It seems Github Actions offers templates for this sort of thing, at least with respect to Python and Node.js - languages like Rust may not have such templating.

Describe alternatives you've considered
There's any number of ways this can be done, however it's probably best to regulate from this repo.

Additional context
N/A

Make change to logging job results

Currently, the Odin-Engine stores job execution logs in a given format using logrus. It seems zap is more performant.

[BUG] Generate without yml extension does not raise exception

Describe the bug
Trying to generate job file using odin cli without yml extension just terminates without any error messages.

To Reproduce

./odin generate -f ./example -l go

Expected behavior
If extension is required, the command should return an error so users know what to expect.

Convert PDF user documentation to markdown

Best move the docs found here to markdown files in the repo.

[Feature Request] Support Cron Scheduling Syntax

For those who are accustomed to writing the cron schedule syntax, the Odin Engine should be able to accept regular cron notation as a means by which to schedule jobs.

This is actually quite a simple thing to do, but issue #33 takes precedence.

Add Documentation on SDK's

Specifically docs should be added in relation to the Node.js and future Bash SDK.

Add Badges

Badges to be added to this project:

Go Report
MIT License

[BUG] Panic during odin log

Describe the bug
Deploying a new job, then pulling logs via the CLI results in a panic (SIGSEGV).

To Reproduce

odin deploy -f test.job.yaml
odin log -i {job_id}

vagrant@vagrant:~/odin$ odin log -i dea194794589
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x79c764]

goroutine 1 [running]:
github.com/theycallmemac/odin/odin-cli/pkg/commands.logJob(0x7ffda66ec793, 0xc, 0x868da6, 0x5)
        /home/vagrant/odin/odin-cli/pkg/commands/log.go:53 +0x154
github.com/theycallmemac/odin/odin-cli/pkg/commands.glob..func7(0xbb6c80, 0xc00000e720, 0x0, 0x2)
        /home/vagrant/odin/odin-cli/pkg/commands/log.go:25 +0xda
github.com/spf13/cobra.(*Command).execute(0xbb6c80, 0xc00000e6e0, 0x2, 0x2, 0xbb6c80, 0xc00000e6e0)
        /home/vagrant/go/pkg/mod/github.com/spf13/[email protected]/command.go:844 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xbb79a0, 0x443a8a, 0xb7f4c0, 0xc000000180)
        /home/vagrant/go/pkg/mod/github.com/spf13/[email protected]/command.go:945 +0x317
github.com/spf13/cobra.(*Command).Execute(...)
        /home/vagrant/go/pkg/mod/github.com/spf13/[email protected]/command.go:885
github.com/theycallmemac/odin/odin-cli/pkg/commands.Execute()
        /home/vagrant/odin/odin-cli/pkg/commands/root.go:33 +0x31
main.main()
        /home/vagrant/odin/odin-cli/cmd/odin/odin.go:6 +0x20

Expected behavior
I believe just no output, until the job runs the first time.

Screenshots
n/a

Desktop (please complete the following information):

OS: Ubuntu 18 LTS (vagrant on macOS)

Odin Engine API Framework Performance

Currently the Odin Engine is built on the Chi framework. This was selected as it was lightweight, idiomatic and very fast to setup.

Now I'm looking at potentially changing the Engine API framework to one of the following:

I will leave further comments in relation to my findings.

Semantic checking of YAML configs

Up until this point its a field that has gone unused, but it was included since the beginning because it was easier to include in from the start, so config files would change much in future versions.

Seeing as v2.0.0 is almost here, the Odin version info found in the generated YAML config can be used for semantic checks during job deployment. It can potentially be used to indicate any deprecation and so forth.

[DOCS] python is not a valid language identifier

The docs indicate -l python should be used, but the source indicates the identifier is actually python3. I would be happy to update the docs, but I wanted to first ask if which identifier is desired/correct (or perhaps both?).

[Automation Request] Add CLI tests to Github Action

Currently the Odin CLI Github Action builds the tool, another job should be run which runs the tests after a successful build.

[Feature Request] Remove Yaml config files

We'd like to remove YAML configuration from jobs in favor of "just" code. This feature is a move away from the need for generating any YAML. All metadata like the name, schedule and so on can be built into the main language file.

This is a very large change to the engine so I myself will take this on, it's sure to touch a lot of areas only I have come in contact with previously.

[Automation Request] Linting and Formatting

Is your feature request related to a problem? Please describe.
Currently the absence of linting/formatting automation and documentation on preferred coding styles is a barrier to entry for new contributors.

Describe the solution you'd like
We can improve automation around linting/formatting with Git Hooks and Github Actions as a way of lowering the barrier to entry for contributors, in turn yielding a lower maintainer burden over these concerns.

Linting and formatting tools need to be implemented and for:

Go (golint and gofmt)
Node.js (eslint and prettier)
Python (pylint and black)
Bash (shellcheck and beautysh)

Along with this, these tools should be outlined in the CONTRIBUTING.md file.

[Feature Request] Provide Docker images

Is your feature request related to a problem? Please describe.
Not a problem per se, but just a thought for lowering the barrier to entry.

Describe the solution you'd like
It'd be convenient to publish a docker image for Odin that can be used for easy deployments and quick installation. Perhaps even a docker-compose.yml could be added to the repository to facilitate setting up Odin + the communication with Mongo and later on for other databases that are supported.

Describe alternatives you've considered
Not much at this time

Additional context
Not much at this time

[Feature Request] Multiple DB support

I have been reading through the reddit post and a common thread I see is not taking to the use of mongodb. I'm not currently aware of the featureset of mongo currently in use, but I'd suspect the usage isn't too far away from the operations of other databases.

I feel creating an abstraction around Odin's database operations, in order to provide the ability to plug in implementations of that abstraction for different databases would be a net win and would cast a larger net on the community.

Remove MongoDB dependencies from Odin SDK's

Is your feature request related to a problem? Please describe.
Currently, writing SDK's requires using a language specific MongoDB library/module. Removing this will make SDK's more lightweight.

Describe the solution you'd like
Currently SDK's access MongoDB directly, using language specific third party code This endpoint would take information from an SDK and handle it apprpriately.

Essentially we are replacing the SDK MongoDB code with a set of HTTP Requests to code at an endpoint of the Odin Engine. This will also make it easier to design SDK's for other language in the future.

Describe alternatives you've considered
None as of yet.

Additional context

Current SDK's to change:

Python3
Go
Node.js

Write tests for Node.js SDK

Currently there are no unit tests in place for the Node.js Odin SDK - this should be part of v2.0.0

[Documentation Request] Comparison with popular distributed schedulers

Would be great if we can document how Odin stands out when compared with popular schedulers like Airflow, Luigi, Dyenin, Resque, etc.

Some areas we can document:

Usability
Configuration
Data Model
Workflow Management
Retry Mechanism
Scaling
Deployment

A stretch goal is if we can benchmark Odin against these solutions.

[Documentation Request] Document deployments

Documentation around deploying to common platforms would be really helpful for a lot of people getting started. A stretch goal would to be to maybe provide some templates or examples?

Some that come to mind

Kubernetes
AWS EC2
Azure
Google Cloud

I'm sure there's even more that could be useful, but just wanted to start a friendly discussion about it

Bash
Ruby
Rust

Add Bash Engine and SDK support

Include Odin Engine support to allow the execution of Bash jobs. Along with this, write a bash command line tool which emulates an Odin SDK which allows for the logging of live values.

Add Node.js Github Action for CI

Something similar to the Python and the Go SDK actions on this project.