eddienko / owl-pipeline-server

Owl is a job scheduler that allows to submit user jobs to a Kubernetes cluster from anywhere.

Owl Job Scheduler

Owl is a job scheduler that allows to submit user jobs to a Kubernetes cluster from anywhere. See documentation for more information.

Discussion

Any feedback, ideas, enhancements, contributions are welcome. Head to the discussion to add yours or submit issues or bugs in the tracker.

LICENSE

GPL. See License File.

owl-pipeline-server's Issues

Allow for walltime in resources

Add a walltime argument to pipeline resources, so pipeline stops if execution time exceeds this.

Allow for custom images to run pipelines

Make sure that we can run pipelines with custom images (bring code from the imaxt version if necessary) and document.
This should work in the pipeline definition file:

resources:
  image: user/name:tag

In order not to run malicious images, we need to whitelist them in the database and check in the code for the whitelist.

Allow email sending when pipeline ends

Support option of sending email when pipeline ends.

Implement Queues

Implement queues and how?

Related: reserve resources for high priority jobs?

Check cluster resources before scheduling jobs

Currently we have only a hard limit on the number of pipelines that can run at the same time. Instead (or together with this) we need to figure out if the cluster has enough resources to allocate the job.

We are already scrapping the metrics from Prometheus if available and this gives us information about nodes availability and requests. Use this information to find out if a job can be allocated.

Support for Slurm jobs

Users want to submit Slurm jobs. We can support this: instead of creating a Dask Cluster, the Pipeline worker can create a slurm compute cluster for each pipeline with the resources needed.