carlos-a-enriquez / dask_k8s

Bioinformatics workflow leveraged through Dask and Kubernetes on a cloud cluster.

Dask_k8s

This repository will contain the files related to the deployment of Dask Helm charts on the OpenStack Kubernetes cluster.

1. Infrastructure setup

Dask cluster setup.

Check here.

Prometheus monitoring setup

Check here for the Promethus stand-alone or here for the Promethus stack (Graphana included).

2. Custom Dask Docker images

The Dask base image and the dask-notebook image were modified to be able to run the "SCENIC" protocol. The corresponding Dockerfiles are accessible here.

3. Example notebooks

To test the execution of pySCENIC, the mouse brain dataset and the PBMC dataset were used. The custom notebooks adapted from the ones that were published in the "SCENIC" protocol repository can be found here and here.

dask_k8s's Issues

ignore this

Adding new libraries to the Dask helm chart

In order to continue with the analysis, new libraries must be added. This issue will be used to trigger new Docker image builds.

Update: Did not work. Scrapped.

The Dask scheduler drops a "close workers" message in DEBUG logs

Dask worker issue

This is the observed issue.

2023-02-24 14:58:32,517 - distributed.scheduler - DEBUG - Suggest closing workers: ['tcp://10.42.3.191:40815', 'tcp://10.42.2.230:35661', 'tcp://10.42.1.217:37249']