danielsc / azureml-and-dask

Shows how to spin up a DASK cluster on AML Compute

Running Dask on AzureML

This repository is no longer maintained. For a simple way of running Dask on an AzureML cluster, please check out the AzureML CLI v2 DASK samples here: https://github.com/Azure/azureml-examples/tree/main/cli

---

This repository shows how to run a Dask cluster on an AzureML Compute cluster. It is designed to run on an AzureML Notebook VM (created after 8/15/2019), but it should work on your local computer, too.

Please follow these setup instructions and then start:

• here for plain DASK interactive scenarios interactive/StartDask.ipynb.
• here for DASK with NVIDIA RAPIDS interactive scenarios rapids_interactive/start_cluster.ipynb.

Setting up the Python Environment

The environment you are running should have the latest version of dask and distributed installed -- run this code in the terminal to make sure:

    conda activate py36  # assuming AzureML Notebook VM
    pip install --upgrade dask distributed

Or, if you want to be on the safe side, create a new conda environment using this environment.yml file like so:

    conda env create -f dask/environment.yml  
    conda activate dask
    python -m ipykernel install --user --name dask --display-name "Python (dask)"

azureml-and-dask's Issues

Store results in blob

This is very promising. I was wondering if there is a way to store the results (let say gridsearchcv results from dask) on blob storage.
Because if I lose my SSH connection, I am not able to retain the results.
Thanks

Unable to get image details Error

Error occurred: Unable to get image details : Unable to fetch workspace resources: The Resource 'Microsoft.ContainerRegistry/registries/14f3d1f5e2

I don't know if it is having trouble pulling the base MPI image from mcr. I am guessing it arises from this line:

myEnv = Environment.from_conda_specification('startDask', './dask/environment.yml')