[Request] Add examples for deploying on AWS spot instances about tune-sklearn

[richardliaw]

Will keep this issue open to make sure we write up a full example though!

[richardliaw]

Great question :) It is entirely possible to deploy tune-sklearn on spot instances as you would do in standard Ray code.

We can put together a full example later (or simply add it to the tune documentation). I think failure handling should work out of the box.

In your script:

# sklearn.py
ray.init(address="auto")
clf = TuneSearchCV(...)
clf.fit()

and

ray up <yaml>
ray submit <yaml> sklearn.py

[rohan-gt]

Ah amazing!

[rohan-gt]

Reopened the issue for the full example. I'm still not completely sure how checkpointing and fault tolerance would be implemented using the tune-sklearn API

[richardliaw]

ah, I think tune-sklearn actually automatically should handle that for you.

[richardliaw]

I just looked into this, and I think there are some missing RayTune parameters to be supported (i.e., tune.run(max_retries=X)), so I'll submit a PR to do this.

Hypothetically, you can do something like this:

1. Start with a YAML

# Save as `ray_cluster.yaml`
cluster_name: ray-tune-cluster
provider: {type: aws, region: us-west-2}
auth: {ssh_user: ubuntu}
min_workers: 3
max_workers: 3

# Deep Learning AMI (Ubuntu) Version 21.0
head_node:
 - InstanceType: c5.xlarge, 
 - ImageId: ami-0b294f219d14e6a82, 
 - InstanceMarketOptions: 
   - MarketType: spot

worker_nodes: 
 - InstanceType: c5.xlarge, 
 - ImageId: ami-0b294f219d14e6a82, 
 - InstanceMarketOptions: 
   - MarketType: spot

2. On any Tune script, you should be able to add 2 lines:

+ray.init(address="auto")
-search = TuneSearchCV(...)
+search = TuneSearchCV(..., max_retries=3)

3. Run this on a Ray cluster. You'll be able to leverage multiple nodes for your search:

$ ray submit ray_cluster.yaml <your_script.py>

The above submit command is equivalent to:

$ ray attach ray_cluster.yaml  # ssh
ubuntu@ip-172-31-24-53:~$ python <your_script.py>

It will also 1. retry failed trials and 2. restart failed nodes. @rohan-gt is this what you're thinking about?

[rohan-gt]

@richardliaw yes this is fine