Infrastructure improvements about raster-vision

[hectcastro]

• I'm not sure introducing all of the overhead around baking an AMI is going to be worth it just to bake in the nvidia-docker dependency. That archive is ~2MB and decompresses to a single binary. Downloading and decompressing that on an EC2 instance should be taking a few seconds end-to-end.

• I think it would be good to make the aws ec2 wait thing use tags vs. keypair names. That looks like an option supported by --filter.

• Terminating the instances from within an EC2 instance could be tricky because I think we'd actually have to terminate the Spot Fleet request (doing that also indirectly messes with Terraform's state of the word).

• Not entirely following the last solutions to processing work in parallel, but as a related note, instanceID is already available via instance metadata.

[lewfish]

I'm not sure introducing all of the overhead around baking an AMI is going to be worth it just to bake in the nvidia-docker dependency. That archive is ~2MB and decompresses to a single binary. Downloading and decompressing that on an EC2 instance should be taking a few seconds end-to-end.

I don't think we need to bake our own AMI with nvidia-docker on it. We can just use an existing AMI that has it. But if it only takes a few seconds to install it, then it doesn't matter.

I think it would be good to make the aws ec2 wait thing use tags vs. keypair names. That looks like an option supported by --filter.

Tags make more sense, but Terraform doesn't let us associate tags with instances created using a spot fleet request. See hashicorp/terraform#3263

Not entirely following the last solutions to processing work in parallel, but as a related note, instanceID is already available via instance metadata.

True, but each worker needs to know its index (ie. a number between 0 and n-1 if n workers) to figure out which batch jobs to run. I don't think we can turn the instanceid into a worker index.

[hectcastro]

Good point about #3263.

Regarding an index for workers, there is also ami-launch-index via instance metadata. Not entirely sure what value that returns when multiple instances get launched from a Spot Fleet though.

I guess my high level concern is that we try to make use of what's already there (if it applies) via instance metadata vs. supplying and managing our own identifiers.

[lewfish]

I'm thinking about using AWS Batch to run lots of experiments in parallel. Does that sound ok? One issue is that Batch uses ECS, and ECS doesn't know about nvidia-docker. There's a workaround to be able to use the GPU even when running using regular docker in ECS: https://blog.cloudsight.ai/deep-learning-image-recognition-using-gpus-in-amazon-ecs-docker-containers-5bdb1956f30e#.mau60bfvo

[lewfish]

I'm moving the conversation about parallelizing experiments to #10