Consider availability zone when picking an instance type about autospotting

[cristim]

Thanks, in this case I'm closing this issue.

[nmeierpolys]

I was able to get past this for our instances by changing the code to skip that instance type and hosting our own binaries. I'll see if I can figure out why it's not handling the case better in general and hopefully put in a PR when I get a chance.

[cristim]

Good catch @nmeierpolys, thanks for reporting this!

Off-topic: I'd like to learn more about your use case for only running in a single region, for reliability concerns that's not something I would normally do, especially with spot instances, and I'm always curious to learn about such interesting use cases happening 'in the wild'.

Regarding the issue you noticed, that sounds like a real problem. As far as I can see in the spot instance pricing history in my AWS console, that instance type is indeed only available in a single AZ in US-East-1(in my account that is labeled as us-east-1a), so I guess the algorithm is somehow not checking against this edge case.

I will try to reproduce this in my own AWS account and I will have a look at the source code to see why does that unavailable instance type appear to be the cheapest.

Somehow related to this issue, I think I could implement a way to track instance launch failures for a given instance type/AZ combination, and somehow temporarily blacklist them for a few hours if the instance fails to launch over multiple autospotting runs in a row. I will create a new issue for implementing this kind of feature.

Since you are the first user to report self-hosting binaries, please consider submitting a PR documenting how you did it, I'd really like to have a bit of documentation for this.

[nmeierpolys]

Thanks @cristim. Right now, we're running in a single AZ because our system and deploys expect everything to be in a single subnet. There's nothing technically preventing us from changing this, it just hasn't been a big enough priority to commit time to it yet.

I'd be happy to add some documentation for self-hosting the binaries. Hopefully I can get a PR for that your way this week sometime.

[cristim]

@nmeierpolys, is this still happening on your environment with the latest version?

[nmeierpolys]

Sorry, but I'm no longer working on the project that used autospotting, so I'm not able to test it out with the latest version.