Resource deletion races with MachineAPI actuators about hive

[dgoodwin]

If we reach into cluster to shut things off we'll need admin kubeconfig available to destroy cluster and Hive, which is probably fine, though it should be an optional thing and if it's given we'll try, but errors would be non-fatal for the cases where we're deleting a broken cluster.

If we could continually scan I'd almost prefer that, wiping out the masters first would help, but it's going to get tricky with the master recovery code coming.

[abhinavdahiya]

If we can model the delete runners to run untill a required confidence level is reached, successfull delete pass increases the confidence, error can reset it...

Or something on the lines so that the delete runners are running multiple passess of success untill we can be sure...

[joelddiaz]

We're currently filtering for instances with state 'running' https://github.com/openshift/hive/blob/master/contrib/pkg/awstagdeprovision/awstagdeprovision.go#L722 , so I believe we can extend this to also catch anything that is just recently launched by adding 'pending'.

[joelddiaz]

I haven't had the pleasure of running into the scenario described here (I'll try to set something up locally here), but it would be great if someone could try this out https://github.com/joelddiaz/hive/tree/catch-pending-instances

[joelddiaz]

So I added the search-for-pending logic and I was still seeing hangs in the uninstall. The issue was that the return logic in deleteInstances() was flipped. I've corrected it and PRed both changes in #58

[joelddiaz]

Can we close this out with #58 being merged?

[crawford]

Probably. I'll reopen if we have trouble again in the future.