Stuck uninstall when deleting a ClusterDeployment before provisioning is complete about hive

[dgoodwin]

@elad661 can you take a look in AWS console and see if you can figure out what's blocking deletion of these objects?

@joelddiaz can you take a look? This keeps popping up, I think I've seen it a few times.

[elad661]

I don't have access to the AWS console. Maybe @cben can help?

[elad661]

@dgoodwin looks like this EIP ID is used for the VPC's NAT Gateway. I don't know why the NAT Gateway is not being deleted.

[joelddiaz]

@elad661 is the EIP tagged with the tectonicClusterID tag?

[joelddiaz]

@elad661 and while we're at it, is the NAT Gateway tagged with tectonicClusterID?

[elad661]

@joelddiaz the EIP is tagged, the network interface it's attached to is not, and the NAT gateway is not tagged either.

This might've been a race condition in which those resources were created but not tagged yet, and then deprovisioning was triggered.

How do you suggest we avoid such cases from our side? Should we disallow deprovisioning before the cluster is fully provisioned? or will Hive be changed to become more robust?

[joelddiaz]

It's okay for the network interface to be untagged as we delete them based on being attached to a particular EIP ( https://github.com/openshift/hive/blob/master/contrib/pkg/awstagdeprovision/awstagdeprovision.go#L588-L597 ).

The untagged NAT Gateways would be a problem as they are found via their tagging. We've been dumping all the objects that are untagged as dependencies in the VPC deletion path ( https://github.com/openshift/hive/blob/master/contrib/pkg/awstagdeprovision/awstagdeprovision.go#L450-L476 ).

I do worry a little bit about having to care about the ordering of the deletions that are piling up in the VPC code path though...

[dgoodwin]

This should be resolved now.