Retrying Install Can Fail on IAM role already existing. about hive

[jhernand]

We are still seeing this problem, and I think it is related to the how the names of the IAM roles are calculated during installation and during uninstallation. During the installation the name of the role is calculated here:

https://github.com/openshift/installer/blob/ac006ae671a645553d58c8a29c676968dfa3d85f/data/data/aws/bootstrap/main.tf#L48

That cluster_name comes from the clusterdeployment.spec.config.clusterID. In my case that is api-tests. As a result the following roles are created:

$ aws iam list-roles | jq -r .Roles[].RoleName | grep api-tests
api-tests-bootstrap-role
api-tests-master-role
api-tests-worker-role

But when Hive tries to delete the cluster it calls the installer, which calculates again this names, this time in the Go code:

https://github.com/openshift/installer/blob/ac006ae671a645553d58c8a29c676968dfa3d85f/pkg/destroy/aws/aws.go#L813

The result of this is different, according to the logs:

time="2018-12-20T15:56:06Z" level=debug msg="Deleting IAM resources (map[openshiftClusterID:1d7eae0b-bb0b-484d-bf05-5b566d37b09e])"
time="2018-12-20T15:56:06Z" level=debug msg="deleting role: api-tests-2k2xt-master-role"

Note that the name here is api-tests-2k2xt-master-role instead of api-tests-master-role. api-tests-2k2xt is the name of the clusterdeployment object. So I believe that the problem is that Hive is passing to the uninstaller a value that is inconsistent with what it passes to the installer.

The net result is that when Hive fails after creating the IAM roles, for whatever the reason, it is impossible to create a cluster with the same name, neither with the automatic retry done by Hive nor manually. The only alternative is to manually remove the IAM roles from AWS.

@joelddiaz @wking @dgoodwin any suggestion?

[dgoodwin]

Looks like another bug in hive not being prepared for the use of GenerateName. Should be a simple fix to use the Cluster name not generate name.

[dgoodwin]

This might explain a lot of failed deprovision jobs in opshive.

[joelddiaz]

We're just blindly using the ungenerated clustername at the uninstall call. If we would simply read the cd.Spec.Config.ClusterID and pass that to the uninstall, i think we'd be okay

I haven't tried this, but I'd think that somewhere around here we can grab/save the data out of the clusterdeployment: https://github.com/openshift/hive/blob/master/contrib/pkg/installmanager/installmanager.go#L164-L186

[dgoodwin]

Haven't seen in awhile.

[jhernand]

I still see it frequently, with the version of Hive that we have in our environment.

[dgoodwin]

https://jira.coreos.com/browse/CO-173