Telegraf Operator prevents pods creation when the secret already exists about telegraf-operator

[wojciechka]

I have reproduced it by deploying telegraf-operator, manually creating the secret with matching name and then deploying a statefulset (statefulset will cause the name of the secret to be predictable, hence this is how I could reproduce it).

However, I can see how this would happen when telegraf-operator was temporarily down when a pod in a statefulset was being deleted.

I think the best long term approach would be to support upgrading the secret when it already exists. This has the danger of overwriting an existing secret that just collided because of same name.

Ideally we should only update the secret if it has a valid annotation - i.e. app.kubernetes.io/managed-by set to telegraf-operator. However, this would break it when upgrading any existing telegraf-operator - since existing secrets would not have the annotation.

So, I propose we always add the annotation and regarding handling of existing secrets:

• by default, always update the secret when needed
• provide an optional CLI flag that will fail if the annotation is not present - the flag could be used for new deployments of telegraf-operator

Additional checks could be made regarding the secret - such as that the secret is of type Opaque and only has one key - telegraf.conf. This will allow us to be safe about accidentally updating an existing secret.

[BondAnthony]

@wojciechka I like the annotation approach of calling out telegraf-operator as the managed-by. I think this would be good regardless so we can easily link secrets with telegraf-operator.

Regards to secret update, the cli fail flag would make sense to help with the upgrade. Once users are beyond this change do we see a need for this going forward if the annotations are present?

[sumo-drosiek]

@wojciechka I tested the PR locally and it solves the issue for us 👍