[Bug] Operator may not trigger galera recovery if pods are restarting about mariadb-operator

[mmontes11]

Hey @djakobczak1 ! Sorry for late response, I was busy with v0.0.26 and v0.0.27 releases, which notably improve Galera support.

Very accurate issue report! I've managed to reproduce, will get back to you with a suggestion soon. 👍🏻

[mmontes11]

@djakobczak1 I think now I understand the root cause of the problem and have a suggestion to solve it.

5. operator sees that wsrep_cluster_size is now above MinClusterSize so it finishes reconciliation
6. started pod fails during e.g. SST and restarts, now operator will not notice that

Indeed, once the cluster reaches healthy state the reconciliation stops and the operator misses that the wsrep_cluster_size goes below the MinClusterSize.

Why?

Currently the controller responsible for this subscribes to the events of the StatefulSet, basically whenever something changes in the StatefulSet a reconciliation is triggered to check whether the cluster is healthy and it sets the GaleraReady accordingly so the other controller performs the recovery..

The problem is that there are situations like this one (but probably more) where the Pods reach a CrashLoopBackOff state, so nothing is changed in the StatefulSet, and therefore no reconciliations are enqueued.

How to solve it?

I've managed to trigger the recovery in the scenario you reported by requeuing reconciliations every 10s even if the cluster is healthy. I suggest we introduce a clusterMonitorInterval, by default set to 10s:

spec:
  galera:
    recovery:
       clusterMonitorInterval: 10s

This way we will allow the user to configure the interval at which the cluster is monitored and we make sure we don't miss any event!

As an alternative, I thought about subscribing to the Pods status, but there might be situations where no reconciliations are triggered for the Pods either, like advanced CrashLoopBackoff state.

What do you think?

[djakobczak1]

Thank you for detailed response.
Suggested solution with configurable clusterMonitorInterval sounds good to me from user perspective.

[mmontes11]

Great @djakobczak1 ! I will raise a PR soon for supporting this.