[Bug] Failover breaks replication if a super privileged user writes data during failover about mariadb-operator

[mmontes11]

Here it is the PR, could you PTAL? 🙏🏻 :

• #311

[mmontes11]

Thanks for reporting this @harunkucuk5 !

Your proposal totally makes sense, FLUSH TABLES WITH READ LOCK; was actually present in the first implementation of replication. We opted to go for a more permissive approach with readonly after facing issues where the operator was struggling to reconcile replication, as it was in an intermediate state where the tables were locked.

I think readonly may be good enough for uses cases where the clients have a restricted amount of permissions, not including SUPER, so they can't actually bypass readonly. That might not be always the case, so I still see the benefit of FLUSH TABLES WITH READ LOCK;

What about exposing this to the user in a new field with propper documentation?:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-repl
spec:
  ...
  replication:
    enabled: true
    primary:
      podIndex: 0
      automaticFailover: true
      lockStrategy: (ReadOnly | ReadLock)

[harunkucuk5]

Hey @mmontes11, read-only may be enough but on the other hand if there are some application users with SUPER privilege this would cause data loss on the new primary along with the broken replication. Ideally, there should not be any application user with SUPER privilege but, unfortunately, when it comes to real-world cases it is not like that in many cases I have seen. I think the operator should provide a reliable auto-failover mechanism regardless of the user privileges. I was unaware that the FLUSH TABLES WITH READ LOCK was already used and caused some issues. I will check the history. But if we implement this and it works without any issues then there should not be any need for the read-only option. Let me check this and get back to you later on

[mmontes11]

Hey @harunkucuk5 !

I agree, consistency and relibability is probably the most important thing during switchover/failover, let's bring this change back in!

This was introduced in v0.0.15 via this PR:

• #127

I will re-apply it and see how it behaves so we can consider to ship it with v0.0.24.

[mmontes11]

Supported by this PR:

• #311