Feature: Add an ability to specify which check type to do when defining the replication hostgroups using proxysql_replication_hostgroups module about community.proxysql

[markuman]

One question which appeared is if it's fine if we update the check_type a bit later? If both the queries are executed in a single transaction, it should be fine. If not, can't this cause issues?

That's fine and by design. You can configure and review before apply the changes to runtime with 'LOAD MYSQL SERVERS TO RUNTIME`
They call it Multi layer configuration system

[markuman]

The main issue is backwards compatiblity: #20 (comment)
But it is a must-have feature.

To summarize it

ProxySQL Version	check_type values
< 2.0.1	check_type column does not exist
>=2.0.1	read_only, innodb_read_only, super_read_only
>=2.0.8	read_only, innodb_read_only, super_read_only, read_only|innodb_read_only, read_only&innodb_read_only

Two options

a) we must handle every variants. #20 (comment)
b) we make a release and introduce a deprecation warning that ProxySQL < 2.0.8 support will be dropped.
2.0.8 was released in Nov 2019.

any ideas or opinions? @zentavr @Andersson007

[zentavr]

Ideally - to support all, but frankly speaking, an example from @markuman what he mean in his comment would be great!

[Andersson007]

I like option a (yes, it can be reasonable to drop support for old things but, imo, only if it's getting too hard).
We check versions a lot in community.mysql and community.postgresql (and even server types (mariadb / mysql) + versions in community.mysql).
It's nasty but i wouldn't say it's hard to implement. Moreover, 2019 doesn't feel like it was a long time ago.

[Andersson007]

maybe try: except ?

[Andersson007]

if the function fails, put the warning meaning "check type failed, check your server version - it must be ... for ..., ... for ..." or something and continue

[Andersson007]

it's a kinda brainstorming ^. @markuman 's table should be in the documentation for the option and try: except approach feels now like an applicable one. But maybe it's lie and illusion :)

[markuman]

try: except approach feels now like an applicable one.

I was already thinking about that :D

In general I would like to avoid this

    def update_repl_group_config(self, cursor):
        if self.support_check_type:
            query_string = \
                """UPDATE mysql_replication_hostgroups
                SET comment = %s, check_type = %s
                WHERE writer_hostgroup = %s
                    AND reader_hostgroup = %s"""

            query_data = [self.comment, self.check_type, self.writer_hostgroup, self.reader_hostgroup]
        else:
            query_string = \
                """UPDATE mysql_replication_hostgroups
                SET comment = %s
                WHERE writer_hostgroup = %s
                    AND reader_hostgroup = %s"""

            query_data = [self.comment,
                          self.writer_hostgroup,
                          self.reader_hostgroup]

        cursor.execute(query_string, query_data)
        return True

It makes is harder (harder to read, harder to maintain and harder to understand) when more columns are introduced.

I'd like to keep the basic functionality as it is and add a function that is handling just the change of the check_type column.

    def update_repl_group_config(self, cursor):
        query_string = \
            """UPDATE mysql_replication_hostgroups
                SET comment = %s
                WHERE writer_hostgroup = %s
                    AND reader_hostgroup = %s"""

        query_data = [self.comment, self.writer_hostgroup, self.reader_hostgroup]

        cursor.execute(query_string, query_data)

        if self.support_check_type and self.check_type:
            # a function that updates the column value check_type separately
            #   if requested and supported
            update_check_type()

        return True

[Andersson007]

:) When I was reading the first example, I thought of the same as you suggested in the second one.. One question which appeared is if it's fine if we update the check_type a bit later? If both the queries are executed in a single transaction, it should be fine. If not, can't this cause issues? I mean in that time gap between the queries. If it's fine, it seems to be a good approach (the second one).

[Andersson007]

That's fine and by design. You can configure and review before apply the changes to runtime with 'LOAD MYSQL SERVERS TO RUNTIME`
They call it Multi layer configuration system

ah, cool, thanks for clarifying!

[markuman]

Ideally, we duplicate the integrationtests with multiple proxysql container to cover and track backwards compatibility as long as we don't drop support for some verion at some random point in the futur.

• proxysql-1.4.7
• proxysql-2.0.0
• proxysql-2.0.1
• proxysql-2.0.8
• proxysql-2.0.18
• proxysql-2.1.0
• proxysql-2.2.0

[Andersson007]

sounds very sensible

[Andersson007]

would be also nice to test against different connector versions

[markuman]

would be also nice to test against different connector versions

Yeah, that should be done in the context of #47

[markuman]

@zentavr do you have some time to adjust your PR?

A separate function that updates/handle only the check_type column via try: except (so we don't make a version check before).
This function can be used in both, create_repl_group_config() and update_repl_group_config()

[markuman]

I just want to move on some brainstorming while start implementing this...

basically the check_repl_group_config() function has a wrong sql query: https://github.com/ansible-collections/community.proxysql/blob/main/plugins/modules/proxysql_replication_hostgroups.py#L153

It queries for writer_hostgroup and reader_hostgroup and (if requested) comment.

But that's wrong, because only the writer_hostgroup is the primary key in this table

CREATE TABLE mysql_replication_hostgroups (
    writer_hostgroup INT CHECK (writer_hostgroup>=0) NOT NULL PRIMARY KEY,
    reader_hostgroup INT NOT NULL CHECK (reader_hostgroup<>writer_hostgroup AND reader_hostgroup>=0),
    check_type VARCHAR CHECK (LOWER(check_type) IN ('read_only','innodb_read_only','super_read_only','read_only|innodb_read_only','read_only&innodb_read_only')) NOT NULL DEFAULT 'read_only',
    comment VARCHAR NOT NULL DEFAULT '', UNIQUE (reader_hostgroup))

In some cases, this will result in a trial of creating a new replication hostgroup - which will fail, because of the uniq primary key writer_hostgroup.

The check if a replication hostgroup exists, must only be

SELECT count(*) AS `repl_groups`
               FROM mysql_replication_hostgroups
               WHERE writer_hostgroup = %s;

that's it. nor search for reader hostgroup nor for comment, nor for check_type.

[markuman]

I guess no one hits this rare condition because most users are using proxysql just in a 1:1 relation ship (_one proxysql is targeting one db cluster).

E.g.
when you request

- name: assign backend servers to replication hostgroups
      proxysql_replication_hostgroups:
        state: present
        writer_hostgroup: 1
        reader_hostgroup: 2
       comment: some comment
        load_to_runtime: yes

and someone is changing this comment manually in your proxysql setup, and you re-run your playbook (because it is immutable), it will fail, because it won't find your replication hostgroup because and the and comment query, and try to create a new replication hostgroup. And that will fail, because id 1 already exists.

[markuman]

the best way to refactor this , is imo to introduce an update function for every column

update_reader_hostgroup()
update_comment()
update_check_type()

[markuman]

https://github.com/ansible-collections/community.proxysql/blob/main/plugins/modules/proxysql_replication_hostgroups.py#L120

This condition is also wrong. reader_hostgroup: 0 is a valid value as long a writer_hostgroup does not use it.

writer_hostgroup: 4323
reader_hostgroup: 0

is valid, but the module will currently fail.

[markuman]

or another demonstration

    - name: create replication hostgroup
      community.proxysql.proxysql_replication_hostgroups:
        login_user: admin
        login_password: admin
        writer_hostgroup: 1
        reader_hostgroup: 2

    - name: change reader hostgroup
      community.proxysql.proxysql_replication_hostgroups:
        login_user: admin
        login_password: admin
        writer_hostgroup: 1
        reader_hostgroup: 3

results in

fatal: [proxysql]: FAILED! => {"changed": false, "msg": "unable to modify replication hostgroup.. (1045, 'ProxySQL Admin Error: UNIQUE constraint failed: mysql_replication_hostgroups.writer_hostgroup')"}

but it is possible

[markuman]

"msg" includes also wrong messages from proxysql_backend_server module...I guess to copy/paste of this modul.