mariadb cannot start after galera cluster creation about meta-akraino

[gaolitao]

The path of the shared library of /usr/lib64/galera/libgalera_smm.so is configured in:

• vars/redhat-7.yml
• vars/suse-42.yml
• vars/ubuntu-16.04.yml

and from the testing, currently, we are identified as redhat-7 now, in which, /usr/lib64/galera/libgalera_smm.so is assigned to galera_wsrep_provider.

In Yocto, we have no /usr/lib64 directory, now libgalera_smm.so has been installed into /usr/lib64/ directly.

• workaround

modify vars/redhat-7.yml:

galera_wsrep_provider: "/usr/lib/libgalera_smm.so"

• Proposed solution:
  add a yocto.yml file for yocto osfamily.

[jackiehjm]

@gaolitao we will use the work around for now since we have faked to be redhat-7 now, we will consider to add yocto osfamily in the future.

[gaolitao]

After 'shared library' issue, we still cannot start mariadb successfully.

Per logging, seems mysqld is shutdown finally for some reason, need to go deeper and figure out the root cause under the hood.

2020-02-20  9:47:08 140193464751936 [Note] Server socket created on IP: '172.20.1.154'.
2020-02-20  9:47:08 140193439565568 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-02-20  9:47:08 140193439565568 [Note] WSREP: REPL Protocols: 9 (4, 2)
2020-02-20  9:47:08 140193439565568 [Note] WSREP: Assign initial position for certification: 0, protocol version: 4
2020-02-20  9:47:08 140193448265472 [Note] WSREP: Service thread queue flushed.
2020-02-20  9:47:08 140193439565568 [Note] WSREP: Synchronized with group, ready for connections
2020-02-20  9:47:08 140193439565568 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-02-20  9:47:08 140193464751936 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.44-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
2020-02-20 10:02:04 140186685826816 [Note] /usr/sbin/mysqld: Normal shutdown
2020-02-20 10:02:04 140186685826816 [Note] WSREP: Stop replication
2020-02-20 10:02:04 140186685826816 [Note] WSREP: Closing send monitor...
2020-02-20 10:02:04 140186685826816 [Note] WSREP: Closed send monitor.
2020-02-20 10:02:04 140186685826816 [Note] WSREP: gcomm: terminating thread
2020-02-20 10:02:04 140186685826816 [Note] WSREP: gcomm: joining thread
2020-02-20 10:02:04 140186685826816 [Note] WSREP: gcomm: closing backend
2020-02-20 10:02:04 140186685826816 [Note] WSREP: view((empty))
2020-02-20 10:02:04 140186685826816 [Note] WSREP: gcomm: closed
2020-02-20 10:02:04 140192262366976 [Note] WSREP: Received self-leave message.
2020-02-20 10:02:04 140192262366976 [Note] WSREP: Flow-control interval: [0, 0]
2020-02-20 10:02:04 140192262366976 [Note] WSREP: Trying to continue unpaused monitor
2020-02-20 10:02:04 140192262366976 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2020-02-20 10:02:04 140192262366976 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 0)
2020-02-20 10:02:04 140192262366976 [Note] WSREP: RECV thread exiting 0: Success
2020-02-20 10:02:04 140186685826816 [Note] WSREP: recv_thread() joined.
2020-02-20 10:02:04 140186685826816 [Note] WSREP: Closing replication queue.
2020-02-20 10:02:04 140186685826816 [Note] WSREP: Closing slave action queue.
2020-02-20 10:02:04 140193439565568 [Note] WSREP: New cluster view: global state: addfcdd1-53eb-11ea-8dd6-6b19124be696:0, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 3
2020-02-20 10:02:04 140193439565568 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-02-20 10:02:04 140193439565568 [Note] WSREP: applier thread exiting (code:0)
2020-02-20 10:02:04 140186687670016 [Note] WSREP: applier thread exiting (code:6)
2020-02-20 10:02:04 140191626974976 [Note] WSREP: applier thread exiting (code:6)
2020-02-20 10:02:06 140193439872768 [Note] WSREP: rollbacker thread exiting
2020-02-20 10:02:06 140186685826816 [Note] Event Scheduler: Purging the queue. 0 events
2020-02-20 10:02:06 140186685826816 [Note] WSREP: dtor state: CLOSED
2020-02-20 10:02:06 140186685826816 [Note] WSREP: apply mon: entered 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: apply mon: entered 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: cert index usage at exit 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: cert trx map usage at exit 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: deps set usage at exit 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: avg deps dist 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: avg cert interval 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: cert index size 0
2020-02-20 10:02:06 140193448265472 [Note] WSREP: Service thread queue flushed.
2020-02-20 10:02:06 140186685826816 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: MemPool(SlaveTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0
2020-02-20 10:02:06 140186685826816 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 0)
2020-02-20 10:02:06 140186685826816 [Note] WSREP: Flushing memory map to disk...
2020-02-20 10:02:06 140186765510400 [Note] InnoDB: FTS optimize thread exiting.
2020-02-20 10:02:06 140186685826816 [Note] InnoDB: Starting shutdown...
2020-02-20 10:02:06 140186685826816 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
2020-02-20 10:02:08 140186685826816 [Note] InnoDB: Shutdown completed; log sequence number 1601209
2020-02-20 10:02:08 140186685826816 [Note] /usr/sbin/mysqld: Shutdown complete

200220 10:02:08 mysqld_safe mysqld from pid file /var/lib/mysql/controller-1.pid ended

[gaolitao]

The issue is that the systemd expecting the pidfile is: /var/lib/mysql/mysqld.pid
However, /usr/bin/mysqld_safe use hostname.pid
this will cause systemd think the service is not ready yet, and finally fails and shutdown the service with the error of 'timeout'

mysql      66528       1  0 00:51 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
mysql      66901   66528  0 00:51 ?        00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/plugin --wsrep_on=ON --wsrep_provider=/usr/lib/libgalera_smm.so --log-error=/var/log/mysql_logs/galera_server_error.log --open-files-limit=65535 --pid-file=controller-1.pid --socket=/var/lib/mysql/mysql.sock --wsrep_start_position=addfcdd1-53eb-11ea-8dd6-6b19124be696:0

• Proposed solution:

/usr/bin/mysqld_safe: line 787
pid_file="mysqld.pid"

[qliu1]

root 32522 32521 0 03:03 pts/2 00:00:00 sh -c while ! mysql --silent --skip-column-names -e 'SHOW STATUS LIKE "wsrep_evs_state"' | grep -wq "OPERATIONAL"; do sleep 5; done
the mysql never reach OPERATIONAL status and it think fail to start new cluster.
MariaDB [(none)]> SHOW STATUS LIKE "wsrep_evs_state";
Empty set (0.00 sec)

[gaolitao]

root 32522 32521 0 03:03 pts/2 00:00:00 sh -c while ! mysql --silent --skip-column-names -e 'SHOW STATUS LIKE "wsrep_evs_state"' | grep -wq "OPERATIONAL"; do sleep 5; done
the mysql never reach OPERATIONAL status and it think fail to start new cluster.
MariaDB [(none)]> SHOW STATUS LIKE "wsrep_evs_state";
Empty set (0.00 sec)

Seems you missed some thing in your environment, i just checked mine with the latest modification:

MariaDB [(none)]> SHOW STATUS LIKE "wsrep_evs_state";
+-----------------+-------------+
| Variable_name | Value |
+-----------------+-------------+
| wsrep_evs_state | OPERATIONAL |
+-----------------+-------------+
1 row in set (0.00 sec)

[qliu1]

/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/plugin --wsrep_on=ON --wsrep_provider=/usr/lib/galera/libgalera_smm.so --log-error=/var/log/mysql_logs/galera_server_error.log --open-files-limit=65535 --pid-file=mysqld.pid --socket=/var/lib/mysql/mysql.sock --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1

--wsrep_start_position=00000000-0000-0000-0000-000000000000:-1 is different from your command line.