adfinis / openshift-mariadb-galera

Kubernetes / OpenShift Images for a MariaDB Galera Cluster

⚠️ This project is no longer being maintained ⚠️

The code in this repository was created as a proof of concept back when StatefulSets where still called PetSets and hasn't seen much love since.

Feel free to use the bootstrap logic and image structure in your project, but we're not going to implement any changes going forward.

If you find the code in this repository helpful, but want to use it under a different license please get in contact with us.

MariaDB Galera cluster on OpenShift

Requirements

• OpenShift Origin 1.3 or Enterprise 3.3 OR
• Kubernetes 1.3+

General informations

Environment variables and volumes

The image recognizes the following environment variables that you can set during initialization by passing -e VAR=VALUE to the Docker run command.

Variable name	Description
MYSQL_USER	User name for MySQL account to be created
MYSQL_PASSWORD	Password for the user account
MYSQL_DATABASE	Database name
MYSQL_ROOT_PASSWORD	Password for the root user (optional)

You can also set the following mount points by passing the -v /host:/container flag to Docker.

Volume mount point	Description
/var/lib/mysql	MySQL data directory

Notice: When mouting a directory from the host into the container, ensure that the mounted directory has the appropriate permissions and that the owner and group of the directory matches the user UID or name which is running inside the container.

Usage in OpenShift

Importing templates

Import the templates into OpenShift with the following commands:

$ oc create -f mariadb-galera-ephemeral-template.yml -n openshift
$ oc create -f mariadb-galera-persistent-template.yml -n openshift

Creating a cluster with templates

• Navigate to the OpenShift web console
• Choose your project and click add to Project
• Choose one of the two mariadb-galera templates

If choosing the persistent template, make sure that the PV's are created in the namespace of your project and match the VOLUME_PV_NAME and VOLUME_CAPACITY parameters of the template.

Manual cluster creation

• Create PV's as cluster admin:

$ oc create -f galera-pv-nfs.yml -n yourproject

• Then create the service and petset itself:

$ oc create -f galera.yml

Manual cluster cleanup

$ oc delete petset mysql
$ oc delete svc galera
$ oc delete pod mysql-0 mysql-1 mysql-2
$ oc delete pv datadir-mysql-0 datadir-mysql-1 datadir-mysql-2

Usage in Kubernetes

This image runs on kubernetes as well.

Create cluster

$ kubectl create -f galera-pv-host.yml
$ kubectl create -f galera.yml

For Kubernetes v1.5 the API endpoints have changed you need to use the definition in galera_k8s_v1.6.yml.

$ kubectl create -f galera-pv-host.yml
$ kubectl create -f galera_k8s_v1.6.yml

Cleanup cluster

$ kubectl delete petset mysql
$ kubectl delete svc galera
$ kubectl delete pod mysql-0 mysql-1 mysql-2
$ kubectl delete pv datadir-mysql-0 datadir-mysql-1 datadir-mysql-2

Building

$ git clone https://github.com/adfinis-sygroup/openshift-mariadb-galera
$ cd k8s-mariadb-galera-centos
$ make

Technical informations how the image works in detail can be found here

Contributions

Contributions are more than welcome! Please feel free to open new issues or pull requests.

License

GNU GENERAL PUBLIC LICENSE Version 3

See the LICENSE file.

openshift-mariadb-galera's Issues

I can't deploy it

it's tell me
/var/run/mysql/mysql.sock' (2 "No such file or directory")

what's happend

Clients connect to unready endpoints

Hi everybody,

due to the annotation: service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" in the service, it seems to me, that the service is distributing requests to nodes, even they are not ready. This leads to connection or SQL errors, when client requests are distributed to nodes which are shutting down.
Therefore I created another service, which has no annotation and use this service for the clients to connect:

# create a service for clients which honors readiness
apiVersion: v1
kind: Service
metadata:
  name: "{{ template "dnsname" . }}-service"
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
spec:
  ports:
  - name: mysql
    port: 3306
  clusterIP: None
  selector:
    app: {{ template "fullname" . }}

Additionally I modified the readiness probe to check for a semaphore file and if the node is in sync:

#!/bin/bash
#
# Adfinis SyGroup AG
# openshift-mariadb-galera: mysqld readinessProbe
#

MYSQL_USER="readinessProbe"
MYSQL_PASS="readinessProbe"
MYSQL_HOST="localhost"

if [ -f "/tmp/wsrep_off" ];then
  exit 1
fi

mysql --protocol=socket --socket=/var/run/mysqld/mysqld.sock -u${MYSQL_USER} -p${MYSQL_PASS} -h${MYSQL_HOST} -e"SHOW DATABASES;"

if [ $? -ne 0 ]; then
  exit 1
fi

SYNCED=$( mysql -s --skip-column-names --protocol=socket --socket=/var/run/mysqld/mysqld.sock -u${MYSQL_USER} -p${MYSQL_PASS} -h${MYSQL_HOST} -e"SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';" | awk '{ print $2 }' )

if [ "${SYNCED}" != "Synced" ];then
  exit 1
else
  exit 0
fi

Then I added a pre_stop command to the stateful set, increased the terminationGracePeriodSeconds to 60, to give the nodes enough time to shut down and set the frequency of the readinessProbe to 10 seconds:

....
terminationGracePeriodSeconds: 60
....
     containers:
        lifecycle:
         preStop:
            exec:
              command:
                - /bin/sh
                - -c
                - touch /tmp/wsrep_off && sleep 20
...
       readinessProbe:
          exec:
            command:
            - /usr/share/container-scripts/mysql/readiness-probe.sh
          timeoutSeconds: 5
          periodSeconds: 10
          failureThreshold: 1

Now the nodes themselves can connect to each other over the galera-mdb-ga service, which tolerates not ready nodes and the clients can connect to the nodes over galera-mdb-ga-service, which distributes requests only to ready nodes.

Configure wsrep_cluster_name

To ensure that accidental communication with other clusters we need to configure wsrep_cluster_name in galera.cnf.

It must be possible to configure this via an environment variable, but fallback to a sane default (MariaDB Galera itself defaults to my_wsrep_cluster according to the documentation)

can not start successful

2017-05-04  9:31:03 140203041720064 [Note] InnoDB: Dumping buffer pool(s) not yet started
2017-05-04  9:31:03 140204026059008 [Note] Plugin 'FEEDBACK' is disabled.
2017-05-04  9:31:03 140204026059008 [Note] Server socket created on IP: '0.0.0.0'.
2017-05-04  9:31:03 140204026059008 [ERROR] Can't start server : Bind on unix socket: Permission denied
2017-05-04  9:31:03 140204026059008 [ERROR] Do you already have another mysqld server running on socket: /var/lib/mysql/mysql.sock ?
2017-05-04  9:31:03 140204026059008 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit

OpenShift templates

Create OpenShift templates for mariadb-galera-petset-ephemeral and mariadb-galera-petset-persistent (nfs)

Better readiness probe

The current readinessProbe is really basic:

mysql -u root -e 'show databases;'

We should use an dedicated monitoring-user configurable over ENV-variables and maybe even check ports. Also move the actual commands from the yaml to an dedicated script in the image.

Compile peer finder from source

A bit worrying the peer finder is currently checked in as a binary. Maybe compile it from source, fetched as part of a multi-staged docker build in the golang container?

Lost+found directory inside /var/lib/mysql

I finally successfully deploy on GKE with SSD volume instead of glusterFS.
I'm able to connect but i noticed a schema named "#mysql50#lost+found and some logs

2017-06-05 20:49:17 140214132201728 [Note] mysqld: ready for connections.
Version: '10.1.20-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
2017-06-05 20:49:21 140214130744064 [ERROR] Invalid (old?) table or database name 'lost+found'
2017-06-05 20:49:31 140214130744064 [ERROR] Invalid (old?) table or database name 'lost+found'
2017-06-05 20:49:41 140214130744064 [ERROR] Invalid (old?) table or database name 'lost+found'
2017-06-05 20:49:51 140214130744064 [ERROR] Invalid (old?) table or database name 'lost+found'

Any idea on this subject ?

Thanks.

Openshift deployment problem: entry point script error

Hi everybody,

I was looking into deploying a mariadb galera cluster into our openshift. Starting from an older article from here, I found this repository.

Looking around and using the galera_k8s_v1.6.yml template, I made my own to test it (attached).
After container creation, I have the following output in the log:

+ CONTAINER_SCRIPTS_DIR=/usr/share/container-scripts/mysql
+ EXTRA_DEFAULTS_FILE=/etc/my.cnf.d/galera.cnf
+ '[' -z cat-risk-amp ']'
+ echo 'Galera: Finding peers'
Galera: Finding peers
++ hostname -f
++ cut -d. -f2
+ K8S_SVC_NAME=mariadb-galera
+ echo 'Using service name: mariadb-galera'
Using service name: mariadb-galera
+ cp /usr/share/container-scripts/mysql/galera.cnf /etc/my.cnf.d/galera.cnf
cp: cannot create regular file '/etc/my.cnf.d/galera.cnf': Permission denied

As I went through the script files, that are used for image creation, the permissions should be set correctly. Could you provide support? Did I overlooked something in the template, that need to be configured? Is this a bug?

The customized template

kind: Template
apiVersion: v1
metadata:
  name: mariadb-stateful-set
  annotations:
    iconClass: "icon-mariadb"
    tags: "database,mariadb,replication"
parameters:
  - name: MYSQL_USER
    value: user
    required: true

  - name: MYSQL_PASSWORD
    value: user1234
    required: true

  - name: MYSQL_DATABASE
    value: userdb
    required: true

  - name: MYSQL_ROOT_PASSWORD
    value: root1234
    required: true

objects:
  - kind: Service
    apiVersion: v1
    metadata:
      annotations:
        service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
      name: mariadb-galera
      labels:
        app: mariadb
    spec:
      ports:
      - port: 3306
        name: mariadb
      clusterIP: None
      selector:
        app: mariadb
  - kind: StatefulSet
    apiVersion: apps/v1beta1
    metadata:
      name: mariadb
    spec:
      serviceName: "mariadb-galera"
      replicas: 2
      template:
        metadata:
          labels:
            app: mariadb
          annotations:
            pod.alpha.kubernetes.io/initialized: "true"
            # found this addition in one of the branches here
            pod.beta.kubernetes.io/init-containers: '[
                { 
                    "name": "mariadb-galera-prepare",
                    "image": "busybox",
                    "command": ["sh", "-c", "chown -R 27:27 /var/lib/mysql"],
                    "volumeMounts": [
                        {
                            "name": "datadir",
                            "mountPath": "/var/lib/mysql"
                        }
                    ]
                }
            ]'
        spec:
          containers:
          - name: mariadb
            image: adfinissygroup/k8s-mariadb-galera-centos:v004
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 3306
              name: mariadb
            - containerPort: 4444
              name: sst
            - containerPort: 4567
              name: replication
            - containerPort: 4568
              name: ist
            readinessProbe:
              exec:
                command:
                - /usr/share/container-scripts/mysql/readiness-probe.sh
              initialDelaySeconds: 15
              timeoutSeconds: 5
            volumeMounts:
            - name: datadir
              mountPath: /var/lib/mysql
              subPath: data
            env:
              - name: MYSQL_USER
                value: "${MYSQL_USER}"
              - name: MYSQL_PASSWORD
                value: "${MYSQL_PASSWORD}"
              - name: MYSQL_DATABASE
                value: "${MYSQL_DATABASE}"
              - name: MYSQL_ROOT_PASSWORD
                value: "${MYSQL_ROOT_PASSWORD}"
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
      volumeClaimTemplates:
      - metadata:
          name: datadir
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi

No route to host

Hi,

using the ephemeral template, I am getting these errors:

+ CONTAINER_SCRIPTS_DIR=/usr/share/container-scripts/mysql
+ EXTRA_DEFAULTS_FILE=/etc/my.cnf.d/galera.cnf
+ '[' -z mysql ']'
+ echo 'Galera: Finding peers'
Galera: Finding peers
++ cut -d. -f2
++ hostname -f
Using service name: galera
+ K8S_SVC_NAME=galera
+ echo 'Using service name: galera'
+ cp /usr/share/container-scripts/mysql/galera.cnf /etc/my.cnf.d/galera.cnf
+ /usr/bin/peer-finder -on-start=/usr/share/container-scripts/mysql/configure-galera.sh -service=galera
2017/03/06 14:38:03 lookup galera on 192.168.178.90:53: read udp 172.17.0.7:59559->192.168.178.90:53: read: no route to host
2017/03/06 14:38:09 lookup galera on 192.168.178.90:53: read udp 172.17.0.7:60500->192.168.178.90:53: read: no route to host
2017/03/06 14:38:15 lookup galera on 192.168.178.90:53: read udp 172.17.0.7:47483->192.168.178.90:53: read: no route to host
2017/03/06 14:38:21 lookup galera on 192.168.178.90:53: read udp 172.17.0.7:38682->192.168.178.90:53: read: no route to host
2017/03/06 14:38:27 lookup galera on 192.168.178.90:53: read udp 172.17.0.7:41022->192.168.178.90:53: read: no route to host
2017/03/06 14:38:38 lookup galera on 192.168.178.90:53: read udp 172.17.0.7:55353->192.168.178.90:53: i/o timeout

Is there anythin I need to configure? I did launch my openshift cluster using oc cluster up, here is details about my ENV:

oc v1.3.1
kubernetes v1.3.0+52492b4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://192.168.178.90:8443
openshift v1.3.1
kubernetes v1.3.0+52492b4

When change in yum.repos.d

Hi team,

First of all thanks for this fantastic cluster setup configuration.
I want your help am using yours configuration to setup same cluster but MYSQL version should be 5.5 or 5.6 whicg comesup with mariadb 5.5.64/65.

So to do it i have change the yum.repos.d in baseurl.
baseurl = http://yum.mariadb.org/5.5.64/centos7-amd64 & create the new image but when i run the cluster from that image facing below error.

====================================================

Running mysql_install_db ...

• echo 'Running mysql_install_db ...'
• mysql_install_db --datadir=/var/lib/mysql
  Neither host 'mysql-0' nor 'localhost' could be looked up with
  '/usr/sbin/resolveip'
  Please configure the 'hostname' command to return a correct
  hostname.
  If you want to solve this at a later stage, restart this script
  with the --force option

===============================================

please help me in this case its urgent

issue with restart nodes

Hi
when i was restart any node that node those not response.
in logs sync successfully pass with another node.
But it seams user table doesn't created.
enviroment: kubernetes

Slave SQL: Error 'Can't find any matching row in the user table' on query. Default database: 'mysql'. Query: 'GRANT ALL PRIVILEGES ON `example_db`.* TO 'example_db'@'%'', Internal MariaDB error code: 1133

mysql-0 0/1 Running 0 5m

pods is running but readinessProbe doesn't pass, because user doesn't exists.
I mount path to host.
But when remove host path, pods run without problem.

Check WSREP status in readiness probe

Hi there.

Is there any way to check if wsrep is ready inside readiness probe? During a pod recovery (die and back again) I'm seeing this error from my client app:

ERROR 1047 (08S01) at line 1: WSREP has not yet prepared node for application use

For what I understand, it is possible to SHOW DATABASES even if wsrep isn't ready yet.

Thanks!

Failed wsrep_sst_xtrabackup-v2 on Openshift 3.7, Kubernetes 1.7

Hi,

the first container gets ready but the second don`t.

Logs of the second container (mysql-1):

2018-03-22 10:04:13 140329350465792 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2018-03-22 10:04:13 140329350465792 [Note] WSREP: restore pc from disk failed
2018-03-22 10:04:13 140329350465792 [Note] WSREP: GMCast version 0
2018-03-22 10:04:13 140329350465792 [Note] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2018-03-22 10:04:13 140329350465792 [Note] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2018-03-22 10:04:13 140329350465792 [Note] WSREP: EVS version 0
2018-03-22 10:04:13 140329350465792 [Note] WSREP: gcomm: connecting to group 'galera', peer 'mysql-0.galera.galera.svc.cluster.local:,mysql-1.galera.galera.svc.cluster.local:'
2018-03-22 10:04:13 140329350465792 [Note] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') connection established to 5f4ff20f tcp://10.131.0.110:4567
2018-03-22 10:04:13 140329350465792 [Warning] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') address 'tcp://10.131.0.110:4567' points to own listening address, blacklisting
2018-03-22 10:04:13 140329350465792 [Note] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') connection established to 49b580d4 tcp://10.130.1.196:4567
2018-03-22 10:04:13 140329350465792 [Note] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2018-03-22 10:04:13 140329350465792 [Note] WSREP: declaring 49b580d4 at tcp://10.130.1.196:4567 stable
2018-03-22 10:04:13 140329350465792 [Note] WSREP: Node 49b580d4 state prim
2018-03-22 10:04:13 140329350465792 [Note] WSREP: view(view_id(PRIM,49b580d4,2) memb {
	49b580d4,0
	5f4ff20f,0
} joined {
} left {
} partitioned {
})
2018-03-22 10:04:13 140329350465792 [Note] WSREP: save pc into disk
2018-03-22 10:04:14 140329350465792 [Note] WSREP: gcomm: connected
2018-03-22 10:04:14 140329350465792 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2018-03-22 10:04:14 140329350465792 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2018-03-22 10:04:14 140329350465792 [Note] WSREP: Opened channel 'galera'
2018-03-22 10:04:14 140329026053888 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2018-03-22 10:04:14 140329350465792 [Note] WSREP: Waiting for SST to complete.
2018-03-22 10:04:14 140329026053888 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2018-03-22 10:04:14 140329026053888 [Note] WSREP: STATE EXCHANGE: sent state msg: 5f9d3238-2db8-11e8-a838-6a2b63375367
2018-03-22 10:04:14 140329026053888 [Note] WSREP: STATE EXCHANGE: got state msg: 5f9d3238-2db8-11e8-a838-6a2b63375367 from 0 (mysql-0)
2018-03-22 10:04:14 140329026053888 [Note] WSREP: STATE EXCHANGE: got state msg: 5f9d3238-2db8-11e8-a838-6a2b63375367 from 1 (mysql-1)
2018-03-22 10:04:14 140329026053888 [Note] WSREP: Quorum results:
	version    = 4,
	component  = PRIMARY,
	conf_id    = 1,
	members    = 1/2 (joined/total),
	act_id     = 0,
	last_appl. = -1,
	protocols  = 0/7/3 (gcs/repl/appl),
	group UUID = 49b5ca35-2db8-11e8-93dd-c3424666cb76
2018-03-22 10:04:14 140329026053888 [Note] WSREP: Flow-control interval: [23, 23]
2018-03-22 10:04:14 140329026053888 [Note] WSREP: Trying to continue unpaused monitor
2018-03-22 10:04:14 140329026053888 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
2018-03-22 10:04:14 140329350146816 [Note] WSREP: State transfer required: 
	Group state: 49b5ca35-2db8-11e8-93dd-c3424666cb76:0
	Local state: 00000000-0000-0000-0000-000000000000:-1
2018-03-22 10:04:14 140329350146816 [Note] WSREP: New cluster view: global state: 49b5ca35-2db8-11e8-93dd-c3424666cb76:0, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 3
2018-03-22 10:04:14 140329350146816 [Warning] WSREP: Gap in state sequence. Need state transfer.
2018-03-22 10:04:14 140328996697856 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address 'mysql-1.galera.galera.svc.cluster.local' --datadir '/var/lib/mysql/'   --parent '1'  '' '
WSREP_SST: [INFO] Streaming with xbstream (20180322 10:04:14.328)
WSREP_SST: [INFO] Using socat as streamer (20180322 10:04:14.330)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20180322 10:04:14.359)
2018-03-22 10:04:16 140329034446592 [Note] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') connection to peer 5f4ff20f with addr tcp://10.131.0.110:4567 timed out, no messages seen in PT3S
2018-03-22 10:04:16 140329034446592 [Note] WSREP: (5f4ff20f, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [ERROR] Possible timeout in receving first data from donor in gtid stage (20180322 10:05:54.364)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20180322 10:05:54.366)
2018-03-22 10:05:54 140328996697856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address 'mysql-1.galera.galera.svc.cluster.local' --datadir '/var/lib/mysql/'   --parent '1'  '' 
	Read: '(null)'
2018-03-22 10:05:54 140328996697856 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address 'mysql-1.galera.galera.svc.cluster.local' --datadir '/var/lib/mysql/'   --parent '1'  '' : 32 (Broken pipe)
2018-03-22 10:05:54 140329350146816 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. Unrecoverable.
2018-03-22 10:05:54 140329350146816 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit

Logs of the first container (mysql-0):

2018-03-22 10:03:36 139761551272192 [Note] WSREP: save pc into disk
--
  | 2018-03-22 10:03:36 139761551272192 [Note] WSREP: gcomm: connected
  | 2018-03-22 10:03:36 139761551272192 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
  | 2018-03-22 10:03:36 139761551272192 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
  | 2018-03-22 10:03:36 139761551272192 [Note] WSREP: Opened channel 'galera'
  | 2018-03-22 10:03:36 139761551272192 [Note] WSREP: Waiting for SST to complete.
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: Starting new group from scratch: 49b5ca35-2db8-11e8-93dd-c3424666cb76
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 49b5cbec-2db8-11e8-b4a7-7a948ae5dbe5
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: STATE EXCHANGE: sent state msg: 49b5cbec-2db8-11e8-b4a7-7a948ae5dbe5
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: STATE EXCHANGE: got state msg: 49b5cbec-2db8-11e8-b4a7-7a948ae5dbe5 from 0 (mysql-0)
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: Quorum results:
  | version    = 4,
  | component  = PRIMARY,
  | conf_id    = 0,
  | members    = 1/1 (joined/total),
  | act_id     = 0,
  | last_appl. = -1,
  | protocols  = 0/7/3 (gcs/repl/appl),
  | group UUID = 49b5ca35-2db8-11e8-93dd-c3424666cb76
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: Flow-control interval: [16, 16]
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: Trying to continue unpaused monitor
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: Restored state OPEN -> JOINED (0)
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: Member 0.0 (mysql-0) synced with group.
  | 2018-03-22 10:03:36 139761226344192 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
  | 2018-03-22 10:03:36 139761550953216 [Note] WSREP: New cluster view: global state: 49b5ca35-2db8-11e8-93dd-c3424666cb76:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
  | 2018-03-22 10:03:36 139761551272192 [Note] WSREP: SST complete, seqno: 0
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
  |  
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: Using mutexes to ref count buffer pool pages
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: The InnoDB memory heap is disabled
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: Compressed tables use zlib 1.2.7
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: Using Linux native AIO
  | 2018-03-22 10:03:36 139761551272192 [Note] InnoDB: Using generic crc32 instructions
  | 2018-03-22 10:03:37 139761551272192 [Note] InnoDB: Initializing buffer pool, size = 128.0M
  | 2018-03-22 10:03:37 139761551272192 [Note] InnoDB: Completed initialization of buffer pool
  | 2018-03-22 10:03:37 139761551272192 [Note] InnoDB: Highest supported file format is Barracuda.
  | 2018-03-22 10:03:37 139761551272192 [Note] InnoDB: 128 rollback segment(s) are active.
  | 2018-03-22 10:03:37 139761551272192 [Note] InnoDB: Waiting for purge to start
  | 2018-03-22 10:03:37 139761551272192 [Note] InnoDB:  Percona XtraDB (http://www.percona.com ) 5.6.36-83.0 started; log sequence number 1616727
  | 2018-03-22 10:03:37 139760517510912 [Note] InnoDB: Dumping buffer pool(s) not yet started
  | 2018-03-22 10:03:37 139761551272192 [Note] Plugin 'FEEDBACK' is disabled.
  | 2018-03-22 10:03:37 139761551272192 [Note] Server socket created on IP: '0.0.0.0'.
  | 2018-03-22 10:03:37 139761551272192 [Note] mysqld: ready for connections.
  | Version: '10.1.31-MariaDB'  socket: '/var/run/mysql/mysql.sock'  port: 3306  MariaDB Server
  | 2018-03-22 10:03:37 139761550953216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  | 2018-03-22 10:03:37 139761550953216 [Note] WSREP: REPL Protocols: 7 (3, 2)
  | 2018-03-22 10:03:37 139761550953216 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
  | 2018-03-22 10:03:37 139761284654848 [Note] WSREP: Service thread queue flushed.
  | 2018-03-22 10:03:37 139761550953216 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 49b5ca35-2db8-11e8-93dd-c3424666cb76:0
  | 2018-03-22 10:03:37 139761550953216 [Note] WSREP: Synchronized with group, ready for connections
  | 2018-03-22 10:03:37 139761550953216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  | 2018-03-22 10:04:13 139761234736896 [Note] WSREP: (49b580d4, 'tcp://0.0.0.0:4567') connection established to 5f4ff20f tcp://10.131.0.110:4567
  | 2018-03-22 10:04:13 139761234736896 [Note] WSREP: (49b580d4, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
  | 2018-03-22 10:04:13 139761234736896 [Note] WSREP: declaring 5f4ff20f at tcp://10.131.0.110:4567 stable
  | 2018-03-22 10:04:13 139761234736896 [Note] WSREP: Node 49b580d4 state prim
  | 2018-03-22 10:04:13 139761234736896 [Note] WSREP: view(view_id(PRIM,49b580d4,2) memb {
  | 49b580d4,0
  | 5f4ff20f,0
  | } joined {
  | } left {
  | } partitioned {
  | })
  | 2018-03-22 10:04:13 139761234736896 [Note] WSREP: save pc into disk
  | 2018-03-22 10:04:13 139761226344192 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
  | 2018-03-22 10:04:13 139761226344192 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 5f9d3238-2db8-11e8-a838-6a2b63375367
  | 2018-03-22 10:04:13 139761226344192 [Note] WSREP: STATE EXCHANGE: sent state msg: 5f9d3238-2db8-11e8-a838-6a2b63375367
  | 2018-03-22 10:04:13 139761226344192 [Note] WSREP: STATE EXCHANGE: got state msg: 5f9d3238-2db8-11e8-a838-6a2b63375367 from 0 (mysql-0)
  | 2018-03-22 10:04:14 139761226344192 [Note] WSREP: STATE EXCHANGE: got state msg: 5f9d3238-2db8-11e8-a838-6a2b63375367 from 1 (mysql-1)
  | 2018-03-22 10:04:14 139761226344192 [Note] WSREP: Quorum results:
  | version    = 4,
  | component  = PRIMARY,
  | conf_id    = 1,
  | members    = 1/2 (joined/total),
  | act_id     = 0,
  | last_appl. = 0,
  | protocols  = 0/7/3 (gcs/repl/appl),
  | group UUID = 49b5ca35-2db8-11e8-93dd-c3424666cb76
  | 2018-03-22 10:04:14 139761226344192 [Note] WSREP: Flow-control interval: [23, 23]
  | 2018-03-22 10:04:14 139761226344192 [Note] WSREP: Trying to continue unpaused monitor
  | 2018-03-22 10:04:14 139761550953216 [Note] WSREP: New cluster view: global state: 49b5ca35-2db8-11e8-93dd-c3424666cb76:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
  | 2018-03-22 10:04:14 139761550953216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  | 2018-03-22 10:04:14 139761550953216 [Note] WSREP: REPL Protocols: 7 (3, 2)
  | 2018-03-22 10:04:14 139761550953216 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
  | 2018-03-22 10:04:14 139761284654848 [Note] WSREP: Service thread queue flushed.
  | 2018-03-22 10:04:16 139761234736896 [Note] WSREP: (49b580d4, 'tcp://0.0.0.0:4567') turning message relay requesting off
  | 2018-03-22 10:05:57 139761234736896 [Note] WSREP: forgetting 5f4ff20f (tcp://10.131.0.110:4567)
  | 2018-03-22 10:05:57 139761234736896 [Note] WSREP: Node 49b580d4 state prim
  | 2018-03-22 10:05:57 139761234736896 [Note] WSREP: view(view_id(PRIM,49b580d4,3) memb {
  | 49b580d4,0
  | } joined {
  | } left {
  | } partitioned {
  | 5f4ff20f,0
  | })
  | 2018-03-22 10:05:57 139761234736896 [Note] WSREP: save pc into disk
  | 2018-03-22 10:05:57 139761234736896 [Note] WSREP: forgetting 5f4ff20f (tcp://10.131.0.110:4567)
  | 2018-03-22 10:05:57 139761226344192 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
  | 2018-03-22 10:05:57 139761226344192 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 9d7549e4-2db8-11e8-a213-ee2e0ed4bdf3
  | 2018-03-22 10:05:57 139761226344192 [Note] WSREP: STATE EXCHANGE: sent state msg: 9d7549e4-2db8-11e8-a213-ee2e0ed4bdf3
  | 2018-03-22 10:05:57 139761226344192 [Note] WSREP: STATE EXCHANGE: got state msg: 9d7549e4-2db8-11e8-a213-ee2e0ed4bdf3 from 0 (mysql-0)
  | 2018-03-22 10:05:57 139761226344192 [Note] WSREP: Quorum results:
  | version    = 4,
  | component  = PRIMARY,
  | conf_id    = 2,
  | members    = 1/1 (joined/total),
  | act_id     = 0,
  | last_appl. = 0,
  | protocols  = 0/7/3 (gcs/repl/appl),
  | group UUID = 49b5ca35-2db8-11e8-93dd-c3424666cb76
  | 2018-03-22 10:05:57 139761226344192 [Note] WSREP: Flow-control interval: [16, 16]
  | 2018-03-22 10:05:57 139761226344192 [Note] WSREP: Trying to continue unpaused monitor
  | 2018-03-22 10:05:57 139761550953216 [Note] WSREP: New cluster view: global state: 49b5ca35-2db8-11e8-93dd-c3424666cb76:0, view# 3: Primary, number of nodes: 1, my index: 0, protocol version 3
  | 2018-03-22 10:05:57 139761550953216 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  | 2018-03-22 10:05:57 139761550953216 [Note] WSREP: REPL Protocols: 7 (3, 2)
  | 2018-03-22 10:05:57 139761550953216 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
  | 2018-03-22 10:05:57 139761284654848 [Note] WSREP: Service thread queue flushed.
  | 2018-03-22 10:06:01 139761234736896 [Note] WSREP: (49b580d4, 'tcp://0.0.0.0:4567') connection established to 5f4ff20f tcp://10.131.0.110:4567
  | 2018-03-22 10:06:01 139761234736896 [Warning] WSREP: discarding established (time wait) 5f4ff20f (tcp://10.131.0.110:4567)
  | 2018-03-22 10:06:02 139761234736896 [Note] WSREP:  cleaning up 5f4ff20f (tcp://10.131.0.110:4567)
  | 2018-03-22 10:06:27 139761234736896 [Note] WSREP: (49b580d4, 'tcp://0.0.0.0:4567') connection established to af9c3439 tcp://10.131.0.110:4567
  | 2018-03-22 10:06:27 139761234736896 [Note] WSREP: (49b580d4, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
  | 2018-03-22 10:06:28 139761234736896 [Note] WSREP: declaring af9c3439 at tcp://10.131.0.110:4567 stable
  | 2018-03-22 10:06:28 139761234736896 [Note] WSREP: Node 49b580d4 state prim
  | 2018-03-22 10:06:28 139761234736896 [Note] WSREP: view(view_id(PRIM,49b580d4,4) memb {
  | 49b580d4,0
  | af9c3439,0
  | } joined {
  | } left {
  | } partitioned {
  | })
  | 2018-03-22 10:06:28 139761234736896 [Note] WSREP: save pc into disk
  | 2018-03-22 10:06:28 139761226344192 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
  | 2018-03-22 10:06:28 139761226344192 [Note] WSREP: STATE_EXCHANGE: sent state UUID: afe96f99-2db8-11e8-a120-7fc34bc2f1d7
  | 2018-03-22 10:06:28 139761226344192 [Note] WSREP: STATE EXCHANGE: sent state msg: afe96f99-2db8-11e8-a120-7fc34bc2f1d7
  | 2018-03-22 10:06:28 139761226344192 [Note] WSREP: STATE EXCHANGE: got state msg: afe96f99-2db8-11e8-a120-7fc34bc2f1d7 from 0 (mysql-0)
  | 2018-03-22 10:06:28 139761226344192 [Note] WSREP: STATE EXCHANGE: got state msg: afe96f99-2db8-11e8-a120-7fc34bc2f1d7 from 1 (mysql-1)
  | 2018-03-22 10:06:28 139761226344192 [Note] WSREP: Quorum results:
  | version    = 4,
  | component  = PRIMARY,
  | conf_id    = 3,
  | members    = 1/2 (joined/total),
  | act_id     = 0,
  | last_appl. = 0,
  | protocols  = 0/7/3 (gcs/repl/appl),
  | group UUID = 49b5ca35-2db8-11e8-93dd-c3424666cb76
  | 2018-03-22 10:06:28 139761226344192 [Note] WSREP: Flow-control interval: [23, 23]

Thanks,
Thomas

Can't work with a custom dns domain

Hi,
I was deploying this statefulset on my local kubernetes(v1.5) cluster using the script galera_k8s_v1.5.yml. But the first pod failed to start. After checking the log serveral times I found it stuck at:

+ /usr/bin/peer-finder -on-start=/uisr/share/container-scripts/mysql/configure-galera.sh -service=galera

I was pretty sure the dns is working perfectly, so I checked the source code of peer-finder and did some debugging. Then I found out where the problem is.
https://github.com/kubernetes/contrib/blob/master/pets/peer-finder/peer-finder.go#L43

	domain    = flag.String("domain", "cluster.local", "The Cluster Domain which is used by the Cluster.")

peer-finder is asking for a domain flag which could be passed by --domain, or it'll use the default setting cluster.local. In my case, I used a custom dns domain test.local instead of the default one cluster.local. Then peer-finder will compare mysql-0.galera.default.svc.cluster.local with mysql-0.galera.default.svc.test.local and then stuck in a infinite loop.
So could anyone add an ENV or something else and pass it to the peer-finder to get it work?

Recover from power failure in DC

If all nodes are ever stopped e.g. from a power failure in a DC (an operational tested scenario), the cluster will fail to start.

It would seem as though the /var/lib/mysql/grastate.dat file never has suitable information for an automated recovery (as documented on galeracluster.com).

This results in the following symptoms:
2018-05-25 13:29:33 140151997229312 [Warning] WSREP: no nodes coming from prim view, prim not possible

And then fail to connect (the same on all nodes):

2018-05-25 13:30:03 140151997229312 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
	 at gcomm/src/pc.cpp:connect():158
2018-05-25 13:30:03 140151997229312 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
2018-05-25 13:30:03 140151997229312 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel 'galera' at 'gcomm://mysql-0.galera.sysdig.svc.cluster.local,mysql-1.galera.sysdig.svc.cluster.local,mysql-2.galera.sysdig.svc.cluster.local': -110 (Connection timed out)
2018-05-25 13:30:03 140151997229312 [ERROR] WSREP: gcs connect failed: Connection timed out
2018-05-25 13:30:03 140151997229312 [ERROR] WSREP: wsrep::connect(gcomm://mysql-0.galera.sysdig.svc.cluster.local,mysql-1.galera.sysdig.svc.cluster.local,mysql-2.galera.sysdig.svc.cluster.local) failed: 7
2018-05-25 13:30:03 140151997229312 [ERROR] Aborting

All the grastate.dat files seem equivalent so any should/could potentially be startable:

$ for i in {0..2} ; do kubectl -n sysdig exec -it mysql-${i} cat -- /var/lib/mysql/grastate.dat ; done
# GALERA saved state
version: 2.1
uuid:    2abe687e-6011-11e8-97dd-c70a88466155
seqno:   -1
safe_to_bootstrap: 0
    # GALERA saved state
version: 2.1
uuid:    2abe687e-6011-11e8-97dd-c70a88466155
seqno:   -1
safe_to_bootstrap: 0
    # GALERA saved state
version: 2.1
uuid:    2abe687e-6011-11e8-97dd-c70a88466155
seqno:   -1
safe_to_bootstrap: 0
$

A bit worried the UUID seems to be the same for all nodes...

See the complete logs here:
mysql-2.log
mysql-1.log
mysql-0.log

Failed to open file '/var/lib/mysql//galera.cache': 1 (Operation not permitted)

I added priviliged scc to default service account.

Full log file is :

• CONTAINER_SCRIPTS_DIR=/usr/share/container-scripts/mysql
  --
    | + EXTRA_DEFAULTS_FILE=/etc/my.cnf.d/galera.cnf
    | + '[' -z dbcluster1 ']'
    | + echo 'Galera: Finding peers'
    | Galera: Finding peers
    | ++ hostname -f
    | ++ cut -d. -f2
    | Using service name: galera
    | + K8S_SVC_NAME=galera
    | + echo 'Using service name: galera'
    | + cp /usr/share/container-scripts/mysql/galera.cnf /etc/my.cnf.d/galera.cnf
    | + /usr/bin/peer-finder -on-start=/usr/share/container-scripts/mysql/configure-galera.sh -service=galera
    | 2018/07/16 19:04:10 Peer list updated
    | was []
    | now [mysql-0.galera.dbcluster1.svc.cluster.local]
    | 2018/07/16 19:04:10 execing: /usr/share/container-scripts/mysql/configure-galera.sh with stdin: mysql-0.galera.dbcluster1.svc.cluster.local
    | 2018/07/16 19:04:10
    | 2018/07/16 19:04:11 Peer finder exiting
    | + '[' '!' -d /var/lib/mysql/mysql ']'
    | + exec mysqld
    | 2018-07-16 19:04:11 140443185940736 [Note] mysqld (mysqld 10.1.31-MariaDB) starting as process 1 ...
    | 2018-07-16 19:04:11 140443185940736 [Note] WSREP: Read nil XID from storage engines, skipping position init
    | 2018-07-16 19:04:11 140443185940736 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
    | 2018-07-16 19:04:11 140443185940736 [Note] WSREP: wsrep_load(): Galera 25.3.22(r3764) by Codership Oy [email protected] loaded successfully.
    | 2018-07-16 19:04:11 140443185940736 [Note] WSREP: CRC-32C: using hardware acceleration.
    | 2018-07-16 19:04:11 140443185940736 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
    | 2018-07-16 19:04:11 140443185940736 [ERROR] WSREP: Failed to open file '/var/lib/mysql//galera.cache': 1 (Operation not permitted)
    | at galerautils/src/gu_fdesc.cpp:constructor_common():142
    | 2018-07-16 19:04:11 140443185940736 [ERROR] WSREP: wsrep::init() failed: 7, must shutdown
    | 2018-07-16 19:04:11 140443185940736 [ERROR] Aborting

How can I fix it?