Mariadb-Galera pods get restarted for 1 time after applying mariadb-galera chart on aws ec2 k8s cluster about mariadb-operator

[vinaykw]

@mmontes11 - Any updates on this issue. Can you please help us here.
Thank you

[mmontes11]

Hey there! Thanks for reporting.

We need more light around why the Pod was restarted in the first place. This error seems unrelated:

ERROR 1396 (HY000) at line 21: Operation CREATE USER failed for 'root'@'%'

Could you provide MariaDB instance events the following?:

kubectl get events --field-selector involvedObject.name=<mariadb-resource-name>

It might have to do with the liveness probe, you can tune it here:

mariadb-operator/examples/manifests/mariadb_v1alpha1_mariadb_full.yaml

Line 75 in 284539a

livenessProbe:

[vinaykw]

@mmontes11 ,

Below is the output of kubectl describe of the pod and it shows restart count as 1. But it does not indicate that it is due to liveness probe failure.

Containers:
  mariadb:
    Container ID:   containerd://f8e4301da5aa5b408fcbc90c7d37670ed3c1f720ad2573ba3ea4ae84b32e2c87
    Image:          669462458354.dkr.ecr.us-east-1.amazonaws.com/mariadb:10.5.17
    Image ID:       669462458354.dkr.ecr.us-east-1.amazonaws.com/mariadb@sha256:4d6380ea98b2db6271ba999f5dc59b74ac73f92b3bb96b25804d62d056dcac85
    Ports:          3306/TCP, 4444/TCP, 4567/TCP, 4568/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Wed, 03 Jan 2024 10:45:10 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 03 Jan 2024 10:45:05 +0000
      Finished:     Wed, 03 Jan 2024 10:45:09 +0000
    Ready:          True
    Restart Count:  1
    Limits:
      memory:  1Gi
    Requests:
      cpu:      300m
      memory:   256Mi
    Liveness:   exec [bash -c mariadb -u root -p"${MARIADB_ROOT_PASSWORD}" -e "SHOW STATUS LIKE 'wsrep_ready'" | grep -c ON] delay=60s timeout=5s period=10s #success=1 #failure=3
    Readiness:  exec [bash -c mariadb -u root -p"${MARIADB_ROOT_PASSWORD}" -e "SHOW STATUS LIKE 'wsrep_ready'" | grep -c ON] delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:
      MYSQL_TCP_PORT:            3306
      MARIADB_ROOT_PASSWORD:     <set to the key 'root-password' in secret 'mariadb'>  Optional: false
      MARIADB_ROOT_HOST:         %
      MYSQL_INITDB_SKIP_TZINFO:  1
      CLUSTER_NAME:              cluster.local
      POD_NAME:                  mariadb-galera-0 (v1:metadata.name)
      MARIADB_DATABASE:          mariadb
      MARIADB_USER:              root
      MARIADB_PASSWORD:          <set to the key 'password' in secret 'mariadb'>  Optional: false
    Mounts:
      /etc/mysql/conf.d from config (rw)
      /etc/mysql/mariadb.conf.d from galera (rw)
      /var/lib/mysql from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from serviceaccount (rw)

I also have the kubectl get events output below:

ubuntu@manager-0:~$ kubectl get events --field-selector involvedObject.name=mariadb-galera-1

LAST SEEN   TYPE      REASON                   OBJECT                 MESSAGE
11m         Warning   FailedScheduling         pod/mariadb-galera-1   0/5 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling..
11m         Normal    Scheduled                pod/mariadb-galera-1   Successfully assigned default/mariadb-galera-1 to worker-0
11m         Normal    SuccessfulAttachVolume   pod/mariadb-galera-1   AttachVolume.Attach succeeded for volume "pvc-13c531e9-5797-4849-a0ee-27869a7602ec"
11m         Normal    SuccessfulAttachVolume   pod/mariadb-galera-1   AttachVolume.Attach succeeded for volume "pvc-f4d0b457-54a6-4663-b973-31bed03e3d83"
11m         Normal    Pulled                   pod/mariadb-galera-1   Container image "669462458354.dkr.ecr.us-east-1.amazonaws.com/ghcr.io/mariadb-operator/init:v0.0.6" already present on machine
11m         Normal    Created                  pod/mariadb-galera-1   Created container init
11m         Normal    Started                  pod/mariadb-galera-1   Started container init
10m         Normal    Pulled                   pod/mariadb-galera-1   Container image "669462458354.dkr.ecr.us-east-1.amazonaws.com/mariadb:10.5.17" already present on machine
10m         Normal    Created                  pod/mariadb-galera-1   Created container mariadb
10m         Normal    Started                  pod/mariadb-galera-1   Started container mariadb
10m         Normal    Pulled                   pod/mariadb-galera-1   Container image "669462458354.dkr.ecr.us-east-1.amazonaws.com/ghcr.io/mariadb-operator/agent:v0.0.3" already present on machine
10m         Normal    Created                  pod/mariadb-galera-1   Created container agent
10m         Normal    Started                  pod/mariadb-galera-1   Started container agent

ubuntu@manager-0:~$ kubectl get events --field-selector involvedObject.name=mariadb-galera

LAST SEEN   TYPE     REASON                 OBJECT                               MESSAGE
12m         Normal   NoPods                 poddisruptionbudget/mariadb-galera   No matching pods found
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Claim storage-mariadb-galera-0 Pod mariadb-galera-0 in StatefulSet mariadb-galera success
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Claim galera-mariadb-galera-0 Pod mariadb-galera-0 in StatefulSet mariadb-galera success
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Pod mariadb-galera-0 in StatefulSet mariadb-galera successful
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Claim storage-mariadb-galera-2 Pod mariadb-galera-2 in StatefulSet mariadb-galera success
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Claim storage-mariadb-galera-1 Pod mariadb-galera-1 in StatefulSet mariadb-galera success
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Claim galera-mariadb-galera-2 Pod mariadb-galera-2 in StatefulSet mariadb-galera success
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Claim galera-mariadb-galera-1 Pod mariadb-galera-1 in StatefulSet mariadb-galera success
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Pod mariadb-galera-2 in StatefulSet mariadb-galera successful
12m         Normal   SuccessfulCreate       statefulset/mariadb-galera           create Pod mariadb-galera-1 in StatefulSet mariadb-galera successful
8m6s        Normal   GaleraClusterHealthy   mariadb/mariadb-galera               Galera cluster is healthy

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[mmontes11]

Hey there @vinaykw !

It is hard to tell with the current information why the Pod gort restarted, neither the logs nor the events show any light. We will need to dig deeper!

This issue is not seen with VM based k8s cluster where we are using glusterfs storage. I have added the podsecurity context to run as user 0 for rook-ceph

Thinking about this now, how a storage type can make a Pod restart?

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[mmontes11]

I encourage you to upgrade first to v0.0.26 and then to v0.0.27, as we've made significant improvements on the Galera support.

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.