Galera node will not come up
I have 3 MySQL databases setup in a Galera cluster. While looking into an issue I noticed that the third node is down. When I run service mysql start it hangs for about a minute and then fails to start. Running journalctl -xe I get the following.
Jun 04 08:20:55 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:00 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:05 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:10 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:15 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:20 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:25 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:30 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:35 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:40 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:45 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:50 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:21:55 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:22:00 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:22:05 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:22:10 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:22:15 bloom-db-3 Keepalived_vrrp[1335]: /usr/bin/clustercheck clustercheck [hidden] 0 exited with status 1
Jun 04 08:22:18 bloom-db-3 systemd[1]: mysql.service: Start operation timed out. Terminating.
Jun 04 08:22:18 bloom-db-3 systemd[1]: mysql.service: Failed with result 'timeout'.
Jun 04 08:22:18 bloom-db-3 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mysql.service has failed.
--
-- The result is RESULT.
When I run /usr/bin/clustercheck clustercheck [hidden] 0 I get the following.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 57
Percona XtraDB Cluster Node is not synced or non-PRIM.
1 Reply
rgerke
Linode Staff
As I'm not yet a Percona expert, I did some digging on this to see what I could find. I want to address the last error that you gave as it looks to be the source of the issue:
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 57
Percona XtraDB Cluster Node is not synced or non-PRIM.
From what I've found, this may have to do with bootstrapping. I'm going to link you to a site on Percona's forums where users are talking about this situation:
Percona: Cluster Node is Not Synced
Since I can't see your setup, I can only surmise that this may be the issue. The gist of the conversation states that you should only be bootstrapping one node and starting the other two normally when utilizing a Galera cluster.
It also advises you check the logs of the one bootstrapped Linode to make sure that it started up because this error seems to indicate that the node is offline.