Здравейте,
Днес ми се случи отпадане на един MySQL сървър, и в резултат на това се получи така наречения MariaDB Galera Cluster Split Brain.
Тъй като използвам 2 нода, което не е препоръчително за Galera Cluster, сутринта работещия нод връщаше следната грешка:
WSREP has not yet prepared node for application use
WSREP has not yet prepared node for application use
WSREP has not yet prepared node for application use
След като оправих проблемния нод (достатъчен беше един рестарт), реших да се справя и с този Split Brain.
Идеята като цяло беше, да добавя трети нод в клъстера, или да включа арбитър.
Първоначалната ситуация изглеждаше така :
mysql> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 2 |
+--------------------+-------+
mysql> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 2 |
+--------------------+-------+
1 row in set (0.01 sec)
mysql> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 2 |
+--------------------+-------+
1 row in set (0.01 sec)
За да увелича размера на клъстера до 3, инсталирах и конфигурирах galera arbitrator на трета машина с минимални ресурси.
apt-get install galera-arbitrator-3
apt-get install galera-arbitrator-3
apt-get install galera-arbitrator-3
Тъй като клъстера ми използва SSL сертификати, се наложи да ги копирам и на тази машина, за да мога да стартирам garbd демона.
Конфигурационният му файл (/etc/default/garb) изглежда така :
# Copyright (C) 2012 Codership Oy
# This config file is to be sourced by garb service script.
# A comma-separated list of node addresses (address[:port]) in the cluster
GALERA_NODES="192.168.168.1:4567 192.168.168.16:4567"
# Galera cluster name, should be the same as on the rest of the nodes.
GALERA_GROUP="cbz_cluster"
# Optional Galera internal options string (e.g. SSL settings)
# see http://galeracluster.com/documentation-webpages/galeraparameters.html
GALERA_OPTIONS="socket.ssl_key=/etc/mysql/ssl/server-key.pem;socket.ssl_cert=/etc/mysql/ssl/server-cert.pem;socket.ssl_ca=/etc/mysql/ssl/ca-cert.pem;socket.ssl_cipher=AES128-SHA"
# Log file for garbd. Optional, by default logs to syslog
# LOG_FILE="/var/log/garbd.log"
# Copyright (C) 2012 Codership Oy
# This config file is to be sourced by garb service script.
# A comma-separated list of node addresses (address[:port]) in the cluster
GALERA_NODES="192.168.168.1:4567 192.168.168.16:4567"
# Galera cluster name, should be the same as on the rest of the nodes.
GALERA_GROUP="cbz_cluster"
# Optional Galera internal options string (e.g. SSL settings)
# see http://galeracluster.com/documentation-webpages/galeraparameters.html
GALERA_OPTIONS="socket.ssl_key=/etc/mysql/ssl/server-key.pem;socket.ssl_cert=/etc/mysql/ssl/server-cert.pem;socket.ssl_ca=/etc/mysql/ssl/ca-cert.pem;socket.ssl_cipher=AES128-SHA"
# Log file for garbd. Optional, by default logs to syslog
# LOG_FILE="/var/log/garbd.log"
# Copyright (C) 2012 Codership Oy
# This config file is to be sourced by garb service script.
# A comma-separated list of node addresses (address[:port]) in the cluster
GALERA_NODES="192.168.168.1:4567 192.168.168.16:4567"
# Galera cluster name, should be the same as on the rest of the nodes.
GALERA_GROUP="cbz_cluster"
# Optional Galera internal options string (e.g. SSL settings)
# see http://galeracluster.com/documentation-webpages/galeraparameters.html
GALERA_OPTIONS="socket.ssl_key=/etc/mysql/ssl/server-key.pem;socket.ssl_cert=/etc/mysql/ssl/server-cert.pem;socket.ssl_ca=/etc/mysql/ssl/ca-cert.pem;socket.ssl_cipher=AES128-SHA"
# Log file for garbd. Optional, by default logs to syslog
# LOG_FILE="/var/log/garbd.log"
Последна стъпка – стартиране на процеса :
web-root:~# /etc/init.d/garb restart . ok [ ok ] Starting /usr/bin/garbd: :.
web-root:~# /etc/init.d/garb restart . ok [ ok ] Starting /usr/bin/garbd: :.
web-root:~# /etc/init.d/garb restart . ok [ ok ] Starting /usr/bin/garbd: :.
И wsrep_cluster_size вече беше 3.
mysql> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
mysql> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.00 sec)
mysql> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.00 sec)
Така при отпадане на един от нодовете, другия ще продължи да работи нормално.
Това е!