Saturday, June 20, 2015
Failover with the MySQL Utilities: Part 2 – mysqlfailover
Failover with the MySQL Utilities: Part 2 – mysqlfailover
In the previous post of this series we saw how you could use
mysqlrpladmin
to perform manual failover/switchover when GTID replication is enabled in MySQL 5.6. Now we will review mysqlfailover
(version 1.4.3), another tool from the MySQL Utilities that can be used for automatic failover.Summary
mysqlfailover
can perform automatic failover if MySQL 5.6’s GTID-replication is enabled.- All slaves must use
--master-info-repository=TABLE
. - The monitoring node is a single point of failure: don’t forget to monitor it!
- Detection of errant transactions works well, but you have to use the
--pedantic
option to make sure failover will never happen if there is an errant transaction. - There are a few limitations such as the inability to only fail over once, or excessive CPU utilization, but they are probably not showstoppers for most setups.
Setup
We will use the same setup as last time: one master and two slaves, all using GTID replication. We can see the topology using
mysqlfailover
with the health
command:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root health
[...]
MySQL Replication Failover Utility
Failover Mode = auto Next Interval = Tue Jul 1 10:01:22 2014
Master Information
------------------
Binary Log File Position Binlog_Do_DB Binlog_Ignore_DB
mysql-bin.000003 700
GTID Executed Set
a9a396c6-00f3-11e4-8e66-9cebe8067a3f:1-3
Replication Health Status
+------------+--------+---------+--------+------------+---------+
| host | port | role | state | gtid_mode | health |
+------------+--------+---------+--------+------------+---------+
| localhost | 13001 | MASTER | UP | ON | OK |
| localhost | 13002 | SLAVE | UP | ON | OK |
| localhost | 13003 | SLAVE | UP | ON | OK |
+------------+--------+---------+--------+------------+---------+
|
Note that
--master-info-repository=TABLE
needs to be configured on all slaves or the tool will exit with an error message:
1
2
|
2014-07-01 10:18:55 AM CRITICAL Failover requires --master-info-repository=TABLE for all slaves.
ERROR: Failover requires --master-info-repository=TABLE for all slaves.
|
Failover
You can use 2 commands to trigger automatic failover:
auto
: the tool tries to find a candidate in the list of servers specified with--candidates
, and if no good server is found in this list, it will look at the other slaves to see if one can be a good candidate. This is the default commandelect
: same asauto
, but if no good candidate is found in the list of candidates, other slaves will not be checked and the tool will exit with an error.
Let’s start the tool with
auto
:
1
|
$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root auto
|
The monitoring console is visible and is refreshed every
--interval
seconds (default: 15). Its output is similar to what you get when using the health
command.
Then let’s kill -9 the master to see what happens once the master is detected as down:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
Failed to reconnect to the master after 3 attemps.
Failover starting in 'auto' mode...
# Candidate slave localhost:13002 will become the new master.
# Checking slaves status (before failover).
# Preparing candidate for failover.
# Creating replication user if it does not exist.
# Stopping slaves.
# Performing STOP on all slaves.
# Switching slaves to new master.
# Disconnecting new master as slave.
# Starting slaves.
# Performing START on all slaves.
# Checking slaves for errors.
# Failover complete.
# Discovering slaves for master at localhost:13002
Failover console will restart in 5 seconds.
MySQL Replication Failover Utility
Failover Mode = auto Next Interval = Tue Jul 1 10:59:47 2014
Master Information
------------------
Binary Log File Position Binlog_Do_DB Binlog_Ignore_DB
mysql-bin.000005 191
GTID Executed Set
a9a396c6-00f3-11e4-8e66-9cebe8067a3f:1-3
Replication Health Status
+------------+--------+---------+--------+------------+---------+
| host | port | role | state | gtid_mode | health |
+------------+--------+---------+--------+------------+---------+
| localhost | 13002 | MASTER | UP | ON | OK |
| localhost | 13003 | SLAVE | UP | ON | OK |
+------------+--------+---------+--------+------------+---------+
|
Looks good! The tool is then ready to fail over to another slave if the new master becomes unavailable.
You can also run custom scripts at several points of execution with the
--exec-before
, --exec-after
, --exec-fail-check
, --exec-post-failover
options.
However it would be great to have a
--failover-and-exit
option to avoid flapping: the tool would detect master failure, promote one of the slaves, reconfigure replication and then exit (this is what MHA does for instance).Tool registration
When the tool is started, it registers itself on the master by writing a few things in the specific table:
1
2
3
4
5
6
|
mysql> SELECT * FROM mysql.failover_console;
+-----------+-------+
| host | port |
+-----------+-------+
| localhost | 13001 |
+-----------+-------+
|
This is nice as it avoids that you start several instances of
mysqlfailover
to monitor the same master. If we try, this is what we get:
1
2
3
4
5
6
|
$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root auto
[...]
Multiple instances of failover console found for master localhost:13001.
If this is an error, restart the console with --force.
Failover mode changed to 'FAIL' for this instance.
Console will start in 10 seconds..........starting Console.
|
With the
fail
command, mysqlfailover
will monitor replication health and exit in the case of a master failure, without actually performing failover.Running in the background
In all previous examples,
mysqlfailover
was running in the foreground. This is very good for demo, but in a production environment you are likely to prefer running it in the background. This can be done with the --daemon
option:
1
|
$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root auto --daemon=start --log=/var/log/mysqlfailover.log
|
and it can be stopped with:
1
|
$ mysqlfailover --daemon=stop
|
Errant transactions
If we create an errant transaction on one of the slaves, it will be detected:
1
2
3
4
5
6
|
MySQL Replication Failover Utility
Failover Mode = auto Next Interval = Tue Jul 1 16:29:44 2014
[...]
WARNING: Errant transaction(s) found on slave(s).
Replication Health Status
[...]
|
However this does not prevent failover from occurring! You have to use
--pedantic
:
1
2
3
4
5
6
|
$ mysqlfailover --master=root@localhost:13001 --discover-slaves-login=root --pedantic auto
[...]
# WARNING: Errant transaction(s) found on slave(s).
# - For slave 'localhost@13003': db906eee-012d-11e4-8fe1-9cebe8067a3f:1
2014-07-01 16:44:49 PM CRITICAL Errant transaction(s) found on slave(s). Note: If you want to ignore this issue, please do not use the --pedantic option.
ERROR: Errant transaction(s) found on slave(s). Note: If you want to ignore this issue, please do not use the --pedantic option.
|
Limitations
- Like for
mysqlrpladmin
, the slave election process is not very sophisticated and it cannot be tuned. - The server on which
mysqlfailover
is running is a single point of failure. - Excessive CPU utilization: once it is running,
mysqlfailover
hogs one core. This is quite surprising.
Conclusion
mysqlfailover
is a good tool to automate failover in clusters using GTID replication. It is flexible and looks reliable. Its main drawback is that there is no easy way to make it highly available itself: if mysqlfailover
crashes, you will have to manually restart it.
Subscribe to:
Posts (Atom)