How to Achieve High Availability with Heartbeat & DRBD on Ubuntu 16

Heartbeat and DRBD can be used effectively to maintain high availability for MySQL databases on Ubuntu 16.04. Heartbeat is a network-oriented tool for maintaining high availability and managing failover. With heartbeat, you can make sure that a shared IP address, is active on one and only one server at a time in your cluster. DRBD (Distributed Replicated Block Device) synchronizes data between two nodes by writing the data to the primary server first and then writing to the secondary server. When used to provide high availability for MySQL databases, heartbeat and DRBD allow for a seamless transition from the primary to the secondary server when the primary server fails. This guide will take you through the steps necessary to create and initiate such a setup.

Getting Started

In order to follow this guide you will need to have the following in place:
• Two nodes (Cloud Server or Dedicated Server) on a single LAN with Ubuntu 16.04 installed. We will call these servers DB1 and DB2 below
• Three network cards on each server to allow for the establishment of all required IP addresses
• A second unpartitioned drive on each server to serve as the DRDB device
• Root access to the nodes

Throughout the tutorial, ALL commands must be done on both servers unless otherwise specified.

Tutorial

IP Assignment

You will need to assign public and private IP addresses to the servers, a floating IP address through which your MySQL databases will be accessed, and IP addresses for DRBD traffic. In this guide, we assume that the IP range for your public network is 173.209.49.66/29 and the IP range for your private network (LAN) is 10.119.0.0/24. We also assume you have a third network card on both servers with a cross-connect between servers for DRBD traffic. The LAN subnet for this cross-connect presumably covers the IP range 192.168.5.0/24.

The networking required for such a setup may be configured in a number of ways. We chose to use the easiest to understand. In the setup described below, the first LAN will be used for floating IP’s and heartbeat while the third network will be reserved for DRBD.

We have assigned the following IP addresses to the components we will use in this tutorial.

Server eth0 (wan) eth1 (lan) eth1 (drbd)
DB1 173.209.49.66 10.119.0.1 192.168.5.1
DB2 173.209.49.67 10.119.0.2 192.168.5.2
Floating None 10.119.0.10 None

Always makes sure your system is up to date before you install any software.

apt-get update
apt-get -y upgrade

Disable Ubuntu’s firewall (if it is installed) before you assign the above IP addresses.

ufw disable

Configure the hostnames for your servers. To do so run the following commands on

On DB1:

echo "db1.mydomain.com" > /etc/hostname
hostname db1.mydomain.com

On DB2:
echo "db2.mydomain.com" > /etc/hostname
hostname db2.mydomain.com

Add the hostnames you have created as entries in the host file:

nano /etc/hosts

Next, add the following entries on both servers. This will bind hostnames to the DRBD IP’s in the 192.168.5.0/24 range:

192.168.5.1 db1.mydomain.com
192.168.5.2 db2.mydomain.com

Your network is now set up.

Install and Configure DRBD and Heartbeat

With your network configuration in place, install DRBD and heartbeat. Configure your nodes so that both DRDB and heartbeat start when the servers boot:

apt-get -y install drbd8-utils heartbeat
systemctl enable drbd
systemctl enable heartbeat

You will need to set up a DRDB device for each server. Check that your second unpartitioned drive is available to use as your DRBD device:

fdisk -l

Below, we can see that we have a 16GB /dev/sdb drive in our setup. We will use this drive for DRBD.

Disk /dev/sda: 16 GiB, 17179869184 bytes, 33554432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcda6f1d3
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 976895 974848 476M 83 Linux
/dev/sda2 976896 4976639 3999744 1.9G 82 Linux swap / Solaris
/dev/sda3 4976640 33552383 28575744 13.6G 83 Linux
Disk /dev/sdb: 16 GiB, 17179869184 bytes, 33554432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

The second drive can easily be partitioned using the simple formula below. This formula will create a single partition that occupies the whole disk.

echo -e 'n\np\n1\n\n\nw' | fdisk /dev/sdb

With the DRBD devices in place, it is time to configure DRBD on both nodes. First, create the DRBD configuration file on each node:

echo "" > /etc/drbd.d/global_common.conf
nano /etc/drbd.d/r0.res

Add the following into both DRBD configuration files:

global {
usage-count no;
}
resource r0 {
protocol C;
startup {
degr-wfc-timeout 60;
}
disk {
on-io-error detach;
}
syncer {
rate 100M;
}
net {
cram-hmac-alg sha1;
shared-secret "wXE8MqVa";
}
on db1.mydomain.com {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.5.1:7789;
meta-disk internal;
}
on db2.mydomain.com {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.5.2:7789;
meta-disk internal;
}
}

We will now edit the various files used to configure our servers for high availability. Remember to edit the files on both servers in exactly the same way except where we indicate there need to be differences.

The first file we will edit is the configuration file /etc/ha.d/ha.cf on both nodes. You may open the file like so:

nano /etc/ha.d/ha.cf

Enter the following parameters in the file. The bcast parameter line is of critical importance. In our case, we have 3 network interfaces. We need to put eth2 on this line. If you are using an existing network setup that is designed differently than our simple example you may need to enter a different value here:

# Check Interval
keepalive 1
# Time before server declared dead
deadtime 10
# Secondary wait delay at boot
initdead 60
# Auto-failback
auto_failback off
# Heartbeat Interface
bcast eth2
# Nodes to monitor
node db1.mydomain.com
node db2.mydomain.com

The next file to edit is the resources file /etc/ha.d/haresources

nano /etc/ha.d/haresources

We will enter only one line in this file. Inspect this entry carefully. It should include the hostname of your main active node (db1), the floating IP (10.119.0.10), the device (/dev/drbd0) and its mount point (/var/lib/mysql):

db1.mydomain.com 10.119.0.10/24 drbddisk::r0 Filesystem::/dev/drbd0::/var/lib/mysql::ext4::noatime

To secure your high availability setup you will need to define and store identical authorization keys on both nodes. To do so, open and edit /etc/ha.d/authkeys

nano /etc/ha.d/authkeys

This file should only contain the two lines below. Use the same password on both nodes. The password is the text immediately after the “sha1” statement.

auth1
1 sha1 e86b38f5075de0318548edad9566436423ada422

Using the partitions created above, create the DRBD disks. Start by entering the following command on DB1:

drbdadm create-md r0
systemctl restart drbd
drbdadm outdate r0
drbdadm -- --overwrite-data-of-peer primary all
drbdadm primary r0
mkfs.ext4 /dev/drbd0
chmod 600 /etc/ha.d/authkeys
mkdir /var/lib/mysql

If you get an error message along the following lines, “The file /dev/drbd0 does not exist and no size was specified,” check that your hostnames have been set properly.

Once the DRBD disk is created on DB1, you may create the DRBD disk on DB2:

drbdadm create-md r0
systemctl restart drbd
chmod 600 /etc/ha.d/authkeys
mkdir /var/lib/mysql

With both disks in place, you may now verify that the DRBD disk is connected and is properly syncing

cat /proc/drbd

The above command should yield the below output. In this output, you should see Primary/Secondary. This output indicates that the DB1 node is the master while the other node is the slave. It also shows that everything is syncing as expected.

root@db1:~# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
srcversion: D496E56BBEBA8B1339BB34A
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:550488 nr:0 dw:135424 dr:416200 al:35 bm:0 lo:0 pe:11 ua:0 ap:0 ep:1 wo:f oos:16233752
[>....................] sync'ed: 3.3% (15852/16380)M
finish: 0:16:45 speed: 16,136 (16,932) K/sec

With properly configured synchronization in place between the primary and secondary nodes, it is time to enable the failover portion of our setup. To do so, we will simply start heartbeat on both nodes

systemctl start heartbeat

The DRBD partition should be mounted on DB1 only. Verify that this is so:

root@db1:/etc/ha.d# mount | grep drbd
/dev/drbd0 on /var/lib/mysql type ext4 (rw,noatime,data=ordered)

Once you verify that the floating IP is only bound to DB1 with the following command, you are ready to install your MySQL-type database service.

root@db1:/etc/ha.d# ip addr show | grep 10.119.0.10
inet 10.119.0.10/24 brd 10.119.0.255 scope global secondary eth1:0

Install MariaDB Database Service

Now is the time to install our database service on both servers. There are several variants of MySQL that were created as forks of the original MySQL source code. In the following, we have opted to install MariaDB because of its performance and good track record.

apt-get install mariadb-server

Percona DB and MySQL are two other options you might choose.

The database service on a given node should only start if that node is designated as the primary node at the time. To assure you don’t end up with MySQL running on both nodes simultaneously, disable auto-start for the MariaDB service on both nodes

systemctl disable mysql

By design, DB1 is primary when our setup is initiated. No database service should be running on DB2. So long as DB1 is primary, databases on DB2 should be created and populated through synchronization with DB1. Therefore, on DB2, you need to stop the MariaDB service and empty /var/lib/mysql. Perform this command ON DB2 ONLY:

systemctl stop mysql
rm -rfv /var/lib/mysql/*

Before you proceed further, configure the root password on DB1. To do so, simply run the wizard. Set a new root password and allow all other options to retain their default values for now:

root@db1:/var/lib# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.
Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] Y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] Y
... Success!
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] Y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!

Copy the MySQL Maintenance configuration file from DB1 to DB2

rsync -av /etc/mysql/debian.cnf root@192.168.5.2:/etc/mysql/debian.cnf

Now we will create a root user for remote management of and access to the databases on the highly available MySQL instance. We will make use of wildcards to do so.

Our cluster architecture is set up so that all other servers on our LAN can reach the database at the floating IP 10.119.0.10. If you wish to enable users outside your LAN to access your highly available database you may bind a public IP in /etc/ha.d/haresources for database access as well, following the pattern set above in editing that file.

In our case, we have set up our high availability database servers to be accessible from other servers on the LAN that share the IP range 10.119.0.0/24.

mysql -u root -p

Enter the following commands to create the root user. Replace “my-password” with the MySQL root password you wish to assign to the remote access user:

MariaDB [(none)]> CREATE USER 'root'@'10.119.0.%' IDENTIFIED BY 'my-password';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.119.0.%' WITH GRANT OPTION;
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> QUIT;

Set the bind address for MySQL on both servers

sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/mariadb.conf.d/*.cnf

Initiate Heartbeat for MySQL Service

Add support for the MySQL service in our heartbeat instances on both servers. First, open the configuration file:

nano /etc/ha.d/haresources

Then, simply add mysql at the end of the line and save the file

db1.mydomain.com 10.119.0.10/24 drbddisk::r0 Filesystem::/dev/drbd0::/var/lib/mysql::ext4::noatime mysql

Once heartbeat is configured we need to restart it on both servers. heartbeat must be started on the primary server (DB1) first. Once heartbeat has started on DB1 allow at least 20 seconds before you restart heartbeat on DB2.

The command sequence is as follows. ON DB1 enter:

systemctl restart heartbeat

Wait 20 seconds or more and then enter the following on DB2:

systemctl restart heartbeat

The delay between initiating the heartbeat stack on DB1 and DB2 will prevent heartbeat from inadvertently initiating failover to DB2 upon startup.

Testing redundancy of our setup

Our goal throughout this tutorial has been to tie together our servers such that MySQL service will not be interrupted if the active server fails. Now that our setup is complete, we will perform a series of tests to verify that heartbeat will actually trigger a transfer from the active server to the passive server when the active server fails in some way. We will also verify that DRBD is properly syncing the data between the two servers.

Tests to Perform on DB1

First, we will verify that DB1 is the primary drbd node

root@db1:~# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
srcversion: D496E56BBEBA8B1339BB34A
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:22764644 nr:256 dw:529232 dr:22248299 al:111 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Next, we will verify that the DRBD disk is mounted

root@db1:~# mount | grep drbd
/dev/drbd0 on /var/lib/mysql type ext4 (rw,noatime,data=ordered)

The floating IP must be bound correctly for the setup to function properly

root@db1:~# ip addr show | grep 10.119.0.10
inet 10.119.0.10/24 brd 10.119.0.255 scope global secondary eth1:0

Check to make sure that MariaDB is running

root@db1:~# ps -ef | grep mysqld
root 7472 1 0 05:52 ? 00:00:00 /bin/bash /usr/bin/mysqld_safe
mysql 7617 7472 0 05:52 ? 00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
root 7618 7472 0 05:52 ? 00:00:00 logger -t mysqld -p daemon error

Use the remote access root user to test the MySQL connection directly on the floating (failover) IP and create a test database.

mysql -h 10.119.0.10 -u root -p
MariaDB [(none)]> create database failtest;
MariaDB [(none)]> quit

Restart heartbeat on DB1.

systemctl restart heartbeat

Heartbeat will interpret this restart as a failure of DB1 and should trigger failover to make DB2 the primary server. Ensure that DRBD is now treating DB1 as the secondary server:

root@db1:~# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
srcversion: D496E56BBEBA8B1339BB34A
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:22764856 nr:388 dw:529576 dr:22248303 al:112 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Having triggered the failover, we will now test that our setup is fully functional with DB2 acting as the primary server.

Tests to perform on DB2

Verify that DB2 is now the primary drbd node

root@db2:~# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
srcversion: D496E56BBEBA8B1339BB34A
0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:412 nr:20880892 dw:20881304 dr:11463 al:7 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Verify that the DRBD disk is mounted on DB2

root@db2:~# mount | grep drbd
/dev/drbd0 on /var/lib/mysql type ext4 (rw,noatime,data=ordered)

Verify the floating IP is now bound to DB2 correctly

root@db2:~# ip addr show | grep 10.119.0.10
inet 10.119.0.10/24 brd 10.119.0.255 scope global secondary eth1:0

Check to make sure that MariaDB is running on DB2

root@db2:~# ps -ef | grep mysqld
root 7568 1 0 06:13 ? 00:00:00 /bin/bash /usr/bin/mysqld_safe
mysql 7713 7568 0 06:13 ? 00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
root 7714 7568 0 06:13 ? 00:00:00 logger -t mysqld -p daemon error

Use the remote access user to connect to the MySQL instance at the floating (failover) IP. If your setup is working properly, the following commands should enable you to view the test database we created earlier while DB1 was the primary server.

mysql -u 10.119.0.10 -u root -p
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| failtest |
| information_schema |
| lost+found |
| mysql |
| performance_schema |
+--------------------+
5 rows in set (0.04 sec)
MariaDB [(none)]> exit

Restart Heartbeat.

systemctl restart heartbeat

As it did before, this restart should trigger failover. To check that it has done so, ensure that DRBD is now secondary on DB2

root@db2:~# cat /proc/drbd
version: 8.4.5 (api:1/proto:86-101)
srcversion: D496E56BBEBA8B1339BB34A
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:508 nr:20881012 dw:20881520 dr:11475 al:7 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Provided your tests yielded the expected results, your high availability MySQL setup should now be fully operational.

Conclusion

Congratulations, the high-availability MySQL setup you have created on your Ubuntu 16.04 servers will make your databases more reliable for mission critical functions. This frees you to create content while your users get nearly continuous data access.