Editing HA cluster with DRBD and Heartbeat
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.
The edit can be undone.
Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
Latest revision | Your text | ||
Line 1: | Line 1: | ||
− | This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines | + | This article shows how to setup a OpenVZ high availability (HA) cluster using the data replication software DRBD and the cluster manager Heartbeat. In this example the two machines builing the cluster run on CentOS 4.3. The article also shows how to do kernel updates in the cluster, including necessary steps like recompiling of new DRBD userspace tools. For this purpose, kernel 2.6.8-022stab078.10 (containing DRBD module 0.7.17) is used as initial kernel version, and kernel 2.6.8-022stab078.14 (containing DRBD module 0.7.20) as updated kernel version. |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
+ | Additional information about clustering of virtual machines can be found in the following paper: http://www.linuxtag.org/2006/fileadmin/linuxtag/dvd/12080-paper.pdf | ||
== Prerequisites == | == Prerequisites == | ||
The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat. | The OpenVZ kernel already includes the DRBD module. The DRBD userspace tools and the cluster manager Heartbeat must be provided seperately. As the API version of the DRBD userspace tools must exactly match the API version of the module, compile them yourself. Also compile Heartbeat yourself, as at the time of this writing the CentOS extras repository only contained an old CVS version of Heartbeat. | ||
− | On a hardware node for production use there should not be any | + | On a hardware node for production use there should not be any applications that are not really needed for running OpenVZ (any things which are not needed by OpenVZ should run in a VE for security reasons). As a result, compile DRBD and Heartbeat on another machine running CentOS 4.3 (in this example I used a virtual machine on a VMware Server). |
=== Compiling Heartbeat === | === Compiling Heartbeat === | ||
Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch. | Heartbeat version 1.2.* has successfully been used in a lot of two-node-clusters around the world. As the codebase used in version 1.2.* is in production use for many years now, the code is very stable. At the time of writing, Heartbeat version 1.2.4 is the current version of the 1.2.* branch. | ||
− | Get | + | Get tar.gz of the current version of the 1.2.* branch from http://linux-ha.org/download/index.html, at the time of this writing this is http://linux-ha.org/download/heartbeat-1.2.4.tar.gz. Use rpmbuild to build the package: |
<pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre> | <pre>rpmbuild -ta heartbeat-1.2.4.tar.gz</pre> | ||
− | After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm | + | After that, you find four rpm packes in /usr/src/redhat/RPMS/i386 (heartbeat-1.2.4-1.i386.rpm, heartbeat-ldirectord-1.2.4-1.i386.rpm, heartbeat-pils-1.2.4-1.i386.rpm, heartbeat-stonith-1.2.4-1.i386.rpm). In this example only heartbeat-1.2.4-1.i386.rpm is needed. |
=== Compiling DRBD userspace tools === | === Compiling DRBD userspace tools === | ||
Line 85: | Line 78: | ||
! other install options | ! other install options | ||
| no firewall, no SELinux | | no firewall, no SELinux | ||
− | | no firewall, no | + | | no firewall, no SELinux |
+ | |- | ||
+ | ! package groups | ||
+ | | deactivated everything, only kept vim-enhanced | ||
+ | | deactivated everything, only kept vim-enhanced | ||
|} | |} | ||
Line 92: | Line 89: | ||
Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat): | Get the OpenVZ kernel and utilities and install them on both nodes, as described in [[quick installation]]. Update grub configuration to use the OpenVZ kernel by default. Disable starting of OpenVZ on system boot on both nodes (OpenVZ will be started and stopped by Heartbeat): | ||
<pre> | <pre> | ||
− | [root@ovz-node1 ~]# chkconfig vz | + | [root@ovz-node1 ~]# chkconfig --del vz |
[root@ovz-node1 ~]# | [root@ovz-node1 ~]# | ||
</pre> | </pre> | ||
Line 99: | Line 96: | ||
== Setting up DRBD == | == Setting up DRBD == | ||
− | + | On each of the two nodes create a partition that acts as underlying DRBD device. The partitions should have exectly the same size (I created a 10 GB partition hda3 using fdisk on each node for this example). Note that it might be necessary to reboot the machines to re-read the partition table. | |
Install the rpm of the DRBD userspace tools on both nodes: | Install the rpm of the DRBD userspace tools on both nodes: | ||
Line 188: | Line 185: | ||
=== Copy necessary OpenVZ files to DRBD device === | === Copy necessary OpenVZ files to DRBD device === | ||
− | + | Currently, ovz-node1 is still Primary of /dev/drbd0. You can now mount it and copy the necessary files to it (only on ovz-node1!): | |
<pre> | <pre> | ||
− | [root@ovz-node1 ~]# | + | [root@ovz-node1 ~]# mount /dev/drbd0 /mnt |
− | [root@ovz-node1 ~]# | + | [root@ovz-node1 ~]# cp -a /vz/* /mnt/ |
+ | [root@ovz-node1 ~]# mkdir -p /mnt/cluster/etc/sysconfig | ||
+ | [root@ovz-node1 ~]# mkdir -p /mnt/cluster/var | ||
+ | [root@ovz-node1 ~]# cp -a /etc/sysconfig/vz /mnt/cluster/etc/sysconfig | ||
+ | [root@ovz-node1 ~]# cp -a /etc/sysconfig/vz-scripts /mnt/cluster/etc/sysconfig | ||
+ | [root@ovz-node1 ~]# cp -a /var/vzquota /mnt/cluster/var | ||
+ | [root@ovz-node1 ~]# umount /dev/drbd0 | ||
[root@ovz-node1 ~]# | [root@ovz-node1 ~]# | ||
</pre> | </pre> | ||
− | + | Afterwards move the original files and replace them with symbolic links (do this on both nodes): | |
− | Afterwards move the | ||
<pre> | <pre> | ||
− | [root@ovz-node1 ~]# mv /etc/vz /etc/vz.orig | + | [root@ovz-node1 ~]# mv /etc/sysconfig/vz /etc/sysconfig/vz.orig |
[root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig | [root@ovz-node1 ~]# mv /etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts.orig | ||
[root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig | [root@ovz-node1 ~]# mv /var/vzquota /var/vzquota.orig | ||
− | [root@ovz-node1 ~]# ln -s /vz/cluster/etc/vz /etc/vz | + | [root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz /etc/sysconfig/vz |
[root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts | [root@ovz-node1 ~]# ln -s /vz/cluster/etc/sysconfig/vz-scripts /etc/sysconfig/vz-scripts | ||
[root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota | [root@ovz-node1 ~]# ln -s /vz/cluster/var/vzquota /var/vzquota | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
[root@ovz-node1 ~]# | [root@ovz-node1 ~]# | ||
</pre> | </pre> | ||
Line 231: | Line 219: | ||
[root@ovz-node1 ~]# | [root@ovz-node1 ~]# | ||
</pre> | </pre> | ||
− | Create the Heartbeat configuration file ha.cf and copy it to | + | Create the Heartbeat configuration file ha.cf and copy it to /etc/ha.d/ha.cf on both nodes. Details about this file can be found at http://www.linux-ha.org/ha.cf. Below is an example configuration which uses the two network connections and also a serial connection for heartbeat packets: |
<pre> | <pre> | ||
# Heartbeat logging configuration | # Heartbeat logging configuration | ||
Line 262: | Line 250: | ||
respawn hacluster /usr/lib64/heartbeat/ipfail | respawn hacluster /usr/lib64/heartbeat/ipfail | ||
</pre> | </pre> | ||
− | Create the Heartbeat configuration file authkeys and copy it to | + | Create the Heartbeat configuration file authkeys and copy it to /etc/ha.d/authkeys on both nodes. Set the permissions of this file to 600. Details about this file can be found at http://www.linux-ha.org/authkeys. Below is an example: |
<pre> | <pre> | ||
auth 1 | auth 1 | ||
1 sha1 PutYourSuperSecretKeyHere | 1 sha1 PutYourSuperSecretKeyHere | ||
</pre> | </pre> | ||
− | Create the Heartbeat configuration file haresources and copy it to | + | Create the Heartbeat configuration file haresources and copy it to /etc/ha.d/haresources on both nodes. Details about this file can be found at http://www.linux-ha.org/haresources. Below is an example: |
<pre> | <pre> | ||
− | ovz-node1 | + | ovz-node1 datadisk::r0 Filesystem::/dev/drbd0::/vz::ext3 vz MailTo::youremail@yourdomain.tld |
</pre> | </pre> | ||
Finally, you can now start heartbeat on both nodes: | Finally, you can now start heartbeat on both nodes: | ||
Line 279: | Line 267: | ||
</pre> | </pre> | ||
− | == | + | == How to do OpenVZ kernel updates when it contains a new DRBD version == |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel. | As mentioned above, it is important to use the correct version of the DRBD userspace tools. When an OpenVZ kernel contains a new DRBD version, it is important that the DRBD API version of the userspace tools matches the API version of the DRBD module that is included in the OpenVZ kernel. The API versions can be found at http://svn.drbd.org/drbd/branches/drbd-0.7/ChangeLog. The best way is to always use the version of the DRBD userspace tools that matches the version of the DRBD module that is included in the OpenVZ kernel. | ||
− | In this example the initial cluster | + | In this example the initial cluster installed contained OpenVZ kernel 2.6.8-022stab078.10, which contains the DRBD module 0.7.17. The steps below should the update procedure to OpenVZ kernel 2.6.8-022stab078.14, which contains the DRBD module 0.7.20. |
In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive): | In the first step build the DRBD userspace tools version 0.7.20 on your buildmachine. Then stop Heartbeat and DRBD on the passive node (hint: you can use 'cat /proc/drbd' to get a hint which node is active and which one is passive): | ||
<pre> | <pre> | ||
Line 373: | Line 332: | ||
Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes: | Ensure after every update of OpenVZ tools that OpenVZ is not started on system boot. To disable starting of OpenVZ on system boot execute on both nodes: | ||
<pre> | <pre> | ||
− | [root@ovz-node1 ~]# chkconfig vz | + | [root@ovz-node1 ~]# chkconfig --del vz |
[root@ovz-node1 ~]# | [root@ovz-node1 ~]# | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
</pre> | </pre> | ||
[[Category: HOWTO]] | [[Category: HOWTO]] |